harness-eval Evaluation Pipeline

Quick / Standard / Full tiers converge on one deterministic scoring engine (평가 파이프라인)

harness-eval Evaluation Pipeline Quick / Standard / Full tiers converge on one deterministic scoring engine (평가 파이프라인) 01 / Entry Point 02 / Deterministic Scripts 03 / Safety Review (Full) 04 / Agent Orchestration (Full) 05 / Design Review (Full) 06 / Results & History Collect Evaluate Synthesize + publish User / CLI · quick·standard·full · Entry Point › Collect · slash commands User / CLI quick·standard·full slash commands scoring.sh · 16 checks · 4 tiers · Deterministic Scripts › Collect · single source scoring.sh 16 checks · 4 tiers single source Static Analysis · static-analysis.sh · Deterministic Scripts › Collect · bash · JSON · hooks Static Analysis static-analysis.sh bash · JSON · hooks Dynamic Checks · runs hooks + tests · Deterministic Scripts › Synthesize + publish · Standard only Dynamic Checks runs hooks + tests Standard only Collector · inventory artifact · Agent Orchestration (Full) › Collect Collector inventory artifact Safety Evaluator · tool scope · secrets · Safety Review (Full) › Evaluate Safety Evaluator tool scope · secrets Completeness · coverage · recovery · Agent Orchestration (Full) › Evaluate Completeness coverage · recovery Design Evaluator · architecture quality · Design Review (Full) › Evaluate Design Evaluator architecture quality Synthesizer · 0.50/0.25/0.25 · en/ko · Agent Orchestration (Full) › Synthesize + publish Synthesizer 0.50/0.25/0.25 · en/ko .harness-eval/ · reports + latest.json · Results & History › Synthesize + publish .harness-eval/ reports + latest.json badge.sh · shields.io · opt-in · Results & History › Synthesize + publish badge.sh shields.io · opt-in standard+ confirm gate quant results Legend Agent logic Policy Context / trace External system

Score Model

  • • 12 dimensions in 3 categories: 0.50 / 0.25 / 0.25
  • • Overall score on a 1.0-10.0 scale
  • • 7 grades from A+ to F
  • • scoring.sh is the single source of truth

Deterministic Core

  • • 16 checks across 4 maturity tiers
  • • Lower-tier pass rates cap higher-tier contribution
  • • Same project always yields the same score
  • • Fixtures verify each maturity band

Execution Safety

  • • Standard dynamic analysis executes hooks and tests
  • • Use --static-only for untrusted repositories
  • • Quick and Full never run target code
  • • Collector failure falls back to Standard results

Workflow diagram • Built with Archify • Create yours ↗ • Hover to trace • R route • Click to focus • +/− zoom • M radar • [/] views • P play story • T theme • E export