Eval Harness

An Eval Harness is the infrastructure that runs a model, prompt, or agent against a fixed set of test cases, collects the outputs, and scores each one with automated graders so behavior can be measured and compared across versions. It typically bundles four parts: a dataset of inputs (often with reference answers or rubrics), a runner that executes the target system over that dataset, a set of graders that turn raw outputs into scores, and a reporting layer that aggregates results into pass rates, per-case diffs, and trend lines. Teams use it to answer a concrete question—did this prompt edit, model swap, or retrieval change make things better or worse?—with numbers instead of impressions. Mature harnesses support multiple grader types, versioned datasets, and per-case tracing so a single regression can be traced back to the exact input that broke.

How it works

The harness loads a dataset, then invokes the target—a prompt, chain, or full agent loop—once per case, capturing outputs plus metadata like latency, token counts, and tool calls. Each output is passed to one or more graders: exact-match or regex checks, programmatic assertions, semantic similarity, or an LLM-as-Judge scoring against a rubric. Scores are aggregated into metrics (accuracy, pass rate, average score) and compared against a stored baseline or threshold. Runs are usually deterministic-friendly—fixed seeds, pinned model versions, cached inputs—so differences reflect the change under test, not noise. Results and traces are persisted for side-by-side inspection.

Why it matters for AI engineers

Prompts and pipelines have no compiler, so a wording tweak or model upgrade can silently degrade quality on cases you already fixed. A harness wired into CI turns those regressions into failing checks before they ship, the same way unit tests guard code. It also quantifies trade-offs—accuracy versus cost, quality versus latency—so model-routing and pipeline decisions rest on evidence rather than one-off spot checks. The recurring costs are dataset curation and grader reliability: a flaky or gameable grader produces confident but wrong verdicts, so graders need their own validation.

Eval Harness vs. alternatives

Related terms