How should teams evaluate AI coding agents in 2026?

Evaluate the whole system: model, reasoning effort, harness, tools, permissions, budget, retries, and review policy. Public benchmarks should shortlist candidates, while internal versioned repository tasks decide what ships.

Is SWE-bench Verified still useful?

Yes, as a historical smoke test and continuity baseline. It should not be the main ranking signal for frontier agents after OpenAI deprecated it in February 2026 because 59.4% of audited tasks had broken or underspecified tests.

Why does the harness matter so much?

The harness controls actions, tools, retries, observations, and stopping rules. Stanford IRIS Lab reported that Claude Opus 4.6 moved from 43% to 76.4% on Terminal-Bench 2.0 when the harness changed.

AI Coding Agent Evaluation Broke at 59.4%; Test the Harness

Q: What metric matters most for production coding agents?

Cost per accepted, reviewed change is the core metric. Track it with wall-clock time, reviewer minutes, regression escapes, and security violations so cheap but risky patches do not look like wins.

AI coding agent evaluation changed on February 23, 2026, when OpenAI deprecated SWE-bench Verified after reporting that 59.4% of audited problems had broken or underspecified tests. A leaderboard row is now a candidate filter; the adoption decision has to test the full model, harness, tools, permissions, budget, and review policy on versioned repository tasks.

AI coding agent evaluation means measuring the complete agent system against reproducible coding tasks, then scoring accepted changes after review, cost, latency, regressions, and security failures. Public coding agent benchmarks qualify candidates. Production coding agent evals decide what ships.

Anthropic's agent eval guidance points in the same direction: teams need their own task distribution and acceptance criteria. A reproducible runner such as the UK AI Security Institute's Inspect framework is now a baseline requirement for serious adoption work.

TL;DR

A leaderboard row is a qualification gate. The production decision requires your repository, your tests, your budget, and your reviewers.

The strongest proof point is the Stanford IRIS Lab Meta-Harness paper: the same Claude Opus 4.6 model moved from 43% to 76.4% on Terminal-Bench 2.0 when the harness changed. Rank coding agents by cost per accepted, reviewed change, with regressions and security failures counted.

Key takeaways

Pin the evaluated system as model + reasoning effort + harness + tools + permissions + budget + retries + review policy.
SWE-bench Verified is now a historical smoke test after OpenAI's February 2026 deprecation.
SWE-bench Pro, SWE-bench Live, Terminal-Bench 2.0, and METR-style time-horizon tasks answer different questions.
Harness choice can move a score by 33.4 points on the same model, according to Stanford IRIS Lab's Meta-Harness paper.
Vendor-reported scores and paper-reported harness results are different measurement objects.
The production metric is cost per accepted change after human review, with escaped regressions and security violations attached.

Why leaderboards broke

The missing field in most leaderboards is the harness commit.

SWE-bench Verified began as a cleaner 500-task subset across 12 repositories, introduced in OpenAI's SWE-bench Verified post. By February 2026, OpenAI said the benchmark no longer separated frontier coding systems.

That matters because a saturated benchmark turns operational details into rank changes. Reasoning effort, retry count, shell permissions, hidden test access, network egress, repository checkout behavior, and patch policy all affect the final row.

OpenAI's deprecation post used a blunt phrase: "SWE-bench Verified no longer measures frontier coding". Anthropic's infrastructure-noise post and its eval guidance both frame agent evaluation as an engineering system problem.

Is SWE-bench Verified still meaningful?

SWE-bench Verified remains useful as a historical smoke test for older Python repository bug fixes. Its role in 2026 should be continuity, because OpenAI deprecated it on February 23 after reporting that 59.4% of audited tasks had broken or underspecified tests and top scores had compressed.

The benchmark still answers a narrow question: can a candidate clear older Python repository bug-fix tasks under a familiar grader? That is useful when a team needs continuity with last year's reports.

It fails as an adoption test because the hard part moved. Teams need to know whether the agent can work inside their repository, respect their tools, stay within budget, produce reviewable patches, and avoid security-damaging shortcuts.

A flawed test suite creates a specific trap. A patch can pass because it adapts to the grader, edits the wrong surface, or hardcodes expected behavior.

SWE-bench Verified belongs in the first pass of the pipeline, beside other legacy signals. It should not sit at the center of procurement.

What should replace SWE-bench Verified?

SWE-bench Verified should be replaced by a layered stack: SWE-bench Pro for harder repository patches, SWE-bench Live for contamination control, Terminal-Bench 2.0 for shell-native execution, METR-style time-horizon tasks for long work, and a versioned internal repository suite for the final adoption decision.

The best SWE-bench alternatives solve different evaluation problems.

SWE-bench Pro expands the task set to 1,865 problems across 41 repositories, with an 11/12/18 public, held-out, and commercial repository split. The research registry records top public scores around 23%, which gives the benchmark headroom compared with the saturated Verified line.

SWE-bench Live targets contamination through post-cutoff tasks. The report records 1,319 tasks across 93 repositories from the NeurIPS 2025 line.

Terminal-Bench 2.0 measures containerized command-line task completion. That matters because real agents spend their time in shells, logs, package managers, test runners, and file edits.

For long-horizon evaluation, use METR-style task construction with care. The captured primary METR source here is the July 14, 2025 domain-variation post, which supports the claim that task horizons vary by domain.

The research report also mentions a 2026 Time Horizon 1.1 registry claim, but no dedicated captured primary URL was provided, so this article does not rely on the exact 2026 revision number.

The evaluation stack

A useful scorecard has to name the system under test.

Use this tuple for every row:

(M, R, H, T, B, P)

M is the model ID. R is reasoning effort, temperature, and sampling configuration. H is the harness. T is the tool and permission surface. B is the token, wall-clock, and retry budget. P is the review and promotion policy.

The harness is the loop. It asks the model for an action, runs a tool, captures stdout, stderr, exit codes, diffs, and test output, then returns that observation to the model.

Anthropic's long-running harness post, OpenAI's harness engineering post, and OpenAI's Codex harness engineering note all treat harnesses as engineering artifacts with design choices, bugs, and diffs.

The Inspect framework is the cleanest open reference point in the research because it is containerized and produces trajectories. The inspect-evals package adds task packs and integrations around Inspect.

How should AI coding agent evaluation work?

AI coding agent evaluation should pin model ID, reasoning effort, harness commit, tools, permissions, token budget, retry policy, and environment digest. The decision metric should combine accepted-task rate with cost, latency, reviewer minutes, escaped regressions, and security failures under the same review policy used for human pull requests.

A production eval should have three layers.

Layer 1 is qualification. Run SWE-bench Pro, Terminal-Bench 2.0, selected METR-style long-horizon tasks, and any domain-specific public benchmark that passes your provenance checks.

Layer 2 is offline repository evaluation. Build at least 50 tasks from your own backlog, pin the repository commit, write executable acceptance tests, and include tasks that require architecture reading.

Layer 3 is shadow and canary. Let finalist agents propose patches in a sandboxed copy of real work, then measure accepted PR rate, reviewer minutes, escaped regressions, and security violations before any production canary.

Use at least five trials per model, harness, and task configuration. Report 95% bootstrap confidence intervals. Classify every failure as compile error, runtime error, wrong test pass, hallucinated test, infinite loop, regression, security violation, or human rejection.

A one-run score on 50 tasks is a demo. A five-trial score with trajectories, pinned containers, confidence intervals, and human adjudication is an evaluation.

Harness effects are first-order

Stanford's Terminal-Bench result is the number to remember.

The paper-reported Meta-Harness result says Claude Opus 4.6 solved 76.4% of Terminal-Bench 2.0 tasks with Stanford IRIS Lab's Meta-Harness, compared with 43% using the stock Cline harness. That is a 33.4-point movement with the same model.

That result should change how teams read every model row. If the harness can move a score by 33.4 points, a model-only ranking is missing the largest control surface in the system.

The practical response is concrete: pin the harness commit and store the trajectory. Without the action log, you cannot tell whether the agent solved the task, got lucky, edited a test, or spent half its budget in a dead loop.

Cost and budget belong in the eval

The price table belongs inside the scorecard.

A model that solves two extra tasks can still lose if it needs high reasoning effort, long trajectories, repeated test runs, and expensive output tokens. As of July 10, 2026, OpenAI's API pricing lists GPT-5.6 Sol at $5 input and $30 output per million tokens, Terra at $2.50 and $15, and Luna at $1 and $6.

Anthropic's Claude Opus 4.8 launch reports vendor-measured 88.6% on SWE-bench Verified and 69.2% on SWE-bench Pro. The research registry records Opus 4.8 at $5 input and $25 output per million tokens with a 1M context window.

Read the next table as a shortlist builder. Vendor-reported SWE-bench scores, paper-reported Terminal-Bench harness results, and internal repository runs are different measurement objects. A row can be useful and still fail as a direct comparison.

Candidate	Price per 1M tokens	Benchmark numbers captured	License / access	Limits captured	Best-fit use case
Claude Opus 4.8	$5 input / $25 output	Vendor-reported 88.6% SWE-bench Verified, 69.2% SWE-bench Pro	Commercial API	1M context in research registry	Complex repository tasks with frontier-model budget
GPT-5.6 Sol	$5 input / $30 output	Frontier coding claim; no comparable public coding row extracted	Commercial API	Context limit not extracted	Teams testing OpenAI's highest GPT-5.6 tier under their own harness
GPT-5.6 Terra	$2.50 input / $15 output	No comparable coding benchmark extracted	Commercial API	Context limit not extracted	Routine engineering work where Sol's cost fails the scorecard
GPT-5.6 Luna	$1 input / $6 output	No comparable coding benchmark extracted	Commercial API	Context limit not extracted	Low-cost triage, test writing, and first-pass patches
GPT-5.2-Codex	See OpenAI docs	Codex-optimized; no comparable score extracted	Commercial API	Context and price depend on docs route	Terminal-native automation and Codex-centered engineering teams
Grok 4.1 Fast	First-party per-million pricing not extracted	Grok coding claim; pricing evidence conflicts across sources	Commercial API	Context limit not extracted	Teams willing to reproduce price and quality before adoption
Llama 4 Scout	Weights or hosted partner pricing, depending on route	No first-party coding score located in the registry	Open-weights route varies by provider	Deployment route dependent	Teams prioritizing control, privacy, and harness ownership

This table should decide what you test. The production scorecard has to normalize by accepted work.

Report cost per accepted task, wall-clock per accepted task, and reviewer minutes per accepted task. Then add regression-escape rate and security-violation rate, because a cheap patch that disables a check is expensive later.

A minimum formula is enough:

text

cost_per_accepted_change =
  total_model_cost_usd / accepted_reviewed_changes

accepted_reviewed_change =
  tests_passed
  and reviewer_accepted
  and security_violation == false
  and regression_escape == false

What's the best AI coding agent in 2026?

The best AI coding agent in 2026 is the model-harness-budget combination that wins on your repository after review. Public rows can shortlist Claude, GPT, Grok, Llama, and Codex paths, while the adoption metric is cost per accepted change with regressions and security failures counted.

For many teams, the first shortlist will include Claude Opus 4.8, GPT-5.6 Sol or Terra, a Codex-specific OpenAI path, and one lower-cost or open-weights candidate. That spread is enough to expose cost and harness sensitivity.

Run each candidate at two effort levels. Low effort tells you whether the agent can handle routine work cheaply. High effort tells you whether the expensive mode buys enough accepted changes to justify itself.

Do not compare vendor claims unless the harness, effort, retries, and tool permissions are disclosed. A vendor-reported 69.2% on SWE-bench Pro and a paper-reported Terminal-Bench run are different measurement objects.

Use public numbers for screening. Use internal tasks for adoption.

A production scorecard schema

Every row needs enough detail for a future engineer to reproduce it.

A useful scorecard is boring on purpose. It records repository state, agent system, execution budget, outcome, and review result.

This is a schema example, not a measured result:

json

{
  "task_id": "checkout-service#BUG-1842",
  "task_source": "internal_backlog",
  "repo": "checkout-service",
  "repo_commit": "4f2c9a1",
  "environment_digest": "sha256:example",
  "model_id": "gpt-5.6-terra-20260709",
  "reasoning_effort": "high",
  "temperature": 0.2,
  "harness": {
    "name": "internal-agent-harness",
    "commit": "91b8d7c",
    "tool_permissions": ["read_files", "edit_files", "run_tests"],
    "network": "disabled"
  },
  "budget": {
    "max_input_tokens": 200000,
    "max_output_tokens": 40000,
    "max_wall_clock_minutes": 20,
    "max_retries": 1,
    "max_usd": 2.00
  },
  "outcome": {
    "accepted": true,
    "tests_passed": true,
    "cost_usd": 1.42,
    "wall_clock_minutes": 11.3,
    "reviewer_minutes": 9,
    "failure_class": null,
    "security_violation": false,
    "regression_escape": false
  },
  "trajectory_uri": "s3://agent-evals/2026-07/task-1842/run-03.json"
}

The environment digest is mandatory. Package mirrors change, base images move, and test suites can become flaky after dependency updates.

The trajectory is mandatory too. Without the action log, the pass/fail bit is almost useless.

For ambiguous cases, use human adjudication. The reviewer should mark whether the patch is acceptable, whether it is minimal enough to merge, and whether it creates follow-up work a normal pull request would avoid.

What this means for you

Start with a 30-day eval build.

In days 0-30, stand up Inspect or a comparable harness runner, add SWE-bench Pro and Terminal-Bench 2.0 as qualification gates, select METR-style long-horizon tasks if relevant, and write at least 50 internal repository tasks with executable acceptance tests.

In days 31-60, run at least three candidates at low and high reasoning effort. Pin every harness commit and environment digest. Publish a scorecard with accepted-task rate, cost per accepted task, wall-clock time, reviewer minutes, regression escapes, and security violations.

In days 61-90, put the two finalists into shadow evaluation on real PRs. Promote one candidate to a 5% canary only after it wins on accepted changes after review.

Leadership will ask why the highest public leaderboard model is not the immediate default. The answer is that leaderboard rank is a qualification signal. Production adoption requires measured savings on your repository under your budget and review policy.

Rerun the eval whenever a vendor changes a model generation, default effort setting, harness, tool policy, or pricing. In 2026, those changes arrive fast enough that stale evals become procurement fiction.

AI Coding Agent Evaluation Broke at 59.4%. Test the Harness.