Agent reliability, priced honestly.
Compare an AI agent today with the version you want to ship. The index combines outcome-verified completion, silent failures, retry-amplified model cost, human review, and p95 latency—then shows the cost per task that actually worked.
Current system vs. proposed change
Use one representative month or eval batch. Estimates are only as credible as your outcome grader and cost inputs.
| Outcome | Current | Proposed | Delta |
|---|
A task is not successful until the outcome agrees.
Agent economics break when the numerator includes retries and reviewers but the denominator is raw requests. This protocol uses verified tasks as the denominator. A coding agent is verified by tests and repository state; a support agent by the resulting ticket, refund, or account state; a research agent by grounded claims, coverage, and source quality.
Why these five inputs
Anthropic’s January 2026 agent-evals guide distinguishes a trial’s transcript from its final outcome and recommends combining deterministic, model-based, and human graders. It also treats the model and harness as one evaluated system. That is why this scorecard asks about the working system—not a model name in isolation. Read Anthropic’s eval guidance.
NIST’s AI Risk Management resources frame testing, evaluation, verification, and validation as an operating discipline and call for metrics tied to the most significant risks. Silent failure belongs here because a plausible success message can hide a wrong real-world state. Open the NIST AI Resource Center.
Use it as a regression protocol
- Start with 20–50 real tasks drawn from incidents and normal work.
- Pin the model, harness, tools, environment, budgets, and grader version.
- Run multiple trials where stochastic variation matters.
- Record outcome verification, attempts, cost, review time, and p95 latency.
- Compare one change at a time, then confirm it in production monitoring.