What is the difference between cost per attempt and cost per accepted task in coding-agent evaluation?

Why Cost per Attempt Is Not the Right Unit for Coding Agents

Cost per attempt is useful for measuring raw execution expense, but it can hide the true economics of agent evaluation. A coding agent that runs cheaply but fails often may consume more budget overall than a pricier agent that reaches acceptance faster. The more decision-relevant unit is cost per accepted task, because it reflects the spend required to produce a result that survives review and is actually usable.

That difference matters most when evaluation includes retries, partial fixes, human review, and rejection cycles. In those settings, a single attempt is only one step in a longer workflow, not the outcome the team is paying for. Teams that optimise only for cheap runs often end up rewarding brittle agents that look efficient in isolation but are expensive to operate at scale.

Practically, cost per attempt answers "What does one run cost?" while cost per accepted task answers "What did it cost to get to a result we can ship?" In practice, teams often discover that the cheapest attempt profile belongs to the most expensive production workflow once review and rework are counted.

How the Metric Changes Evaluation in Practice

Cost per accepted task treats evaluation as an end-to-end system, not a single inference call. The denominator is the accepted outcome, so every retry, failed branch, and human correction becomes part of the measured spend. That makes it better suited to coding agents, where the real unit of value is a correct, approved change rather than an isolated generation.

A practical evaluation loop usually needs to include:

• all agent runs for a task, including unsuccessful ones
• human time spent reviewing, correcting, or rejecting outputs
• tooling or orchestration costs tied to the workflow
• the final acceptance event, which defines the denominator

This metric is especially useful when comparing agent strategies. An agent that explores broadly may raise attempt count, but still lower accepted-task cost if it converges faster on a reviewable solution. By contrast, an agent that appears efficient per run may create hidden labour in debugging, cleanup, or repeated review. For that reason, accepted-task cost is usually the better metric for procurement, scaling decisions, and model selection.

The distinction is also a guardrail against misleading benchmarks. If a benchmark stops at first-pass output, it rewards speed over reliability. If it measures accepted-task cost, it captures the actual operational burden of getting useful code through the full workflow. That is why evaluation teams should define acceptance criteria before comparing systems, or the metric will drift into whatever outcome is easiest to generate.

One useful way to sanity-check the metric is to ask whether the cost model still makes sense if the review step doubles in length. If that changes the ranking of agents, attempt cost alone was never describing the real operational picture. These controls tend to break down when acceptance is subjective or inconsistent across reviewers, because the denominator becomes unstable.

Common Variations and Edge Cases

Tighter acceptance criteria often increase review overhead, so teams must balance measurement precision against operational cost. That tradeoff is real, especially in coding-agent pilots where review standards vary by task type or team maturity.

In some environments, cost per attempt is still useful as a diagnostic metric. It can help isolate prompt efficiency, tool-use overhead, or model latency before the broader workflow is layered in. But it should be treated as a component metric, not the headline number, when the question is economic viability.

The hardest edge case is comparing tasks with very different acceptance difficulty. A task that needs deep human verification may naturally show a higher accepted-task cost than a routine refactor, even if the agent performs well. That does not mean the agent is worse, only that task complexity has to be normalised before conclusions are drawn. Another common pitfall is counting only failed model outputs while ignoring rejected-but-usable drafts that still consumed reviewer time.

Risk and Threat Considerations

When teams optimise coding agents on cost per attempt alone, they can create a distorted incentive structure, cheaper runs, more retries, and less visibility into the real cost of getting trustworthy output. The exposure is operational and governance-related rather than purely financial: bad incentives can push teams toward brittle automation that looks efficient in benchmarks but creates more review burden and more residual error in production.

Failure mechanism: The control failure appears when the evaluation metric excludes retries, rejection handling, or human validation. That lets agents appear cost-effective even when they are repeatedly failing in ways that only become visible after review. The same pattern can also hide quality regression, because the organisation measures generation cost instead of the cost of reaching an accepted outcome.

Impact: Teams may underprice agent deployment, overstate ROI, and scale systems that are expensive to supervise. In the worst case, they approve workflows that keep producing low-quality code while the true labour cost shifts to reviewers and incident responders.

Practitioner Guidance

What to prioritise: Use cost per accepted task as the primary business metric, and keep cost per attempt as a diagnostic signal for model efficiency. That split helps separate workflow economics from generation mechanics.

What to verify: Confirm that the acceptance definition is stable, reviewer time is counted consistently, and failed runs are included in the same cost model. If those inputs vary by team, the metric will not support fair comparison.

Decision rule: If two agents have similar per-attempt cost but different retry or review rates, treat the one with lower accepted-task cost as the stronger operational choice. If acceptance is inconsistent, fix the review process before trusting the comparison.

Practitioner takeaway: The right metric is the one that matches the real unit of value, and for coding agents that is usually the accepted task, not the isolated attempt.

Related resources from NHI Mgmt Group

• What is the difference between task-based and autonomous AI agent identity risk?
• What is the difference between agent evaluation and runtime control?
• What is the difference between tracing and evaluation in agent governance?
• What is the difference between task oriented and workflow oriented agent benchmarks?