Cost per successful task is the right metric for AI model routing

At a glance

What this is: Arize's benchmark argues that AI routing should be judged by cost per successful task, not token price alone, because retries, failures, and task difficulty change the real economics.

Why it matters: This matters for AI and IAM practitioners because routing, escalation, and verification policies determine whether autonomous work is economically viable and governable at scale.

By the numbers:

• Arize and Fireworks tested 10 open and closed models across 40 real agent tasks in 2,400 runs.
• The benchmark used 6 trials per task-model cell, creating 2,400 runs in total.
• The study recorded $626 of API spend across the benchmark.
• gpt-oss-120b finished a task for $0.054 while passing only 33% of the time.

Context

Cost per token is easy to price but incomplete for agentic AI operations. A model that looks cheap on paper can become expensive once retries, tool failures, timeout loops, and verification overhead are included in the actual work outcome.

The article is really about routing governance for AI systems, not just model selection. Once task success becomes the unit of value, teams have to think about coverage, escalation, and evaluation as control points rather than treating inference spend as a simple procurement decision.

Key questions

Q: What breaks when model routing is based only on token price?

A: Token-only routing hides retries, tool failures, and timeout loops, so the cheapest-looking model can become expensive once you count failed attempts. It also obscures capability gaps, because a model that is cheap per token may still fail too often to finish real work. That is why cost per successful task is the more accurate operating metric.

Q: When should teams switch from a frontier model to a lower-cost model?

A: Teams should switch when the cheaper model can meet the same task success threshold on their own evaluations. The decision should be based on measured outcome quality, not on vendor claims or token price alone. If performance stays within policy bounds, a lower-cost model is usually the safer operating choice.

Q: What are the signs that an AI routing ladder is failing?

A: Common signs include repeated retries, budget exhaustion, non-zero tool exits, and a rising cost per success even when token prices stay flat. Another warning is poor coverage, where the system only solves a narrow subset of tasks and quietly drops the rest. Those signals show the ladder is miscalibrated.

Q: How should security and AI teams govern escalation between models?

A: Govern escalation as a controlled decision with clear triggers, measured fallback behaviour, and an outcome definition for success. If you can verify results directly, use that check. If you cannot, rely on trace-based distress signals and task-class routing so escalation adds capability without turning into blind spend.

Technical breakdown

Why token price misses the real economics of agentic work

Token pricing captures only the inference layer, not the full cost of getting a task across the finish line. In agentic workflows, the model may fail, retry, call tools repeatedly, or time out, and all of those attempts still consume budget. Cost per successful task therefore treats failed runs as part of the economic picture, which is closer to how production systems behave. This also explains why a model with a low per-token rate can still be a poor operating choice if its success rate is weak or its retry tax is high.

Practical implication: Track cost against completed work, not just token consumption, before approving a model for production use.

Why routing beats one-model buying decisions

Routing changes the unit of decision from model preference to task fit. Easy tasks can be sent to cheaper models, while hard tasks are escalated to frontier models that can handle them reliably. This works because the benchmark shows meaningful separation by task difficulty, with some models strong on easy work and others only competitive on the hard tail. The technical point is that routing is not merely cost reduction, it is workload classification plus controlled escalation, with success criteria attached to each rung.

Practical implication: Build routing rules around workload classes and escalation thresholds instead of standardising on one default model.

Why verification changes whether escalation is safe

Escalation only works cleanly when success or failure can be judged. In benchmark settings, deterministic tests make that straightforward, but many production agent flows lack an oracle. In those cases, teams need proxy signals such as non-zero tool exits, repeated commands, schema validation failures, or budget exhaustion to detect distress early. The important distinction is between noisy failure signals and silent wrong answers. If you cannot verify the outcome at all, routing becomes much harder to govern because escalation may simply add cost without improving trust.

Practical implication: Pair routing with the strongest available verifier, then use trace signals to trigger escalation when no direct test exists.

NHI Mgmt Group analysis

Cost per successful task is becoming the right governance unit for agentic AI. Token price is a procurement metric, not an operating metric. Once retries, tool calls, and verification overhead are included, the economic picture changes enough to alter model choice, routing policy, and control design. Practitioners should treat task success as the governed outcome, not tokens as the governed input.

Routing introduces a new control problem that looks a lot like policy enforcement for AI systems. The issue is not just picking a cheaper model, it is deciding when a request should stay with a low-cost model and when it should be escalated to a stronger one. That means model routing increasingly resembles access and privilege decisions in IAM: the system must grant more capability only when the task warrants it.

Cost efficiency and coverage are different governance questions. A model can be extremely efficient per success because it only solves a narrow set of tasks, which makes it unsuitable as a universal replacement. The benchmark shows that coverage must sit beside unit economics in programme reporting. Teams should avoid optimising for the lowest successful-task cost if that quietly shrinks what the system can actually do.

Task-class routing is the practical middle ground between static model choice and uncontrolled escalation. The article shows that easy work and hard work belong on different rungs, and that a poorly designed ladder can cost more than doing nothing at all. For practitioners, the governance implication is simple: define the ladder, instrument the transitions, and measure whether each rung is earning its keep.

From our research library:

• 13% of organisations suffered breaches of their AI models, according to IBM's Cost of a Data Breach 2025 report.

What this signals

Task-level economics are now a governance issue, not just a benchmarking curiosity. Once retries and tool failures are included, the real question becomes whether the AI system is being managed for productive completion or for apparently cheap inference. Programmes that ignore this distinction will overestimate the value of low token prices and underestimate the operational cost of weak models.

Cost per successful task creates a clearer control boundary for model routing. It forces teams to decide which work belongs on efficient models, which work belongs on frontier models, and which work should never be automated without a verifier. That is the right foundation for policy, procurement, and runtime oversight.

Routing policies need trace visibility to be safe at scale. Without telemetry on repeated commands, failure loops, and budget exhaustion, escalation becomes guesswork and cost control collapses into hope. The programme implication is that observability and evaluation must sit beside model selection, not after it.

For practitioners

• Define success at the task level Use outcome-based evaluation, not token counts, as the primary measure for AI agent economics. Include retries, failed tool calls, and timeout cases in the cost model so the metric reflects completed work.
• Build a difficulty-based routing policy Classify requests into workload bands and send each band to the cheapest model that can complete it reliably. Reserve frontier models for the hard tail where smaller models cannot finish the task consistently.
• Instrument escalation with trace signals Use telemetry such as repeated commands, non-zero tool exits, schema failures, and budget exhaustion to identify distress before a run is irrecoverable. Tie those signals to controlled escalation instead of uncontrolled retries.
• Report coverage alongside cost Track how many task classes each model can solve reliably, not just the mean cost per success. A cheap model with narrow coverage can save money while breaking product capability.

Key takeaways

• The article shows that token price alone is a weak proxy for real AI operating cost because retries, failures, and timeouts change the economics.
• Routing by task difficulty can improve both cost and coverage, but only when the ladder is designed around measured workload fit.
• Coverage and unit cost are separate decisions, so teams should report both instead of treating the cheapest model as the best default.

Key terms

• Cost Per Task: Cost per task is the total expense required to complete one discrete unit of work. It is calculated by combining labor, infrastructure, software, and overhead costs, then dividing by the number of tasks completed. In identity and security operations, it helps measure efficiency, automation impact, and operational scalability.
• Retry Tax: The extra spend created when a model needs multiple attempts to finish one task. It is a practical way to express hidden inefficiency in agentic workflows, where low per-attempt pricing can still produce expensive outcomes if the system frequently repeats work or burns budget before succeeding.
• Task Routing: The practice of sending different work items to different models based on difficulty, risk, or required capability. In AI operations, routing is a policy decision as much as a performance tactic because it controls which systems may handle which classes of tasks and when escalation is allowed.
• Coverage: Coverage is the proportion of the relevant asset estate that a security tool or process can actually see and track. In ASM, coverage quality is more important than raw discovery volume because incomplete or stale views distort prioritisation and ownership decisions.

Deepen your knowledge

NHI Mgmt Group's NHI Foundation Level course covers NHI governance, agentic AI identity, machine identity security, IAM, and secrets management. It helps practitioners connect identity controls to the operational realities of autonomous and semi-autonomous systems.