Cost per successful task is the right metric for AI model routing

At a glance

What this is: This benchmark shows that token price is a poor proxy for production value, while cost per successful task better captures retries, failures, and coverage across real agent work.

Why it matters: For IAM and security teams, the same lesson applies to AI governance and NHI controls: you need to measure end-to-end outcomes, not just the cheapest apparent unit cost, before deciding what to automate or trust.

By the numbers:

• Arize and Fireworks tested 10 open and closed models across 40 real agent tasks and 2,400 runs.
• The benchmark used 6 trials per task-model cell, giving roughly a ±6 point 95% confidence interval on pass rates.
• The study recorded $626 of API spend across the benchmark runs.

👉 Read Arize's benchmark of AI model routing and cost per successful task

Context

Cost per successful task is a better production metric than token price because it captures the full cost of retries, failures, timeouts, and judge overhead. In AI operations, the cheapest attempt is not always the cheapest outcome, and that distinction matters wherever models are used to complete business tasks rather than generate text.

For identity and security practitioners, the governance parallel is straightforward: a system that looks efficient in isolation can still be expensive, fragile, or risky when measured against actual completion. That is especially relevant as AI agents, workflow automation, and NHI-style service accounts increasingly depend on model selection, routing, and escalation decisions.

The article’s core finding is typical of mature operational benchmarking: apparent price leadership often collapses once success, coverage, and retry behaviour are measured together. That is the right starting point for teams evaluating AI in production.

Key questions

Q: How should teams decide which AI model to use for a workflow?

A: Teams should choose models based on cost per successful task, not on token price alone. The right model is the one that completes the task reliably at the lowest end-to-end cost. That means measuring retries, failures, timeouts, and coverage on the actual workload before you standardise a routing policy.

Q: Why is routing AI tasks across multiple models often better than using one model everywhere?

A: Routing works because different tasks need different capability levels. Cheap models can handle simple work efficiently, while harder tasks need frontier models with higher success rates. A deliberate routing ladder reduces wasted spend and preserves capability where it matters, instead of paying premium prices for every request.

Q: What signals show that an AI model is becoming too expensive to run in production?

A: Look for repeated retries, budget exhaustion, identical tool calls, rising latency, and a growing gap between attempts and successful completions. Those signs mean the model is burning compute without finishing work. Trace data is the most reliable way to see whether the hidden cost is scaling faster than value.

Q: How do security and IAM teams govern AI routing decisions safely?

A: Treat routing rules like policy. Define which tasks each model tier may handle, when escalation is allowed, who can change the rules, and what evidence is required for high-risk actions. If AI systems can trigger access, workflow, or secret handling steps, routing becomes part of governance, not just optimisation.

Technical breakdown

Why token price hides the real cost of AI work

Token price measures input consumption, not whether a model completed a task. Cost per successful task adds every attempt, including retries, failed tool calls, malformed outputs, and runs that end in timeout or budget exhaustion. That makes it a better unit for agentic workflows, where the model is not just generating text but trying to produce a verifiable outcome. In practice, a low per-token rate can still become expensive if the model churns through multiple attempts before succeeding. Practical implication: benchmark AI models on task completion economics, not just API price sheets.

Practical implication: Measure total spend against successful task completion before selecting a model for production routing.

How routing changes model economics in production

Routing works because not all tasks need the same capability. Some tasks are easy and should go to a lower-cost model, while harder tasks justify a frontier model with stronger success rates. The key is to separate task class from model choice, then escalate only when a lower rung cannot complete the work. This reduces waste without assuming a single model can serve every use case. The article shows that even a naive escalation ladder can outperform a single-model strategy if the ladder is designed carefully. Practical implication: classify tasks by difficulty and map each class to an appropriate model tier.

Practical implication: Use difficulty-based routing to reserve expensive models for the tasks that actually need them.

What traces reveal that averages cannot

Aggregate pass rates and averages conceal whether failures are loud or silent. Traces expose whether a model is exhausting its budget, repeating the same command, or finishing confidently but incorrectly. That distinction matters because the governance question is not only whether a model failed, but how it failed and whether the failure mode is detectable. For AI operations, tracing is the difference between a cost report and an audit trail. Practical implication: instrument model runs with trace-level telemetry so you can see retry tax, failure shape, and hidden cost drivers.

Practical implication: Add trace telemetry to distinguish recoverable churn from silent failure before you route more work.

NHI Mgmt Group analysis

Cost per successful task is becoming the more useful governance metric for AI operations. Token-based pricing encourages teams to optimise consumption, but not completion. In agentic workflows, that creates a blind spot because retries, tool failures, and timeouts are part of the real operating cost. For security and identity programmes, the same logic applies to automation: a control that looks cheap but fails repeatedly is not efficient. The practitioner conclusion is to govern for outcome cost, not input cost.

Task routing is now an identity-adjacent control problem, not only an AI procurement problem. Once AI systems are used to execute work, the question becomes which model gets which authority, with what escalation path, and under what verification. That makes model selection part of access governance for agentic systems, especially where AI actions touch secrets, workflows, or operational privileges. Practitioners should treat routing rules as policy, not just optimisation.

Coverage matters as much as cost, and that is where simplistic consolidation stories fail. A model can be cheap because it only solves a narrow slice of tasks, not because it is broadly efficient. That is analogous to identity programmes that celebrate low friction while leaving exception paths unmanaged. The named concept here is completion economics: the real unit of control is the cost to finish work successfully, including failure handling. Teams should measure capability coverage before standardising on any model tier.

Tracing is becoming the audit layer for AI decisioning. The benchmark’s traces show why failures happened, not just how often they happened. That matters for governance because models that burn budget, repeat tool calls, or fail silently create different risk profiles. In broader AI security and NHI governance, observability is no longer optional once machine actions affect business outcomes. Practitioners should require traceability for model routing, escalation, and exception handling.

Frontier capability still matters, but only where the task actually needs it. The study shows that hard work remains hard and that cheap models do not become capable through repeated attempts alone. That is the correct framing for enterprise AI adoption: use low-cost models where they reliably solve the task, but reserve frontier models for the hard tail. The practitioner conclusion is to separate baseline automation from escalated execution and govern both explicitly.

What this signals

Completion economics is the governance shift here: AI programmes need to measure the cost of finishing work, not the cost of starting it. That means routing logic, escalation thresholds, and traceability must sit alongside model selection as first-class controls, especially where AI touches secrets, workflows, or access decisions.

Teams should expect greater pressure to justify model choice with workload evidence rather than brand preference. The practical standard is moving toward trace-backed routing, where policy, observability, and verifiable outcomes determine which model handles which task.

For identity-led organisations, the broader implication is that AI execution paths are becoming policy surfaces. When a model can drive actions that affect accounts, credentials, or approvals, the control question is no longer only about model quality. It is about who can delegate what, to which system, under what audit trail.

For practitioners

• Benchmark on completed work, not token spend Measure cost per successful task across your own representative workloads, including retries, failures, and timeout cases, before selecting a production model tier.
• Build a task-class routing policy Route simple, verifiable work to lower-cost models and reserve frontier models for tasks that demonstrably need higher capability or broader coverage.
• Instrument model traces end to end Capture token counts, tool calls, latency, retries, and failure reasons so you can distinguish budget exhaustion from silent wrong answers.
• Define escalation criteria before production use Use deterministic checks where possible, and when checks do not exist, escalate on distress signals such as repeated tool failures or abnormal spend.
• Treat routing policy as governance Document who can change model tiers, what tasks each tier may handle, and which escalation paths are allowed for higher-risk workflows.

Key takeaways

• Token price is not a reliable indicator of AI production cost because retries, failures, and timeouts materially change the economics.
• Routing by task difficulty can reduce spend while preserving capability, but only if teams measure their own workloads instead of assuming labels tell the story.
• Traceability turns model selection from a procurement choice into a governable operational control, which is the right posture for agentic systems.

Key terms

• Cost per completed task: A workload metric that measures the total cost of finishing a task successfully, including retries, context use, and orchestration overhead. It is more useful than token price alone because it reflects the real operational cost of getting useful output from an AI system.
• Retry Tax: The extra spend created when a model needs multiple attempts to finish one task. It is a practical way to express hidden inefficiency in agentic workflows, where low per-attempt pricing can still produce expensive outcomes if the system frequently repeats work or burns budget before succeeding.
• Task Routing: The practice of sending different work items to different models based on difficulty, risk, or required capability. In AI operations, routing is a policy decision as much as a performance tactic because it controls which systems may handle which classes of tasks and when escalation is allowed.
• Silent Failure: A silent failure is a fault that does not crash the system but still corrupts output, state, or learning signals. These failures are dangerous in AI pipelines because dashboards can remain green while the model, trainer, or gateway drifts away from intended behaviour.

What's in the full report

Arize's full article covers the operational detail this post intentionally leaves for the source:

• Full benchmark tables for all 10 models across the 40-task harness, including pass-rate spread and cost per successful task.
• The routing and escalation simulation logic used to compare single-model versus multi-model strategies.
• Trace-level observations showing where retries, token caps, and silent failures changed the economics.
• Task-by-task examples that show why easy work and hard work produce very different cost curves.

👉 The full Arize article includes the task-level benchmark detail, routing simulation, and trace evidence behind the findings.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, secrets management, and machine identity security in operational terms. It is designed for practitioners who need to connect identity controls to real-world automation risk.