Notifications
Clear all
Topic starter
18/08/2026 2:47 pm
TL;DR: Cost per successful task can diverge sharply from token price, and routing by task difficulty can beat single-model buying decisions, according to Arize and Fireworks benchmarking 10 AI models across 40 real agent tasks and 2,400 runs. The lesson is operational: measure success end to end, then route and escalate deliberately.
NHIMG editorial — based on content published by Arize: Cost per successful task: Benchmarking Kimi K3, GPT-5.5, and 8 more AI models
By the numbers:
- The benchmark used 6 trials per task-model cell, giving roughly a ±6 point 95% confidence interval on pass rates.
- The study recorded $626 of API spend across the benchmark runs.
Questions worth separating out
Q: How should teams decide which AI model to use for a workflow?
A: Teams should choose models based on cost per successful task, not on token price alone.
Q: Why is routing AI tasks across multiple models often better than using one model everywhere?
A: Routing works because different tasks need different capability levels.
Q: What signals show that an AI model is becoming too expensive to run in production?
A: Look for repeated retries, budget exhaustion, identical tool calls, rising latency, and a growing gap between attempts and successful completions.
Practitioner guidance
- Benchmark on completed work, not token spend Measure cost per successful task across your own representative workloads, including retries, failures, and timeout cases, before selecting a production model tier.
- Build a task-class routing policy Route simple, verifiable work to lower-cost models and reserve frontier models for tasks that demonstrably need higher capability or broader coverage.
- Instrument model traces end to end Capture token counts, tool calls, latency, retries, and failure reasons so you can distinguish budget exhaustion from silent wrong answers.
What's in the full report
Arize's full article covers the operational detail this post intentionally leaves for the source:
- Full benchmark tables for all 10 models across the 40-task harness, including pass-rate spread and cost per successful task.
- The routing and escalation simulation logic used to compare single-model versus multi-model strategies.
- Trace-level observations showing where retries, token caps, and silent failures changed the economics.
- Task-by-task examples that show why easy work and hard work produce very different cost curves.
👉 Read Arize's benchmark of AI model routing and cost per successful task →
Cost per successful task in AI routing: what should teams optimise?
Explore further
18/08/2026 3:30 pm
Cost per successful task is becoming the more useful governance metric for AI operations. Token-based pricing encourages teams to optimise consumption, but not completion. In agentic workflows, that creates a blind spot because retries, tool failures, and timeouts are part of the real operating cost. For security and identity programmes, the same logic applies to automation: a control that looks cheap but fails repeatedly is not efficient. The practitioner conclusion is to govern for outcome cost, not input cost.
A question worth separating out:
Q: How do security and IAM teams govern AI routing decisions safely?
A: Treat routing rules like policy. Define which tasks each model tier may handle, when escalation is allowed, who can change the rules, and what evidence is required for high-risk actions. If AI systems can trigger access, workflow, or secret handling steps, routing becomes part of governance, not just optimisation.
👉 Read our full editorial: Cost per successful task is the right metric for AI model routing
