Topic starter
18/08/2026 2:47 pm
TL;DR: Cost per successful task can diverge sharply from token price, and routing by task difficulty can beat single-model buying decisions, according to Arize and Fireworks benchmarking 10 AI models across 40 real agent tasks and 2,400 runs. The lesson is operational: measure success end to end, then route and escalate deliberately.
Editorial analysis by NHI Mgmt Group, based on content published by Arize: “Cost per successful task: Benchmarking Kimi K3, GPT-5.5, and 8 more AI models”.
By the numbers:
- The benchmark used 6 trials per task-model cell, creating 2,400 runs in total.
- The study recorded $626 of API spend across the benchmark.
- gpt-oss-120b finished a task for $0.054 while passing only 33% of the time.
Key questions
Q: What breaks when model routing is based only on token price?
A: Token-only routing hides retries, tool failures, and timeout loops, so the cheapest-looking model can become expensive once you count failed attempts.
Q: When should teams switch from a frontier model to a lower-cost model?
A: Teams should switch when the cheaper model can meet the same task success threshold on their own evaluations.
Q: What are the signs that an AI routing ladder is failing?
A: Common signs include repeated retries, budget exhaustion, non-zero tool exits, and a rising cost per success even when token prices stay flat.
Practitioner guidance
- Define success at the task level Use outcome-based evaluation, not token counts, as the primary measure for AI agent economics.
- Build a difficulty-based routing policy Classify requests into workload bands and send each band to the cheapest model that can complete it reliably.
- Instrument escalation with trace signals Use telemetry such as repeated commands, non-zero tool exits, schema failures, and budget exhaustion to identify distress before a run is irrecoverable.
Bottom line: The article shows that token price alone is a weak proxy for real AI operating cost because retries, failures, and timeouts change the economics.
Explore further
View Full Forum → | NHI Foundation Course → | Our Services → | Read the full analysis →
18/08/2026 3:30 pm
Cost per successful task is becoming the right governance unit for agentic AI. Token price is a procurement metric, not an operating metric. Once retries, tool calls, and verification overhead are included, the economic picture changes enough to alter model choice, routing policy, and control design. Practitioners should treat task success as the governed outcome, not tokens as the governed input.
A few things that frame the scale:
- 13% of organisations suffered breaches of their AI models, according to IBM's Cost of a Data Breach 2025 report.
A question worth separating out:
Q: How should security and AI teams govern escalation between models?
A: Govern escalation as a controlled decision with clear triggers, measured fallback behaviour, and an outcome definition for success. If you can verify results directly, use that check. If you cannot, rely on trace-based distress signals and task-class routing so escalation adds capability without turning into blind spend.
👉 Read our full editorial: Cost per successful task is the right metric for AI model routing