Notifications
Clear all
Topic starter
18/08/2026 2:18 pm
TL;DR: Qwen3.8-Max looks cheaper on paper, but TruFoundry's check found it cost more per completed task than GPT-5.6 Sol because it used far more tokens on harder prompts. The practical lesson is that model-selection decisions should optimise for cost per correct answer, not advertised token price.
NHIMG editorial — based on content published by TruFoundry: We Checked Alibaba's Math: Qwen3.8-Max's Price Advantage Doesn't Survive Contact With Real Tasks
By the numbers:
- Qwen3.8-Max completed the hard tasks with an average of 5,364 tokens per answer, compared with 729 for Kimi K3 and 282 for GPT-5.6 Sol.
- GPT-5.6 Sol landed at $0.0067 to $0.0070 per solved task, while Qwen3.8-Max landed at $0.0144 to $0.0284.
Questions worth separating out
Q: How should teams choose between AI models when per-token pricing looks misleading?
A: Choose models on solved-task cost, not token price alone.
Q: Why do AI gateway metrics matter for model governance?
A: Gateway metrics show what actually happened during execution, including spend, latency, completion length, retries, and tool usage.
Q: What do security teams get wrong about AI cost control?
A: They often treat cost as a finance-only issue and overlook the identity layer that drives usage.
Practitioner guidance
- Benchmark on solved-task cost Compare models using complete task success, latency, and total token consumption on your own workload.
- Instrument the AI gateway as the control plane Capture per-request spend, completion length, retries, and downstream tool calls in the gateway so routing decisions reflect actual production behaviour.
- Set bounded-output policies for hard tasks Define maximum completion lengths, retry thresholds, and tool-step limits for workflows that are prone to verbose or spiralling responses.
What's in the full article
TruFoundry's full analysis covers the operational detail this post intentionally leaves for the source:
- The full task-by-task benchmark breakdown, including the hand-written test suite and run-by-run variance.
- Per-model token consumption patterns on the harder prompts, which show why cost curves diverged.
- The raw timing and output-length comparisons that explain the 5-minute versus blink-level gap.
- Context on how the TrueFoundry AI Gateway was used to route and observe the models during testing.
👉 Read TruFoundry's analysis of Qwen3.8-Max pricing versus real task cost →
Model pricing vs real task cost: what should teams optimise?
Explore further
18/08/2026 3:11 pm
Cost-per-correct-answer is the only model selection metric that survives contact with production. Per-token pricing is a procurement signal, not an operational control. If a model uses 10x or 20x more tokens to complete the same task, the apparent discount is irrelevant to platform economics and can mislead governance teams. Practitioners should evaluate model choice as a unit-cost problem tied to successful execution, not a pricing brochure problem.
A question worth separating out:
Q: How can organisations keep AI workflows from becoming expensive to govern?
A: Set explicit limits on output length, retry behaviour, and tool-step depth, then monitor whether those limits hold in production. If a workflow regularly exceeds its bounds, treat that as a governance issue, not just a tuning problem. The goal is predictable execution cost and predictable control exposure.
👉 Read our full editorial: Token cost per correct answer matters more than model price
