Braintrust’s GPT-5.6 eval shows routing matters more than raw model size

At a glance

What this is: Braintrust’s eval shows that task family, not just model size, determines which GPT-5.6 model is best for exact, machine-checkable work.

Why it matters: For IAM and agentic AI teams, the finding matters because routing decisions, not only model choice, determine reliability when exact outputs, structured data, and deterministic substeps affect control quality.

Context

This evaluation is about routing exact, structured tasks to the right model, not about open-ended reasoning or creative output. When outputs must match a required format exactly, small failures in arithmetic, rule-following, or structured transformation can break an agent workflow just as surely as a bad credential can break an access flow. That makes model routing a governance problem, not just a model-quality question.

The identity angle is indirect but real: agentic systems often use models as decision steps inside workflows that depend on reliable output, predictable latency, and bounded retries. If the orchestration layer cannot distinguish between a task that needs a flagship model and one that can be handled cheaply, it creates operational risk through overconfidence in the wrong substep.

The article’s starting position is typical for production AI programmes: teams often over-index on broad benchmark scores and underweight the exact task family that actually fails in deployment.

Key questions

Q: What breaks when an agent routes exact tasks to the wrong model?

A: The workflow breaks when the task needs exact structure, ordered rules, or a single correct value and the chosen model cannot reliably satisfy that contract. In practice, the failure shows up as malformed output, repeated retries, or silent downstream errors. Routing has to reflect task fragility, not just general capability.

Q: Why do refusal rates matter as much as accuracy in model routing?

A: Because a refusal is a failed task even when the model might have been capable of producing a correct answer. In production, refusals create the same operational burden as wrong outputs if the workflow depends on immediate completion. Teams should measure accepted work, not just correctness on completed work.

Q: How should security teams decide whether a cheaper AI model is worth using for cyber work?

A: Teams should compare models by cost per successful outcome, not by raw benchmark score or lowest price. A cheaper model can become expensive if it needs many more iterations, produces weaker results, or increases analyst review time. The right test is whether the model delivers reliable value inside the workload and budget band you actually operate in.

Q: How should organisations split planner and executor roles in agentic AI?

A: Use the planner to decompose work and choose the right path, then assign executors based on the output burden of each subtask. Exact, brittle steps may need a stronger model, while routine transforms can often use a cheaper one. The split should be validated against observed failure modes, latency, and retry cost.

Technical breakdown

Why exact-match scoring changes the model decision

This eval uses machine-graded outputs with no partial credit, which is the right lens for tasks where downstream software consumes the response directly. Exact-match scoring treats formatting, field completeness, and ordering as part of the control surface, not cosmetic details. That matters because many agent substeps are not about being approximately right. They are about emitting one valid answer, one valid object, or one valid rewrite. In those settings, a model that reasons well but misses the required structure is operationally wrong. The routing implication is simple: evaluate models against the failure mode your workflow actually punishes, not against general intelligence scores.

Practical implication: Benchmark models on the exact output contract your workflow enforces, not on broad benchmark prestige.

Task-family routing beats one-size-fits-all model selection

The results separate cleanly by task family. Data transforms are comparatively forgiving, symbolic rules are much stricter, and arithmetic behaves differently again because some superficially easy tasks are actually multi-field reconciliations. That means the right model choice depends less on a single overall score and more on the combination of structure, determinism, and tolerance for retries. In practice, routing is a control plane for cost, latency, and reliability. The best model for one subtask can be the wrong model for the next if the output shape changes. For agentic systems, this is a governance issue because the planner must know when a cheap executor is good enough and when exactness requires escalation.

Practical implication: Build routing rules by task family and difficulty class, not by one global model ranking.

Refusals can distort the apparent quality picture

The Anthropic rows in this eval show why raw headline scores can mislead. A model can look weak overall because it refuses a large share of tasks, even while it performs well on the attempts it does complete. That creates a governance problem for production routing: refusal rate is not a side metric, it is part of the effective service level. In an agent workflow, a refusal can be as disruptive as a wrong answer if the orchestration layer does not know how to recover. The operational question is not just whether a model can answer, but how often it will accept the work and keep the workflow moving.

Practical implication: Treat refusal rate as a first-class routing metric alongside accuracy and cost.

NHI Mgmt Group analysis

Task routing is becoming a governance control, not an engineering convenience. When exact outputs feed downstream systems, the question is no longer which model is smartest in the abstract. It is which model can satisfy the specific contract of the subtask without forcing retries, manual correction, or workflow collapse. That shifts routing into the same category as policy enforcement. Practitioner conclusion: govern model selection as part of control design, not just prompt design.

Structured work exposes a model’s real operating boundary. Benchmarks that mix open-ended and exact tasks blur the line between capability and suitability. This evaluation shows that a model can be broadly strong and still be a poor fit for symbolic rules or strict transforms. That is especially relevant for agentic AI, where the planner may decompose work into steps that look similar but have different error costs. Practitioner conclusion: classify substeps by failure tolerance before assigning a model.

Latency and refusal behaviour are part of the same reliability problem. A fast model that refuses less often may outperform a slower flagship in production even if the flagship is stronger on paper. The reverse is also true when a task is brittle and one wrong field matters more than response time. That makes reliability a multi-variable decision that spans model quality, response time, and workflow recovery design. Practitioner conclusion: define the acceptable tradeoff per subtask, not per vendor model.

Exactness, not just capability, should shape agent architecture. In agentic systems, the planner and executor should not be assumed to share the same quality threshold. One model may be best at deciding the next step while another is best at emitting the exact artifact. That means organisations need separate governance for orchestration quality and execution quality. Practitioner conclusion: split evaluation by role and measure each role against its actual output burden.

What this signals

Exactness routing is the new control point: when model outputs feed software rather than humans, the governance question shifts from “which model is best” to “which model can satisfy this task contract every time.” That is especially relevant in agentic workflows where one substep may need deterministic formatting while another can tolerate approximation.

Named concept: routing reliability budget. The practical limit is not only how capable a model is, but how much retry, refusal, and latency overhead a workflow can absorb before the agent becomes operationally brittle. Teams should budget reliability at the subtask level and treat recovery design as part of model governance.

For practitioners

• Define routing thresholds by task family Set explicit reliability bars for arithmetic, symbolic rules, and data transforms so model selection follows task risk rather than general benchmark rank.
• Measure refusal rates separately from accuracy Track non-answers, refusals, and malformed outputs as distinct failure modes because a model that declines work can still disrupt production workflows.
• Use a flagship model for strict rule-following Route symbolic or order-sensitive steps to the strongest model when one wrong token or missing field would invalidate the whole downstream result.
• Reserve cheaper models for forgiving transforms Assign lower-cost models to data reshaping and other structured tasks only when the output contract tolerates low retry cost and minimal ambiguity.
• Test latency against workflow recovery Measure end-to-end response time together with retry overhead so you know whether a slower but more accurate model actually improves completion time.

Key takeaways

• Exact-output work exposes the gap between benchmark performance and production suitability, because a model that is broadly capable can still fail a workflow by missing format or order requirements.
• Task family is the decisive variable in routing decisions, with forgiving transforms, brittle symbolic rules, and mixed arithmetic reconciliations each demanding different thresholds.
• The operational answer is to set subtask-specific reliability bars, track refusals separately, and reserve flagship models for the exact steps where retries are expensive or unacceptable.

Key terms

• Model Routing Layer: The model routing layer is the policy and orchestration logic that decides which model handles a request, when to escalate, and what tools the request can reach. In AI programmes, it behaves like a control plane because it shapes data exposure, privilege boundaries, and auditability.
• Exact-Match Scoring: A scoring method that counts only perfectly correct outputs as success. In structured model evaluation, any missing field, wrong number, or malformed format is a failure because downstream systems usually need machine-consumable precision, not approximate correctness.
• Refusal Rate: The share of prompts a model declines to answer. In AI security evaluation, it is a useful but incomplete signal because a model can refuse unsafe tasks while still failing to produce correct answers on permitted tasks.
• Structured Output Contract: A structured output contract is the required format, fields, and ordering that a model must satisfy for its response to be usable. These contracts matter whenever software, not a human, consumes the output, because the wrong shape can fail even if the content is semantically close.

Deepen your knowledge

The NHI Foundation Level course covers NHI governance, agentic AI identity, and workload identity as part of the industry's only accredited NHI security programme. It helps practitioners align identity controls with the realities of agentic systems, model routing, and operational governance.