Braintrust’s GPT-5.6 eval shows routing matters more than raw model size

At a glance

What this is: This Braintrust eval compares GPT-5.6 family models and Anthropic models on exact, code-graded tasks, finding that task type drives routing decisions more than generic benchmark scores.

Why it matters: It matters because IAM-adjacent automation, agent workflows, and other security operations increasingly depend on small deterministic model steps where one formatting miss can break downstream control logic.

By the numbers:

• Braintrust tested 225 procedurally generated, code-graded tasks across three task families and three difficulty levels.
• Sol and Terra were both around 83% overall, while Luna trailed at about 68% in the GPT-5.6 family.

👉 Read Braintrust's evaluation of GPT-5.6 family routing for exact tasks

Context

Structured model evaluation matters because many AI failures are not about reasoning in the abstract. They are about exact outputs, strict formats, and deterministic handoffs where a single missing field or wrong number can break the next control in the chain. For security teams, that makes routing and reliability more important than model branding.

The article’s core finding is that model choice should be driven by task family, not by a blanket assumption that one flagship is always the right answer. That is directly relevant to agentic AI governance, where models increasingly act as subcomponents inside automated workflows and need to be selected, tested, and monitored by function rather than reputation.

Key questions

Q: How should teams route models for exact, machine-checked tasks?

A: Route by task family, difficulty, and acceptable error cost. Use cheaper models for structured transforms only when they consistently clear the quality bar, then reserve higher-capability models for rule-heavy or high-consequence steps. The decision should be driven by measured solve rate, refusal behaviour, latency, and fallback cost, not by general benchmark reputation.

Q: Why do model rankings change so much across task types?

A: Because model strengths are not uniform across all work. A model that performs well on structured data transforms may be weaker on symbolic rule-following or reconciliation tasks, while a slower flagship may buy reliability on the hardest cells. Security teams should therefore evaluate the actual workload, not an average score that hides where failures occur.

Q: What do security teams get wrong about AI model evaluation?

A: They often collapse quality into a single score and ignore output format, refusals, and latency. In real workflows, those factors determine whether the result can be consumed safely by downstream systems. Good governance treats evaluation as an operational control, with separate thresholds for correctness, completion, and timing.

Q: How should teams decide whether a cheaper model is good enough?

A: Compare unique vulnerabilities found per dollar, not just per-run recall. If a mid-tier model repeated three times matches or beats a flagship once, it may be the better operational choice. The real decision is about stable yield, analyst effort, and how much variance your programme can absorb.

Technical breakdown

Exact-match scoring changes how model quality should be judged

This eval uses code-checked exact match, so a model only succeeds if its output is both correct and usable in the required format. That is a different standard from loose semantic scoring, and it mirrors production workflows where downstream systems parse the result directly. In practical terms, a model that “knows” the answer but formats it incorrectly still creates operational failure. That makes task design, output schemas, and retry logic part of governance, not just engineering convenience.

Practical implication: validate models against machine-consumable outputs, not just human-judged answers.

Task family matters more than headline model rank

The eval separates arithmetic, symbolic rules, and data transforms because these are not interchangeable workloads. Data transforms are often tolerant of cheaper models, while symbolic rules expose strict instruction-following limits and arithmetic can fail in unexpectedly hard reconciliation patterns. This is why a single enterprise model standard often underperforms in agent pipelines. The right control is routing by work type, then measuring whether the chosen model still meets the reliability bar under realistic constraints.

Practical implication: route exact tasks by family and difficulty instead of standardising on one model for everything.

Refusals can distort apparent model performance

The Anthropic results show that low headline accuracy can reflect refusal behaviour rather than weak problem solving. On completed attempts, the ranking changes materially, which means security and governance teams should not treat failure rates as a single scalar. In agentic systems, refusal, malformed output, and wrong output are different failure modes with different risk implications. Governance should therefore track both answer quality and response completion, especially where models support operational workflows or automated controls.

Practical implication: separate refusals from incorrect answers when setting acceptance thresholds and escalation rules.

NHI Mgmt Group analysis

Task-level model routing is becoming an identity-adjacent governance problem. As AI systems move into operational workflows, the model is no longer just a text generator. It becomes a decision component inside chains that may touch access reviews, policy triage, and automated evidence handling. That means the control question is not only which model is best, but which model is acceptable for each step of a governed workflow. The practitioner conclusion is that routing policy now belongs in the control plane.

Exact-output work exposes the weakness of “one model fits all” thinking. The eval shows that structured tasks reward different capabilities than open-ended reasoning, and that cost-efficient models can be adequate for some subtasks while failing hard on others. This is especially relevant where AI agents operate as software entities with delegated actions. The named concept here is routing fragility: the gap between apparent capability and the narrow reliability required by a specific workflow. The practitioner conclusion is to classify tasks before assigning models.

Latency is a control variable, not just an operations metric. The faster models in the eval are not simply cheaper substitutes; they define the feasible shape of a workflow. When response time changes retry behaviour, orchestration design, or the point at which a human has to intervene, latency becomes part of security and resilience planning. That matters for AI-assisted SOC, GRC automation, and identity operations alike. The practitioner conclusion is to budget latency as part of workflow risk.

Benchmarks for agentic systems should be evaluated as control evidence, not marketing evidence. The article’s split between solve rate, cost per call, and latency is the right direction because it aligns with operational governance rather than abstract model scoring. For teams responsible for agentic AI, the question is whether a model can be trusted in a specific role, under specific error conditions, with auditable outputs. The practitioner conclusion is to require role-specific acceptance criteria before deployment.

What this signals

Model routing will become part of AI governance baselines as soon as organisations embed models inside controlled workflows. The practical challenge is to define which tasks can tolerate lightweight models and which tasks require higher assurance, then enforce that policy through orchestration.

Routing fragility: the next failure mode for agentic AI is not always model hallucination. It is selecting a model that is only apparently capable of a task, then discovering too late that exact output, timing, or completion reliability was outside tolerance.

For identity and security programmes, the takeaway is to treat model selection as a governance decision with measurable control objectives. That is where frameworks such as the NIST AI Risk Management Framework and OWASP Agentic AI Top 10 become operational rather than theoretical.

For practitioners

• Define routing rules by task family Separate exact, structured work such as arithmetic, rule following, and data transforms from open-ended tasks, then assign acceptance thresholds per family rather than per vendor model. Use the cheapest model that clears the bar for the specific workload.
• Track refusals separately from wrong answers Measure refusal rate, malformed output rate, and incorrect-answer rate as distinct control signals so you can tell whether a model is failing safely or simply failing to produce usable output.
• Build retry logic around hard cells Assume some task and difficulty combinations will not clear your reliability bar on the first pass. Where exact outputs matter, add bounded retries, schema validation, and fallback routing before the workflow depends on the result.
• Set latency budgets for orchestration steps Treat response time as part of workflow design, especially where the model sits inside a multi-step agent or decision pipeline. If a slower model changes the human handoff point or retry window, it changes operational risk.

Key takeaways

• Exact model evaluation changes the decision from “which model sounds best” to “which model reliably completes this workflow step.”
• Task family, latency, and refusal behaviour explain more of the performance gap than broad benchmark headlines do.
• Agentic AI governance should route and measure models as controlled workflow components, not as interchangeable chat interfaces.

Key terms

• Exact-Match Scoring: A scoring method that counts only perfectly correct outputs as success. In structured model evaluation, any missing field, wrong number, or malformed format is a failure because downstream systems usually need machine-consumable precision, not approximate correctness.
• Task-Family Routing: The practice of assigning different models to different classes of work based on measured performance. It recognises that arithmetic, rule following, and structured transformation have different reliability profiles, so model choice should follow the task rather than a single global ranking.
• Refusal Behaviour: The tendency of an AI model to decline requests that it judges risky, harmful, or policy-violating. Refusal is useful in production, but it can limit safety research if the same model is expected to generate the scenarios needed to test itself.
• Routing Fragility: The gap between a model that appears capable in general and one that is reliable enough for a specific workflow step. It becomes visible when exact output, timing, or completion requirements are stricter than the model’s average benchmark performance.

What's in the full report

Braintrust's full blog post covers the operational detail this post intentionally leaves for the source:

• The per-cell decision map showing which model clears the reliability bar for each task family and difficulty level.
• The latency and cost breakdown behind the routing recommendations, including the trade-off between Sol and Terra for time-sensitive paths.
• The statistical model used to test whether Terra is meaningfully worse than Sol, which matters if you need evidence for model selection policy.
• The refusal analysis that separates non-answers from incorrect answers, useful when building acceptance criteria for automated workflows.

👉 Braintrust's full post covers the task-by-task decision map, refusal analysis, and latency comparisons.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management in the contexts that security teams actually manage. It helps practitioners align identity controls with the operational reality of automated systems and delegated workflows.