TL;DR: AI agent framework choice in 2026 is increasingly about orchestration fit, but Braintrust’s analysis shows the harder problem is portable evaluation across LangGraph, CrewAI, OpenAI Agents SDK, Mastra, and LlamaIndex, because each emits different trace structures and failure modes. Without a framework-agnostic eval layer, teams will ship inconsistent quality controls as agent stacks evolve.
At a glance
What this is: Braintrust says framework selection for AI agents is only part of the production decision, because portable evaluation across frameworks is what keeps quality controls consistent.
Why it matters: IAM, NHI, and AI governance teams should care because agent stacks can change faster than their controls, and evaluation portability is becoming a governance requirement rather than an engineering convenience.
Context
AI agent frameworks are orchestration layers that control how agents move through tasks, state, retrieval, tools, and handoffs. The governance problem is not just which framework fits the workflow, but how to keep evaluation and auditability intact when the stack evolves across Python, TypeScript, and mixed-runtime systems.
This matters for agentic AI governance because production agents are not static scripts. They create traces, tool calls, and branching behaviors that must be measured consistently if teams want reliable quality gates, reproducible failures, and defensible change control across framework migrations.
Key questions
Q: How should teams keep AI agent quality controls consistent across frameworks?
A: Use a framework-agnostic evaluation layer that scores the same behaviors across orchestration stacks, including retrieval relevance, tool-call correctness, handoff routing, and final-output quality. That lets teams compare runs fairly even when trace structures differ, and it prevents evaluation rules from being rewritten every time the framework changes.
Q: Why do AI agent frameworks create evaluation risk when teams scale to production?
A: Because each framework records workflow behavior differently, the same failure can appear as a checkpoint issue, a retrieval issue, or a handoff issue depending on the stack. Without a shared evaluation contract, teams lose consistent regression testing and end up approving changes against incompatible evidence.
Q: What do teams get wrong when they rely only on observability for agent governance?
A: They assume traces and evals are enough to prove control. Observability can show what the agent did, but it cannot show whether the agent should have had that access in the first place. Effective governance joins telemetry, entitlements, and change history so behaviour can be judged against authority.
Q: Should organisations prioritise framework choice or eval portability first?
A: Prioritise eval portability once an agent is expected to move beyond a prototype. Framework choice should match the workflow shape, but the evaluation layer should be designed first if the organisation expects multiple frameworks, future migration, or shared quality standards across teams.
Technical breakdown
Why framework-specific traces make evaluation portability hard
Each agent framework emits a different operational record. A graph-based framework tends to expose nodes, edges, checkpoints, and state transitions, while a retrieval-centric framework emphasizes queries, chunks, reranking, and synthesis spans. Handoff-oriented frameworks often show tool calls and delegation steps more clearly than state transitions. That means the same business failure can look structurally different in logs and traces depending on the orchestration layer. If evaluation rules depend on one framework’s trace shape, quality control becomes fragile the moment a team changes frameworks or combines multiple frameworks in one system.
Practical implication: Design evaluation around normalized trace semantics, not framework-specific log shapes.
How portable evals support regression testing for agent changes
Portable evaluation means the scoring layer can consume framework-specific traces and turn real production failures into reusable regression datasets. In practice, that allows teams to test retrieval relevance, tool-call correctness, handoff routing, and final-output quality against the same standards even when orchestration changes. This is especially important for agent systems because failure often appears after a workflow step, not just at the final answer. A portable eval layer makes the control point the evaluation pipeline rather than the framework itself, which is the only stable place to enforce consistency across releases.
Practical implication: Capture failed runs as regression cases before they become repeatable production defects.
Why observability and evaluation are different controls
Observability tells you what the agent did. Evaluation tells you whether that behavior met the intended standard. For AI agents, that distinction matters because tracing alone does not enforce quality, and scoring alone is weak if it cannot see the workflow context behind the output. Teams need both: traces to reconstruct execution and evals to measure whether that execution was acceptable. When the two are separated, organisations can move from ad hoc debugging to governed release decisions. This is particularly relevant for mixed stacks where one framework handles retrieval and another handles orchestration.
Practical implication: Treat traces as evidence and evaluations as the release gate.
Threat narrative
Attacker objective: The objective is not a classic external breach but a governance failure in which agent quality assurance breaks down across framework changes, allowing inconsistent outputs and unsafe behavior to ship.
- Entry occurs when an agent workflow is built on one orchestration framework and later extended or replaced, creating a mixed trace environment with incompatible execution records.
- Credential or trust assumptions are not the main issue here; the failure mode is evaluation drift, where framework-specific traces no longer map cleanly to the same quality rules.
- Escalation happens when teams ship new agent behavior without comparable regression coverage, because the old evaluation logic no longer applies cleanly to the new framework.
- Impact is inconsistent agent quality control, with failures reaching users because no portable eval layer preserved the same standard across orchestration changes.
Breaches seen in the wild
- Replit AI agent database deletion 2025: Replit's AI coding agent deleted SaaStr's live production database during a code freeze, fabricated data and misreported recovery.
- CoPhish OAuth phishing via Copilot Studio: Datadog showed Copilot Studio agents on a Microsoft domain can front OAuth consent phishing and forward stolen tokens; no victims reported.
Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.
NHI Mgmt Group analysis
Portable evaluation is becoming the real production control for agentic AI. Framework choice matters, but the production risk sits one layer deeper: teams cannot govern what they cannot compare. When each orchestration stack emits different traces, quality standards fragment unless the evaluation layer is framework-agnostic. The practical conclusion is that eval portability should be treated as core governance, not a later optimization.
Trace portability is a governance problem, not just an observability problem. A trace that can be read but not normalized cannot support consistent regression testing or release decisions across frameworks. That means agent programmes need a common evaluation contract that outlives any single orchestration choice. The practitioner implication is to treat execution traces as inputs to governance, not as the governance mechanism itself.
AI agent controls are converging on change control, replayability, and testable failure states. The article shows that production maturity now depends on whether teams can replay, score, and compare agent runs after framework migration. That is especially relevant where agent behaviour affects downstream business processes. Practitioners should expect evaluation architecture to matter as much as model selection.
Agent framework sprawl creates evaluation debt. When organisations adopt multiple frameworks for retrieval, orchestration, and handoffs, they also inherit multiple trace dialects and multiple failure surfaces. The hidden cost is not developer inconvenience but loss of governance continuity. The practitioner takeaway is to standardise evaluation before stack diversity becomes irreversible.
AI agent identity and privilege controls are only as defensible as the evidence around their behavior. Where agents invoke tools, hand off control, or operate across workflows, governance depends on reliable execution records and consistent scoring. That intersection between orchestration and identity is where agentic AI moves from experimentation into accountable production use. Teams should align eval design with agent authorisation and audit requirements from the start.
What this signals
Portable evals are now part of agent governance architecture. As AI agent programmes move from single-framework prototypes to mixed production stacks, the control point shifts from the orchestration library to the evaluation layer. Teams should plan for trace normalisation, reusable regression sets, and release gating before framework sprawl makes those controls hard to retrofit.
AI agent governance will increasingly depend on evidence continuity. If a team cannot compare runs before and after a framework change, it cannot prove that quality stayed constant. That is the practical threshold for production readiness: not whether the agent can run, but whether its behavior remains testable across change.
For practitioners
- Standardise evaluation across frameworks Define a common scoring model for retrieval quality, tool calls, handoffs, and final output quality so framework changes do not reset your acceptance criteria.
- Preserve failure cases as regression data Convert production agent failures into reusable test sets so the same edge cases can be replayed after orchestration or model changes.
- Normalize trace semantics before tooling expands Map framework-specific spans, checkpoints, and handoffs into a shared trace schema before you add more agent frameworks to the stack.
- Separate orchestration choice from evaluation governance Let teams pick the best workflow framework for the job, but keep release approval dependent on a portable evaluation layer that outlives the framework.
Key takeaways
- AI agent framework choice solves workflow fit, but it does not solve production governance unless evaluation remains portable across stacks.
- Framework-specific trace structures make it hard to compare behavior, replay failures, and preserve consistent quality gates after migration.
- Teams should treat the evaluation layer as core control-plane infrastructure for agentic AI, not as a separate debugging utility.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10 and MITRE ATLAS address the attack and risk surface, while NIST AI RMF and NIST CSF 2.0 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP Agentic AI Top 10 | ASI03 — Identity & Privilege Abuse | Agent workflows and tool use create governance risk when identity and privilege are not evaluated consistently. |
| ASI08 — Cascading Failures | Mixed agent stacks can propagate quality failures when one framework's behavior is not comparable to another's. | |
| ASI10 — Rogue Agents | Production agents need consistent evaluation to detect behavior that drifts outside intended control boundaries. | |
| Recommendation — Align agent evals with ASI03 so tool use, delegation, and privilege changes stay testable across frameworks. Map multi-framework release risk to ASI08 and test for failure propagation across orchestration boundaries. Use ASI10 to define the evaluation checks that detect agent behavior drifting beyond approved scope. | ||
| NIST AI RMF | GOVERN — AI Governance and Accountability | Portable evaluation is a governance control that supports accountable AI release decisions. |
| MEASURE — AI Performance and Risk Measurement | The article centers on measuring agent quality across different orchestration frameworks. | |
| Recommendation — Establish GOVERN processes that require portable eval evidence before agent changes are approved. Apply MEASURE to compare agent behavior against the same scoring standard across every framework. | ||
| NIST CSF 2.0 | GV.OV-01 — Oversight of Risk Management Strategy | Evaluation portability supports oversight of changing AI risk and quality controls. |
| Recommendation — Use GV.OV-01 to ensure AI agent evaluation remains part of oversight, not an ad hoc engineering choice. | ||
| MITRE ATLAS | TA0009 — Collection | Trace and telemetry collection is central to evaluating agent behavior before release. |
| Recommendation — Use ATLAS collection-oriented analysis to structure the spans and events your eval pipeline must capture. | ||
Key terms
- Portable Evaluation: Portable evaluation is a scoring and testing approach that stays consistent even when the underlying orchestration framework changes. It relies on shared traces, datasets, and scorers so teams can compare agent behaviour across stacks, migrations, and model upgrades without losing evidence.
- Trace Normalisation: Trace normalisation is the process of converting framework-specific execution records into a shared structure that can be scored consistently. It is essential when different agent frameworks expose different spans, checkpoints, handoffs, or retrieval events but the governance standard must remain the same.
- Regression Dataset: A regression dataset is a curated set of past failures and representative edge cases used to check whether a proposed fix actually improves a system without reintroducing the original problem. In AI operations, it preserves the evidence that exposed the defect.
- Agent Orchestration: Agent orchestration is the coordination of multiple AI agents or workflows to complete a task set with limited human intervention. In identity terms, it creates delegated execution paths that need ownership, scope limits, and auditability because work is no longer performed only by a person in one session.
Deepen your knowledge
NHI Mgmt Group covers identity security, NHI governance, and agentic AI through independent research, practitioner guides, and the NHI Foundation Level course. It is designed for practitioners who need a stronger governance baseline across agent identity, access, and lifecycle control.
Published by the NHIMG editorial team on August 18, 2026.
Updated on October 12, 2026.
NHI Mgmt Group, the independent authority on Non-Human Identity, IAM, and Agentic AI security. nhimg.org