Framework-neutral tracing makes AI evals portable across stacks

At a glance

What this is: This is a best-practices post on using Braintrust to standardise tracing, evals, and debugging across heterogeneous AI frameworks and model providers.

Why it matters: It matters because AI teams increasingly operate mixed model and agent stacks, and identity, access, and governance controls need consistent visibility into how those systems call tools, handle prompts, and move data.

By the numbers:

• Only 44% of developers are reported to follow security best practices for secrets management, exposing a significant developer behaviour gap.
• When AWS credentials are exposed publicly, attackers attempt access within an average of 17 minutes.

👉 Read Braintrust's guide to framework-neutral AI tracing and evals

Context

Framework-neutral tracing matters because AI estates are no longer built on a single agent framework or a single model provider. The operational problem is not only observability, but governance consistency: if traces, tool calls, and model calls are captured differently across stacks, teams cannot compare behaviour, evaluate risk, or investigate failures with the same control model. In agentic systems, that visibility gap quickly becomes an identity and access gap when tools, secrets, and delegated actions are involved.

Braintrust is positioning instrumentation as the unifying layer across SDKs, OpenTelemetry, and multiple agent frameworks, which makes the topic relevant to AI governance, workload identity, secrets handling, and runtime accountability. For IAM and NHI practitioners, the intersection is real because the same pipelines that evaluate model quality also reveal how credentials, tool permissions, and execution boundaries are being used in practice.

Key questions

Q: How should security teams govern AI connectivity across multiple models and providers?

A: Security teams should govern AI connectivity with a central policy layer that handles authentication, authorisation, logging, redaction, and quota enforcement across all providers. The key is consistency. If every team implements its own controls, auditability breaks down and AI traffic becomes impossible to govern at enterprise scale.

Q: Why do AI agent pipelines create new governance problems for identity teams?

A: Because agent pipelines often combine model calls, tool execution, and delegated access in one runtime path. That means secrets, APIs, and downstream systems can be touched within the same session, so identity governance must follow the runtime path, not just the infrastructure boundary. The control issue is who or what was allowed to act, and when.

Q: What breaks when AI traces are not normalised across systems?

A: Comparability breaks first, then accountability. If one service logs model calls one way and another logs them differently, teams cannot reliably compare performance, investigate failures, or prove which tool calls occurred. In practice, this makes evaluation noisy and weakens the audit trail needed for security and compliance decisions.

Q: How do teams know whether AI governance is actually working?

A: Look for evidence that every AI interaction can be traced end to end, from identity and intent to output and enforcement. If auditors can ask for a transaction and receive a complete record in hours, not weeks, the programme is producing usable control evidence rather than just documentation.

Technical breakdown

Why framework-neutral tracing changes AI governance

Framework-neutral tracing means the telemetry format stays stable even when the orchestration layer changes. That matters because agent stacks are increasingly composable, with one team using LangGraph, another CrewAI, and a third a custom loop or workflow engine. When traces, spans, token metrics, and tool calls are normalised, organisations can compare outputs, latency, and failure behaviour across systems without rebuilding their evaluation logic each time the stack shifts. The real control value is governance portability, not convenience.

Practical implication: standardise trace capture early so evaluation and investigation survive framework churn.

How OpenTelemetry becomes the common control plane

OpenTelemetry gives AI teams a vendor-neutral path for collecting spans, but AI governance needs more than raw telemetry. GenAI semantic conventions add structure to prompts, outputs, metadata, and token data, which allows AI-specific analysis instead of generic service tracing. When the same trace context can cross services and be enriched consistently, teams get a single evidence trail for model calls, retrieval steps, and tool execution. That creates a better foundation for monitoring delegated actions and identifying when an AI workflow is touching privileged assets.

Practical implication: map AI tracing to OpenTelemetry early so observability can support governance, audit, and incident review.

What evals reveal about model and agent risk

Evals turn trace data into repeatable assessment. The important architecture point is that the task under test can be any function, whether it wraps a model call, a retrieval chain, or an agent loop, while scorers measure behaviour consistently across runs. That makes variance visible, which is essential when AI systems use tool calls, external data, or multiple providers. Without repeatable evals, teams can see failures after deployment but cannot compare whether a change in framework, prompt, or provider altered control behaviour.

Practical implication: tie evals to trace data so drift, variance, and unsafe tool use are measurable before rollout.

NHI Mgmt Group analysis

Framework-neutral observability is becoming an AI governance requirement, not a developer preference. When teams mix providers and orchestration layers, the absence of a common trace model creates blind spots in accountability and change control. The same problem appears in identity programmes when every system expresses access, delegation, or privilege differently. Practitioners should treat normalised tracing as part of the control fabric, not an optional engineering convenience.

Trace portability is the named governance gap here: the ability to carry evidence across frameworks without losing meaning. In practical terms, this is the difference between a telemetry system that records activity and one that supports audit, comparison, and incident investigation across changing agent stacks. That matters for NHI governance because delegated actions, tool calls, and credential use need to remain intelligible even as frameworks are swapped or combined. Practitioners should insist on evidence portability before stack sprawl makes review impossible.

AI evals are now a control validation mechanism. Once evals can run repeatedly against the same dataset and task, they become a way to test whether prompt, model, or framework changes altered behaviour in ways that matter to security. This is especially relevant where model calls can trigger tool use or access downstream systems. The governing question is no longer whether the system can run, but whether it behaves consistently enough to trust in production. Practitioners should link release decisions to measurable eval stability.

Identity controls will increasingly need to follow the AI runtime, not just the workload boundary. The post shows how distributed tracing can span heterogeneous services and provider calls, which is exactly where entitlement, secret, and execution-boundary questions surface. If an AI system can invoke tools, traverse services, and change providers while preserving a trace, then access governance must be equally continuous. Practitioners should align runtime evidence with least-privilege and delegated-access reviews.

The market signal is clear: AI observability is converging with AI governance. Tracing, evaluation, and debugging are no longer separate operational activities when AI systems are making decisions at runtime. The organisations that succeed will treat observability data as governance evidence, especially where agentic systems can touch secrets, APIs, or production tools. Practitioners should prepare for governance teams to ask for trace-backed proof of control, not just dashboards.

What this signals

AI observability is turning into governance infrastructure. Once traces, evals, and tool calls are preserved consistently, security teams can start treating runtime evidence as a control input rather than an after-the-fact log stream. That shift matters where delegated access, secrets, and workload identity intersect with AI execution.

Trace portability will expose weak ownership models. If a team cannot explain which service, provider, or agent path touched a credential or downstream system, the problem is not just observability. It is lifecycle governance. Linking AI telemetry to identity review, secret rotation, and access certification will become a practical requirement for mature programmes.

For teams managing AI-linked secrets, the control question is now evidence quality. The same data that supports debugging should also help verify whether privileged tool access was necessary, temporary, and correctly bounded. That is where the overlap between AI operations and NHI governance becomes operationally important.

For practitioners

• Instrument once across all AI stacks Adopt a single trace format across SDK-based, OpenTelemetry-based, and custom agent paths so provider changes do not break auditability or evaluation.
• Link eval runs to release gates Require repeated evals on the same dataset before model, prompt, or framework changes reach production, and record variance alongside the trace evidence.
• Preserve tool-call context in traces Capture retrieval steps, tool inputs and outputs, retries, and spans around privileged actions so investigations can reconstruct delegated behaviour end to end.
• Apply identity review to AI execution paths Review which secrets, APIs, and downstream systems each agent path can reach, then map those privileges back to the trace segments that exercised them.
• Use OpenTelemetry conventions for AI telemetry Normalise gen_ai attributes and trace context so AI observability can support cross-service correlation, governance review, and incident triage.

Key takeaways

• Mixed AI stacks need a shared trace model or governance fragments as frameworks and providers change.
• Repeatable evals matter because they convert runtime behaviour into evidence that can support release decisions and incident review.
• Identity teams should follow AI execution paths, because tool calls, secrets, and downstream access now move through the same runtime.

Key terms

• Framework-Neutral Tracing: A tracing approach that records AI activity in a consistent format regardless of the orchestration framework or model provider. It lets teams compare behaviour, debug failures, and preserve evidence across mixed stacks without reworking their observability model every time the architecture changes.
• OpenTelemetry GenAI semantic conventions: A shared set of attributes for representing model calls, token usage, tool invocations, and related metadata in a portable trace format. They let different observability backends read the same AI execution data without custom schemas or one-off instrumentation.
• Evaluation Variance: The degree to which an AI system produces different results when the same dataset, task, and scorer are run repeatedly. High variance is a governance concern because it can reveal unstable prompts, non-deterministic model behaviour, or hidden dependence on external tools and retrieval inputs.
• Control Portability: Control portability is the ability of a governance control to keep working when the application architecture changes. In clean core programmes, portable controls survive release cycles, integrations, and cleanup of custom code, which makes them more reliable than controls that only exist inside legacy extensions.

What's in the full article

Braintrust's full blog covers the operational detail this post intentionally leaves for the source:

• SDK-specific setup guidance for Python, TypeScript, Go, Ruby, Java, and .NET instrumentation
• OpenTelemetry export configuration and gen_ai semantic convention mapping for AI traces
• Framework-by-framework tracing examples for LangGraph, CrewAI, LangChain, and custom agent loops
• Gateway and custom provider configuration details for teams standardising model access

👉 The full Braintrust post covers SDK setup, OpenTelemetry routing, and provider-specific examples.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, workload identity, secrets management, and agentic AI identity. It is designed for practitioners who need to connect identity controls to modern runtime systems and AI-driven access paths.