Notifications
Clear all
Topic starter
20/08/2026 5:54 pm
TL;DR: Teams can instrument AI systems once through SDKs or OpenTelemetry and then keep traces, evals, and debugging consistent across LangGraph, CrewAI, custom loops, and multiple model providers, according to Braintrust. The governance lesson is that portable observability only matters when it becomes a control surface for repeatable evaluation, not just a logging layer.
NHIMG editorial — based on content published by Braintrust: How to use Braintrust with any framework or provider
By the numbers:
- When AWS credentials are exposed publicly, attackers attempt access within an average of 17 minutes.
Questions worth separating out
Q: How should security teams govern AI connectivity across multiple models and providers?
A: Security teams should govern AI connectivity with a central policy layer that handles authentication, authorisation, logging, redaction, and quota enforcement across all providers.
Q: Why do AI agent pipelines create new governance problems for identity teams?
A: Because agent pipelines often combine model calls, tool execution, and delegated access in one runtime path.
Q: What breaks when AI traces are not normalised across systems?
A: Comparability breaks first, then accountability.
Practitioner guidance
- Instrument once across all AI stacks Adopt a single trace format across SDK-based, OpenTelemetry-based, and custom agent paths so provider changes do not break auditability or evaluation.
- Link eval runs to release gates Require repeated evals on the same dataset before model, prompt, or framework changes reach production, and record variance alongside the trace evidence.
- Preserve tool-call context in traces Capture retrieval steps, tool inputs and outputs, retries, and spans around privileged actions so investigations can reconstruct delegated behaviour end to end.
What's in the full article
Braintrust's full blog covers the operational detail this post intentionally leaves for the source:
- SDK-specific setup guidance for Python, TypeScript, Go, Ruby, Java, and .NET instrumentation
- OpenTelemetry export configuration and gen_ai semantic convention mapping for AI traces
- Framework-by-framework tracing examples for LangGraph, CrewAI, LangChain, and custom agent loops
- Gateway and custom provider configuration details for teams standardising model access
👉 Read Braintrust's guide to framework-neutral AI tracing and evals →
AI observability across frameworks: what changes for governance teams?
Explore further
20/08/2026 6:35 pm
Framework-neutral observability is becoming an AI governance requirement, not a developer preference. When teams mix providers and orchestration layers, the absence of a common trace model creates blind spots in accountability and change control. The same problem appears in identity programmes when every system expresses access, delegation, or privilege differently. Practitioners should treat normalised tracing as part of the control fabric, not an optional engineering convenience.
A question worth separating out:
Q: How do teams know whether AI governance is actually working?
A: Look for evidence that every AI interaction can be traced end to end, from identity and intent to output and enforcement. If auditors can ask for a transaction and receive a complete record in hours, not weeks, the programme is producing usable control evidence rather than just documentation.
👉 Read our full editorial: Framework-neutral tracing makes AI evals portable across stacks
