AI evals and observability need shared evidence to build trust

At a glance

What this is: This is a Braintrust blog post arguing that AI feature trust depends on combining evals, traces, and operational metrics into shared views.

Why it matters: It matters to IAM and security practitioners because AI governance is now tied to how evidence is collected, reviewed, and acted on across teams, especially where agentic AI and human decision-making intersect.

By the numbers:

• Only 44% of developers are reported to follow security best practices for secrets management, exposing a significant developer behaviour gap.

👉 Read Braintrust's post on earning stakeholder trust with evals and observability

Context

AI observability is the discipline of making model behaviour legible through traces, metrics, and evaluative signals. The governance problem is not whether teams have data, but whether they can connect quality, cost, latency, and failure modes into a decision-ready view that leadership and operators trust.

For identity and access programmes, the relevance is indirect but real: AI systems are increasingly part of the operational stack, and the same governance questions apply to who can see data, who can query production systems, and who is accountable when AI-generated outputs influence access or security decisions. That makes shared evidence a control issue, not just a reporting preference.

Key questions

Q: How should teams make AI evals and observability useful for leadership reviews?

A: Use a small set of shared metrics that answer whether the feature works, what it costs, and whether quality is changing. Pair headline scores with trend lines and keep the review artifact simple enough that product, engineering, and leadership can all interrogate the same evidence without translation.

Q: Why does fragmented AI visibility create governance problems?

A: Fragmented visibility creates governance problems because teams cannot reliably determine whether a discovered tool is authorised, risky, or tied to sensitive data. Separate tools produce partial truths, which leads to inventory gaps and inconsistent policy enforcement. Governance depends on context, and context is lost when discovery is split across disconnected sources.

Q: How can security teams govern natural-language access to production data?

A: Treat it like any other privileged data access path. Define which datasets can be queried, log every request, review outputs for sensitive leakage, and restrict escalation from ad hoc questions into broad production visibility. Convenience is acceptable only when access boundaries stay explicit.

Q: What should organisations do when AI traces are too complex for non-engineers to review?

A: Create simplified trace views that mirror the product flow and expose the decision, the outcome, and the quality signal in plain language. If stakeholders cannot understand one run, they cannot challenge systemic issues, so the review interface itself becomes part of governance.

Technical breakdown

How dashboards turn scattered AI signals into governance evidence

AI teams usually split evidence across experiments, logs, and engineering dashboards, which makes each signal useful but incomplete. A dashboard layer normalises these signals into time series, top lists, and headline metrics so leaders can compare quality, cost, and latency in one place. The important mechanism is not the chart itself, but the shared data model underneath it. Once traces and evals are queryable together, teams can examine trends by model, user segment, deploy version, or topic without re-creating reports for each audience.

Practical implication: build one governed metrics layer before adding more dashboards, or every team will keep arguing from different numbers.

Why custom trace views matter more than raw traces

Raw traces are technically rich but hard to interpret outside engineering because they expose span-by-span execution detail. Custom trace views translate that JSON-heavy structure into a form that matches how stakeholders reason about the product, such as a support ticket, decision summary, or workflow card. This is a governance pattern as much as a UX pattern because it reduces interpretation drift. When non-technical reviewers can read the same interaction shape as engineers, review quality improves and disputes about what the model did become evidence-based rather than anecdotal.

Practical implication: standardise a small set of trace views for recurring review audiences, especially when AI outputs affect risk, access, or customer decisions.

How ad hoc query tools change operational accountability

Natural-language query tools over production data close the gap between a question and the evidence needed to answer it. Instead of waiting for an engineer to write SQL, a reviewer can ask a question, generate a filtered dataset, and inspect the result in context. The architectural shift is important: investigation becomes part of routine governance, not an exceptional task. That makes the quality of the underlying permissions, data access boundaries, and auditability more important, because ad hoc analysis can quickly become de facto production access if controls are weak.

Practical implication: pair self-service AI queries with explicit access boundaries and audit logging so convenience does not become uncontrolled data reach.

NHI Mgmt Group analysis

Shared evidence is now a governance control, not a reporting convenience. AI programmes fail stakeholder alignment when evals, traces, and operational metrics remain separate artefacts. That fragmentation creates versioned truth, where product, engineering, and leadership each see a different reality. In practice, the control failure is not missing telemetry, but missing synthesis. Teams should treat the evidence layer as part of AI governance and review it alongside decision rights.

Trace interpretability is becoming a trust boundary. If non-engineers cannot understand a run, they cannot meaningfully challenge or approve the behaviour. Custom trace views reduce that gap by translating execution detail into domain language. This matters for AI governance because legibility is what turns technical observability into accountable review. The practical conclusion is that trace design should be built for oversight, not only debugging.

Natural-language querying introduces a new access governance problem. When anyone can ask questions against production data, the risk shifts from manual report bottlenecks to uncontrolled investigative reach. That does not make the capability unsafe by default, but it does require clearer permission scoping, logging, and review of who can interrogate which datasets. The field should treat ad hoc AI analytics as a governed access pattern, not a productivity feature alone.

AI governance debt: this post shows how quickly technical signal quality turns into organisational debt when teams delay shared review structures. The longer evals, traces, and business metrics stay isolated, the harder it becomes to explain failures, prioritise fixes, or defend model decisions. For practitioners, the lesson is to make governance reviewable before the next model release.

Observability becomes a shared language for AI risk management. The strongest organisations will not just monitor model health; they will standardise how different functions read it. That is where AI governance starts to resemble other control domains such as IAM and PAM, where common evidence and common interpretation matter as much as the raw control itself. Practitioners should align observability with accountability early.

What this signals

AI governance debt: when review evidence is split across experiments, traces, and dashboards, organisations accumulate process friction that shows up later as slower approvals and weaker accountability. The practical response is to treat evidence synthesis as a standing control, not an afterthought, and anchor that control in standards such as the NIST SP 800-53 Rev 5 Security and Privacy Controls.

The same pattern appears in identity-adjacent AI programmes where sensitive data can leak through operational telemetry. A recent NHIMG finding showed that 43% of security professionals worry about AI systems learning and reproducing sensitive information patterns from codebases, which is a strong signal that observability must be paired with data minimisation and access review.

For practitioners, the next step is not more dashboards but clearer governance around who can query, who can interpret, and who can approve AI behaviour. The more AI systems influence customer outcomes or internal decisions, the more their evidence layer starts to resemble a control surface that needs documented ownership.

For practitioners

• Create a single evidence layer for AI reviews Unify eval scores, trace data, and production metrics so leadership, product, and engineering use the same source of truth for AI quality decisions.
• Design trace views for non-technical oversight Build simplified trace interfaces that show the decision path, outcome, and key metrics in language reviewers can understand without reading raw spans.
• Constrain self-service production queries Limit who can query production logs, require audited access, and define which datasets ad hoc tools may reach before they become part of normal review.
• Standardise review metrics by audience Use headline quality, cost, and latency for leadership, regression and deploy-version cuts for engineering, and segmentation by user task for product governance.

Key takeaways

• AI trust depends on merging evals, traces, and operational metrics into one reviewable evidence model.
• Custom trace views and self-service queries make AI behaviour legible, but they also create new governance and access-control obligations.
• The organisations that scale AI safely will standardise how different functions read model evidence before the next release cycle.

Key terms

• AI observability: AI observability is the ability to see how AI systems are being used, what information they process, and what actions they trigger. In security programmes, it extends beyond uptime or model quality to runtime visibility, policy enforcement, and audit evidence across human and agent-driven use cases.
• Evaluation Score: An evaluation score is a measured output that represents how well an AI system performed against defined criteria such as accuracy, safety, or task completion. It is useful only when paired with context about the dataset, audience, and failure mode it was intended to represent.
• Custom Trace View: A custom trace view is a tailored interface that restructures raw execution traces into a format easier for humans to read. It helps non-engineers review AI behaviour without parsing low-level spans, which makes oversight more practical and less dependent on specialist interpretation.
• Governed Ad Hoc Querying: Governed ad hoc querying is the controlled use of natural-language or self-service analysis tools against production data. It gives teams flexibility to investigate new questions while keeping permissions, audit logging, and dataset boundaries explicit enough to prevent uncontrolled access.

What's in the full article

Braintrust's full blog post covers the operational detail this post intentionally leaves for the source:

• How to configure dashboard charts for time series, top lists, and big-number views across evals, latency, cost, and token usage.
• How to use custom trace views to transform raw spans into stakeholder-friendly artefacts for product, leadership, and engineering reviews.
• How Loop translates natural-language questions into SQL over production data and promotes one-off answers into reusable charts.
• How to build segmentation by user segment, task type, and deploy version so regressions are visible in review meetings.

👉 The full Braintrust post shows how to turn traces, evals, and dashboards into reusable review artefacts.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and identity lifecycle fundamentals. It is a fit for practitioners who need to connect governance evidence to broader security and access programmes.