Braintrust reviews expose a bigger gap in AI governance

At a glance

What this is: This is an independent analysis of Braintrust reviews and the platform boundary between AI evaluation and AI governance.

Why it matters: It matters because IAM, PAM, and AI governance teams need to separate post-inference observability from request-path control, especially where agents, tools, and internal access decisions are involved.

By the numbers:

• 80% of organisations report their AI agents have already performed actions beyond their intended scope, including accessing unauthorised systems, sharing sensitive data, or revealing access credentials.

👉 Read TruFoundry's analysis of Braintrust reviews and AI governance boundaries

Context

Braintrust Dev is an AI evaluation and observability platform, but evaluation alone does not answer the security questions enterprises now face around AI gateways, tool access, and identity-aware request control. In practice, the governance gap appears when teams assume that better tracing equals better enforcement, which is not true for model access, spending caps, or MCP tool boundaries.

The primary IAM and NHI lesson is that AI systems create a control stack, not a single product decision. Teams may use evaluation tooling to improve model quality, but they still need separate controls for request authorization, secret handling, and runtime policy enforcement. That distinction is especially important as agentic workflows expand and the boundary between application identity and non-human identity becomes harder to ignore.

Key questions

Q: What breaks when AI governance is built only around approved tools?

A: Tool-only governance fails when employees shift to new or personal AI services faster than policy can update. It also misses the bigger issue that the same sensitive data can travel through multiple interfaces. Without data-aware enforcement, organisations end up policing names of tools instead of controlling exposure.

Q: When should organisations prioritise an AI gateway over better observability?

A: They should prioritise the gateway whenever model calls involve internal data, regulated workflows, agents, or external tools. Observability is valuable, but it cannot replace request authorization, least privilege, or runtime policy enforcement. If a system can act, spend, or reach tools, the preventive control has to sit in front of the call.

Q: What do security teams get wrong about agent autonomy and governance?

A: They often confuse autonomy with safe authority. An agent can act independently in execution without being trusted to decide everything. The governance mistake is giving broad standing access because the system appears intelligent, when the safer model is bounded authority with tightly scoped privileges and clear escalation.

Q: Who is accountable when an AI system makes a harmful decision?

A: Accountability should follow the identity chain that authorized, configured, or triggered the action, including the human owner, the platform team, and any delegated agent or tool account. If the organisation cannot name that chain, the governance model is too weak for regulated AI use.

Technical breakdown

Why evaluation platforms sit downstream of inference

Evaluation and observability tools capture traces, scores, and metadata after a model call has already happened. They are built to measure output quality, compare prompts, and replay experiments, which is useful for software development but not enough for security control. Once inference is complete, the tool can explain what happened, but it cannot decide whether a request should have been allowed in the first place. That makes it a downstream analytics layer, not a policy enforcement point.

Practical implication: place request authorization, budget enforcement, and tool control in the gateway or access layer, not in the eval platform.

AI gateway governance and request-path control

An AI gateway enforces policy before a model call reaches a provider or internal model. That can include user identity checks, service account restrictions, model allowlists, routing policies, rate limits, and audit logging. This is a different function from trace inspection because it acts on the live request path. For enterprises using agents or MCP-connected tools, the gateway becomes the control boundary where identity, privilege, and data access converge.

Practical implication: use gateway controls to make least privilege operational at the point of model invocation.

MCP tool governance creates a new identity boundary

The Model Context Protocol connects agents to tools and data sources, which expands the attack surface beyond text generation. A tool call may read files, trigger workflows, or update systems, so the question is no longer only what the model said but what the agent was allowed to do. Logging a tool call after the fact is useful for forensics, but it does not stop overreach or misuse in real time. That is why MCP needs identity and authorization controls at the boundary.

Practical implication: treat MCP-connected tools as privileged integrations and control them with explicit authorization, scope checks, and audit.

NHI Mgmt Group analysis

Evaluation quality is not governance. Braintrust-style platforms help teams measure output quality, but measurement does not substitute for access control, budget limits, or tool authorization. In AI programmes, the core mistake is treating post-inference telemetry as if it were preventive security. Practitioners should separate observability from enforcement and align each to the right control owner.

AI gateways are becoming the identity control plane for model traffic. As enterprises connect internal users, service accounts, and agents to multiple models, the real governance decision moves to the request path. That is where identity, privilege, and cost policy must be applied consistently. For teams managing NHI and IAM programmes, this shifts the centre of gravity from dashboards to live authorization.

Request-path blind spot: this is the gap created when teams can inspect traces but cannot block, scope, or budget the call before execution. The pattern matters because agentic systems can act quickly enough that after-the-fact review is too late. That means governance should be designed around prevention, not only evidence collection.

Agentic workloads make tool control a privileged-access problem. Once an AI system can call external tools, it inherits a new form of non-human privilege that needs lifecycle management, scope review, and offboarding. The organisation that can audit traces but not bound tool use has not solved agent governance. Practitioners should treat agent tool access as a PAM-adjacent control domain with identity at the centre.

Vendor reviews often blur product categories, which weakens buyer judgement. Braintrust review pages mix evaluation, observability, and recruiting feedback because two unrelated products share a name. That creates noisy market signals and can distort procurement decisions. Security and identity teams should map requirements to control functions first, then compare products only inside the correct category.

What this signals

Request-path control is becoming the decisive governance layer for AI programmes. Teams that can only inspect traces will continue to miss the point where authorization, cost, and tool use must be decided. The practical response is to align AI gateway policy with IAM and PAM review cycles, then test those controls against real agent workflows and MCP connections.

Non-human identity management now extends beyond secrets to runtime authority. The operational question is no longer only whether a model or agent has credentials, but whether those credentials are allowed to invoke the right tools at the right time. That is where identity governance, access review, and revocation discipline intersect with AI security.

The more agentic the workflow becomes, the less useful passive visibility is on its own. Security teams should expect governance failures to show up first as overbroad tool access, unmanaged spend, or unreviewed service-account paths rather than as obvious model compromise.

For practitioners

• Separate observability from enforcement Assign evaluation platforms to scoring, traces, and regression checks only. Put request authorization, spend limits, and tool approvals in a gateway or policy layer that sits before inference, not after it.
• Map AI requests to identity and privilege Document which human users, service accounts, and agents can call which models, with what scopes, and under which conditions. Review that mapping as a governed entitlement set, not as an informal engineering setting.
• Treat MCP connections as privileged integrations Require explicit approval for every tool connection, define scope per tool, and maintain audit evidence for who can modify or revoke those connections. Re-check access when the agent or workflow changes.
• Use evaluation data as a control signal, not a control substitute Feed trace anomalies, failed evals, and cost spikes into security operations, but do not rely on them to stop harmful requests. Build escalation paths from evaluation findings into gateway policy and access review.

Key takeaways

• Braintrust is best understood as an evaluation and observability layer, not as the control point for model authorization or tool governance.
• Agentic AI introduces identity and privilege decisions that must be enforced before inference, not merely reviewed afterward.
• Enterprises should separate quality measurement from security enforcement and assign each to the correct layer of the AI stack.

Key terms

• AI Gateway: A control point that sits between AI applications and the models, tools, or data they call. In practice, it can authenticate requests, enforce policy, inspect runtime behaviour, and stop unsafe actions before they spread into connected systems.
• Request-path Governance: Request-path governance is the set of policies that apply before an AI request executes. It covers who can call a model, which tools can be reached, what data can move, and whether the request should be blocked, throttled, or routed differently.
• Mcp Install Boundary: The MCP install boundary is the point where a user reviews and accepts a tool's configuration before it is written into a workspace. In security terms, it is supposed to separate visible intent from persisted runtime state. If hidden values can bypass that review, the boundary no longer governs trust.
• Non-Human Identity (NHI): A digital identity assigned to a non-human entity such as a software application, service account, API key, bot, machine, or AI agent that enables it to authenticate and interact with systems without direct human involvement. NHIs now outnumber human identities in most enterprises by 25 to 50 times.

What's in the full article

TruFoundry's full analysis covers the operational detail this post intentionally leaves for the source:

• Pricing and tier differences that matter for Enterprise buyers evaluating RBAC, SSO, retention, and deployment controls.
• The platform boundary between eval tooling and request-path governance, including the limits of trace-based oversight.
• Detailed comparisons with alternative AI gateway and observability patterns for teams deciding where policy enforcement belongs.
• Feature-level capabilities around traces, datasets, experiments, and replay workflows that support implementation planning.

👉 TruFoundry's full article covers pricing tiers, governance gaps, and platform comparisons in more operational detail.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management. It helps security and identity practitioners translate governance requirements into operational controls.