Hidden failure patterns in AI agent traffic expose observability gaps

At a glance

What this is: This is an analysis of how production trace clustering can uncover AI agent failure modes that scorers and dashboards miss, including stale retrieved context and repeated tool retries.

Why it matters: It matters because identity, access, and governance controls around AI agents only work when teams can see the behaviours those agents actually exhibit in production, not just the failures already pre-defined in evaluation.

By the numbers:

• Only 44% of developers are reported to follow security best practices for secrets management, exposing a significant developer behaviour gap.
• 96% of organisations store secrets outside of secrets managers in vulnerable locations including code, config files, and CI/CD tools.
• Only 5.7% of organisations have full visibility into their service accounts.

👉 Read Braintrust's guide to discovering hidden failure patterns in AI agent production traffic

Context

AI agent observability fails when teams only measure the failures they already know how to name. A scorer can catch unsupported answers and a latency alert can catch slow responses, but neither will detect a fluent answer built on stale retrieved context unless that failure mode has been defined first.

The governance gap is broader than model quality alone. Production traces often reveal hidden behaviour patterns that matter for control design, including unresolved user requests, repeated tool retries, and stale source data, and that intersects with identity governance when agents rely on secrets, service accounts, and tool permissions to complete work.

Key questions

Q: What breaks when AI agent failures are only tracked with predefined scorers?

A: Teams miss behaviours they never described in advance, including stale retrieved context, repeated retries, and unresolved workflows. The agent can look healthy in dashboards while users experience wrong outcomes. Discovery has to start with trace clustering and manual confirmation, then move into scoring once the failure pattern is real and repeatable.

Q: Why do local AI agents complicate identity and access management?

A: They can retain legitimate permissions while changing timing, prioritisation, and action sequence outside human presence. That means the visible identity may remain stable even as the operational behaviour becomes autonomous. IAM teams then lose the simple link between user session, authorisation, and accountability.

Q: How do security teams know whether a hidden agent failure is real?

A: Validate the cluster at trace level, measure how much traffic it represents, and check whether the pattern correlates with user frustration or unresolved requests. A real failure is repeated, visible in the underlying traces, and large or harmful enough to justify a permanent scorer or review path.

Q: Who should own AI agent failure discovery and response?

A: Ownership should sit across AI engineering, security, and identity governance. Engineering can confirm the behaviour, security can assess risk and impact, and identity teams can verify whether access scope, secrets, or tool permissions enabled the failure. The control gap is shared, so the accountability model must be shared too.

Technical breakdown

Why predefined scorers miss hidden AI agent failures

Predefined scorers are effective only when the failure shape is already known. They work like narrow test cases: useful for unsupported answers, slow responses, or other explicit conditions, but blind to emergent behaviours such as stale retrieval, partial task completion, or repeated retries that still look acceptable at a metric level. In production, the agent may appear healthy while systematically failing a workflow. That is why behaviour discovery must precede control design. Clustering traces by what happened in the run, rather than by a preset metric, exposes patterns that can later become evaluation criteria.

Practical implication: Treat hidden-failure discovery as a control design input, not a reporting layer.

How trace clustering turns behaviour into an operational signal

Trace clustering groups similar runs by their content, then surfaces unusual clusters for review. The technical value is that the grouping happens on actual execution behaviour, not on a team-defined KPI. When a cluster contains many traces with the same task path, issue label, or user tone, it can reveal an unmodelled failure mode. Once confirmed, the pattern can be converted into a deterministic scorer, a labelled evaluation set, or a human review workflow. That creates a closed loop from observation to enforcement. For AI systems operating with tools, retrieval, or delegated access, this is essential because failure often appears first as behaviour drift, not as a hard exception.

Practical implication: Use clustering to promote unknown behaviours into checks that can be enforced repeatedly.

Why observability for AI agents has an identity dimension

AI agents are not just inference engines. They also consume credentials, call tools, and act through service identities, which means production trace analysis often intersects with NHI governance. When a hidden failure pattern involves stale context, failed tool calls, or repeated retries, the root issue may sit in authorization scope, secret handling, or tool access rather than in the model itself. That makes the observability problem an identity problem as well. If the agent can reach the wrong data, keep retrying with the same privilege, or continue after an unresolved request, the control gap is in the runtime trust model as much as in the evaluation layer.

Practical implication: Review agent traces alongside tool permissions, secret scope, and service identity boundaries.

Threat narrative

Attacker objective: The objective is not classical intrusion but persistent misuse of an AI agent's trusted runtime so it keeps producing wrong or unreconciled outcomes at scale.

1. Entry occurs when the agent receives production traffic and retrieves or invokes external context that may already be stale or incomplete.
2. Escalation happens when the agent repeatedly retries tools, reuses outdated context, or continues a workflow without a correction signal.
3. Impact follows when the system returns a fluent but wrong answer, leaves a user request unresolved, or propagates bad decisions into downstream automation.

NHI Mgmt Group analysis

Hidden failure discovery is now a governance control, not just an observability feature. If teams only score the failures they can name before release, they will miss the behaviour that customers actually experience in production. The practical lesson is to treat trace clustering, cluster review, and scorer creation as part of the control stack, not as optional analytics.

AI agent observability has a named failure mode: trace-defined blind spots. This is the gap created when dashboards measure predefined metrics while behaviour clusters reveal unknown defects, especially around stale retrieval and repeated tool retries. Security and AI teams should recognise that blind spots form whenever runtime behaviour outpaces the evaluation taxonomy.

Agent governance now depends on the trustworthiness of the tool and identity layer. When an agent can act through service accounts, access APIs, or pull retrieved context, the boundary between model error and governance failure becomes thin. That is why NHI oversight belongs in AI operations, not only in IAM or PAM reviews.

Confirming failure patterns before automation prevents false control confidence. A cluster is only useful once it has been validated against trace-level evidence and measured for user impact. Teams that jump directly from dashboards to policy enforcement risk automating the wrong failure model.

For the market, the next frontier in AI assurance is behavioural coverage. The category is moving beyond post-hoc scoring toward systems that discover what was not anticipated, then convert it into durable checks. Practitioners should expect evaluation, tracing, and governance to converge around the same production evidence.

What this signals

Trace clustering gives teams a way to see the failure modes that scorecards were never designed to catch. For programmes operating agents with tool access, the real question is whether unusual behaviour is visible soon enough to become a control, not whether the model passed a benchmark. That is why production evidence now needs to feed evaluation, review, and governance together.

AI agent runtime risk increasingly sits at the intersection of behaviour and identity. When the agent uses service identities, secrets, or delegated tools, trace anomalies should trigger an access review as readily as a quality review. Practitioners should align this work with the OWASP Agentic AI Top 10 and the NIST AI Risk Management Framework.

The operational signal is simple: if clusters keep surfacing the same retry, stale-context, or unresolved-task patterns, the programme has a discovery problem, not just a scoring problem. Teams should expect evaluation coverage to expand continuously as production traffic changes, or hidden failures will reappear under new labels.

For practitioners

• Build a cluster-review loop for production traces Group agent traces by behaviour, open the unusual clusters manually, and confirm whether the label matches the underlying conversations before promoting anything into policy or scoring. Link this review to the same workflow used for evaluation datasets so discoveries become repeatable checks.
• Convert confirmed failures into deterministic scorers When a cluster has a stable label, turn it into a scorer that flags future traces with the same pattern. Keep the scorer tied to concrete trace classifications such as task, issue, and sentiment so it can run continuously without a human every time.
• Add human review for subtle or high-impact clusters Route clusters that are low-volume, ambiguous, or user-visible into reviewer queues before automation hardens the wrong behaviour. Use this step for cases where a judge model would struggle to distinguish a minor anomaly from a material workflow failure.
• Check tool permissions alongside trace anomalies For any cluster involving retries, stale answers, or unresolved requests, inspect the agent's tool scope, service account access, and secret exposure window. The failure may be caused by an identity boundary problem rather than a model-quality issue.

Key takeaways

• Predefined scorers catch only known failure shapes, so hidden agent failures can stay invisible until production traces are clustered.
• The identity layer matters because AI agents fail through tool access, service identities, and secrets as well as through model behaviour.
• The right control response is a closed loop of clustering, validation, scoring, and review, not dashboards alone.

Key terms

• Trace Clustering: Trace clustering groups similar execution traces into behaviour-based clusters so teams can spot repeated patterns that single metrics miss. In AI agent operations, it helps expose emergent failure modes such as stale context, tool retry loops, and unresolved workflows before they become a formal scorer.
• Hidden Failure Pattern: A hidden failure pattern is a repeatable agent behaviour that causes incorrect or incomplete outcomes but is not covered by existing dashboards or scorers. It usually appears first in production traces, where the behaviour is visible only when traces are reviewed together rather than as isolated events.
• LLM-as-a-judge: A control pattern where one language model evaluates another model's prompts, tool calls, or outputs against policy. It is not content moderation alone. In practice, it acts as a runtime decision layer that can allow, block, redact, or escalate based on semantic context and organisational rules.
• Service Identity Boundary: A service identity boundary is the set of permissions, credentials, and tool scopes that constrain what an agent can access at runtime. If that boundary is too broad or poorly monitored, a model failure can become a governance failure because the agent can keep operating through trusted access.

What's in the full article

Braintrust's full research covers the operational detail this post intentionally leaves for the source:

• Daily pipeline mechanics for clustering traces into topics and facet summaries
• Example scorer code and LLM-as-a-judge prompt structure for confirmed failure patterns
• Custom facet design for tracking product-specific failure dimensions such as citation quality and tool reliability
• Trace review workflows for moving clusters into datasets or human assignment

👉 The full Braintrust article covers trace clustering steps, scorer examples, and review workflows in implementation detail.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, agentic AI identity, machine identity security, and secrets management. It helps security and identity practitioners connect runtime access decisions to durable governance controls.