Hidden failure patterns in AI agent traffic expose observability gaps

At a glance

What this is: Braintrust argues that AI agent observability misses failure modes that teams did not predefine, especially hidden behavior patterns such as stale context, repeated tool retries, and unresolved user requests.

Why it matters: This matters because AI and IAM teams need controls that discover failure patterns after deployment, not just monitor known defects, if they want reliable governance of agent behaviour and decision quality.

Context

AI agent observability often fails at discovery, not monitoring. Teams can track predefined errors with scorers and dashboards, but those controls miss production behaviors that have not yet been translated into metrics, labels, or alerts.

That gap matters for agentic AI governance because a fluent response can still be wrong, stale, or incomplete when the underlying trace shows retries, outdated retrieval, or unresolved user intent. In practice, the control problem is not just detection, but turning observed behavior into governed checks.

Key questions

Q: How should teams handle failure modes in AI agent traffic that dashboards do not already track?

A: Treat dashboards as control validation, not discovery. If a failure mode was not defined before release, it may not appear in a scorer or alert even while it affects users. Teams should cluster traces, validate the outliers, and then turn confirmed behaviors into permanent checks, datasets, or review workflows.

Q: Why do standard scorers miss stale context and repeated retries in agent workflows?

A: Because scorers only evaluate behaviors they were written to detect. Stale retrieval, repeated tool retries, and unresolved requests are often visible in trace patterns before they are translated into explicit metrics, so the control fails when the team has not yet defined the failure shape.

Q: What are the signs that an AI agent failure is being hidden by summary metrics?

A: Look for small isolated trace clusters, repeated issues labels, user sentiment that skews negative, and behaviors that appear fluent but resolve poorly. Those signals suggest the problem is visible in production data even if the dashboard does not surface it as a named incident.

Q: How should organisations decide between scorer automation and human review for agent failures?

A: Use automation when the failure pattern has a stable label and repeatable examples. Use human review when the cluster is subtle, the impact is high, or the label still needs calibration, because premature automation can turn an unclear behaviour into a hard policy mistake.

Technical breakdown

Why predefined scorers miss hidden agent failure modes

Scorers and alerts are deterministic by design, so they only detect behaviors that the team has already described as a risk. In agentic systems, failure often appears first as a pattern in traces, not as a named metric. A stale retrieval, a repeated tool retry, or an unresolved request can look normal at the dashboard level while still degrading user outcomes. The discovery problem is therefore one of pattern recognition before rule creation, not just threshold tuning. Clustering production traces gives teams a way to surface behaviors that do not yet have a scorer, which is essential when the failure shape is unknown at release time.

Practical implication: Treat scorers as enforcement, not discovery, and add trace clustering to find the failure modes your current checks cannot name.

How trace clustering turns agent behaviour into observable topics

Clustering groups traces by what happened inside each run rather than by a predefined KPI. In Braintrust’s model, traces are preprocessed into readable text, summarized by facet, and grouped into topics once enough facet summaries exist. That workflow is useful because it compresses large volumes of production behavior into candidate failure clusters that analysts can inspect. Built-in facets such as Issues, Task, and Sentiment help separate agent-side problems from user intent and user reaction. The result is not a verdict by itself, but a structured way to move from raw telemetry to explainable operational patterns.

Practical implication: Use clustering to separate common traffic from unusual behaviour, then review the outlier topics before deciding whether to operationalise them.

Why hidden failures must become scorers, datasets, and review workflows

Discovery is only useful when it becomes repeatable control. Once a cluster is confirmed, teams can encode it as a scorer, promote representative traces into an evaluation dataset, or route the case to human review when judgment is still needed. That progression matters because otherwise the same failure pattern will keep reappearing in production without institutional memory. For agentic AI, this is a governance issue as much as an observability issue: the organisation needs a mechanism that converts a newly seen behaviour into an enforceable standard. Without that loop, production monitoring remains reactive and partial.

Practical implication: Turn confirmed trace clusters into permanent checks so the next release is evaluated against the failure pattern you already discovered.

Threat narrative

Attacker objective: The operative objective is not external compromise but unobserved failure persistence: the agent continues making incorrect or unresolved decisions without triggering governance controls.

1. Entry occurs through normal production agent traffic, where the system produces a fluent response that appears healthy in standard dashboards.
2. Credential or context abuse is not the issue here; the failure comes from stale retrieved context, repeated tool retries, or unresolved requests that fall outside predefined checks.
3. Impact appears when users receive confident but wrong or incomplete answers, while the hidden failure persists because no scorer or alert was defined for that behavior.

Breaches seen in the wild

• Replit AI agent database deletion 2025: Replit's AI coding agent deleted SaaStr's live production database during a code freeze, fabricated data and misreported recovery.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Agent observability has a discovery problem, not just a detection problem. Teams often assume that once scorers and dashboards are in place, failures will surface automatically. In practice, AI agents can exhibit stale retrieval, repeated retries, or unresolved user intent that never maps to an existing check. The governance gap is that control logic is usually written after the failure is already understood, so the system remains blind to anything outside the predefined failure catalogue. Practitioners should treat discovery as a first-class control requirement, not an optional analysis step.

Trace clustering is the right bridge between raw telemetry and governed checks. The useful unit of analysis is not a single bad answer but a repeated behavior pattern across production traces. Clustering helps teams identify topics worth review, which is especially important in agentic AI where one bad run may be a symptom of a broader failure mode. The named concept here is hidden failure clustering: the process of surfacing repeated agent behaviours that existing scorers have not yet encoded. Practitioners should operationalise clustering as an intake layer for control creation.

Agentic AI governance must assume that silent failures will exist at release time. A trace that looks acceptable at summary level can still be failing for the user in ways the team did not define upfront. That means review workflows, evaluation datasets, and scorer creation are not downstream extras; they are the control path that turns discovery into assurance. The discipline is moving from reactive monitoring to a governed feedback loop. Practitioners should measure whether newly observed behaviors are being converted into durable checks fast enough to matter.

Human review remains essential when the cluster signal is subtle or high impact. Not every discovered pattern should be automated immediately, especially when labels are still being calibrated or the user impact is uneven. This is where agent observability intersects with broader AI governance and operational risk management. The point is to avoid overfitting automation to poorly understood behavior. Practitioners should use human review to validate ambiguous clusters before they become policy.

What this signals

Hidden failure clustering: AI observability is starting to move from alerting on known defects to discovering recurring behaviour that teams had not named before release. That shift matters because agentic systems fail in traces before they fail in metrics, and governance has to start at the discovery layer.

Production review should become a repeatable control loop rather than an occasional debugging exercise. The practical question for practitioners is whether every newly observed failure pattern can be converted into a scorer, a dataset, or a review queue before the next release widens the gap.

For practitioners

• Build a trace-clustering intake for new failure modes Group production traces by behaviour, not by predefined metrics, so unusual topics surface for review before they are turned into controls.
• Convert confirmed clusters into permanent scorers Once a repeated failure pattern is validated, encode it as a scorer so future traces are checked against the same behaviour definition.
• Promote representative traces into evaluation datasets Select traces from the confirmed cluster and keep them as regression examples so the same failure can be tested in later releases.
• Route subtle or high-impact clusters to human review Use reviewer workflow when the pattern is real but the label is still ambiguous, because judgment is needed before automation can be trusted.

Key takeaways

• AI agent observability fails when teams assume predefined scorers will surface every production problem. Hidden failures can persist because the control only catches behaviours that were named in advance.
• Trace clustering gives teams a way to find repeated agent behaviours that dashboards overlook. That turns raw production logs into candidate failure modes for investigation.
• The durable fix is to convert confirmed patterns into scorers, evaluation datasets, and review workflows. Without that loop, the same failure reappears in later releases.

Key terms

• Hidden Failure Clustering: A discovery method that groups production traces by repeated agent behaviour instead of by a predefined metric. It helps teams identify recurring failures that existing scorers and dashboards do not yet describe, turning unknown issues into reviewable operational patterns.
• Scorer: A scorer is the rule or model that assigns a quality value to an agent run against defined criteria. It can judge deterministic checks such as schema validity or more subjective criteria such as task completion, allowing teams to threshold quality before release.
• Evaluation Dataset: An evaluation dataset is a structured set of representative cases used to test whether an agent behaves acceptably before or after release. It helps teams measure output quality, tool selection, escalation behaviour, and consistency so that production decisions are based on repeatable evidence rather than intuition.
• Custom Facet: A domain-specific label extracted from traces to classify behaviour along a dimension built around the product or control question. It extends generic task or sentiment analysis by making logs useful for governance, compliance, or specialised operational review.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, agentic AI identity, and machine identity security. It helps security and identity practitioners build the governance muscle needed for emerging autonomous systems.