What breaks when agent evals are built only from synthetic examples?

Synthetic examples tend to reproduce the inputs teams already anticipated, so they miss the strange paths and combined failures that appear in real use. The result is false confidence: the suite looks complete, but production behaviour still escapes it. Teams need live traces to capture unknown trajectories and to turn real failure patterns into useful test cases.

Why This Matters for Security Teams

Synthetic-only evals fail at the exact point where agentic systems become risky: the gap between a curated test prompt and an unpredictable production trajectory. Synthetic sets are useful for bootstrapping, but they usually reflect known intents, tidy workflows, and assumptions about tool use. That leaves blind spots in prompt injection handling, tool misuse, state drift, and cascading errors across multi-step actions. Guidance in the OWASP Agentic AI Top 10 and the NIST AI Risk Management Framework both point toward continuous measurement of real-world behaviour, not just lab success.

The operational risk is that teams optimise for benchmark cleanliness instead of resilience. A model or agent can score well on synthetic tasks while still failing under messy context, adversarial inputs, or ambiguous user intent. That matters for governance too: leaders may believe they have validated safety, while the system has only been validated against scenarios the builders already imagined. In practice, many security teams encounter these failures only after an agent has already executed the wrong action or leaked sensitive context, rather than through intentional pre-production discovery.

How It Works in Practice

Effective eval design usually needs a layered approach. Synthetic examples still have value for coverage, regression testing, and safe reproduction of known failure classes, but they should be treated as one input stream rather than the source of truth. Live traces, red-team transcripts, incident logs, and production telemetry reveal the branching behaviour that synthetic suites often miss. That is especially important for agents with tool access, memory, retrieval, or delegated execution authority, because failure often emerges from the interaction between components rather than from a single model response.

Practitioners should map evals to the actual risk surface, then tune the test corpus to capture both expected and emergent behaviour. The MITRE ATLAS adversarial AI threat matrix is useful for thinking about attack patterns, while the CSA MAESTRO agentic AI threat modeling framework helps teams reason about agent workflows, trust boundaries, and tool exposure. In practice, a strong workflow includes:

• Recording real prompts, tool calls, and outcomes, then sanitising and curating them into a replay set.
• Classifying failures by impact, such as unsafe action, policy bypass, hallucinated action, or data exposure.
• Adding adversarial variants that mutate successful production traces rather than inventing isolated examples.
• Re-running the suite whenever tools, policies, retrieval sources, or system prompts change.

This approach works best when the test corpus is tied to operational events and reviewed by both security and product owners. These controls tend to break down when agents operate over highly dynamic external tools or rapidly changing knowledge sources because the observed behaviour becomes stale before the eval suite is refreshed.

Common Variations and Edge Cases

Tighter evaluation coverage often increases data handling, review effort, and privacy overhead, requiring organisations to balance realism against operational cost. That tradeoff is especially acute when teams want to use production traces but must avoid exposing personal data, secrets, or regulated content. In those cases, current guidance suggests sanitisation, selective sampling, and strong access controls rather than a blanket ban on real traces.

There is no universal standard for how much synthetic data is enough, because the answer depends on the agent’s autonomy, tool scope, and blast radius. A customer-support chatbot with limited retrieval is not the same as an agent that can send emails, modify records, or trigger workflows. For higher-risk systems, the bar should rise toward scenario replay, adversarial mutation, and live monitoring aligned to NIST AI Risk Management Framework practices. For more advanced agentic environments, the OWASP Top 10 for Agentic Applications 2026 is a practical reference for where synthetic-only testing is most likely to miss unsafe execution paths.

The biggest edge case is when teams confuse model quality with system safety. A strong base model can still fail badly once retrieval, tools, permissions, and human handoffs are added. That is why synthetic-only evals are least reliable in agentic environments where the security question is not just what the model says, but what the system does next.

Related resources from NHI Mgmt Group

• What breaks when agent connectivity is built without a runtime control layer?
• What breaks when agent memory is built only from retrieval and vector storage?
• What breaks when ownership for an agent is tied to the person who built it?
• What breaks when sandbox validation does not match actual execution in agent systems?