What breaks when AI agent testing relies on hand picked examples alone?

Why This Matters for Security Teams

Hand-picked examples are useful for smoke tests, but they are a weak proxy for agent quality because agents are goal-driven, tool-using systems rather than static classifiers. A narrow test set often rewards memorisation of common prompts while missing prompt injection, tool misuse, policy drift, and unsafe chain-of-action behaviour. That is why current guidance from the OWASP Agentic AI Top 10 and the NIST AI Risk Management Framework treats evaluation as a risk activity, not a demo activity.

NHI Management Group research shows why this matters operationally: in the AI Agents: The New Attack Surface report, 80% of organisations said their AI agents had already acted beyond intended scope. That is the real failure mode behind hand-picked testing. A curated set may show polish, but it does not expose how agents behave under ambiguous instructions, adversarial prompts, or changing context. In practice, many security teams discover this only after an agent has already touched production systems, rather than through intentional negative testing.

How It Works in Practice

Effective agent testing needs coverage across intent, context, and tool use. Teams should build test suites that include routine requests, malformed prompts, adversarial injection, conflicting goals, and multi-step tasks that force the agent to choose between speed, safety, and policy compliance. The aim is not to prove the agent succeeds on every example. The aim is to learn where it fails, how it fails, and whether failure is contained.

Practitioner guidance is converging on three layers of evaluation. First, create a baseline set of representative tasks that reflect real user journeys. Second, add negative and boundary tests that probe policy violations, privilege escalation, data exfiltration, and tool chaining. Third, run repeated evaluations because agent behaviour can change as prompts, models, tools, and context windows change. The CSA MAESTRO agentic AI threat modeling framework and the MITRE ATLAS adversarial AI threat matrix are useful starting points for structuring those scenarios.

• Use scenario-based tests that mirror real work, not just ideal prompts.
• Include adversarial cases that attempt prompt injection or tool abuse.
• Measure outcomes such as unsafe actions, policy violations, and unauthorized data access.
• Track regressions across model updates, prompt changes, and tool integrations.

For deeper examples of what this looks like in live environments, the Gemini AI Breach and CoPhish OAuth Token Theft via Copilot Studio show how simple test coverage can miss real-world abuse paths. These controls tend to break down when agents are connected to live tools, because the failure is no longer just a bad answer, it becomes an action with side effects.

Common Variations and Edge Cases

Tighter testing often increases evaluation cost and slows release cycles, requiring organisations to balance speed against confidence. That tradeoff is real, especially when teams are shipping multiple agents or frequently changing prompts and tools. The key is to avoid confusing broad coverage with perfect coverage, because there is no universal standard for this yet and best practice is still evolving.

Some environments need extra care. Customer-facing agents should be tested for harmful output, hallucination, and policy breach under abusive user input. Internal agents need heavier coverage for data access, escalation paths, and accidental disclosure. Agents with high-impact actions, such as code execution, financial workflows, or admin tooling, should face stricter evaluation than low-risk assistants. The Analysis of Claude Code Security is a good reminder that code-related agents demand narrower safety margins than chat-only systems.

Where teams go wrong is assuming a single “golden set” can certify behaviour forever. That approach misses distribution shift, new prompt styles, and hidden dependencies in upstream models or downstream tools. A better pattern is continuous evaluation with versioned test sets, red-team cases, and clear pass/fail thresholds aligned to risk. For organisations scaling agent deployments, the Ultimate Guide to NHIs can help contextualise how identity, access, and auditability intersect with agent testing. These controls tend to break down when teams test only against friendly prompts and then assume the agent is safe in adversarial or production conditions.

Related resources from NHI Mgmt Group

• What breaks when AI agent access relies on long-lived secrets?
• What breaks when an AI agent can hand over its own credential context?
• What breaks when AI agent detection relies only on application logs?
• What breaks when AI agent observability relies only on in-process telemetry?