Agent breaker shows how GenAI security testbeds model real attacks

At a glance

What this is: Agent Breaker is Lakera’s GenAI security playground that turns real attack patterns into scored red-teaming challenges.

Why it matters: It matters because IAM and security teams now have to govern agent behaviour, tool access, and guardrails as part of the identity perimeter, not as an afterthought.

👉 Read Lakera's analysis of Agent Breaker and GenAI attack mechanics

Context

GenAI security failures are not limited to bad prompts. Once an agent can read retrieved content, call tools, or act on external documents, the real problem becomes what it is allowed to ingest, infer, and do under pressure. That shifts the governance question toward agent identity, tool trust, and the controls surrounding delegated access.

Agent Breaker frames those failures as repeatable test cases rather than abstract risk statements. For teams building or governing AI-enabled workflows, the useful question is not whether a model can be tricked in theory, but which control failed first when the system was given realistic inputs and layered defenses.

Key questions

Q: How should security teams test GenAI agents for prompt injection risk?

A: Security teams should test GenAI agents with both direct prompts and indirect inputs such as documents, webpages, and tool metadata. The goal is to see whether the agent treats untrusted content as instruction, leaks protected context, or performs unintended tool actions. Effective testing measures partial success, not only total compromise, because small leaks and minor tool misuse are often the first signs of control failure.

Q: Why do AI agents create identity governance problems that standard app controls miss?

A: AI agents can ingest content, choose actions, and invoke tools inside the same runtime, so the security question becomes who or what is authorised to influence those actions. Standard app controls often stop at authentication or input filtering, but agent risk lives in delegated authority, dynamic context, and runtime decision-making. That is why identity governance must extend into tool access and execution boundaries.

Q: What do teams get wrong about defending against indirect prompt injection?

A: Teams often treat indirect prompt injection as a content safety issue when it is really a trust-boundary issue. The problem is not only that malicious text exists, but that the agent is allowed to interpret that text as operationally relevant. Defences need to isolate untrusted input from execution authority, otherwise the model can be manipulated through the very content it is supposed to process.

Q: How can organisations measure whether GenAI guardrails are actually working?

A: Organisations should measure whether guardrails prevent partial leaks, partial tool misuse, and objective drift, not just whether they block obvious jailbreaks. A useful program looks for how far an attack progressed, what information escaped, and whether the agent still behaved inside its intended scope. If the system only tracks binary success, it will miss the most common real-world failure states.

Technical breakdown

Threat snapshots as agentic attack models

Agent Breaker packages each level as a threat snapshot, meaning a compact model of a real GenAI application with a defined objective, vector, and scenario. That structure matters because it mirrors how production systems fail: through a combination of model context, retrieval content, tool exposure, and user interaction. The article’s important technical point is that the attack surface is not just the prompt. It is the whole runtime assembly around the model, including what it can see, what it can call, and how its output is scored.

Practical implication: treat agent security as an end-to-end system problem, not a prompt-filtering problem.

Indirect prompt injection through retrieved content and tools

The article distinguishes direct attacks from indirect prompt injection, where malicious instructions hide inside content the agent ingests, such as a document, webpage, or tool description. That is a crucial architectural distinction because the model is not being persuaded only by the user. It is being influenced by untrusted context that the system itself chose to trust. In practice, this means the attack path often enters through data flows, then converts into tool misuse or objective hijacking once the agent processes that content.

Practical implication: classify retrieved content and tool metadata as attack inputs, not passive configuration.

Scoring partial success is how GenAI risk becomes measurable

Agent Breaker scores outcomes from 0 to 100 using a mix of exact match, lexical overlap, semantic similarity, and classifier-based checks. That approach reflects a real GenAI security reality: attacks are often partially successful, not binary. A model may leak some of a system prompt, reveal one tool, or drift partway toward a malicious objective. The useful insight is that evaluation in agentic systems must capture gradations of failure, because operational risk often begins before a fully catastrophic output appears.

Practical implication: measure partial compromise states, not only complete exploit success.

Threat narrative

Attacker objective: The attacker aims to manipulate a GenAI agent into leaking protected context, abusing tools, or executing a malicious objective.

1. Entry occurs through direct user prompts or indirect prompt injection embedded in external content that the agent ingests.
2. Credential or control abuse follows when the agent treats untrusted instructions as context and proceeds to reveal tools, leak prompts, or misuse functions.
3. Impact lands as objective hijacking, tool poisoning, or sensitive information extraction that changes the agent’s behaviour in production-like settings.

Breaches seen in the wild

• Moltbook AI agent keys breach — Moltbook breach exposed 1.5M AI agent keys.
• AI LLM hijack breach — attackers used stolen AWS access keys to hijack Anthropic LLM models on Bedrock.

Read our 52 NHI Breaches Analysis report for a comprehensive view of breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Agentic security testing is now an identity governance problem, not just an LLM safety exercise. The article shows that once a model can read external context and call tools, the meaningful control surface becomes delegated access. That moves the discussion from prompt quality to authority boundaries, which is exactly where identity teams operate. Practitioners should treat the agent runtime as an identity-bearing system with constrained permissions, observable actions, and explicit trust boundaries.

Indirect prompt injection is really a trust-boundary failure between the agent and the content it consumes. A webpage, PDF, or tool description becomes dangerous when the system treats it as operationally authoritative rather than untrusted input. That means the security model fails at the ingestion layer before the model ever decides anything. Practitioners should separate content ingestion from execution authority and verify where untrusted text can still influence tool use.

Scored partial success creates a better mental model for GenAI risk than all-or-nothing compromise. The article’s metric design recognizes that leaking half a prompt or nudging a tool call is still meaningful failure. That is important for governance because many review processes only register fully visible incidents. Practitioners should build control thresholds around partial compromise states, not just final exploit outcomes.

Runtime guardrails are only as strong as the assumptions behind them. The level progression from weak defenses to classifier checks and final filtering shows how attackers adapt to each layer. That is a reminder that policy controls need to assume hostile inputs, not cooperative users, and that the effective question is whether the system can remain safe when context itself is adversarial. Practitioners should validate controls against live attack patterns, not synthetic happy paths.

Agent breaker-style testbeds are becoming the fastest way to translate AI risk into governance evidence. Because the levels mirror real production setups, they expose which controls fail first under realistic pressure. That gives security, IAM, and red-team functions a shared language for discussing privilege, delegation, and abuse pathways. Practitioners should use testbeds to map failure modes back to identity and access decisions.

From our research:

• The average estimated time to remediate a leaked secret is 27 days, despite 75% of organisations expressing strong confidence in their secrets management capabilities, according to LLMjacking: How Attackers Hijack AI Using Compromised NHIs.
• 43% of security professionals are concerned about AI systems learning and reproducing sensitive information patterns from codebases, which shows the governance issue is already affecting design assumptions.
• That concern connects to Ultimate Guide to NHIs , 2025 Outlook and Predictions, where the shift toward agentic systems makes runtime identity and access control a first-order problem.

What this signals

Agentic AI expands the attack surface faster than most governance programmes can adapt. The practical problem is not just malicious prompts, but the combination of delegated tools, external context, and runtime decision-making. Teams should expect red-team findings to map less to model quality and more to identity, access, and trust-boundary decisions.

With 43% of security professionals already worried about AI systems learning sensitive patterns from codebases, the concern is no longer hypothetical. That figure points to a broader programme issue: controls around context ingestion and output leakage must be designed as part of identity governance, not left to application teams alone.

Runtime trust will become the decisive control concept for GenAI programmes. As agents begin to act on retrieved content and tool metadata, organisations need to know where untrusted input can still become authority. That is where frameworks such as the OWASP Agentic AI Top 10 and the NIST AI Risk Management Framework become operationally useful.

For practitioners

• Separate ingestion trust from execution trust Mark retrieved documents, webpages, and tool descriptions as untrusted until they pass a policy check that is independent from the model’s reasoning layer. This reduces the chance that hidden instructions become operational commands.
• Scope agent tool access to the smallest viable action set Inventory every tool an agent can call and remove broad permissions that are not required for the task. Reassess those grants as soon as the agent’s use case changes, especially where external content can influence execution.
• Test for partial compromise, not just complete jailbreaks Use red-team exercises and scoring criteria that detect partial leaks, partial tool misuse, and partial objective drift. Treat those outcomes as governance failures because they often precede a full compromise.
• Map agent controls to identity governance reviews Include agent runtime behaviour, tool delegation, and content ingestion paths in access reviews and design reviews. If a control cannot explain who can cause the agent to act, it is incomplete.

Key takeaways

• Agent Breaker turns GenAI risk into a testable model of how agents fail under pressure, which makes identity and access decisions visible.
• The strongest security signal in the article is not prompt cleverness but the repeated failure of trust boundaries, scoring, and delegated tool use.
• Practitioners should treat GenAI guardrails as governance controls that need adversarial testing, partial-compromise metrics, and clear authority boundaries.

Key terms

• Threat Snapshot: A threat snapshot is a compact model of a real application attack surface, built to reproduce a specific failure pattern. In GenAI systems, it combines the objective, vector, scenario, and defenses so teams can test how the runtime behaves under realistic pressure.
• Indirect Prompt Injection: Indirect prompt injection is a technique where malicious instructions are hidden inside content an AI system ingests, such as a webpage, document, or tool description. The model then follows the injected instructions instead of the user’s intent because the system treated untrusted context as authoritative.
• Partial Compromise: Partial compromise is a state where an attack does not fully succeed, but still causes meaningful leakage, drift, or misuse. In GenAI security this matters because a model can reveal fragments of a prompt, expose a single tool, or shift behavior enough to create real operational risk.
• Runtime Guardrail: A runtime guardrail is a control that evaluates or constrains model behaviour while the system is running, not just at build time. For agentic systems it may inspect input, output, tool calls, or intent, but it only works if the trust boundary is enforced outside the model itself.

What's in the full article

Lakera's full article covers the technical design detail this post intentionally leaves at a higher level:

• How the ten mock agentic apps are structured and how each threat snapshot maps to a real production pattern
• The scoring logic for partial success, including how BLEU, ROUGE, embedding similarity, and classifiers are combined
• Why guardrails change from Level 1 to Level 5 and how the intent classifier, LLM judge, and final filter interact
• Examples of the attack objectives used in the playground, such as tool extraction, prompt extraction, and toxicity injection

👉 Lakera's full article covers the threat snapshot design, scoring model, and guardrail progression in more detail

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity lifecycle are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are responsible for identity security strategy or NHI governance in your organisation, it is worth exploring.