Agent breaker shows how GenAI security testbeds model real attacks

At a glance

What this is: This is Lakera’s explanation of Agent Breaker, a public red teaming testbed for GenAI systems that models realistic attacks and scores how well they succeed.

Why it matters: It matters because practitioners need to test agentic AI, RAG, and tool-using systems against measurable attack paths instead of relying on prompt-only safeguards.

Context

GenAI security risk is no longer limited to prompts typed into a chatbot. Once systems ingest external content, call tools, retain memory, or browse the web, they inherit attack surfaces that look more like application security than simple prompt engineering.

Agent Breaker is Lakera’s attempt to make those failure modes visible in a controlled setting. The article frames the testbed as a way to model realistic attack scenarios, score outcomes, and show how agentic AI systems can be manipulated when defenders focus on prompts instead of behaviour.

Key questions

Q: How should security teams use AI pentesting to test real attack paths?

A: Treat AI pentesting as a way to validate whether an attacker can move from entry to impact, not as a way to count more findings. Define the business-critical paths you care about, then require the test to prove exploitability, chaining, and control failure across those paths. The output should support remediation prioritisation, not just reporting.

Q: Why do partial-success scores matter in GenAI security testing?

A: Partial-success scores matter because many AI attacks do not fail cleanly. A model may leak part of a prompt, follow only some malicious instructions, or expose enough state to make the next exploit easier. Graded scoring shows whether defenses are reducing attack impact, not just whether they are blocking the most obvious case.

Q: What are the signs that GenAI guardrails are not working as intended?

A: Common warning signs include unsafe outputs reaching users, policy violations slipping through, slow or inconsistent enforcement, and operational friction that pushes teams to bypass controls. If AI systems still create reputational, compliance, or customer trust problems despite guardrails, the protection layer is not aligned to actual production behavior and needs re-tuning.

Q: How can teams evaluate whether a red team testbed is useful?

A: A useful testbed reproduces realistic application behaviour, gives clear success criteria, and scores more than simple pass or fail outcomes. It should help teams compare model behaviour under stress, identify recurring exploit paths, and turn findings into repeatable security tests that fit the production environment.

Technical breakdown

How threat snapshots model real GenAI attack paths

A threat snapshot is a compact representation of a real GenAI application and the way it can fail under attack. It combines the application’s purpose, architecture, system context, and the attack objective the player is trying to achieve. In practice, this means a testbed can mirror RAG pipelines, browsing tools, memory-enabled chatbots, and other agentic patterns without exposing real production systems. The technical value is not the game itself, but the repeatable way it turns abstract GenAI risk into a concrete attack surface that can be probed, measured, and compared across levels.

Practical implication: Treat GenAI security testing as a system-level exercise, not a prompt-only review.

Why scoring matters in adversarial AI evaluation

GenAI attacks are rarely binary. A prompt may partially extract a system prompt, partially poison tool output, or only partially hijack an objective. Agent Breaker’s scoring logic captures that uncertainty by combining metrics such as string similarity, lexical overlap, semantic similarity, and classifier-based judgments into a 0 to 100 score. That matters because many AI failures are incremental before they are catastrophic. If teams only track success or failure, they miss the intermediate states where an attacker is already learning the system’s boundaries and weakening the guardrails around it.

Practical implication: Use partial-success scoring to identify attack degradation before a full compromise occurs.

How layered guardrails change the attack surface

The article describes layered defenses that sit before and after model output, including intent classifiers, LLM judges, and Lakera Guard at the final stage. That structure is important because each layer changes what an attacker must do to succeed. Instead of a single prompt vector, the adversary now has to evade multiple gates while still achieving the objective. This is closer to real production conditions, where agents are constrained by filters, classifiers, and policy checks. The result is a more realistic test of whether a defence actually alters attack behaviour or merely raises the bar slightly.

Practical implication: Test controls as a chain of gates, not as isolated filters.

Threat narrative

Attacker objective: The objective is to manipulate a GenAI system into revealing, changing, or misusing information and tools in ways that demonstrate exploitable failure modes.

1. Entry occurs through direct prompt injection or indirect prompt injection hidden inside content that the agent ingests and processes.
2. The attacker then uses the model’s context, tools, or memory to steer the agent toward a harmful objective such as tool leakage, prompt extraction, or poisoned output.
3. Escalation happens when layered guardrails are bypassed or insufficiently aligned to the attack objective, allowing partial or full success.
4. Impact is measured as leaked instructions, manipulated tool use, hijacked purpose, or other adversarial outcomes that reveal how the agent can be controlled.

Breaches seen in the wild

• iOS apps leaking hard-coded secrets: Cybernews found 71% of 156,080 iOS apps leak hard-coded secrets, with open cloud storage and Firebase databases exposing user data.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Agentic security needs measurable attack realism, not prompt etiquette. The article shows that real GenAI risk comes from how systems ingest content, call tools, and generate actions under pressure. That shifts the discipline from prompt cleanliness to adversarial testing of architecture, behaviour, and guardrail interplay. Practitioners should treat attack simulation as a core security control, not a research novelty.

Scoring partial success is the right way to evaluate GenAI failure. Binary pass or fail thinking misses the way many model exploits unfold in stages. A model that only partially leaks a prompt or partially follows malicious instructions is already revealing boundary weakness, and that weakness is operationally meaningful. Security teams should measure degradation, not just catastrophe.

Threat snapshots are a useful named concept because they map system design to exploitability. By packaging the application, state, model context, guardrails, and scoring function together, the testbed makes the failure pattern legible to builders and red teamers. That is more valuable than abstract risk labels because it ties each weakness to a concrete mechanism. Teams should build similar threat-centric views of their own agentic systems.

Layered guardrails only work when they change attacker economics. The article’s progression from intent classifiers to LLM judges to final-stage blocking reflects the reality that no single defence covers every vector. If a control does not materially change the attack path, it is just another obstacle, not governance. Practitioners should evaluate whether each layer forces the adversary into a different and weaker tactic.

Distributed red teaming will become a default pattern for GenAI assurance. The article points to a model where many testers probe many applications at scale, producing data that manual review cannot match. That matters because GenAI systems evolve too quickly for sporadic testing to be enough. Security programmes should expect more continuous, crowd-assisted validation of agentic and tool-using systems.

From our research library:

• Generative AI use specifically increased from 33% in 2023 to 79% in 2025, according to McKinsey’s Global Surveys on the State of AI.

What this signals

Attack-aware evaluation is becoming a requirement for agentic AI programmes. Teams that rely on prompt review alone will miss the real failure surface, because the exploit often lives in the relationship between external content, tools, memory, and scoring. That means testing has to move closer to production behaviour and away from isolated prompt experiments.

GenAI assurance should be built around exploit reproducibility. If a weakness can be triggered once, it should be turned into a repeatable test case that survives model updates, guardrail changes, and new tool integrations. Otherwise, the same failure mode will keep reappearing under a different name.

For practitioners

• Define threat snapshots for your own GenAI apps Document the application purpose, model context, tool set, memory behaviour, and likely attack objectives so testers can reproduce realistic adversarial conditions.
• Measure partial success, not just complete compromise Use scoring or graded evaluation to capture prompt leakage, tool misuse, and partial policy bypasses before those states turn into full exploit paths.
• Test indirect prompt injection paths Review external documents, webpages, logs, and retrieved content that your agent ingests, then verify those sources cannot redirect behaviour through hidden instructions.
• Evaluate guardrails as a layered chain Assess whether each classifier, judge, or filter actually changes attacker behaviour, rather than assuming that stacking controls automatically improves security.
• Create repeatable red team scenarios Turn the highest-value attack paths into reusable test cases so engineers can compare control changes over time and track whether the same exploit still works.

Key takeaways

• GenAI testbeds are most useful when they model realistic attack paths rather than abstract prompt tricks.
• Partial success is still a meaningful security outcome because it shows where controls are weakening before full compromise.
• Security teams need repeatable adversarial tests for tool-using and retrieval-enabled systems, not one-off prompt reviews.

Key terms

• Threat Snapshot: A threat snapshot is a focused test case that freezes one attack moment, one system state, and one objective so security can be measured consistently. It lets teams compare how a model behaves under the same adversarial pressure rather than relying on broad, hard-to-reproduce agent simulations.
• Indirect Prompt Injection: Indirect prompt injection is an attack where malicious instructions are hidden inside content that an AI system reads later. The model may treat that content as context rather than as hostile input, which can influence tool use, data access, or workflow actions if controls are weak.
• Partial Success Scoring: Partial success scoring measures how far an attack progresses when the outcome is not fully binary. In GenAI testing, this is important because a model can leak, deviate, or comply only partly while still revealing a real security weakness that matters operationally.
• Guardrail Layering: Guardrail layering is the practice of combining multiple independent controls so that one failure does not expose the full system. In AI security, that usually means pairing cloud configuration controls, model behaviour checks and identity restrictions across the same workflow.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity lifecycle are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are building or maturing an IAM programme, it is worth exploring.