LLM security validation is shifting from theory to live testing

At a glance

What this is: SafeBreach has added AI Security Validation scenarios for OWASP LLM risks, turning prompt injection, exfiltration, and system prompt leakage into testable attack paths.

Why it matters: IAM and security teams need this because LLMs and AI agents inherit identity, data, and access assumptions that traditional vulnerability tools do not assess well.

👉 Read SafeBreach's coverage of OWASP Top 10 for LLM validation

Context

LLM security has moved from a theoretical governance issue to a live control problem because production models are now connected to data, users, and increasingly to operational workflows. The primary gap is simple: most traditional scanners are not built to test language-driven abuse, guardrail bypass, or disclosure through model responses, which leaves a gap between AI adoption and AI assurance.

That gap matters to identity and access programmes because LLMs are not isolated systems. When they can read data, return sensitive content, or eventually call tools, they become part of the access surface and the identity surface at the same time. That is where NHI governance, workload identity, and policy enforcement begin to intersect with AI security validation, rather than sitting in separate silos.

Key questions

Q: How should security teams test LLM guardrails across multiple languages?

A: Security teams should test the same harmful intent in every major language and in code-switched variants, then compare block, warn, and allow outcomes. The goal is to prove that policy enforcement is intent-based rather than English-based. If equivalent prompts produce different outcomes, the control is inconsistent and should not be trusted in production.

Q: Why do connected LLMs create new access-control risk?

A: Because the model sits between users and sensitive data, it can become a disclosure path even when no infrastructure component is breached. Access control must therefore cover retrieval scope, prompt handling, and output behaviour. If the model can see and speak sensitive data, then its response path is part of the access model.

Q: What do security teams get wrong about prompt injection defence?

A: They often assume better blocklists will solve the problem, but obfuscation simply changes the shape of the payload. Real defence requires examining meaning across the full interaction, including retrieved content and model responses. If the control cannot interpret intent, it will keep missing the attack class it is meant to stop.

Q: How should organisations govern external tools used by AI agents?

A: Organisations should review external tools as security inputs, not convenience features. Each tool needs ownership, approval, metadata inspection, and ongoing monitoring for hidden instructions or unexpected behaviour. If an AI agent can act on a tool, then the tool’s provenance and control status should be governed like any other sensitive integration.

Technical breakdown

Why prompt injection bypasses normal application testing

Prompt injection is a control-bypass problem, not a classic code exploit. The attacker shapes input so the model follows malicious instructions instead of the system prompt or policy intent. Because the model interprets language probabilistically, the abuse happens inside the same channel used for legitimate requests, which makes static scanning and conventional WAF-style controls insufficient. The issue is not just that the model can be tricked. It is that trust boundaries in LLM systems are soft, stateful, and highly dependent on prompt design, retrieval content, and tool permissions.

Practical implication: test the model’s instruction hierarchy and refusal behaviour as a control, not just the hosting environment.

Sensitive information disclosure in connected LLMs

Sensitive information disclosure occurs when a model returns data it should not reveal, either because the prompt elicits it, the retrieval layer exposes it, or the model has been connected to content it should not answer from. In enterprise settings, the model often sits between users and sensitive stores, so leakage can happen without any obvious infrastructure breach. That is why the risk is governance-heavy: access scope, retrieval boundaries, and response filtering all matter. If the model can see it, and the model can speak it, the containment model must treat output as an access decision.

Practical implication: limit retrieval scope and classify model outputs as part of your data access control model.

Why agentic AI raises the ceiling on LLM risk

Once an LLM can call tools, retain memory, or chain actions, the same weaknesses become much more consequential. A prompt injection that only produces a bad answer in a chatbot can become a tool misuse or data movement event in an agent. That is because the model is no longer only generating text. It is participating in decisions, executing steps, and extending trust across systems. In identity terms, the agent begins to behave like a non-human identity with delegated authority, which means its privilege boundaries must be explicit, short-lived, and continuously validated.

Practical implication: treat agent tool access like privileged access and constrain it to the narrowest task scope possible.

Threat narrative

Attacker objective: The attacker wants to extract sensitive information or force the model and its connected tools to perform actions outside intended policy boundaries.

1. Entry begins when an attacker reaches an LLM through a normal user or application interface and manipulates the input language.
2. Escalation occurs when the model follows malicious instructions, reveals sensitive content, or executes an unsafe tool-linked action.
3. Impact is the disclosure of guarded data or the misuse of connected systems through the model’s delegated access.

NHI Mgmt Group analysis

LLM security validation is becoming a control discipline, not a product feature. The article shows why reading the OWASP taxonomy is not enough when models are already in production. Security teams need empirical evidence about what the model will do under attack, because guardrails, retrieval scope, and output policies fail in different ways depending on deployment. The practical conclusion is that AI assurance belongs in the testing program alongside other control validation.

Prompt injection is really a trust-boundary failure inside the application stack. The model is not vulnerable in the same way as a library or endpoint is vulnerable. Instead, the boundary between user intent, system instructions, and delegated model behaviour becomes contestable. That matters for identity governance because the same boundary will later govern AI agents acting as non-human identities. Practitioners should start treating instruction privilege as a security object.

System prompt leakage and sensitive disclosure define a named concept: the LLM trust gap. This is the gap between what the business assumes the model will withhold and what the model can actually reveal when prompted or misconfigured. Once a model is connected to data, the confidence that “nothing was hacked” becomes misleading, because disclosure can happen through authorised pathways. The practical conclusion is that data access reviews must extend to model response paths.

Agentic AI will turn today’s LLM risk into delegated-access risk. The article correctly flags that autonomy amplifies severity. When an LLM can call tools, the question changes from “what can it say?” to “what can it do with the privileges it inherits?” That is where NHI governance becomes central. Security teams will need task-scoped, time-bound, and continuously verified permissions for agents, not permanent assumptions about model trust.

Security validation will become the common language between AI teams, IAM teams, and assurance teams. OWASP gives the taxonomy, but validation gives the evidence. That combination is likely to become the default way enterprises separate AI experimentation from production assurance. Practitioners should expect AI testing to join existing control validation workflows, especially where models touch sensitive data or operational systems.

What this signals

LLM assurance will increasingly overlap with identity governance. As models move from chat to action, the boundary between AI security and NHI control becomes operational rather than theoretical. Teams that already manage service accounts, secrets, and delegated access will be better placed to govern AI systems that inherit similar trust patterns. The practical signal is that AI security teams and IAM teams will need shared validation criteria, especially where models touch production data or tools.

Prompt injection testing should become part of change control for AI deployments. A model update, prompt change, retrieval source change, or tool integration can alter risk without any visible infrastructure defect. That means release management for AI needs the same discipline as other privileged systems, including pre-production validation and post-change re-testing. If the deployment is reachable, it is part of the attack surface.

Agentic expansion will make short-lived privilege the safer default. The more autonomy a model gains, the less defensible persistent access becomes. Practitioners should plan for task-scoped permissions, explicit revocation, and monitoring that can prove what the agent accessed and why. This is where NHI governance, workload identity, and AI assurance will converge in the same programme.

For practitioners

• Validate prompt refusal paths Run repeatable tests against system prompts, guardrails, and retrieval prompts to see whether malicious instructions are blocked or followed under realistic abuse conditions.
• Constrain retrieval to least privilege Limit which documents, records, and knowledge sources a model can access, then review whether output filtering prevents sensitive information disclosure when those sources are queried.
• Treat agent permissions as NHI permissions If an LLM can call tools or actions, assign narrow, task-specific access and review it like a non-human identity with explicit expiry and revocation rules.
• Add AI scenarios to assurance cycles Include prompt injection, exfiltration, and system prompt leakage in recurring validation so AI security moves from one-time review to continuous control testing.

Key takeaways

• LLM risk is now a validation problem, not just a policy problem, because attackers can test guardrails against the live model and its connected data.
• Prompt injection, disclosure, and prompt leakage expose a trust gap between what organisations think the model will refuse and what it can actually reveal.
• As agents gain tool access, the right control model shifts toward short-lived, task-scoped permissions managed like non-human identity privileges.

Key terms

• Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads — causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.
• Sensitive Information: Sensitive information is a higher-risk category of personal data that attracts stricter handling requirements because misuse can create greater harm. In Australia this includes biometrics, health information, political opinions, and criminal history, which means controls for collection, access, disclosure, and transfer need tighter governance than ordinary personal data.
• System Prompt: The system prompt is the core instruction set that defines an LLM’s behaviour, boundaries, and response style during a session. When attackers influence or override it, they are not merely changing text. They are trying to change the model’s governing authority.
• AI Security Validation: AI security validation is the practice of testing model behaviour against realistic attack scenarios to see whether safeguards actually hold. It turns AI risk from a theoretical checklist into an observable control outcome, especially for prompt injection, data leakage, and tool-use abuse.

What's in the full article

SafeBreach's full coverage covers the operational detail this post intentionally leaves for the source:

• Attack scenario descriptions for malicious general actions, malicious cyber actions, exfiltration, and system prompt leakage
• Coverage specifics for AWS Bedrock first and Azure OpenAI as the follow-on platform
• Blocked or not blocked result handling and how to interpret each test outcome
• The relationship between the Hacker's Playbook and AI Security Validation in the CTEM workflow

👉 SafeBreach's full post covers the test categories, platform coverage, and guardrail implications in more detail.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management. It helps practitioners connect identity controls to emerging AI and automation risks across the programme.