AI systems increase risk when they can access databases, APIs, or business workflows with more privilege than the user or the task requires. Prompt injection can steer the model into exposing sensitive data or taking unintended actions. The risk grows as AI becomes connected to real systems, because one successful manipulation can affect many downstream operations.
Why This Matters for Security Teams
AI agents and LLM applications change the access model. A chat interface may look harmless, but the underlying system often has live connections to email, files, code repositories, ticketing platforms, cloud APIs, and internal knowledge stores. That means a single model interaction can move from text generation into data retrieval, action execution, and credential use. Current guidance from the NIST AI Risk Management Framework is clear that organisations need to treat AI as a governed system, not just a model.
The security issue is not only direct exfiltration. Prompt injection, tool abuse, insecure retrieval, and excessive delegation can all cause the agent to surface records it should not see or perform actions beyond the task request. When the AI has access to sensitive business systems, the blast radius can become much larger than a conventional user session because the agent may operate faster, across more systems, and with fewer human checks. This is especially important where the application handles customer data, source code, financial records, or privileged workflows.
In practice, many security teams encounter the weakness only after an agent has already exposed sensitive data or approved an action that no human intended.
How It Works in Practice
The main failure mode is privilege mismatch. An AI agent is granted broad tool access so it can “help,” but the task only requires a narrow slice of that access. If the model is tricked, misled, or simply behaves unpredictably, it can query too much data, pass secrets into prompts, or chain actions across systems. The problem becomes more severe when retrieval-augmented generation, plugins, browser tools, or workflow automation are attached to the same session.
Security teams should think in terms of trust boundaries and execution paths. The model is not just producing text; it is consuming untrusted input, deciding what to retrieve, and sometimes issuing commands. That creates multiple points where sensitive data can leak:
- Training or prompt content can contain hidden instructions that override policy.
- Retrieval systems may return documents that were not intended for that user or task.
- Tool calls may expose tokens, API responses, or internal metadata into logs.
- Agent memory can persist sensitive context beyond the immediate task.
Frameworks such as the OWASP Agentic AI Top 10 and the MITRE ATLAS adversarial AI threat matrix are useful because they map these failure paths to practical threat scenarios. CSA MAESTRO agentic AI threat modeling framework is also helpful when teams need to model where the agent starts, what it can touch, and how outputs are validated before action is taken.
Operationally, good controls usually include least privilege for tools, per-task scoping, output filtering, human approval for high-impact actions, secrets isolation, and strong logging that separates user input from model instructions. These controls tend to break down when the agent is wired directly into production systems with shared credentials and no transaction-level approval because the model can then act faster than the review process can intervene.
Common Variations and Edge Cases
Tighter agent controls often increase build complexity and workflow friction, requiring organisations to balance automation benefits against governance overhead. That tradeoff becomes visible in environments that want fast self-service while also handling regulated, confidential, or safety-sensitive data.
Best practice is evolving for agents that act on behalf of a user versus agents that act as system operators. There is no universal standard for this yet, but the practical rule is simple: the more autonomy and data access the agent has, the more it needs containment, validation, and auditability. For example, an internal support chatbot may only need read-only search access, while a procurement agent may need approval gates before creating or modifying records.
Edge cases often include shared service accounts, long-lived memory, and multi-agent workflows. These patterns can reintroduce OWASP Non-Human Identity Top 10 concerns because the agent is effectively operating as a non-human identity with secrets, permissions, and audit requirements. Where agent outputs feed downstream automation, validation should happen before the output is trusted, not after the action has already executed. That is also why the NIST AI 600-1 Generative AI Profile and the NIST AI Risk Management Framework are best used together: one focuses attention on generative AI specifics, while the other anchors broader governance and control design.
The cleanest implementations assume the model may be manipulated at any point and design for containment first. That approach is especially important where the agent can reach privileged data stores, because the damage from one compromised interaction is often broader than teams expect.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10, MITRE ATLAS and OWASP Non-Human Identity Top 10 address the attack and risk surface, while NIST AI RMF and NIST AI 600-1 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP Agentic AI Top 10 | Directly maps common agent abuse paths like tool misuse and prompt injection. | |
| MITRE ATLAS | Covers adversarial AI tactics that drive data leakage and unsafe actions. | |
| NIST AI RMF | Frames AI as a governed risk domain requiring mapping, measurement, and management. | |
| NIST AI 600-1 | Generative AI guidance is relevant to prompt, output, and retrieval risks. | |
| OWASP Non-Human Identity Top 10 | Agents often operate with credentials and secrets like a non-human identity. |
Model likely AI attack paths and test whether the agent resists manipulation and exfiltration attempts.
