The main failure is that the agent can still complete authorised actions even when the instruction is malicious. If you rely only on text filtering or static policy, the attack moves into valid tool use, where normal access checks still pass. Without a runtime baseline, you cannot reliably tell whether the action sequence is expected or coerced.
Why This Matters for Security Teams
Prompt injection becomes materially more dangerous when an AI agent has tool access, because the failure is no longer confined to a bad answer. The agent can be nudged into taking legitimate-looking steps that still produce harmful outcomes, such as sending data, changing records, or chaining calls across systems. Guidance from the NIST AI Risk Management Framework is useful here because it frames AI risk as a lifecycle issue, not just a content-filtering problem.
Security teams often underestimate how quickly this becomes an identity and authorisation issue. If the agent inherits broad permissions, prompt injection can turn a language problem into a privilege problem, especially when there is no runtime baseline for expected tool calls, sequence order, or destination boundaries. Static allowlists help, but they do not prove intent or detect coercion in the live execution path. The practical failure is that the system looks compliant at the policy layer while behaving unsafely at runtime.
In practice, many security teams encounter the blast radius only after an agent has already used legitimate credentials to complete an unauthorised workflow, rather than through intentional security testing.
How It Works in Practice
A runtime baseline defines what “normal” looks like for a specific agent, task, and context. That usually includes the approved tool set, expected call ordering, allowed data scopes, sensitive action thresholds, and the usual timing or dependency pattern for a given workflow. When the agent departs from that baseline, the platform can slow, challenge, isolate, or stop execution before the tool chain causes damage. This aligns with the control intent described in the OWASP Agentic AI Top 10, where prompt injection, excessive agency, and insecure tool use are treated as distinct risks.
Operationally, teams usually need layered controls rather than a single detector. A practical baseline often combines:
- Task-scoped permissions so the agent can only reach the minimum necessary tools and data.
- Action logging that records prompts, tool calls, outputs, and intermediate decisions for later review.
- Policy checks on high-impact actions such as payment changes, admin operations, or data exfiltration routes.
- Human approval for unusual sequences, especially when the agent tries to branch outside its normal workflow.
Runtime baselines are most effective when they are tied to the actual workflow rather than the model alone. That means profiling the normal behaviour of a specific agent in a specific business process, then comparing live execution against that profile. MITRE’s MITRE ATLAS adversarial AI threat matrix is useful for mapping how prompt injection and related techniques can bend model behaviour toward harmful action sequences. These controls tend to break down when agents are granted broad, persistent credentials across many business systems because the baseline becomes too generic to distinguish normal variation from coerced behaviour.
Common Variations and Edge Cases
Tighter runtime control often increases friction, requiring organisations to balance safety against automation speed and operator workload. That tradeoff is especially visible in customer support, IT operations, and security orchestration, where agent behaviour can vary a lot by ticket, incident, or user request. There is no universal standard for this yet, so current guidance suggests starting with the highest-risk actions and expanding baselines from observed production use rather than trying to define every possible path in advance.
Edge cases matter. Retrieval-augmented generation workflows can still be exposed if malicious content enters the retrieved corpus, even when the base model is well governed. Multi-agent systems are harder still, because one compromised agent can influence another through shared context or delegated tasks. The CSA MAESTRO agentic AI threat modeling framework is helpful for thinking about these dependencies, while the Anthropic report on AI-orchestrated cyber espionage shows how tool use, persistence, and execution authority can be combined in real operations.
Where runtime baselines become brittle is in highly dynamic environments with frequent workflow changes, shared service accounts, or poorly segmented tools. In those conditions, security teams need to treat baseline drift as a governance issue, not just a tuning problem, because unsafe exceptions quickly become the new normal.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10, MITRE ATLAS and CSA MAESTRO address the attack and risk surface, while NIST AI RMF and NIST AI 600-1 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST AI RMF | GOV | Runtime baselines need governance, accountability, and risk ownership. |
| OWASP Agentic AI Top 10 | A2 | Prompt injection and unsafe tool use are core agentic AI failure modes. |
| MITRE ATLAS | AML.T0001 | Prompt injection maps to adversarial techniques that steer model behaviour. |
| CSA MAESTRO | TRM | Agentic workflows need threat modelling across tools, delegation, and runtime trust. |
| NIST AI 600-1 | GenAI Profile | Generative AI profiles emphasise secure deployment and output validation. |
Add deployment-time checks for unsafe outputs, tool calls, and policy drift in GenAI systems.
