AI workloads can act on instructions, invoke tools, and change state in ways classic web services do not. That creates a broader attack surface that includes hidden prompts, tool use, external links, and model-driven decisions. Security teams need to verify what the system is allowed to do, not only what it is expected to answer.
Why This Matters for Security Teams
AI agent and LLM workloads change the trust model because the system is no longer just producing text. It may call tools, retrieve data, trigger workflows, and carry forward hidden state. That means offensive testing has to examine instruction handling, action boundaries, and identity trust, not only prompt quality. Guidance from the NIST AI Risk Management Framework is useful here because it frames AI risk as a lifecycle governance problem, not a one-time validation task.
Security teams often miss that the real issue is delegated authority. If an AI agent can open tickets, query systems, send messages, or reach sensitive records, then a harmless-looking prompt can become a control failure. Offensive discovery must therefore test what the model is permitted to do, what it is tricked into doing, and what it can reach through integrations. That is why current best practice is evolving toward threat modeling that includes agent permissions, tool exposure, and provenance of inputs. In practice, many security teams encounter the trust gap only after an agent has already touched a real system through an over-permissive integration, rather than through intentional testing.
How It Works in Practice
Effective testing starts by mapping the agent’s operational envelope. That includes the model, the orchestration layer, external tools, retrieval sources, and any identity used for execution. A useful baseline is to separate conversational risk from action risk: a model may be acceptable at answering questions while still being unsafe if it can modify records, approve requests, or invoke shell-like tools. The OWASP Agentic AI Top 10 is helpful for structuring those tests around prompt injection, excessive agency, data leakage, and insecure tool use.
Practitioners usually test four layers together:
- Prompt and context controls, including hidden instructions and retrieval contamination.
- Tool and API authorization, including least privilege and approval gating.
- Identity and workload trust, including service identity, token scope, and session lifetime.
- Output validation, including whether model output can drive unsafe automation.
Workload identity matters because an agent that authenticates as a generic service account can blur accountability and expand blast radius. The SPIFFE workload identity specification is relevant when organizations need verifiable, short-lived identity for machine-to-machine trust. Offensive discovery should also check whether retrieval sources, browser plugins, and external connectors can be abused to smuggle instructions into the agent’s decision path. For threat pattern mapping, the MITRE ATLAS adversarial AI threat matrix helps teams distinguish model abuse from ordinary application abuse.
These controls tend to break down when the agent operates across fragmented environments with shared credentials, uncatalogued plugins, or legacy automation that was never designed for per-action authorization.
Common Variations and Edge Cases
Tighter agent governance often increases latency and integration overhead, requiring organisations to balance safer action controls against delivery speed and user experience. That tradeoff is real, especially where teams want autonomous workflows but still need approval for sensitive steps.
Some environments are easier to secure than others. Closed internal copilots with a narrow tool set can often be tested with standard red-team methods, while open-ended agents connected to email, browsers, code repositories, and operational systems require broader discovery. There is no universal standard for how many tools is too many, but current guidance suggests reducing agency wherever the business does not need full autonomy.
Edge cases also appear in mixed-trust workflows. An LLM that drafts a response may be low risk, but the same model can become high risk if downstream automation executes its output without validation. Another common issue is identity overlap: if one workload identity is reused across multiple agents, attribution and containment become weak. Offensive testing should therefore verify per-agent identity, per-tool scope, and per-step authorization rather than assuming one control covers the whole chain.
Where external data is involved, teams should treat retrieved content as potentially hostile until validated. That is especially important in environments with untrusted documents, browser access, or third-party connectors. For a broader governance lens, the NIST AI 600-1 Generative AI Profile and the CSA MAESTRO agentic AI threat modeling framework both reinforce that discovery should cover provenance, tool trust, and operational guardrails, not just model behavior.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10 and MITRE ATLAS address the attack and risk surface, while NIST AI RMF, NIST AI 600-1 and NIST CSF 2.0 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST AI RMF | AI risk governance applies to delegated actions and model-driven decisions. | |
| OWASP Agentic AI Top 10 | Agentic app risks include prompt injection, excessive agency, and unsafe tool use. | |
| MITRE ATLAS | ATLAS models adversarial tactics against AI systems and their orchestration layers. | |
| NIST AI 600-1 | GenAI profile emphasizes provenance, validation, and operational safeguards. | |
| NIST CSF 2.0 | PR.AC-4 | Least privilege is central when agents can invoke tools and reach systems. |
Map offensive tests to ATLAS tactics to cover model abuse, data poisoning, and manipulation paths.
