Teams miss the AI-specific attack surface, especially prompt handling, output manipulation, retrieval poisoning, and tool misuse. A standard pen test can prove infrastructure exposure, but it cannot reliably measure whether the model will follow malicious instructions or misuse delegated access. That leaves a major assurance gap for agentic systems.
Why This Matters for Security Teams
Traditional penetration testing is designed to find exploitable weaknesses in infrastructure, applications, and identity paths. ai red teaming has a different target: the model, its prompts, its retrieval layer, its tool permissions, and the policies that constrain output. When teams treat those as the same exercise, they often validate the wrong thing and miss the failure modes that matter most for safety and trust.
That distinction is especially important for agentic systems, where a model can take actions through APIs, plugins, or workflow tools. A clean network scan says little about whether a prompt injection can redirect behaviour, whether retrieved content can poison answers, or whether the system will over-disclose sensitive data under adversarial input. Current guidance from NIST AI Risk Management Framework treats these as governance and risk problems, not just technical vulnerability checks.
In practice, many security teams encounter AI red teaming only after a model has already produced unsafe actions or leaked sensitive context, rather than through intentional assurance design.
How It Works in Practice
AI red teaming should exercise the complete AI interaction path, not just the hosting environment. That means testing system prompts, conversation state, retrieval sources, agent tools, policy filters, output validation, and human override paths. It also means using adversarial scenarios that are specific to AI, such as prompt injection, jailbreak attempts, data exfiltration through model responses, retrieval poisoning, and tool abuse.
For model-centric testing, teams should compare benign and hostile prompts to see how the system changes behaviour under pressure. For retrieval-augmented systems, they should inspect whether hostile documents or poisoned knowledge entries can change answers without triggering review. For agentic workflows, the focus must include whether the model can reach actions it should not perform, especially where delegated access is broad or poorly scoped. The MITRE ATLAS knowledge base is useful for structuring these adversarial techniques, while the OWASP Top 10 for LLM Applications helps map common failure patterns.
- Test prompt handling, instruction hierarchy, and refusal behaviour.
- Probe retrieval layers for poisoned or misleading content.
- Verify tool permissions, approval gates, and action logging.
- Check whether outputs expose secrets, personal data, or policy-controlled content.
- Validate fallback behaviour when the model is uncertain or under attack.
The right result is not just a list of bugs. It is a view of whether the AI system remains within approved behaviour when exposed to adversarial input, malformed context, and unsafe downstream actions. These controls tend to break down when an AI system is tightly integrated into live business workflows because operational teams assume the surrounding application controls are enough.
Common Variations and Edge Cases
Tighter AI controls often increase testing overhead and can reduce model flexibility, requiring organisations to balance assurance depth against delivery speed. That tradeoff becomes sharper in production systems where the model must support multiple use cases, external data sources, and tool access at the same time.
Best practice is evolving, but one point is already clear: a conventional penetration test may still be valuable for infrastructure, authentication, and exposure analysis, yet it does not replace AI-specific adversarial testing. Some teams also overcorrect by testing only the model prompt and ignoring the wider system. That misses the real edge cases, such as poisoned retrieval data, indirect prompt injection through third-party content, and unsafe action execution in tool-enabled agents.
For higher-risk deployments, the question is not whether the model can be attacked in a laboratory sense. It is whether the organisation can detect, contain, and govern unsafe model behaviour across the full lifecycle. NIST’s AI RMF resources and Anthropic’s Claude Mythos technical analysis both reinforce that red teaming should measure system behaviour, not just isolated technical exposure.
Where this guidance breaks down most clearly is in highly custom agent stacks with weak telemetry, because testers cannot reliably distinguish model judgment from tool-chain side effects without strong logging and replay capability.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
MITRE ATLAS and OWASP Agentic AI Top 10 address the attack surface, NIST AI RMF and NIST AI 600-1 set the technical controls, and EU AI Act define the regulatory obligations.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST AI RMF | GOV | AI red teaming must map to governance, accountability, and risk acceptance. |
| MITRE ATLAS | ATLAS catalogues adversarial ML tactics like prompt injection and poisoning. | |
| OWASP Agentic AI Top 10 | Agentic AI introduces tool misuse and unsafe action execution paths. | |
| NIST AI 600-1 | GenAI profiles emphasise model behaviour, misuse, and output controls. | |
| EU AI Act | Article 9 | High-risk AI requires documented risk management and control verification. |
Assign ownership, define risk thresholds, and require review before AI systems go live.
Related resources from NHI Mgmt Group
Deepen Your Knowledge
Reviewed and updated by the NHIMG editorial team on August 2, 2026.
NHI Mgmt Group — the #1 independent authority on Non-Human Identity, IAM, and Agentic AI security. nhimg.org
