Which frameworks should guide AI red teaming and application pentesting?

Why This Matters for Security Teams

ai red teaming and application pentesting are often treated as separate exercises, but AI-enabled systems blend model behavior, application logic, and identity controls into one attack surface. That means a tester can find a prompt injection path without finding the privilege escalation that makes it exploitable, or the reverse. NIST AI RMF gives structure to adversarial testing of model behavior, while application security frameworks help verify the surrounding control plane and enforcement layer. The NIST Cybersecurity Framework 2.0 is useful for organizing governance, protection, detection, response, and recovery around the wider system.

Teams also need to distinguish between testing a model’s outputs and testing the security of the application that hosts it. A red team may need to probe prompt injection, tool misuse, data leakage, jailbreak resilience, and unsafe autonomous action. A pentest may need to verify authentication, session handling, authorization boundaries, logging, rate limits, and secrets handling. Where AI agents or MCP-connected workflows are involved, identity and authorization become part of the security question, not a separate concern. In practice, many security teams encounter the real weakness only after a harmless-looking model response has already been chained into an unauthorized action or data exposure.

How It Works in Practice

The most effective approach is to define two overlapping test scopes. The first is AI behavior security, which covers whether the model can be manipulated into unsafe, misleading, or policy-breaking outputs. The second is application and infrastructure security, which covers whether the system properly enforces access, validates inputs, protects secrets, and logs activity. NIST AI RMF helps teams structure the first scope around governance, measurement, and risk treatment. For the second scope, NIST SP 800-53 Rev. 5 remains the practical baseline for control testing across access control, audit, configuration, and incident response.

A useful test plan usually includes:

• Prompt injection and indirect prompt injection tests against user-facing and retrieval-augmented workflows.
• Tool and function-call abuse tests to see whether the system can be driven to misuse external actions.
• Authorization tests for tenant isolation, role boundaries, and service-to-service permissions.
• Secrets exposure tests for API keys, tokens, prompts, embeddings, and logs.
• Detection tests to confirm that risky behavior is visible in SIEM or SOAR workflows.

For AI-specific attack patterns, the Anthropic Frontier Red Team writeups are useful because they show how model misuse, instruction hierarchy issues, and tool access can combine into a real exploit path. That is especially important when an AI agent can act on behalf of a user or service principal. The pentest should verify both the model’s susceptibility to manipulation and the platform’s ability to stop the resulting action. These controls tend to break down when the model is tested in isolation, because the exploit only becomes material once tool access, broad permissions, or weak logging are present.

Common Variations and Edge Cases

Tighter testing scope often increases coordination overhead, requiring organisations to balance realism against production stability. Best practice is evolving for autonomous agents, and there is no universal standard for this yet, especially when a model can plan, call tools, and persist state across sessions. In those environments, a traditional web application pentest is necessary but not sufficient, because the highest-risk failure may be policy drift, unauthorized delegation, or unsafe action chaining rather than a classic injection flaw.

Edge cases appear when the system uses RAG, third-party tools, or multiple model providers. A model may be secure in a clean lab but vulnerable once exposed to untrusted documents, external APIs, or shared memory. Identity matters here too: service accounts, workload identities, and secret scopes often determine whether a prompt injection becomes a minor defect or a full incident. For governance, many teams align reporting to NIST AI RMF and map operational control evidence to NIST SP 800-53 Rev. 5, then use the NIST Cybersecurity Framework 2.0 to communicate broader risk treatment across leadership.

Where regulated or high-stakes systems are involved, the question is not whether the red team found a clever jailbreak. The real question is whether the organisation can prove that model behavior, authorization boundaries, and monitoring were tested together under realistic operating conditions.

Related resources from NHI Mgmt Group

• What is the difference between prompt testing and red-teaming agentic AI?
• What is the difference between red teaming an AI system and proving it is safe?
• How should security teams use AI red teaming results in production governance?
• Why do AI agents create a different red teaming problem from ordinary AI applications?