They should test both the model and the surrounding system. Use AI red teaming to see whether prompts can steer the model into unsafe behavior, and use penetration testing to verify that APIs, permissions, and workflows block unauthorized execution. The important control is the seam between output and action, because that is where an attacker turns influence into impact.
Why This Matters for Security Teams
AI systems that can trigger real actions change the risk profile from content quality to operational impact. A model that drafts an unsafe response is a problem; a model that can call tools, move funds, change records, or approve workflows can become an execution path. Testing therefore has to cover prompt influence, tool access, permission boundaries, logging, and human approval steps. Guidance in NIST SP 800-53 Rev 5 Security and Privacy Controls maps well to this because it treats access control, auditability, and system integrity as operational controls, not abstract policy statements.
The common mistake is to test the model in isolation and assume the surrounding system will absorb failure. In practice, the model may be only one decision point in a chain that includes RAG retrieval, orchestration logic, tool execution, and downstream business systems. If any one of those layers trusts the model output too much, prompt injection or manipulated context can become real-world action. That is why security teams need scenario-based testing, not just static review. In practice, many security teams encounter dangerous execution paths only after a workflow has already been automated, rather than through intentional pre-production abuse testing.
How It Works in Practice
Effective testing starts by separating model behavior from actionability. AI red teaming should probe whether the system can be induced to reveal secrets, bypass policy, mishandle instructions, or produce tool calls that violate intended constraints. Penetration testing should then verify that the environment rejects those actions even when the model attempts them. The relevant question is not just “Can the model be tricked?” but “Can a tricked model still do damage?”
A practical test plan usually covers:
- Prompt injection against system prompts, tool instructions, and retrieved content.
- Authorization checks on APIs, connectors, and agent tools.
- Step-up approval for sensitive actions such as payments, deletions, or privilege changes.
- Audit logging that captures the original input, model output, tool invocation, and final action.
- Rate limits, circuit breakers, and deny-by-default handling for uncertain outputs.
For AI-specific threat modeling, MITRE ATLAS is useful for mapping adversary behavior against model and agent abuse patterns, while OWASP Top 10 for Large Language Model Applications helps teams test common failure modes such as prompt injection and insecure tool usage. If the system uses agents, OWASP guidance for agentic AI is especially relevant for tracing how one compromised instruction can cascade into multiple actions.
Testing should be repeatable and tied to release gates. Security teams need test cases that prove whether the model can be manipulated, whether policy enforcement occurs outside the model, and whether human approval is required at the right points. These controls tend to break down when legacy automation gives the model broad API credentials because the system treats the AI as a trusted operator instead of an untrusted input source.
Common Variations and Edge Cases
Tighter action controls often increase operational friction, requiring organisations to balance automation speed against abuse resistance. That tradeoff is most visible in environments where the AI supports customer service, IT operations, or financial workflows, because business teams want low-latency execution while security teams need a clear review boundary.
Current guidance suggests there is no universal standard for how much autonomy is acceptable, so testing should reflect the sensitivity of the action, not just the sophistication of the model. A summarisation assistant that never leaves the chat surface can often be tested for information safety alone. An AI agent that can approve refunds, modify IAM roles, or deploy code needs stronger controls, including explicit allowlists, human-in-the-loop approval, and segregation of duties.
Edge cases appear when the model operates through third-party tools, external MCP servers, or nested orchestration layers. In those settings, the key question is provenance: where did the instruction come from, who can change it, and what identity actually executes the action. Teams should also test failure handling, because a blocked action that falls back to a default path can be just as risky as a successful one. If the workflow spans multiple systems and trust boundaries, assurance can erode quickly because no single team owns the full execution chain.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
MITRE ATLAS and OWASP Agentic AI Top 10 address the attack and risk surface, while NIST AI RMF, NIST CSF 2.0 and NIST SP 800-53 Rev 5 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST AI RMF | AI risk management should govern testing of model behavior and downstream impact. | |
| MITRE ATLAS | ATLAS models adversarial tactics against AI systems and agent workflows. | |
| OWASP Agentic AI Top 10 | Agentic AI guidance covers tool abuse and unsafe autonomous execution paths. | |
| NIST CSF 2.0 | PR.AC-4 | Least privilege is central when AI can initiate actions through APIs. |
| NIST SP 800-53 Rev 5 | AC-6 | Least privilege and enforcement controls support safe action boundaries. |
Use the AI RMF to define, measure, and govern risks from model outputs that can trigger actions.
Related resources from NHI Mgmt Group
- How should security teams test MCP-connected AI systems for real risk?
- How should teams govern Slack-native AI workflows that can trigger real actions?
- How should security teams apply NIST 800-53 to AI systems with autonomous actions?
- How should security teams test for visual prompt injection in multimodal AI systems?
