Start with the actions, not the model. Identify where AI outputs can trigger payments, approvals, account changes, or disclosures, then test whether those actions require independent verification from a trusted human or system. Include the identities, tokens, and permissions that let the workflow execute, because those controls define whether a prompt becomes a real-world event.
Why This Matters for Security Teams
AI workflows become materially risky when a generated output is allowed to move money, change access, publish content, or reveal data. The security question is no longer whether the model can be manipulated in isolation, but whether a downstream system trusts that output as if it were a verified instruction. That makes the blast radius a business issue, not just a model quality issue.
Red teaming should therefore focus on the full action chain: prompt, context, tool call, approval step, and execution path. This is where governance, identity, and workflow security meet. The most useful control lens is whether the action can be independently validated before it is committed, especially where the AI agent or orchestration layer has token access or delegated authority. Guidance from NIST SP 800-53 Rev 5 Security and Privacy Controls is still relevant here because it emphasizes access control, auditability, and separation of duties across systems that actually perform the action.
Teams often get this wrong by testing model answers for accuracy while leaving the real control gap untouched. In practice, many security teams encounter the failure only after an AI-generated instruction has already been accepted by a connected workflow, rather than through intentional action-path testing.
How It Works in Practice
Effective red teaming starts by mapping every AI-enabled business action to the identity and permissions behind it. That includes service accounts, API keys, delegated OAuth grants, session tokens, approval bots, and any agentic workflow that can call external tools. The test plan should not stop at prompt injection. It should ask whether the attacker can cause the workflow to:
- submit a payment or invoice
- approve a request or override a queue
- reset credentials or modify entitlements
- send sensitive data to an external destination
- create, update, or delete records in a system of record
From there, the red team should validate whether there is a trusted verification step before execution. That may be a human approval, a policy engine, a second system of record, or a cryptographic or workflow signature that proves the instruction is authorized. Current guidance suggests that strong controls include least privilege, just-in-time access, and explicit separation between content generation and action execution. For AI-specific attack patterns, MITRE ATLAS is useful for structuring tests around prompt injection, indirect manipulation, and adversarial influence over model behavior.
Practitioners should also test logging and alerting. If an AI workflow tries to perform an abnormal business action, the security team should be able to reconstruct who or what initiated the request, which context was supplied, what tools were invoked, and whether any approval step was bypassed. That is where OWASP Top 10 for LLM Applications helps translate model risk into concrete test cases, especially around prompt injection, insecure output handling, and excessive agency. These controls tend to break down when the workflow is deeply embedded in legacy automation because identity boundaries, audit trails, and approval logic are often inconsistent across systems.
Common Variations and Edge Cases
Tighter action controls often increase latency and operational overhead, requiring organisations to balance speed against assurance. That tradeoff becomes more visible when business teams want an AI workflow to act in real time, but the security team needs a verification gate before any external change is made.
There is no universal standard for this yet. Best practice is evolving toward tiered authorization, where low-risk actions can proceed automatically but high-impact actions require independent verification. For example, drafting a refund request is not the same as executing the refund, and summarizing an account change is not the same as applying it. The red team should test for confusion between those states, especially when the interface makes generated content look like approved instruction.
Identity is often the hidden edge case. If an agent inherits a broad service account, or if multiple workflows reuse the same token, the test surface expands beyond prompt injection into credential misuse and privilege chaining. This is where NIST Zero Trust Architecture is relevant because every action should be authorized based on current context, not assumed trust. The same applies when humans review output but do not actually re-validate the underlying intent or recipient.
Another important exception is shadow automation. Some teams connect AI tools to spreadsheets, chat systems, or internal macros without central governance. Those environments can be hard to instrument, so red teaming should include discovery of unsanctioned paths, not just the formally approved workflow. Where sensitive data or regulated payments are involved, the risk is not merely model failure but unauthorized real-world execution.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
MITRE ATLAS and OWASP Agentic AI Top 10 address the attack and risk surface, while NIST CSF 2.0, NIST AI RMF and NIST Zero Trust (SP 800-207) set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST CSF 2.0 | PR.AC-4 | AI workflows need least privilege before outputs can trigger real actions. |
| MITRE ATLAS | ATLAS models adversarial techniques against AI systems and agents. | |
| OWASP Agentic AI Top 10 | Agentic workflows expand risk when models can call tools and act autonomously. | |
| NIST AI RMF | GOVERN | AI governance is central when model outputs can cause business actions. |
| NIST Zero Trust (SP 800-207) | AL | Zero trust supports continuous verification before workflow actions execute. |
Restrict workflow permissions so the AI can only request actions, not perform unrestricted changes.
