They often treat model-level safety as the boundary, when the real boundary is the application layer that controls tools, data access, and execution. A model can refuse harmful text and still trigger unsafe behaviour through a tool call. Security has to be enforced before execution, not only during generation.
Why model safety is the wrong boundary in agentic systems
Security teams often evaluate the model as if its refusal behavior defines the whole control boundary. In an agentic system, that is too narrow. The risk sits in the application layer that decides what the model can reach, which tools it can invoke, what data it can see, and whether an output becomes an executed action.
A model that declines to produce harmful text can still help drive unsafe behaviour if the surrounding system turns a benign-looking completion into a tool call, file change, API request, or privilege-bearing workflow. The practical question is not only what the model will say, but what the application will let it do.
Where the real control boundary sits
The meaningful boundary is the orchestration and policy layer around the model. That layer should enforce identity, authorization, data scoping, action gating, and execution constraints before the request reaches a tool or external system. If those controls are absent, model safety becomes a helpful signal, not a reliable security control.
This is why agentic risk is different from plain content moderation. A model can be harmless in a chat transcript and still be dangerous once it is attached to email, files, databases, ticketing systems, code execution, or cloud APIs. The security decision must be made at the point of action, not only at the point of generation.
For a broader view of how autonomy changes the security surface, see AI Agents vs Agentic AI and the Agentic AI Security Guide, which both frame the shift from model output to governed execution.
What security teams should design instead
The safer pattern is to treat the model as one component inside a constrained execution system. That means per-action authorization, scoped tool access, short-lived credentials, explicit approval for sensitive operations, and logging that ties an agent action back to a principal and a decision path. In other words, the control plane must be able to say yes or no independently of the model’s wording.
That is also why least privilege matters more than prompts, jailbreak tests, or refusal tuning. If the agent cannot reach a system, cannot call a tool, or cannot use a privilege without policy approval, then the model’s occasional mistake is contained. If it can, model safety alone is just a last-mile filter.
Teams that want a concrete authorization pattern should compare their design to AI Agent Authorisation Guide and, for systems that rely on delegated access, Zero Trust for AI Agents.
Risk and Threat Considerations
The main failure mode is trust inversion: teams assume a safe model response means safe system behavior. Attackers do not need the model to generate disallowed text if they can steer the agent into a permitted action that has harmful consequences, such as data exposure, overbroad access, or an unsafe tool invocation.
Failure mechanism: The application layer accepts model output as sufficient authorization, or passes the request into a tool, connector, or workflow without independent policy enforcement. A constrained model response then becomes an unsafe real-world action.
Impact: This can produce unauthorized data access, unintended state changes, privilege abuse, and harder-to-detect compromises because the model appeared to behave safely while the surrounding system executed the harmful step.
Adversarial techniques in agentic systems are documented in OWASP Agentic AI Top 10, which is useful here because it separates model behavior from tool, identity, and orchestration risks.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10 addresses the attack and risk surface, while NIST SP 800-53 Rev 5, NIST Zero Trust (SP 800-207) and OWASP ASVS set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP Agentic AI Top 10 | ASI03 — Identity & Privilege Abuse | Agent safety here hinges on preventing the model from driving privileged actions through the app layer. |
| Recommendation — Enforce per-action authorization and least privilege before any agent tool invocation. | ||
| NIST SP 800-53 Rev 5 | AC-6 — Least Privilege | Agent execution becomes risky when the application allows more access than the task requires. |
| IA-5 — Authenticator Management | Agentic systems depend on controlled credentials, tokens, and secrets to prevent unsafe execution paths. | |
| Recommendation — Constrain agent credentials and tool scopes to the minimum required for each task. Rotate and govern agent credentials so tool access is short-lived and revocable. | ||
| NIST Zero Trust (SP 800-207) | 4.1 — Zero Trust principles | The answer centers on verifying each request before execution rather than trusting model output. |
| Recommendation — Verify each agent request before allowing access to data or tools. | ||
| OWASP ASVS | V8 — Authorization | The core issue is whether application-level authorization, not model output, governs the action. |
| Recommendation — Require explicit authorization checks for every sensitive agent action. | ||
Practitioner Guidance
What to prioritise: Put policy enforcement, tool gating, and execution boundaries ahead of model safety tuning. If an action can change data, spend money, call external systems, or touch privileged resources, the decision must be checked outside the model.
What to verify: Confirm that every agent action is authorized against the current principal, scope, and destination resource before execution. Also verify that denied or modified actions are visible in logs so teams can explain why the agent was allowed or stopped.
Common mistake: Treating prompt filters, refusal behavior, or red-team success against the model as proof that the system is safe. That test misses the application and orchestration path, which is where most material harm occurs.
Practitioner takeaway: In agentic systems, model safety is only advisory until the application layer constrains action, access, and execution. If you cannot independently control what the agent may do, the model’s refusal behavior is not a security boundary.