Because safety and security are different failure modes. A model can avoid generating harmful language and still be manipulated into taking harmful actions once it is connected to tools, memory, and APIs. The risk rises when the agent’s execution path is driven by untrusted inputs that the model mistakes for instructions.
Why safe-looking models still fail once tools change the game
A model can look safe in isolation because its output is filtered for harmful language, while still being unsafe in an agentic system. The failure mode shifts from what the model says to what it is allowed to do. Once the model can call tools, write memory, or trigger API actions, instruction-following becomes an execution problem, not just a content-safety problem.
That is why agent safety depends on the full action path, not only the model’s text output. Untrusted web pages, emails, documents, prompts, or retrieved context can be interpreted as instructions, and the model may act on them with legitimate credentials or permissions. The result is often a confused deputy pattern: the model remains “well behaved” conversationally while the surrounding system carries out harmful side effects.
In practice, the most important distinction is between harmless generation and harmful authority. An agent can refuse toxic requests and still leak data, change records, send messages, approve tasks, or exfiltrate secrets if those actions are reachable through connected tools. Good security therefore has to bound what the agent can access, what it can change, and what it can do without an additional check.
Where the failure comes from in agentic workflows
The weak point is usually not the base model, but the orchestration around it. Agents often receive a mix of system instructions, user requests, tool outputs, memory, and retrieved content, then choose actions from that blended context. If the policy layer does not clearly separate trusted instructions from untrusted content, the model can mis-rank attacker-supplied text as part of the job to be done.
That becomes dangerous when tool permissions are broad. A model that can read inboxes, browse internal systems, create tickets, or execute code is already operating with delegated authority, so prompt injection or context poisoning can turn a benign workflow into a live control-plane issue. NHIMG’s Agentic AI Security Guide frames this as a layered problem across inputs, memory, tools, orchestration, and identity.
Safe-looking models also fail when teams assume one guardrail covers every risk. Output filters do little against tool misuse, overbroad scopes, or unsafe automation paths. For example, a model may decline to produce credentials in text but still retrieve them from memory, forward them through a connector, or use them indirectly if the workflow permits it. That is why the control objective is to constrain action, not just language.
Two practical mechanisms deserve special attention: AI Agent Authorisation Guide and Zero Trust for AI Agents. The first addresses per-action policy decisions and just-in-time access, while the second makes clear that each request, principal, and tool call should be verified before authority is granted. Together, they show why agentic failure is usually about excessive agency, not just model quality.
Why trust boundaries matter more than model “safety”
In agentic systems, trust is structural. The model may be trustworthy enough to classify, summarize, or draft responses, but the surrounding platform can still be unsafe if it allows untrusted inputs to steer privileged actions. When the same runtime can both interpret content and execute commands, the trust boundary has to sit around the action gate, not around the model output alone.
That is why identity and observability become operationally important. If an agent can act on behalf of a person or service, teams need to know which principal authorized the action, which tool was used, and what evidence exists for later review. NHIMG’s AI Agent Observability, Audit and Incident Response Guide is useful here because it treats attribution, logging, and kill-switch design as part of containment, not as an afterthought.
Tool and protocol choice also changes the exposure. MCP Security Guide is relevant because protocol-level trust, token handling, and local server access can create hidden paths from benign-looking prompts to real operations. Likewise, the browser and computer-use pattern is risky because the agent may inherit a live human session and act with the user’s existing trust context.
The deeper lesson is that “safe” model behavior is only one layer of assurance. In agentic environments, the security question is whether the model’s authority is narrow, observable, and revocable enough that a mistaken interpretation cannot become a material action. If the answer is no, the model can still fail even when its text output looks carefully aligned.
Risk and Threat Considerations
Agentic systems expand the blast radius of a model failure because they convert misinterpretation into execution. An attacker does not need to make the model generate obviously malicious text; it is often enough to embed instructions in untrusted content and let the agent carry them out through approved tools, data sources, or sessions.
Failure mechanism: The model confuses attacker-controlled content with legitimate task context, then uses granted permissions to perform side effects such as data access, exfiltration, account actions, or workflow manipulation.
Impact: The result can be unauthorized action, data exposure, privilege abuse, or persistence through trusted workflows, even when the model appears safe from a content-moderation perspective.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10 and MITRE ATT&CK address the attack and risk surface, while NIST SP 800-53 Rev 5, NIST Zero Trust (SP 800-207) and OWASP ASVS set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP Agentic AI Top 10 | ASI03 — Identity & Privilege Abuse | Agent failure here centers on overbroad authority and misuse of delegated access. |
| ASI02 — Tool Misuse | The question is about harmful actions taken through tools, not harmful text generation. | |
| ASI06 — Memory & Context Poisoning | Untrusted inputs can be mistaken for instructions when they enter agent context. | |
| Recommendation — Constrain agent authority per action and require step-up approval for sensitive tool use. Restrict tool access to approved workflows and validate each tool invocation. Separate trusted instructions from untrusted context and harden memory write rules. | ||
| NIST SP 800-53 Rev 5 | AC-6 — Least Privilege | Agentic failure is amplified when connected tools grant excessive permissions. |
| AU-2 — Event Logging | Agent actions need attributable logs to detect misuse and support response. | |
| Recommendation — Limit agent permissions to the minimum scope needed for each task. Log agent requests, tool calls, and resulting actions with clear attribution. | ||
| NIST Zero Trust (SP 800-207) | AC-6 — Least Privilege | Zero trust is directly relevant to verifying each agent request before action. |
| Recommendation — Verify each request and grant only per-request access to the needed resource. | ||
| OWASP ASVS | V8 — Authorization | The key failure is not content generation but unauthorized execution through connected functions. |
| Recommendation — Enforce authorization checks on every privileged action the agent can trigger. | ||
| MITRE ATT&CK | T1204 — User Execution | Socially engineered or injected content can cause trusted execution paths to run attacker-chosen actions. |
| Recommendation — Map untrusted-content-to-action paths and hunt for execution triggered by deceptive inputs. | ||
Practitioner Guidance
What to verify: Verify that every tool call, write action, and external side effect has an explicit authorization boundary and a logged principal. If you cannot trace the action back to a request, a policy decision, and a callable scope, the agent is too trusted.
Decision rule: If the agent can cause a state change, treat that action as security-sensitive even when the underlying model output is harmless. Add step-up approval, tighter scopes, or read-only mode before increasing autonomy.
Common mistake: Do not rely on prompt wording, content filters, or “the model refused bad output” as evidence that the workflow is safe. That only measures one failure mode, not the one that matters in agentic execution.
Practitioner takeaway: The safest model can still be the riskiest agent if authority, tool access, and untrusted inputs are allowed to converge without strong action controls.
Related resources from NHI Mgmt Group
- How should security teams govern machine identity credentials in agentic AI environments?
- Why do temporary access models still fail in enterprise environments?
- What are the core risks identified by the OWASP Agentic Top 10?
- When does just-in-time access reduce risk for agentic AI, and when does it fall short?