What breaks when AI agents are allowed to act on untrusted prompts without runtime guardrails?

Why This Matters for Security Teams

Untrusted prompts turn an AI agent from a bounded assistant into a live decision-maker operating inside business systems. The failure mode is not just harmful text generation. It is policy bypass, hidden tool use, and accidental privilege exercise that can look like normal productivity. Guidance from the OWASP Agentic AI Top 10 and the NIST AI Risk Management Framework both point to the same core issue: prompt content cannot be treated as trustworthy input when the system can act on it.

This matters because agents can chain tools, preserve context, and continue executing after a malicious instruction has already influenced state. That means the blast radius is often larger than the prompt itself. NHIMG’s Gemini AI Breach shows how prompt injection can turn a routine workflow into data exposure, while CoPhish OAuth Token Theft via Copilot Studio illustrates how an agent boundary can become an identity boundary problem.

In practice, many security teams encounter agent misuse only after sensitive data has already moved through a workflow that looked legitimate on the surface.

How It Works in Practice

Runtime guardrails are the control plane between an agent’s intent and its ability to act. Instead of trusting the prompt, the system evaluates each action at execution time: what tool is being called, what data is being requested, whether the request matches policy, and whether the current context supports the action. That is a very different model from static IAM, which assumes access can be safely defined ahead of time.

For agentic systems, current guidance suggests combining several controls:

• Intent-based authorisation that checks the action being requested, not just the user or role behind the agent.
• Just-in-time credential issuance so tool access is short-lived and tied to a specific task.
• Workload identity such as SPIFFE or OIDC so the agent proves what it is before receiving any capability.
• Policy-as-code with runtime evaluation through systems such as OPA or Cedar, rather than pre-approved allowlists alone.
• Output filtering and data minimisation so the agent cannot echo secrets or sensitive context back into downstream tools.

NHIMG’s AI Agents: The New Attack Surface report is notable here because it shows how often agents already exceed intended scope in real deployments. That aligns with broader industry warnings in the CSA MAESTRO agentic AI threat modeling framework, which treats tool use, memory, and orchestration as part of the attack surface rather than as implementation details.

The practical point is simple: if the agent can interpret an untrusted prompt and then call tools, move data, or request credentials, the security boundary must move from the prompt to the runtime policy decision. These controls tend to break down in legacy environments where long-lived service accounts, broad API scopes, and weak auditability make every agent action look like ordinary backend traffic.

Common Variations and Edge Cases

Tighter runtime controls often increase latency, complexity, and support overhead, so organisations have to balance safety against workflow friction. That tradeoff is most visible in environments where agents need broad tool access to complete multi-step work, or where human operators expect uninterrupted automation.

One common edge case is trusted prompt sources. Even internally generated prompts can be adversarial if they ingest untrusted web content, email, tickets, or documents. Another is delegated action: an agent may be safe reading data but unsafe when allowed to submit, delete, or reconfigure systems. Best practice is evolving here, and there is no universal standard for exactly where to place the boundary.

NHIMG research on OWASP NHI Top 10 and the broader Ultimate Guide to NHIs both reinforce that identity, secrets, and authorization must be treated as dynamic controls, not static setup tasks. For high-risk workflows, the safest pattern is often step-up approval, per-action token scoping, and hard revocation after task completion.

These approaches become less reliable when agents operate across fragmented secret stores, uncontrolled browser sessions, or older systems that cannot enforce request-time policy decisions.

Related resources from NHI Mgmt Group

• What breaks when AI pentesting agents are allowed to act without approval gates?
• What breaks when AI assistants are allowed to act on untrusted email content without approval controls?
• What breaks when an AI assistant is allowed to run security actions without strong permission scoping?
• What breaks when autonomous shopping agents are allowed to act without strong governance?