Start by limiting what the model can do, not just what it can say. Constrain tools, permissions, and external actions to the smallest necessary scope, then validate inputs and outputs before they reach downstream systems. Add human approval for sensitive actions, isolate untrusted content, and red team the application so indirect prompts do not turn into unauthorized execution.
Why Guardrails Matter When LLMs Can Act, Not Just Respond
Guardrails are the boundary between a useful LLM application and one that can be steered into unsafe execution. Prompt injection matters because the model may encounter hostile instructions inside emails, web pages, tickets, or files that look like ordinary data but compete with the system prompt. excessive agency matters because the same model can be asked to recommend, approve, or trigger actions beyond the original user intent. OWASP’s OWASP Agentic AI Top 10 is useful here because it frames the issue as control over tool use, autonomy, and trust boundaries rather than output quality alone.
The practical mistake is treating prompt injection as a text-filtering problem when it is really an authority-management problem. If the application can search, send, delete, summarise, or update records, then the guardrail must decide whether the request is safe to execute, not merely whether the answer sounds plausible. In practice, many security teams discover the problem only after an agent has already been allowed to act on untrusted content rather than through deliberate testing of its decision boundary.
What Effective Guardrails Look Like in an LLM Workflow
Good guardrails separate reasoning from execution. The model may interpret a request, but it should not directly inherit unrestricted access to tools, systems, or data stores. Security teams should treat each tool call as a privilege decision, each external document as untrusted input, and each high-impact action as a point where policy must intervene. NIST’s NIST AI Risk Management Framework is relevant because it emphasises governance, mapping, measurement, and management of AI risk across the lifecycle.
- Limit tool scope to the smallest viable set of actions and records.
- Classify inputs by trust level so retrieved content cannot silently override instructions.
- Require confirmation, approval, or a separate policy check for irreversible or sensitive actions.
- Log prompts, tool calls, policy decisions, and overrides so you can reconstruct why the system acted.
A useful design pattern is to make the model propose and the control plane dispose. That means the LLM can draft an email, suggest a ticket change, or identify a suspicious record, while a policy layer decides whether the action is permitted. For higher-risk workflows, teams should also segment privileges by task class, so a support assistant cannot become an administrative automation engine simply because it handles the same conversation. Where retrieval is involved, the application should distinguish evidence from instruction; otherwise a malicious document can redirect the workflow without ever breaching the model itself. This guidance breaks down when teams allow the model to hold broad standing access and then rely on post hoc review to catch misuse.
Where Prompt Injection and Excessive Agency Usually Break the Design
Tighter control often increases friction, so organisations must balance automation value against the cost of additional checks. The right balance depends on whether the workflow is reversible, whether the data is sensitive, and whether a mistaken action can be contained. In mixed-trust environments, the same LLM may handle both benign summaries and adversarial content, which makes context separation more important than raw model capability.
One edge case is indirect prompt injection through retrieved content. A model that is allowed to browse, summarise, or ingest documents can be manipulated even when the user prompt is clean, because the hostile instruction arrives inside a trusted-looking source. Another edge case is over-broad delegation inside agentic workflows, where the model can chain multiple low-risk actions into a high-risk outcome. NIST’s NIST AI 600-1 Generative AI Profile is especially useful when the application is built around generative AI rather than a fully autonomous agent, because it helps teams distinguish model-centric risks from workflow-centric ones.
Where there is no reliable way to separate instruction from content, or no meaningful human checkpoint before impactful actions, the guardrail design is too weak for production use.
Risk and Threat Considerations
Prompt injection creates a control-bypass problem, while excessive agency creates an authority-abuse problem. Together they let untrusted content influence decisions and then turn those decisions into external action, which can lead to data exposure, unauthorised changes, or misuse of internal tools.
Failure mechanism: The attacker places instructions inside content the system is expected to read, then relies on weak trust boundaries, broad tool permissions, or missing action approval to convert model compliance into execution.
Impact: The application may leak information, perform unintended actions, alter records, send malicious messages, or create a durable path from content ingestion to operational compromise.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10 and MITRE ATLAS address the attack and risk surface, while NIST AI RMF and CIS Controls v8 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP Agentic AI Top 10 | A1 — Excessive Agency | Directly addresses over-privileged agent actions and unsafe autonomy. |
| A2 — Prompt Injection | Directly covers hostile instructions embedded in untrusted content. | |
| A4 — Tool Misuse | Applies where model tool access can be abused for unintended actions. | |
| Recommendation — Constrain agent permissions and require approval before impactful tool use. Isolate untrusted inputs and validate instruction boundaries before execution. Restrict tools to least privilege and log every action-bearing call. | ||
| NIST AI RMF | GOV — Govern | Applies to governance, accountability, and AI risk ownership decisions. |
| MAP — Map | Applies to identifying context, inputs, outputs, and deployment boundaries. | |
| MAN — Manage | Applies to ongoing treatment of model and workflow risks through controls. | |
| Recommendation — Define accountable ownership for AI risk decisions and approval thresholds. Map trust boundaries, data flows, and model actions before enabling deployment. Manage residual AI risk with policy checks, monitoring, and escalation rules. | ||
| MITRE ATLAS | AML.TA0001 — Reconnaissance | Prompt injection attacks often probe model behavior and control boundaries. |
| AML.TA0004 — Execution | Covers adversarial attempts to make the model carry out malicious actions. | |
| Recommendation — Hunt for probing patterns that reveal available tools and trust gaps. Detect and block prompts that induce unauthorized execution or tool use. | ||
| CIS Controls v8 | 5.1 — Establish and Maintain an Inventory of Enterprise Assets | Relevant where LLM tools and connected systems need explicit inventory and ownership. |
| 6.3 — Require MFA for Externally-Exposed Applications | Relevant when LLM applications expose administrative or externally reachable actions. | |
| Recommendation — Inventory LLM-connected tools, data sources, and approval points before enabling use. Protect externally reachable LLM admin paths with strong authentication. | ||
Practitioner Guidance
What to prioritise: Focus first on the actions the model can trigger, not on polishing prompts or adding more restrictive wording. If a tool call can change state, disclose data, or reach outside the application boundary, that action needs a separate control decision.
What to verify: Confirm that every sensitive tool is explicitly allowlisted, every untrusted input path is isolated from instructions, and every high-impact action has a policy check or human approval path. Teams should also verify that logs capture enough context to explain why a request was allowed, blocked, or escalated.
Practitioner takeaway: The strongest guardrail is not a better prompt, but a narrower authority model that treats the LLM as an advisory component unless and until a separate control layer authorises execution.
Related resources from NHI Mgmt Group
- How should security teams handle prompt injection in production LLM applications?
- How should security teams implement runtime guardrails for LLM applications in production?
- How should security teams scan LLM application code for prompt injection risks in pull requests?
- How should security teams implement prompt versioning in production LLM applications?
Deepen Your Knowledge
Reviewed and updated by the NHIMG editorial team on September 9, 2026.
NHI Mgmt Group — the #1 independent authority on Non-Human Identity, IAM, and Agentic AI security. nhimg.org
