Because the application often executes model output with the same authority it uses for normal business logic. If the assistant can reach databases, shell commands, or internal APIs, unsafe output can trigger actions the user should never have been able to perform. That is an authorisation design problem, not only an injection problem.
Why untrusted LLM output becomes an authorisation problem
Untrusted LLM output is risky when the application treats it as an instruction source instead of plain text. The authorisation boundary is crossed when model output can influence privileged actions, expand the user’s effective permissions, or steer backend tools without a separate access decision.
That is why the failure is not just “bad text entered the system.” It is that the application may act on that text with the authority of a trusted workflow, which turns a content problem into a control problem.
Where the authorisation boundary is actually broken
The key design issue is whether the model output is confined to display, or whether it can reach code paths that execute database queries, invoke APIs, trigger workflows, or launch shell commands. If the output can change what the system does, then the model is effectively participating in access control decisions even though it is not an authoriser.
That risk is strongest in systems that mix natural-language generation with tool use, agentic actions, or retrieval over sensitive content. A model that can see internal data and then produce a command, filter, or structured request can become a conduit for privilege misuse if the application does not re-check who is allowed to do what before execution.
This is also why permission-aware retrieval matters. If a system can surface content the user cannot normally access, the output may be valid language while still causing an authorisation failure in retrieval and downstream action. The same logic applies when the output drives connectors, not just search.
Why prompt injection and tool use raise the stakes
Untrusted output becomes more dangerous when the application forwards it into another trust boundary. The model may be instructed, manipulated, or simply overconfident, but the real issue is that the surrounding system may grant it execution paths that normal users do not have. At that point, the model is not merely generating unsafe text, it is influencing an access-bearing workflow.
Agentic systems make this sharper because tools often operate with standing credentials, service tokens, or delegated authority. In that setting, even a small instruction confusion can translate into a high-impact action if the application does not separate “what was generated” from “what is authorised.” The agentic AI security model and the OWASP Agentic AI Top 10 both treat identity and privilege abuse as core design risks, not edge cases.
External guidance also aligns with this view. NIST’s AI RMF GenAI profile focuses on content provenance, testing, and governance around generated outputs, while OWASP’s API Security Top 10 remains relevant where model output can drive broken authorisation at the API layer.
Risk and Threat Considerations
When untrusted LLM output can reach privileged execution paths, the security failure is often horizontal expansion of access rather than obvious compromise. The model may be exploited to issue requests, select tools, or reshape parameters in ways that bypass the user’s actual permissions, which is especially dangerous in applications with internal APIs, admin functions, or automation hooks.
Failure mechanism: The application trusts generated output as if it were an approved instruction, then executes it under the application’s own authority, or under a privileged service identity, without a fresh authorisation check.
Impact: Attackers can trigger unintended actions, read restricted data, modify business records, or pivot into internal systems through a path the user should never have had.
In practice, this is where prompt injection, over-broad tool access, and weak function-level controls converge. A model that is “only” untrusted text can still become an abuse channel if the surrounding code maps its output directly into sensitive operations.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP API Security Top 10 and OWASP Agentic AI Top 10 address the attack and risk surface, while NIST AI RMF and NIST SP 800-53 Rev 5 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP API Security Top 10 | API5 — Broken Function Level Authorization | LLM output can trigger functions the user cannot invoke directly. |
| Recommendation — Enforce function-level checks before any model-driven action executes. | ||
| OWASP Agentic AI Top 10 | ASI03 — Identity & Privilege Abuse | Agentic workflows can misuse delegated authority through generated output. |
| Recommendation — Bound tool authority and reauthorize sensitive agent actions. | ||
| NIST AI RMF | Generative AI risk management | GenAI output needs governance, testing, and provenance around downstream use. |
| Recommendation — Assess generated-output risks before allowing execution paths. | ||
| NIST SP 800-53 Rev 5 | AC-6 — Least Privilege | Privileged execution paths should not inherit broad authority from model output. |
| IA-9 — Service Identification and Authentication | Backend tools and services need strong identity checks before acting on output. | |
| Recommendation — Limit every tool and service account to the minimum required privilege. Authenticate services before trusting model-triggered requests. | ||
Practitioner Guidance
What to verify: Check every place where model output is consumed by code, especially tool calls, SQL generation, API requests, workflow triggers, and command execution. The question is not whether the output looks safe, but whether the downstream action is authorised independently of the model.
Decision rule: If the output can cause a state change, data disclosure, or privileged action, require an explicit policy gate and a user or system authorisation check before execution. If you cannot express the check clearly, the model is too close to the trust boundary.
Common mistake: Teams often add content filtering and assume the problem is solved, but content filtering does not replace access control. The secure design pattern is to treat model output as untrusted input until the final decision point.
Practitioner takeaway: Untrusted LLM output is safe only when it is informational, not authoritative, and every privileged action still passes through a separate, enforceable authorisation decision.