Both, but application security comes first because the failures are concrete injection and execution problems. AI governance sets the rules for acceptable behaviour, while application security enforces those rules at the URL, path, query, template, and command boundary. If the boundary is weak, governance statements alone will not prevent exploitation.
Why this is an application security question first
LLM framework risk becomes real where user-controlled input can alter execution paths, retrieve unintended data, or invoke tools and templates in unsafe ways. That is an application security problem before it is an abstract policy issue. The failure mode is usually concrete: prompt injection, unsafe routing, broken authorization, or command execution at a boundary that should have been enforced in code.
Security teams should treat the framework as software that processes untrusted input, not as a policy document that merely emits text. That means the review target is the path from request to action, including URLs, prompt assembly, retrieval layers, tool calls, and any server-side function that trusts model output. OWASP ASVS is the clearest control reference for that boundary-focused review.
For teams testing web and API surfaces around the model, the practical question is whether the framework can change what the application does, not just what it says. OWASP Web Security Testing Guide and OWASP Top 10 remain useful because they anchor testing in input handling, authorization, and server-side trust failures.
Where AI governance fits without taking over
AI governance is still necessary, but it sets the operating rules rather than the runtime defenses. It defines what behaviour is acceptable, which data classes may be exposed, who can approve deployment, and how high-risk use cases should be reviewed. That matters for model use, disclosure, accountability, and acceptable interaction patterns, especially where the system can influence users or downstream systems.
The limitation is that governance cannot compensate for a weak execution boundary. A policy saying “do not execute unsafe instructions” does not stop a model from following malicious content if the application lets that content reach a command, query, or tool call. Governance therefore needs enforcement points, and those points usually live in application controls, not in the policy layer alone.
NIST AI Risk Management Framework is relevant where the organisation needs a formal way to define risk ownership, acceptable behaviour, and monitoring expectations for AI use. ISO/IEC 42001:2023 AI Management System Standard is also useful when governance must be auditable and repeatable across teams and deployments.
How practitioners should split the work
The cleanest operating model is to assign application security to the mechanics of exploitation, and AI governance to the guardrails around use. Application security owns prompt injection resistance, tool authorization, input validation, output handling, and isolation between users, sessions, and contexts. AI governance owns policy, risk appetite, escalation, and approved use cases.
That split works only if the boundary is explicit. If the model can trigger actions, then the application layer must constrain those actions with allowlists, permission checks, and bounded execution. If the model can read data, then retrieval and routing must enforce least privilege before the prompt is assembled. In practice, the safest programmes test the app layer as they would any other internet-facing service, then layer governance on top for approval and oversight.
NIST AI 600-1 GenAI Profile helps when teams need generative-AI specific controls around testing, provenance, and disclosure. NIST IR 8596 Cyber AI Profile is useful when the organisation wants to map AI-specific risks back into broader cybersecurity functions.
Risk and Threat Considerations
LLM frameworks create a security risk when organisations assume policy language will stop technical abuse. Attackers do not need to defeat governance statements if they can exploit injection paths, manipulate retrieval, or steer tool invocation into unsafe execution. The risk increases when the framework is connected to internal data, external APIs, or privileged workflows.
Failure mechanism: Untrusted text or inputs reach an execution boundary, then the application treats model output as instruction, authority, or routing logic. That can expose data, trigger unintended actions, or bypass intended control checks.
Impact: Confidential data exposure, unauthorised actions, privilege misuse, and trust erosion across the AI stack. In higher-risk deployments, a single weak boundary can turn a benign assistant into a path for data exfiltration or workflow abuse.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP ASVS, NIST AI RMF and NIST SP 800-53 Rev 5 set the technical controls, while ISO/IEC 42001:2023 defines the regulatory obligations.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP ASVS | V4 — API and Web Service | LLM framework abuse often lands at request, routing, and action boundaries. |
| V8 — Authorization | The core failure is model-driven action crossing an access boundary. | |
| V15 — Secure Coding and Architecture | Safe LLM integrations depend on secure prompt, tool, and execution architecture. | |
| Recommendation — Apply V4 checks to validate input, routing, and server-side trust before actions execute. Enforce V8 to separate model output from authorization decisions and privileged actions. Use V15 to isolate untrusted model input from code paths that cause side effects. | ||
| NIST AI RMF | Govern Map Measure and Manage | AI governance is needed to define acceptable LLM use and oversight expectations. |
| Recommendation — Adopt AIRMF to define AI risk ownership, controls, and monitoring across the lifecycle. | ||
| ISO/IEC 42001:2023 | AI Management System | LLM risk needs auditable governance, accountability, and repeatable operating controls. |
| Recommendation — Use ISO-42001 to formalise approvals, accountability, and continual AI risk review. | ||
| NIST SP 800-53 Rev 5 | AC-6 — Least Privilege | LLM tools and workflows must not inherit unnecessary privilege. |
| IA-5 — Authenticator Management | AI systems often fail when secrets and credentials are mishandled at integration boundaries. | |
| SI-10 — Information Input Validation | Prompt injection and unsafe input handling are central to LLM framework risk. | |
| Recommendation — Apply AC-6 to constrain model-linked components to the minimum required access. Use IA-5 to govern credential storage, rotation, and invalidation for AI integrations. Apply SI-10 to validate untrusted inputs before they affect prompts, tools, or queries. | ||
Practitioner Guidance
What to prioritise: Put the first control effort into the application boundary, especially input handling, output-to-action mapping, and tool authorization. If a model can reach a side effect, validate that side effect independently of the model’s reasoning.
What to verify: Check that no prompt, retrieval result, or user-provided field can directly control a URL, file path, query, template, or command without a separate policy check. Also verify that the same request cannot inherit more privilege just because it came through the AI flow.
Practitioner takeaway: Treat AI governance as the rulebook and application security as the enforcement layer, because only the latter can reliably block boundary abuse in live systems.
Related resources from NHI Mgmt Group
- Why is single-provider AI agent governance not enough for enterprise security?
- How should security teams handle risks from AI browser extensions?
- What breaks when organisations treat AI governance as a separate security program?
- Should organisations treat AI governance and AI security as the same thing?