They miss the main risk surface. GenAI systems can be manipulated through language, retrieved content and tool outputs, so a clean code scan does not prove safe behaviour. If teams rely only on static controls, they will miss prompt injection, model drift and unauthorised actions that appear only at runtime.
Why ordinary application controls miss the GenAI attack surface
GenAI systems do not fail only where code is malformed or infrastructure is weak. Their risk surface includes the prompt layer, retrieved context, model responses, and any connected tools that can execute actions on the user’s behalf. Treating them like ordinary applications encourages teams to inspect code and configuration while ignoring runtime behaviour, where prompt injection, unsafe tool use, and context poisoning actually emerge. The relevant control question is not just whether the software is patched, but whether the system can be steered into producing harmful output or taking unauthorised action.
NIST’s NIST AI 600-1 GenAI Profile is useful here because it frames generative AI risk as an operational and governance problem, not only a software hygiene problem. In practice, many security teams discover the gap only after a model has already been influenced by hostile instructions or a connected tool has already acted on untrusted content.
How the failure shows up in practice
Ordinary application security assumes that the important boundary is the application itself: input validation, code review, access control, and patching. GenAI systems add a second boundary inside the runtime interaction. The model may accept natural language instructions, incorporate external retrieval results, and decide whether to call tools, summarize content, or trigger downstream workflows. Each of those steps can be manipulated even when the underlying application code is technically sound.
That is why static assurance often gives a false sense of safety. A code scan can confirm that no obvious injection exists in the host application, but it cannot prove that the model will ignore malicious instructions buried in user input, web content, documents, or retrieved data. The same problem appears with tool use: a workflow may be correctly authenticated and still be unsafe if the model can be induced to request actions the operator never intended.
- Prompt injection targets instruction following, not syntax errors.
- Retrieved content can carry hostile instructions or misleading context.
- Tool outputs can become a new trust boundary if they are fed back into the model.
- Model drift can change behavior after deployment without any code change.
The practical consequence is that security testing must include runtime abuse paths, not just application defects. A model that behaves acceptably in a controlled test may still fail once exposed to adversarial phrasing, contaminated context, or chained tool interactions. That is why GenAI security needs evaluation of behavior, authorization, and containment together, rather than a code-first review alone. This guidance breaks down when the system has no retrieval, no tool access, and no meaningful autonomy, because then the AI-specific runtime abuse surface is much smaller.
Where the common comparison to ordinary apps stops working
Tighter control over GenAI behavior often increases operational overhead, requiring organisations to balance flexibility against containment. The main ambiguity is that some teams treat every AI feature as if it were either a harmless chatbot or a fully autonomous agent, when many systems sit between those extremes. Guidance and consensus are not fully aligned on how much autonomy is acceptable in each use case, so the correct control depth depends on whether the model can merely draft text or can also retrieve data, invoke tools, or commit changes.
Another edge case is that some failures are not strictly malicious. A model can still produce unsafe or unreliable outcomes through hallucination, stale retrieval, or prompt ambiguity even without an attacker. That means the security model must account for misuse, abuse, and ordinary unreliability together. When a GenAI system is embedded in a customer-facing or operational workflow, the risk becomes material even if no single prompt looks dangerous on its own.
Organisations also underestimate how quickly trust shifts once a model is allowed to act. A plain application may be trusted to display data, while a GenAI system may be trusted to interpret data, summarise policy, draft responses, or trigger actions. That shift changes the control objective from protecting records to governing decisions. The control stack has to reflect that difference, or teams end up securing the container while leaving the behaviour ungoverned.
Risk and Threat Considerations
When GenAI is secured like an ordinary application, the material risk is runtime abuse of a system that can interpret untrusted language and execute delegated actions. The exposure is not limited to data leakage; it also includes unauthorized action, policy bypass, and hidden influence over downstream workflows.
Failure mechanism: An attacker or malicious input can manipulate the model through prompt injection, contaminated retrieval content, or hostile tool outputs, causing the system to ignore intended instructions or invoke actions outside the operator’s intent.
Impact: The organisation can lose control over output integrity, access decisions, and automated actions, which may expose data, corrupt business processes, or create unauthorized changes in connected systems.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10 address the attack and risk surface, while NIST AI RMF and NIST AI 600-1 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST AI RMF | GOVERN — Govern | GenAI risk here is governance of runtime behavior, not just software hygiene. |
| Recommendation — Define GenAI oversight for model behavior, autonomy, and acceptable risk boundaries. | ||
| NIST AI 600-1 | MAP — Map | The question centers on identifying GenAI-specific risk surfaces and contexts. |
| MANAGE — Manage | Runtime misuse, drift, and unsafe actions require active risk treatment. | |
| MEASURE — Measure | Static checks cannot prove safe GenAI behavior; evaluation must test behavior. | |
| Recommendation — Map prompts, retrieval paths, and tool actions to the system's real risk surface. Manage runtime abuse cases with controls for instruction, retrieval, and action safety. Measure model behavior under adversarial prompts and tool-use scenarios. | ||
| OWASP Agentic AI Top 10 | A2 — Prompt Injection | The core failure mode is manipulation through language and retrieved content. |
| Recommendation — Test prompt and context injection paths before allowing autonomous actions. | ||
Practitioner Guidance
What to prioritise: Treat tool access, retrieval, and action authority as the first-order control surfaces, not as optional extras. If a GenAI system can do more than draft text, the security review has to include what it can reach, what it can change, and what it can be tricked into trusting.
What to verify: Confirm that the system is evaluated under adversarial prompts, hostile retrieved content, and failure conditions that occur only at runtime. A clean static review is useful, but it is not evidence that the model will remain aligned once it is exposed to real users, external content, or chained automation.
Practitioner takeaway: The key mistake is assuming GenAI risk is mostly a software-quality problem; once a model can interpret content and take action, security must govern behavior as well as code.
Related resources from NHI Mgmt Group
Deepen Your Knowledge
Reviewed and updated by the NHIMG editorial team on September 7, 2026.
NHI Mgmt Group — the #1 independent authority on Non-Human Identity, IAM, and Agentic AI security. nhimg.org
