What breaks when GenAI systems are secured like ordinary applications?

They miss the main risk surface. GenAI systems can be manipulated through language, retrieved content and tool outputs, so a clean code scan does not prove safe behaviour. If teams rely only on static controls, they will miss prompt injection, model drift and unauthorised actions that appear only at runtime.

Why This Matters for Security Teams

GenAI breaks the assumption that security can be validated by code review, endpoint hardening, or a clean application scan alone. The risk surface includes prompts, retrieved content, model outputs, plugins, and tool calls, which means the system can be manipulated at runtime even when the surrounding application looks compliant. NIST’s NIST AI 600-1 GenAI Profile treats these as distinct operational risks, not just software defects.

That matters because attackers do not need to break the model directly to cause harm. They can inject instructions into content the model consumes, steer the model into unsafe tool use, or induce it to reveal data that should never have been in scope. NHIMG’s DeepSeek breach coverage illustrates how exposed data, sensitive records, and backend credentials can turn a language system into an attack path rather than a bounded application.

Teams often miss that GenAI behaviour is probabilistic and context-sensitive, so “secure by deployment” is not the same as secure by design. In practice, many security teams encounter prompt injection and unauthorised tool use only after the model has already acted on attacker-controlled input, rather than through intentional testing.

How It Works in Practice

Securing GenAI like an ordinary application usually means the wrong controls are placed at the center. Traditional app security focuses on known binaries, stable input schemas, and predictable privilege boundaries. GenAI systems are different because the runtime may synthesize actions from natural language, retrieved documents, and external tools. That is why current guidance suggests testing the full interaction path, not just the codebase.

Operationally, strong programmes separate the model from the actions it can trigger. They constrain tool access, validate retrieved content before it is fed back into the model, and treat every model-generated request as untrusted until policy approves it. The NIST AI 600-1 GenAI Profile is useful here because it pushes teams to map governance, measurement, and monitoring to real runtime behaviour instead of static policy statements.

• Require explicit allowlists for tools, APIs, and connectors the model can invoke.
• Inspect prompt and retrieval inputs for instruction injection and data exfiltration patterns.
• Log model decisions, tool calls, and policy denials so investigations can reconstruct the chain of action.
• Use least privilege for secrets and tokens exposed to agentic workflows.

NHIMG research on DeepSeek breach shows why this matters: once sensitive records and credentials are exposed, the model layer becomes another route to reach them, not a safeguard around them. These controls tend to break down when the GenAI system can freely chain tools across multiple backend services because each step expands the attack surface faster than static review can track.

Common Variations and Edge Cases

Tighter control over GenAI often increases latency, integration effort, and false positives, so organisations have to balance runtime safety against developer velocity. There is no universal standard for this yet, and best practice is evolving as vendors and standards bodies refine guidance.

RAG systems are a common edge case. They may look safe because the model itself is not retrained, but malicious retrieval content can still poison the answer path or redirect tool use. Another edge case is hybrid systems where a chatbot is only one component in a larger workflow; the surrounding automation may be the real risk because it can execute downstream actions with broader authority than the model should ever hold.

In environments with long-lived tokens, broad API permissions, or loosely governed connectors, ordinary application controls fail fastest. The right question is not whether the code passes inspection, but whether the GenAI system can be tricked into taking an action that the original user was never authorised to request.

Related resources from NHI Mgmt Group

• What breaks when AI systems are governed like static applications?
• What breaks when AI gateway controls are treated like ordinary API security?
• What breaks when AI agents are managed like ordinary machine identities?
• What breaks when autonomous agents are managed like ordinary NHIs?