Why do Gen AI deployments fail when organisations focus only on the model and infrastructure?

Why This Matters for Security Teams

Gen AI failures are usually governance failures disguised as technical ones. A strong model stack and resilient infrastructure can still produce unsafe outcomes if users enter confidential data, if prompts are not monitored, or if approval paths are unclear. The real risk is that organisations assume model quality equals deployment safety, when the operating model around the system is often what determines exposure.

This is why security teams should evaluate Gen AI through a control lens, not only a platform lens. The NIST Cybersecurity Framework 2.0 is useful here because it forces attention onto governance, protection, detection, and response, not just build-time security. That framing matters when employees can access copilots, agents, or retrieval tools that touch regulated data, customer records, or internal IP. If the deployment model does not define acceptable use, logging, escalation, and ownership, the organisation inherits shadow AI behaviour almost immediately.

Security teams also need to recognise that Gen AI changes the blast radius of ordinary mistakes. A single prompt can expose data, trigger an unauthorised workflow, or surface content that creates legal or reputational risk. In practice, many security teams encounter Gen AI misuse only after sensitive data has already been exposed, rather than through intentional governance and access design.

How It Works in Practice

Effective Gen AI deployment starts with the use case, not the model. Teams need to decide what the system is allowed to do, what data it may access, which users may invoke it, and which outputs require validation. That usually means combining security architecture with policy, training, and operational controls. For example, a customer-service assistant should not have the same data reach, retention rules, or approval workflow as an internal drafting tool.

Current guidance from frameworks such as OWASP Top 10 for Large Language Model Applications and the NIST AI Risk Management Framework points toward layered risk treatment: reduce exposure to prompt injection, prevent unsafe retrieval, validate outputs before downstream action, and maintain traceability for decision-making. For higher-risk deployments, organisations should also map the human workflow around the model, including who can approve actions, override safeguards, and review incidents.

A practical rollout usually includes:

• data classification and filtering before prompts reach the model
• least-privilege access to tools, connectors, and enterprise content
• logging for prompts, responses, and tool actions, with privacy review
• human review for high-impact outputs and external communications
• clear policies for acceptable use, prohibited content, and escalation

This is also where AI-specific attack patterns matter. Model poisoning, prompt injection, and retrieval abuse can all undermine a deployment even when the infrastructure is hardened. Security teams should test those conditions explicitly, not assume standard application controls are enough. These controls tend to break down when multiple business units connect the same model to different data sources without a single owner for policy enforcement, because inconsistent permissions and monitoring create blind spots.

Common Variations and Edge Cases

Tighter Gen AI governance often increases friction for users, requiring organisations to balance speed of adoption against control assurance. That tradeoff is real, especially in fast-moving business units that want immediate productivity gains. Best practice is evolving, and there is no universal standard for exactly where to draw the line between autonomy and oversight.

Some deployments are relatively low risk, such as internal drafting assistants with no external actions and tightly scoped data access. Others are materially more dangerous, especially agentic systems that can query systems, execute transactions, or interact with customers. In those cases, the question is not only whether the model is accurate, but whether the surrounding workflow can prevent an incorrect output from becoming an operational action. The distinction matters because a harmless hallucination in a chat window becomes a security incident when it triggers access changes, payments, or data disclosure.

Identity and privilege governance become especially important when Gen AI tools act on behalf of users or services. If the system uses shared credentials, weak approval chains, or poorly governed service accounts, then the model becomes a new route to privilege misuse rather than a productivity layer. Organisations should therefore treat AI access as a governed identity problem as much as a model risk problem, and align controls to both operational reality and the business impact of failure.

Related resources from NHI Mgmt Group

• Should organisations build separate controls for AI agent deployments?
• How should organisations handle privileged access when workloads and AI systems are part of the model?
• When should organisations add containment controls to AI agent deployments?
• Should organisations use just-in-time access for AI model operations?