Why do AI systems create risk even when an organisation has formal governance and compliance in place?

Why This Matters for Security Teams

Formal governance can create a false sense of control if it is treated as a documentation exercise rather than an operational discipline. AI systems are especially risky because they combine data access, model behaviour, and user intent in ways that traditional policy reviews do not fully capture. A policy may approve a use case, yet still leave prompts, connectors, training data, or output handling exposed to misuse. That is why security teams need to test the control path, not just approve the control design, using a framework such as the NIST Cybersecurity Framework 2.0 to connect governance with operational safeguards.

The practical issue is that AI failures rarely look like a single control gap. They emerge when users submit sensitive material into approved tools, when access rights are broader than intended, or when the organisation assumes vendor guardrails are enough. Formal compliance can reduce audit findings, but it does not by itself prevent prompt injection, data leakage, or unsafe automation. In practice, many security teams encounter these failures only after an AI workflow has already exposed data, produced an unsafe action, or bypassed the intended approval path, rather than through intentional control testing.

How It Works in Practice

Effective AI risk management starts with mapping where the system can receive, transform, and emit sensitive information. That includes the model, the user interface, plugin or API integrations, retrieval layers, logs, and downstream systems that consume AI output. Organisations should treat each of those points as a control boundary and verify that governance requirements are actually enforced there. The NIST SP 800-53 Rev 5 Security and Privacy Controls is useful here because it forces teams to translate policy into access control, auditability, configuration management, and incident response requirements.

Operationally, strong programmes usually include:

• data classification rules that define what may enter prompts, retrieval stores, and fine-tuning pipelines
• role-based approval for connecting AI tools to internal systems or secrets repositories
• logging and review of prompts, tool calls, and high-risk outputs, with privacy boundaries defined in advance
• validation of AI output before it is used in customer, financial, legal, or privileged workflows
• periodic testing for prompt injection, data exfiltration, and unsafe automation paths

That testing matters because governance documents often assume users will follow the approved process. Real environments are messier. An employee may paste customer records into a generative tool, an agent may call an internal API with excessive privileges, or a retrieval system may surface information from the wrong repository. Best practice is evolving, but current guidance suggests pairing policy with technical enforcement, monitoring, and human review. These controls tend to break down in fast-moving SaaS environments with many unvetted integrations because the effective attack surface changes faster than the governance review cycle.

Common Variations and Edge Cases

Tighter AI controls often increase operational overhead, requiring organisations to balance speed of adoption against the need for demonstrable safety. That tradeoff is especially visible when teams want broad experimentation but also expect sensitive data protection and reproducible oversight. Current guidance suggests that high-risk use cases need stronger validation, while lower-risk internal assistants may tolerate lighter controls if data exposure is tightly constrained.

There is no universal standard for every AI deployment pattern yet, so the right answer depends on how autonomous the system is, what data it can reach, and whether it can trigger actions. A customer-facing chatbot with retrieval over regulated data needs a different control model than an internal drafting assistant. Where AI is used for identity proofing, payments, or fraud workflows, organisations should also consider the control expectations reflected in the ISO/IEC 27001:2022 Information Security Management and ISO/IEC 27002:2022 Information Security Controls families, especially around access discipline, supplier risk, and monitoring. For AI-enabled customer onboarding or financial screening, the FATF Recommendations - AML and KYC Framework can also be relevant where model decisions influence trust, identity, or transaction decisions.

The main edge case is over-reliance on vendor assurances. If an organisation treats a model as compliant because it is hosted by a reputable provider, it may miss local misconfiguration, over-permissioned connectors, weak output review, or shadow AI use outside approved channels.

Related resources from NHI Mgmt Group

• When does AI create more governance risk than traditional data systems?
• Why do AI model servers create NHI governance risk even when deployed locally?
• Why do autonomous AI systems create new IAM risk even when no attacker is involved?
• Why do runtime context requests create new governance risk for AI systems?