A prohibited AI system is an AI capability that an organization bans because it creates unacceptable security, legal, privacy, or governance risk. In practice, this includes systems that process sensitive data without approval, operate outside policy controls, or introduce unmanaged agent, NHI, or model access into enterprise environments.
What makes a prohibited AI system prohibited?
A prohibited AI system is not simply an unapproved tool, it is a capability that crosses an organisational line because its expected use creates unacceptable security, privacy, legal, or governance exposure. The core issue is that the system’s design or operating model conflicts with policy, law, or risk appetite before it is ever allowed into routine production use.
That distinction matters because prohibition is a governance decision, not just a technical one. An organisation may allow limited experimentation with some AI tools, but still prohibit deployment when the system handles sensitive data without approval, bypasses oversight, or introduces uncontrolled access paths into enterprise workflows.
Why prohibited AI systems become a governance boundary
Prohibited AI systems usually sit at the point where innovation, compliance, and security controls stop aligning. The most common trigger is not model capability alone, but the combination of autonomy, data access, and unclear accountability. Once a system can act on sensitive information or make decisions without the required controls, the risk shifts from acceptable experimentation to prohibited use.
That is why many organisations define prohibited use cases at the policy level. The boundary helps separate permitted AI from workloads that may be useful but still cannot be operated safely, lawfully, or transparently under current controls. This is especially important when teams can spin up tools quickly and bypass the normal intake and review process.
Common characteristics of prohibited AI use
Prohibited AI systems often share a few traits: they process restricted data without approved safeguards, rely on shadow deployments, or connect to identities, tokens, or APIs that were never reviewed for that purpose. They may also embed human trust assumptions that the organisation cannot verify, such as unlogged agent actions or undocumented decision paths.
In practice, the phrase also covers systems that create unmanaged agent, NHI, or model access inside the enterprise. If a tool can read, write, call services, or influence business decisions without a clear owner and control boundary, it is no longer just a convenience feature, it becomes a governance and exposure problem.
How organisations identify and control prohibited AI use
Teams usually identify prohibited AI systems through use case approval, data classification, vendor review, and inventory control. The question is not only whether the model is powerful, but whether the deployment can be governed, monitored, and reversed if needed. An AI use case that lacks those basics should be treated as a stop condition, not as a candidate for informal exception handling.
For policy definition and enforcement, the AI Acceptable Use Policy Template is useful for translating approval boundaries into day-to-day rules, while the AI Governance Guide helps connect those rules to oversight, ownership, and use case approval. Where the issue is specifically prohibited practices and compliance scope, the EU AI Act Compliance Guide provides a practical map of restricted and regulated AI categories.
What to do when a system is already in use
When an AI system is discovered to be prohibited, the response should focus on containment, removal, and evidence preservation. That may mean disabling the integration, revoking access, and stopping any automated actions before the system can create further data exposure or policy breach. The goal is to prevent continued use while preserving enough information to understand how the system was introduced and what it touched.
Where the prohibited condition involves autonomous or semi-autonomous behaviour, the AI Agent Kill Switch and Containment Guide is directly relevant because it frames practical containment steps such as token revocation, tool cut-off, and quarantine. For risk assessment and sign-off, the AI Impact Assessment Template supports documenting why a use case crosses the threshold from reviewable to prohibited.
Risk and Threat Considerations
Prohibited AI systems matter because they can create immediate exposure even before an obvious incident occurs. The risk is usually not the model itself, but the combination of sensitive data, uncontrolled access, weak oversight, and hidden persistence of credentials, connectors, or agent privileges.
Failure mechanism: A banned system can still be used through shadow deployment, unsanctioned integrations, or reused credentials, allowing data to flow or actions to execute outside approved controls.
Impact: The result can be privacy leakage, policy breach, unauthorised access, and hard-to-audit business decisions, especially when the system operates with long-lived access or touches regulated data.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10 and OWASP Non-Human Identity Top 10 address the attack surface, NIST AI RMF sets the technical controls, and ISO/IEC 27001:2022 and EU AI Act define the regulatory obligations.
| Framework | Control / Reference | Relevance |
|---|---|---|
| ISO/IEC 27001:2022 | A.5.15 — Access control | Prohibited AI systems hinge on controlling who can access data and services. |
| Recommendation — Define and enforce access rules that prevent banned AI tools from reaching sensitive systems. | ||
| NIST AI RMF | GOVERN — Govern | Prohibited AI systems are governed through policy, accountability, and risk appetite decisions. |
| MAP — Map | Mapping use cases and context is necessary to identify when an AI system crosses into prohibited use. | |
| MANAGE — Manage | Managing risk includes stopping or constraining AI systems that cannot be operated safely. | |
| Recommendation — Set governance thresholds that block AI use cases with unacceptable risk. Inventory AI use cases and classify where they introduce unacceptable exposure. Implement controls that restrict, contain, or retire AI systems that fail risk review. | ||
| EU AI Act | Article 5 — Prohibited AI practices | The term directly concerns AI uses an organisation should forbid because they are unacceptable. |
| Article 9 — Risk management system | Risk management is the mechanism used to determine when AI use becomes unacceptable. | |
| Recommendation — Align internal prohibition lists to banned AI practices and block them from deployment. Use a documented risk process to identify and stop prohibited AI deployments. | ||
| OWASP Agentic AI Top 10 | ASI03 — Identity & Privilege Abuse | Prohibited AI systems often become dangerous when agents gain unmanaged authority or access. |
| ASI10 — Rogue Agents | Unapproved autonomous behaviour is a direct analogue of prohibited AI use. | |
| Recommendation — Restrict agent privileges so prohibited systems cannot act with excess authority. Detect and stop rogue agent behaviour before it reaches production data or tools. | ||
| OWASP Non-Human Identity Top 10 | NHI-05 — Overprivileged NHI | The term includes unmanaged agent and NHI access, which becomes risky when overprivileged. |
| NHI-02 — Secret Leakage | Prohibited AI systems often persist through exposed secrets, tokens, or API keys. | |
| Recommendation — Reduce non-human privileges so banned AI services cannot reach sensitive resources. Protect and rotate secrets that could keep prohibited AI access alive. | ||
Practitioner Guidance
Governance implication: Treat “prohibited” as an operational control status, not a vague policy label. If a use case is prohibited, it should be blocked in intake, procurement, and deployment workflows so teams do not keep rediscovering the same risk in production.
What to watch for: Hidden connectors, unmanaged tokens, ambiguous ownership, and requests to “just pilot it” are common signs that a prohibited system is being introduced through exception paths. The practical test is whether the organisation can explain who approved it, what data it can reach, and how it would be turned off.
Related resources from NHI Mgmt Group
- What breaks when an organisation continues using an AI system that falls within the EU AI Act prohibited practices?
- What happens when an AI system in the EU is deployed with prohibited behavior?
- When should organisations treat an AI agent as a privileged system?
- When should organisations treat an AI system as a non-human identity?