What breaks when an organisation continues using an AI system that falls within the EU AI Act prohibited practices?

Why This Matters for Security Teams

Continuing to operate an AI system that falls within prohibited practices creates a control failure that goes beyond model quality or minor policy drift. It means the organisation is relying on a system that should have been removed from service, which turns AI governance into an active compliance exposure. For security, legal, and risk teams, the key issue is that a prohibited system can undermine trust in every downstream approval, exception, and assurance decision tied to it. The EU AI Act is designed to force this kind of escalation early, before deployment becomes entrenched. If a business keeps using the system anyway, it may also create contradictory obligations across privacy, procurement, vendor oversight, and incident response. That makes the problem operational, not just regulatory. In practice, many security teams encounter this only after a product owner has already embedded the system into a workflow and removal now affects customers, operations, and reporting lines.

How It Works in Practice

When an AI system is classified as prohibited under the eu ai act, the expected response is not compensating control tuning. The correct move is to stop the use case, document the decision, and assess whether any related data, integrations, or downstream decisions must also be unwound. That can include suspending API access, disabling automation, freezing model updates, and preserving evidence for legal and internal review. Where the system sits inside a broader platform, teams also need to separate the prohibited capability from adjacent lawful functions so they do not accidentally leave the risky feature active.

Operationally, security and governance teams should coordinate around four tasks:

• Identify the specific prohibited practice and the business process it supports.
• Confirm whether the system is still running in production, shadow mode, or test environments.
• Map decision impact, especially where the AI output influenced access, hiring, scoring, profiling, or monitoring.
• Record remediation actions in the same control evidence chain used for assurance and audit.

This is also where control frameworks matter. A mature programme will align shutdown, escalation, and evidence handling with broader security controls such as policy enforcement, logging, and asset governance. The most useful comparison is not “can the model be made safer,” but “can the organisation prove it has ceased the prohibited use.” Guidance from the NIST control family, including NIST SP 800-53 Rev 5 Security and Privacy Controls, is helpful here because it reinforces disciplined control ownership, traceability, and incident-style handling. These controls tend to break down when the AI system is embedded in a customer-facing workflow with weak asset inventory, because no one can quickly confirm where the prohibited capability is still active.

Common Variations and Edge Cases

Tighter AI shutdown controls often increase business disruption, requiring organisations to balance legal compliance against operational continuity. The hardest cases are rarely the obvious ones. They appear when the prohibited practice is one component inside a larger model stack, when a vendor is hosting the system, or when internal teams disagree on whether the use case crosses the legal threshold. Current guidance suggests that organisations should treat those disagreements as governance issues, not as reasons to keep operating by default.

Edge cases also arise when the AI output has already been consumed by other systems. If a prohibited system fed risk scoring, identity decisions, or screening logic, the organisation may need to reassess affected records and not just switch the model off. That becomes especially sensitive where identity signals or account decisions were involved, because follow-on actions may have been taken automatically. In those scenarios, the question is not only whether the model was noncompliant, but whether the downstream control chain now carries tainted decisions. If the implementation is split across multiple jurisdictions, the organisation may also face different legal thresholds and documentation duties. There is no universal standard for this yet, so the safest approach is to privilege documented cessation, evidence preservation, and legal review over partial remediation. For identity-heavy deployments, this is also where poor linkage to identity assurance can magnify the damage; if identity claims were part of the AI workflow, the issue may intersect with NIST SP 800-63 Digital Identity Guidelines on assurance and trust decisions.

Related resources from NHI Mgmt Group

• What breaks when AI system inventory is incomplete under the EU AI Act?
• What breaks when technical documentation is incomplete for EU AI Act conformity assessment?
• What breaks when multiple AI agents can act on the same authentication system?
• Who is accountable when an AI system misses EU AI Act requirements?