EU AI Act compliance is becoming an ongoing AI governance test

At a glance

What this is: This is an analysis of the EU AI Act's requirements for GenAI systems, with the central finding that compliance now depends on continuous governance across the full AI lifecycle.

Why it matters: It matters to IAM practitioners because GenAI systems increasingly behave like governed digital actors, so access, oversight, audit, and policy enforcement must extend beyond human users to the AI system itself.

By the numbers:

• The EU AI Act can impose fines of up to €35 million or 7% of global annual revenue for prohibited practices.
• Other violations can trigger penalties of up to €15 million or 3% of global annual revenue.

👉 Read ActiveFence's analysis of EU AI Act compliance for GenAI systems

Context

The EU AI Act changes the governance problem for GenAI from model choice to system control. Once a chatbot, copilot, or agent is exposed to users, the organisation is responsible for the full stack around it, including prompts, retrieval layers, tools, output rules, monitoring, and the evidence needed to prove those controls are operating.

That matters because many enterprise AI programmes still treat compliance as a document set rather than a runtime control set. Where GenAI systems can influence decisions, disclose data, or interact with external tools, the control boundary starts to look like identity and access governance for software actors, not just policy review for a model catalogue.

Key questions

Q: What breaks when GenAI compliance is treated as a documentation exercise?

A: Documentation without runtime controls creates a gap between declared policy and actual behaviour. GenAI systems change through prompts, tools, retrieval, and user context, so compliance has to be proven in operation. If organisations cannot show testing, monitoring, and enforcement over time, they can satisfy a checklist while still failing the underlying regulatory duty.

Q: Why do GenAI systems need both red teaming and guardrails?

A: Red teaming finds the ways a system can fail, while guardrails block unsafe behaviour in production. One without the other is incomplete. Testing reveals prompt injection, data leakage, and tool abuse paths, but only runtime enforcement stops those issues from reaching users or external systems.

Q: What do security teams get wrong about AI agent scope control?

A: Teams often assume a task description is enough to define privilege. In reality, scope control fails when an agent moves from a narrow informational task into operational action, such as modifying records or sending requests. The fix is not more intent language, but tighter action-level boundaries.

Q: Who is accountable when a GenAI system exposes sensitive data or generates harmful content?

A: Accountability sits with the organisation that designed, approved, and operated the workflow, including the teams responsible for identity, data, model governance, and compliance. In regulated environments, the control owner must be able to show logs, policy decisions, and remediation evidence.

Technical breakdown

Why system-level AI compliance is harder than model-level approval

The article's core point is that compliance cannot stop at the foundation model. A deployed GenAI system includes the model, retrieval, system prompts, user interface, tool permissions, output filters, and human override paths. Each of those components can change the risk profile, especially when the system influences hiring, support, or regulated decisions. That is why the article frames the AI Act as lifecycle regulation, not a one-time launch checklist. In practice, the governance unit is the AI system in production, not the model artifact in isolation.

Practical implication: inventory the full AI system, not just the model, before assigning risk ownership or approval.

How adversarial testing and guardrails work together

Adversarial testing, often called red teaming, is the structured attempt to break a system before users or attackers do. Guardrails are the runtime controls that block harmful prompts, unsafe outputs, or policy-violating actions while the system is operating. The article links the two because testing finds failure modes and guardrails enforce policy in real time. For GenAI, that includes prompt injection, harmful content, data leakage, and misuse of connected tools. The important governance point is that testing without enforcement leaves residual risk, while enforcement without testing gives false confidence.

Practical implication: pair pre-deployment adversarial testing with runtime policy enforcement and log both outcomes.

Why observability is a compliance control, not just a monitoring feature

The article treats observability as evidence, not convenience. For AI governance, observability means preserving a trace of what the system saw, what it returned, what policy fired, and how a decision was handled. That matters for regulatory inspection, incident review, and internal assurance. In identity terms, the same logic applies to non-human systems that can act, call tools, or produce outputs with business impact. Without auditable traces, organisations cannot explain why the system behaved as it did or whether human oversight was effective.

Practical implication: require audit trails that tie each AI action to policy, input, output, and resolution.

Threat narrative

Attacker objective: The attacker wants to influence the GenAI system into revealing information, bypassing policy, or producing business-impacting outputs that the organisation cannot safely explain or contain.

1. Entry begins when a public-facing GenAI system is exposed to users, retrieval sources, or external tools that expand its attack surface.
2. Escalation occurs when prompt injection, unsafe outputs, or tool misuse lets the system act beyond the intended policy boundary.
3. Impact follows when the system leaks data, produces harmful decisions, or forces rollback and regulatory scrutiny because controls were not enforced continuously.

NHI Mgmt Group analysis

AI compliance is becoming a runtime governance problem, not a document exercise. The article is right to frame the EU AI Act around continuous risk management, adversarial testing, and monitoring. For security teams, that means the control objective is sustained evidence of safe operation, not a one-off approval package. The practical conclusion is that AI governance now needs operational ownership, not just legal review.

System-level inventory is the missing control layer in most GenAI programmes. The article correctly distinguishes the model from the assembled product around it. Retrieval, tools, prompts, users, and override paths can all create new risk even when the base model stays unchanged. That is the same governance lesson identity teams learned with service accounts and API keys: what matters is the effective system in motion, not the component in isolation. Practitioners should inventory the entire AI system boundary.

AI observability is a trust and accountability control. The requirement to document what happened, why a safeguard triggered, and how the event was resolved is not administrative overhead. It is the only way to prove policy enforcement, investigate misuse, and support incident response. In identity and governance terms, auditability is what turns an AI feature into a controlled enterprise service. The practitioner takeaway is simple: if you cannot trace it, you cannot govern it.

Named concept: compliance drift in GenAI. The article points to a recurring failure mode where a system starts within policy but becomes misaligned as prompts, tools, users, or outputs change over time. That drift is especially dangerous for high-risk deployments because the legal obligation persists even when the product evolves. Practitioners should treat change management, testing, and approval refresh as part of the control itself, not as optional maintenance.

Identity and AI governance are converging at the point of delegated action. When a chatbot or agent can access data, call tools, or influence decisions, it behaves like a non-human actor that needs boundaries, oversight, and revocation paths. The article reinforces a broader market shift toward treating AI systems as governed entities. Security teams should expect IAM, GRC, and AI governance to share more control language, not less.

What this signals

Compliance drift in GenAI is now a programme risk, not just an AI team concern. As prompts, tools, retrieval sources, and user populations change, the system can move out of its original approval boundary even when the model itself does not change. Teams that already maintain identity lifecycle discipline will recognise the pattern: change management and re-certification need to follow the workload, not the project plan.

The control conversation is also shifting toward delegated action. Once a GenAI system can access data or execute tools, it needs boundaries, traceability, and revocation paths similar to other non-human actors. That makes identity governance, AI governance, and audit evidence part of the same operating model, especially where enterprises want alignment with the NIST AI Risk Management Framework and policy-aligned runtime controls.

For practitioners

• Map the full AI system boundary Document the model, retrieval sources, tool permissions, prompts, outputs, human override points, and affected users for every GenAI deployment.
• Make adversarial testing a release gate Run pre-deployment red teaming for prompt injection, data leakage, unsafe actions, and tool misuse, then repeat testing after material changes.
• Enforce runtime guardrails with policy mapping Tie output filters and tool restrictions to specific legal and internal policies so blocked actions are explainable and auditable.
• Require audit trails for every AI action Preserve traces of inputs, policy hits, outputs, and remediation decisions so incidents and compliance checks can be reconstructed later.
• Reassess risk after every material change Reclassify the system when new tools, new users, or new decision contexts change the compliance profile, even if the model stays the same.

Key takeaways

• The EU AI Act turns GenAI governance into a continuous control problem that spans design, deployment, and monitoring.
• Enterprises that cannot trace AI behaviour, test it adversarially, and enforce runtime policy will struggle to prove compliance.
• AI systems that can act on data or tools need identity-style boundaries, revocation, and auditability, not just model-level review.

Key terms

• High-Risk AI System: A high-risk AI system is one whose outputs can materially affect a person’s rights, opportunities, or safety. These systems need stronger oversight because errors, bias, or unauthorized actions can create legal exposure as well as security and trust problems.
• Adversarial Testing: A testing approach that tries to break a policy by using hostile or unexpected inputs. For PBAC and AI access controls, that means probing for prompt injection, role crossover, leakage, and connector drift so the organisation can see whether the policy still holds under pressure.
• Policy-Aligned Guardrails: Runtime controls that constrain what an AI system can say, access, or execute according to enterprise policy and regulatory requirements. They translate abstract obligations into enforceable behaviour and create evidence when a control blocks an action.
• Compliance Drift: Compliance drift is the gap between what a policy says, what the procedure requires, and what the organisation actually does. It usually appears when ownership is unclear, version control is weak, or evidence is collected too late to prove control operation.

What's in the full article

ActiveFence's full blog covers the operational detail this post intentionally leaves for the source:

• The article's breakdown of the 4-byte cache poisoning problem and how Python .pyc caches can be abused in GenAI-adjacent supply chains.
• The proof-of-concept discussion showing why review workflows and scanners miss the unchecked hash mode risk.
• The operational mapping between EU AI Act obligations and the vendor's WonderSuite controls for red teaming, guardrails, and observability.
• The specific examples of policy-aligned blocking for text, image, audio, and video outputs across multiple languages.

👉 ActiveFence's full blog covers cache poisoning, proof of concept details, and control mapping for high-risk GenAI deployments.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, secrets management, and agentic AI identity. It helps practitioners build the control thinking needed across identity, access, and delegated system behaviour.