EU AI Act compliance is becoming an ongoing AI governance test

At a glance

What this is: The article says the EU AI Act is forcing GenAI teams to treat compliance as an ongoing governance test, with continuous risk management, adversarial testing, guardrails and auditability for high-risk systems.

Why it matters: This matters because IAM, NHI and AI governance teams need evidence that the whole AI system is controlled over time, not just that a model was selected or a policy was written.

By the numbers:

• The Act came into force in August 2024, but its full AI safety architecture requirements will not apply until August 2026.
• Prohibited practices can trigger fines of up to €35 million or 7% of global annual revenue, whichever is higher.
• Other violations can trigger fines of up to €15 million or 3% of global revenue.
• Supplying false or incomplete information can trigger fines of up to €7.5 million or 1% of global revenue.

Context

The EU AI Act is not a one-time documentation exercise. It is a lifecycle governance regime for AI systems that can change behaviour after deployment, especially when organisations add retrieval layers, tools, policies, user interfaces and decision logic around a model. That is why compliance for GenAI is increasingly an operational question about system boundaries, oversight and change control.

For IAM, NHI and agentic AI programmes, the important issue is not only who can use the system, but what the system can do, what it can reach and how those permissions are governed over time. Once copilots or agents start acting through tools and external systems, the compliance problem becomes an identity and authorisation problem as much as a model-risk problem.

Key questions

Q: How should teams govern GenAI systems under the EU AI Act?

A: Teams should govern the entire AI system, not only the base model. That means documenting ownership, purpose, data flows, tool access, monitoring and human oversight, then revisiting those controls whenever the system changes. Compliance is continuous because the risk profile can change after launch, especially when agents or retrieval layers expand what the system can do.

Q: Why do GenAI tools become higher risk when they can call external systems?

A: They become higher risk because tool access turns model output into operational action. Once a chatbot or agent can query data, update records or trigger workflows, the control problem shifts from content safety to authorisation, logging and oversight. That creates more ways for prompt injection, misuse or unintended delegation to create real-world impact.

Q: What are the signs that AI governance is failing in the enterprise?

A: Common warning signs include rapid growth in AI use without matching policy coverage, sensitive files being copied into personal accounts, and a large share of AI apps carrying high or critical risk. Another indicator is weak visibility into who is using which tools and what data they are sending. If teams cannot answer those questions, governance is not working as intended.

Q: What happens when AI development environments are not continuously monitored and controlled?

A: When AI environments are not continuously monitored, misconfigurations and privilege violations can persist long enough to cause data leakage, unsafe model exposure, or compliance failures. Shadow assets such as forgotten notebooks and rogue datasets can stay active without oversight. Attackers and insiders gain more room to exploit weak access controls and unnoticed workflow drift.

Technical breakdown

Why GenAI compliance is a system problem, not a model-selection problem

The article’s core point is that a GenAI product is not just the foundation model. It is the surrounding system of prompts, retrieval, tools, policies, users and decision logic that creates the actual risk profile. Two deployments using the same model can have very different obligations if one influences hiring, customer decisions or access to sensitive data. That is why regulators care about the operating system around the model, not just the model card. For identity teams, that boundary definition is critical because permissions, delegation and human oversight are part of the compliance surface.

Practical implication: Define the governed AI system, not just the model, and assign ownership for its data, tools and decision paths.

How adversarial testing and guardrails work together

Adversarial testing, or red teaming, is the pre-deployment and post-deployment process of trying to break the AI’s safety and policy assumptions. Guardrails are the runtime controls that stop harmful outputs, unsafe tool use or policy violations before users see them. The article treats these as complementary controls rather than substitutes. Testing finds failure modes, while guardrails enforce the live policy boundary. For agentic systems, that distinction matters because a model can be technically compliant in the lab but still unsafe once it can call tools, fetch data or chain actions.

Practical implication: Test the full AI workflow, then enforce those findings with runtime guardrails tied to policy and access scope.

What continuous monitoring means for high-risk AI

The EU AI Act pushes compliance into post-deployment monitoring because risk changes as systems, data and use cases change. Continuous monitoring is about detecting drift, new misuse patterns, policy violations and changes in how the system influences decisions. In practice, this means evidence trails, alerting and change review need to cover the AI’s inputs, outputs, tool calls and human override paths. For AI agents, that also creates an identity governance requirement: if the agent’s authority expands or its integrations change, the risk classification can change with it.

Practical implication: Track changes to model behaviour, tool access and user impact as part of the compliance control set.

Threat narrative

Attacker objective: The objective is to exploit weak governance around the AI system so that harmful or non-compliant behaviour reaches users and regulators can no longer treat the deployment as controlled.

1. Entry occurs when a GenAI system is deployed with tools, retrieval and user-facing workflows that extend beyond the base model’s original risk profile.
2. Credential_harvested or delegated access is abused when the system can reach external data, internal services or policy exceptions through approved integrations.
3. Escalation follows when unsupervised outputs, prompt injection or unsafe tool chaining let the system act beyond the intended governance boundary.
4. Impact is regulatory, operational and reputational, including fines, forced rollbacks, feature freezes and unsafe decisions reaching users.

Breaches seen in the wild

• CoPhish OAuth phishing via Copilot Studio: Datadog showed Copilot Studio agents on a Microsoft domain can front OAuth consent phishing and forward stolen tokens; no victims reported.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

EU AI Act compliance is becoming a control-operations problem, not a paperwork problem. The article correctly frames the shift from one-off documentation to ongoing proof of safe operation across the AI lifecycle. That matters because compliance evidence now has to survive model updates, tool changes and new use cases. Practitioners should treat the AI system as a governed service with living control boundaries.

System-level inventory is the new governance baseline. The article’s strongest insight is that model selection alone cannot describe the compliance risk. Once retrieval, policies, interfaces and decision logic are added, the real subject is a governed AI service, not a model. That is especially relevant where agentic systems can reach tools or data sources, because identity, authorisation and oversight become part of the compliance boundary. Practitioners should inventory AI systems the same way they inventory other critical services, with ownership and lifecycle control.

Adversarial testing has moved from assurance activity to regulatory expectation. The article shows that red teaming is no longer a specialised enhancement for mature AI teams. It is part of demonstrating that a high-risk system can be challenged before users or regulators do it for you. That shifts the standard from “can this model work?” to “can this system be shown not to fail in predictable ways?” Practitioners should build testing evidence into governance records.

Runtime guardrails are now the practical expression of AI policy. A policy that cannot be enforced at the point of inference is not much help for conversational or agentic systems. The article’s guardrail discussion points to a wider industry pattern: compliance is moving into runtime authorisation, output control and traceable logging. For identity programmes, that means AI policy and access policy are converging. Practitioners should align guardrails with access scope, logging and human override paths.

Named concept: AI governance drift. This article illustrates how a system can move out of its original compliance posture as features, integrations and audiences change. That drift is what makes static reviews inadequate for GenAI deployments that evolve after launch. The practitioner response is to treat every material change as a governance revalidation event, not a minor release note.

From our research library:

• 92% of organisations expose NHIs to third parties, raising concerns about supply chain security, according to the Ultimate Guide to NHIs.
• Generative AI use specifically increased from 33% in 2023 to 79% in 2025, according to McKinsey’s Global Surveys on the State of AI.
• Read next: AI Agent Observability, Audit and Incident Response Guide

What this signals

AI governance drift: once an organisation adds retrieval, tools or policy layers, the compliance question changes from model assurance to change control. That creates a governance burden familiar to IAM teams: the control has to be revalidated whenever the operating context changes, not just when the model changes.

The most important programme implication is that GenAI oversight has to be tied to access governance, logging and human approval paths. Where agentic systems can act through tools, identity and authorisation controls become part of the AI compliance story, not a separate security workstream.

For practitioners

• Define the governed AI system boundary Document the model, retrieval layer, tools, policies, users and decision logic as one controlled service. Reassess scope whenever features or integrations change.
• Operationalise adversarial testing before launch Run red-team scenarios against prompt injection, unsafe tool calls, policy bypass and data exposure, then retain the findings as compliance evidence.
• Map agent and copilot permissions to explicit approval paths Limit tool and data access to the minimum needed for the use case and require human review where outputs can affect rights, safety or access decisions.
• Build runtime guardrails with audit trails Enforce policy at inference time, log what was blocked or overridden, and preserve traces that show why a response or action was allowed.
• Treat feature changes as compliance events Revalidate risk, oversight and documentation whenever a system starts serving a new user group, new tools or a higher-stakes workflow.

Key takeaways

• The EU AI Act pushes GenAI compliance toward continuous operational proof, not static policy statements.
• The most material governance boundary is the full AI system, including retrieval, tools, prompts, users and oversight.
• Teams that cannot revalidate change, testing and runtime control after launch will struggle to defend high-risk AI deployments.

Key terms

• High-Risk AI System: A high-risk AI system is one whose outputs can materially affect a person’s rights, opportunities, or safety. These systems need stronger oversight because errors, bias, or unauthorized actions can create legal exposure as well as security and trust problems.
• Adversarial Testing: A testing approach that tries to break a policy by using hostile or unexpected inputs. For PBAC and AI access controls, that means probing for prompt injection, role crossover, leakage, and connector drift so the organisation can see whether the policy still holds under pressure.
• Runtime Guardrail: A control applied while an AI agent is operating, not just during configuration or review. Guardrails can block dangerous tool calls, require approval for sensitive actions, or stop data leakage before it reaches systems or users.
• API Governance Drift: API governance drift is the gradual mismatch between how APIs are supposed to be controlled and how they are actually used. It happens when policies, authentication rules, schemas, ownership, or logging expectations change unevenly across teams and environments, creating blind spots, inconsistent enforcement, and higher exposure to misuse or unauthorized access.

Deepen your knowledge

NHI Mgmt Group's NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, agentic AI identity and machine identity security. It is designed for practitioners who need to connect identity control to modern AI and access governance.