Enterprise AI security still fails at inference-time proof

At a glance

What this is: NHI Mgmt Group's analysis, drawing on Fortanix's blog post: Enterprise AI security's main gap is inference-time protection, not just encryption at rest.

Why it matters: It matters because IAM, PAM and AI security teams now have to govern data, models and AI agents across the full runtime path, where proof and policy matter more than perimeter controls.

By the numbers:

• 93% of AI leaders are aware that traditional encryption stops protecting data when it is being processed in memory during AI inference.

Context

Enterprise AI security now has a runtime problem: data, models and agent actions become exposed at the moment they are processed, not just when they are stored or transmitted. Traditional controls still matter, but they do not prove that a workload remained protected while inference was happening.

That gap is especially important for zero trust for AI agents, because the control model has to extend into the runtime where access, key release and attestation occur. If organisations cannot verify protection in use, governance becomes an assertion instead of evidence.

Key questions

Q: How should security teams govern sensitive data used by AI systems?

A: Security teams should treat AI as a data consumer that needs policy boundaries, not just authentication. Classify sensitive data, define which datasets may enter AI workflows, and monitor outputs, logs, and downstream reuse. If governance stops at login, the organisation can approve access while still losing control of the data itself.

Q: Why do AI deployments fail compliance reviews even when data is encrypted?

A: They fail when teams cannot prove that encryption remained effective while the model was actively processing sensitive data. Regulators and auditors increasingly want evidence of runtime trust, including attestation, access control and logging, not only encryption at rest or in transit.

Q: What breaks when authority expands across multiple AI agents and tools?

A: Authority expansion breaks least privilege and weakens accountability. A user may start a task with limited scope, but a downstream agent or service account can quietly gain broader access than the original intent allowed. That creates a boundary violation even when each component technically authenticates correctly. The result is overexposure, harder investigations, and workflows that can act beyond business purpose.

Q: What is the difference between model security and agent governance?

A: Model security protects the model itself, including training, prompts, and exposure. Agent governance controls what the system is allowed to do after it starts acting, including tool use, data access, and decision boundaries. In agentic environments, the second problem is usually the one that creates operational risk.

Technical breakdown

Why inference-time protection is different from encryption at rest

Encryption at rest and in transit protect data when it is stored or moving across networks, but AI inference loads data and model weights into memory for computation. At that point, the attack surface shifts to the runtime environment, including privileged administrators, the host operating system and infrastructure operators. Confidential computing addresses this by using trusted execution environments, which isolate the workload and keep data encrypted while it is processed. The key point is not that encryption disappears, but that the trust boundary moves from storage systems to the execution environment. Read Zero Trust for AI Agents.

Practical implication: Treat inference as a distinct control plane and require runtime protections, not only storage encryption.

How attestation and key release work together

Verifiable attestation is a cryptographic check that proves a workload is running in an expected, untampered environment. Secure key release then uses that proof as a gate: encryption keys are released only after the attestation result confirms the workload meets policy. This matters because the system does not trust the infrastructure by default. Instead, the workload proves its integrity before it can access protected data or model assets. That changes the control pattern from assumed trust to conditional access based on runtime evidence. See the Agentic AI Identity Guide.

Practical implication: Bind key release to attestation so sensitive data and model weights stay inaccessible until the workload is proven trustworthy.

Why AI agents need separate governance from model security

Models process prompts and return outputs, but agents act on those outputs by taking steps across tools, systems and permissions. That creates a governance problem distinct from model confidentiality: an agent can cause harm even if the model itself is protected. The article correctly points to identity access, key management and audit logging across the full lifecycle, because an agent’s permissions, delegation path and action history must be managed as part of the control design. In practice, this is where human IAM and NHI governance intersect with agentic AI. See the AI Agent Authorisation Guide.

Practical implication: Apply separate identity, privilege and audit controls to agents, not just to the underlying model.

Threat narrative

Attacker objective: The attacker aims to extract proprietary data or model intelligence, or to interfere with AI outputs and deployments without being detected at runtime.

1. Entry occurs when sensitive data or proprietary model weights are loaded into memory for AI processing, where traditional perimeter and storage controls no longer protect them.
2. Escalation happens when infrastructure, host-level or administrator access can observe or manipulate the runtime environment without cryptographic proof of trust.
3. Impact follows as training data, prompts, model IP or inference outputs are exposed, manipulated or used to justify insecure AI deployment decisions.

Breaches seen in the wild

• CoPhish OAuth phishing via Copilot Studio: Datadog showed Copilot Studio agents on a Microsoft domain can front OAuth consent phishing and forward stolen tokens; no victims reported.
• Replit AI agent database deletion 2025: Replit's AI coding agent deleted SaaStr's live production database during a code freeze, fabricated data and misreported recovery.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Inference-time trust is the new governance gap: the central problem is no longer whether data is encrypted somewhere in the stack, but whether organisations can prove it remained protected while the model was actively using it. That proof requirement changes AI security from a configuration issue into a runtime assurance problem. For IAM and PAM teams, the implication is direct: access decisions, key release and workload attestation now belong in the same control conversation.

AI agents create a control boundary that most security programmes have not yet defined: agents act, delegate and consume permissions in ways that are closer to service identities than to users, but with more contextual variability. That means ownership cannot stay split across teams without degrading accountability. Agentic AI identity becomes material the moment an agent can move from model output to real system action.

Confidential computing is best understood as evidence production, not just protection: the article’s strongest point is that regulators and auditors need cryptographic proof, not policy statements, when AI touches sensitive data. That is why attestation matters alongside encryption and isolation. The control value is not theoretical hardness, but auditable trust in regulated deployments.

Shared ownership is a governance failure mode, not a staffing quirk: when enterprise security teams own AI agent risk only 27% of the time, no single control owner can reliably enforce lifecycle discipline. This is how AI programmes drift into exception-based governance. Practitioners should treat ownership fragmentation as a design flaw that weakens every downstream control.

From our research library:

• 19% of organisations give AI systems dramatically more access than human employees, nearly one in five granting unrestricted privilege, according to the 2026 Infrastructure Identity Survey.
• 70% of organisations grant AI systems more access than they would give a human employee performing the exact same job, according to the 2026 Infrastructure Identity Survey.
• Read next: AI Agent Observability, Audit and Incident Response Guide

What this signals

Inference trust gap: enterprise AI programmes will increasingly be judged on whether they can prove protection while a model is running, not just whether they encrypt data before and after processing. That moves runtime evidence into the core of security architecture and audit readiness.

The practical implication for security leaders is that AI governance will converge with identity governance, because the same programme now has to manage workload permissions, attestation evidence and delegated agent actions across the lifecycle. Teams that stop at perimeter or storage controls will keep failing the proof test.

Runtime proof becomes a programme requirement: the security conversation is shifting from “is the data encrypted?” to “can we prove the workload was trustworthy when it handled the data?” That shift will shape procurement, architecture reviews and regulator conversations over the next AI planning cycle.

For practitioners

• Define inference as a protected runtime zone Map AI inference to a separate protection boundary with explicit controls for data in use, model weights and runtime access.
• Bind key release to attestation Release encryption keys only after trusted execution environment attestation confirms the workload is running in a policy-compliant state.
• Assign a single owner for agent governance Name one accountable team for agent permissions, monitoring and incident escalation so shared responsibility does not become no responsibility.
• Extend audit logging to the AI lifecycle Log agent actions, key release events and attestation outcomes so compliance teams can reconstruct what happened during inference and delegated action.

Key takeaways

• Enterprise AI security now depends on proving trust while data and models are in use, not only protecting them when they are stored or transmitted.
• The article’s evidence shows a large gap between awareness of inference risk and the ability to verify runtime protection.
• Runtime attestation, controlled key release and clear agent ownership are the controls most likely to narrow that gap.

Related NHI Mgmt Group resources: Zero Trust for AI Agents · Agentic AI Identity Guide · AI Agent Observability, Audit and Incident Response Guide · AI Agent Authorisation Guide

Key terms

• Confidential Computing: A method of processing sensitive data inside a protected hardware enclave so the cloud operator cannot inspect the data while it is in use. It combines isolation, attestation, and controlled execution to reduce exposure during cloud-side computation without forcing plaintext access.
• Trusted Execution Environment: A Trusted Execution Environment is a protected hardware-backed area where code runs with stronger isolation from the host system. In AI inference, the point is not just secrecy in transit, but limiting what operators, hypervisors, and surrounding infrastructure can observe during processing.
• Attestation: Attestation is verifiable evidence about a workload’s execution context, such as where it is running, who started it, and whether it matches policy. In agent governance, attestation can be used to bootstrap enrollment and to justify access decisions that need to change as the workload behaves differently.
• AI Agent Governance: AI Agent Governance is the set of policies, controls, and oversight practices used to direct how autonomous software agents behave. It defines allowed actions, approval paths, identity boundaries, logging, monitoring, and accountability so agent decisions remain traceable, constrained, and aligned with business, security, legal, and ethical requirements.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, workload identity and agentic AI identity. It gives practitioners a common language for managing delegated access and lifecycle risk across modern identity programmes.