Securing the human-ai attack surface now needs dual governance

At a glance

What this is: This whitepaper argues that AI-enabled work has created a human-AI attack surface that needs discovery, monitoring and policy controls alongside conventional security measures.

Why it matters: It matters because IAM, PAM and security teams now have to govern both human decision-making and AI agent behaviour, especially where trust, access and automation intersect.

By the numbers:

• 92% agree governing AI agents is critical to enterprise security, yet only 44% have implemented any policies to do so.
• 80% of organisations report their AI agents have already performed actions beyond their intended scope, including accessing unauthorised systems (39%), inappropriately sharing sensitive data (31%), and revealing access credentials (23%).
• 72% of organisations have experienced or suspect they have experienced a breach of non-human identities , 46% confirmed, 26% suspected.

👉 Read KnowBe4's whitepaper on securing the human-AI attack surface

Context

The core problem is not that AI has entered the workplace. The problem is that most security models still assume people and systems interact through clearly bounded workflows, while AI copilots, assistants and autonomous agents now sit inside those workflows and influence decisions, content and access paths in real time. That creates a human-AI attack surface that spans identity, trust and operational control.

For IAM and security teams, the governance issue is the boundary between human intent and machine execution. When an AI agent can draft, recommend, retrieve or trigger actions, it becomes part of the control plane in practice even if it is not part of the formal identity model. That is why discovery, monitoring and policy enforcement now need to cover AI-assisted work as well as conventional user access.

The whitepaper's starting point is becoming typical rather than exceptional: most large environments are already dealing with some form of AI-enabled collaboration, but few have designed controls for prompt injection, AI impersonation or automation bias.

Key questions

Q: How should security teams govern AI-enabled workflows that can act on their own?

A: Treat them as identity-governed execution paths, not just software features. Assign a named owner, define least-privilege access, log every tool call, and require revocation paths for credentials and tokens. If the workflow can touch production systems or sensitive data, its permissions must be reviewed with the same discipline used for privileged machine identities.

Q: Why do AI agents increase non-human identity risk?

A: AI agents increase non-human identity risk because they can execute many actions quickly once they inherit a credential or tool permission. That speed expands blast radius, shortens attacker dwell time, and makes weak delegation more dangerous. The remedy is tighter scoping, continuous verification, and strict separation between observation and execution privileges.

Q: What do organisations get wrong about AI prompt injection risk?

A: Organisations often treat prompt injection as a text-only problem, when it is really an execution problem. The issue is not only manipulated output, but whether that output can trigger sensitive data access or downstream actions. Effective defence requires monitoring the entire live interaction path.

Q: Who is accountable when an AI agent causes a security incident?

A: Accountability should sit with the business owner, the system owner, and the security function together, because agent behaviour crosses operational boundaries. Organisations need a defined owner for approval, monitoring, and retirement, plus audit evidence that shows what the agent accessed and why.

Technical breakdown

Human-AI attack surface: how trust becomes an exploit path

The human-AI attack surface emerges when users rely on AI systems to draft, summarise, retrieve or act on information inside operational workflows. That creates a trust channel attackers can manipulate through prompt injection, impersonation and misleading context. The issue is not only model output quality. It is that people may treat AI-generated content as authoritative, while the system may also pull data or trigger actions with limited human scrutiny. In identity terms, the control gap is around delegated influence, not just authentication.

Practical implication: classify AI-assisted workflows as governed interaction paths and apply explicit approval, logging and segregation controls where outputs can affect access or business action.

Prompt injection and AI impersonation as social engineering at machine speed

Prompt injection exploits the instructions and context an AI system consumes, causing it to ignore boundaries or reveal data it should not expose. AI impersonation uses synthetic or manipulated content to create false legitimacy around requests, especially in channels where employees expect rapid responses. Both attack types work by shaping the decision environment rather than breaking encryption or authentication. For security teams, this means the adversary is now targeting the interface between identity, language and automation, which traditional phishing training alone will not fully address.

Practical implication: add content filtering, tool-use restrictions and monitored delegation paths to reduce the chance that manipulated prompts or impersonated messages drive unsafe actions.

Discovery and monitoring for AI usage need to include identity and privilege context

Visibility into AI usage is not the same as merely knowing which tools are installed. Effective discovery must identify where AI systems are embedded, what data they can touch, which accounts or tokens they use, and whether their actions are attributable to a person, service account or autonomous workflow. That is where AI governance intersects directly with IAM and NHI control. Without that mapping, organisations cannot tell whether a risky action came from a human, an assistant or an agent operating under standing privilege.

Practical implication: inventory AI-enabled workflows with their backing identities, data scopes and permissions so access reviews can include both human and non-human actors.

Threat narrative

Attacker objective: The attacker aims to hijack the trust relationship between employees and AI systems so that manipulated outputs, disclosures or actions create operational compromise.

1. Entry occurs through a trusted human-AI workflow where a prompt, message or synthetic instruction reaches the assistant or agent inside a legitimate business process.
2. Escalation follows when the system treats the manipulated context as valid and exposes data, generates harmful output or performs an action under the permissions it inherits from the surrounding workflow.
3. Impact is achieved when the attacker uses the resulting trust failure to influence decisions, leak information or trigger unauthorised actions at scale.

NHI Mgmt Group analysis

AI governance is becoming an identity problem, not just a model-risk problem. Once AI assistants are embedded in production workflows, they influence actions, data access and trust decisions in ways that belong in the identity control plane. That means IAM, PAM and NHI teams cannot treat AI usage as an adjacent concern. They need governance for who or what can act, on whose behalf, and with what data scope. The practitioner conclusion is clear: AI oversight must be mapped into identity governance, not parked beside it.

Human-AI attack surface is the right concept for this category. The article usefully frames the risk as interaction-layer abuse, which is more precise than generic AI security language. Attackers are not only targeting models. They are targeting the collaborative space where human judgement, agentic behaviour and enterprise context overlap. That makes prompt injection, impersonation and automation bias part of a single governance problem. The practitioner conclusion is to manage the interaction surface as a first-class security domain.

Discovery without attribution is incomplete for AI-enabled environments. Knowing that AI exists in the estate is not enough if teams cannot map the backing identity, privilege scope and data path behind each workflow. This is where NHI governance and AI governance converge. A model that can read, summarise or act on business data needs a traceable operational identity, even when a human initiated the session. The practitioner conclusion is to make identity attribution a prerequisite for AI workflow approval.

Automation bias is now a control failure mode. When users over-trust AI-generated output, the organisation inherits a new class of human-factor weakness that can bypass otherwise sound technical controls. That matters because security programmes often measure phishing resilience but do not measure over-reliance on AI-generated recommendations. The practitioner conclusion is to test whether staff can challenge AI output before it becomes an access or decision event.

Policy controls for AI-enabled environments must separate recommendation from execution. The article points toward scalable governance, but the real governance requirement is stricter: systems that advise should not be the same systems that act without boundary checks. This aligns with broader zero-trust principles and with NHI governance where standing privilege is the real problem. The practitioner conclusion is to constrain execution paths even when AI assistance remains broad.

What this signals

The practical signal for security programmes is that AI governance cannot live only in policy language. Teams need to know which assistants and agents are active, which identities they use, and where human approval still matters. Interaction-layer governance is emerging as the missing control domain for AI-enabled work, especially where a recommendation can become an action.

For identity teams, the next step is to extend lifecycle thinking to AI-enabled access paths. That includes discovery, ownership, data scope and revocation for assistants that behave like users but are not managed like users. The same discipline that reduced NHI risk in other contexts now needs to apply to the human-AI collaboration layer.

Programmes that already use zero-trust language should anchor AI controls in the same logic: verify the actor, constrain the action, and limit the blast radius of every privileged path. The control objective is not to eliminate AI from work. It is to make AI participation auditable, bounded and reversible.

For practitioners

• Map AI-enabled workflows to their backing identities Inventory every assistant, copilot and agent that can access enterprise data, then bind each to the service account, token or user context it uses. Require owners, data scopes and approval paths before the workflow is allowed to influence business action.
• Separate advice from execution in AI-assisted processes Allow AI systems to draft, summarise or recommend, but require explicit guardrails before they can submit, modify or trigger actions that affect access, records or transactions. Log each transition from recommendation to execution.
• Add prompt-injection and impersonation controls Use content filtering, tool-use restrictions and channel validation to reduce the chance that manipulated prompts or synthetic messages can drive unsafe behaviour in AI systems or employee workflows. Pair this with targeted awareness training on AI-assisted deception.
• Extend access reviews to AI-driven data paths Review not only the human user but also the AI service, model or agent that can read, transform or transmit sensitive data. Where standing privilege exists, tighten scope and move toward task-bounded access with clear audit trails.

Key takeaways

• AI copilots and autonomous agents create a human-AI attack surface that sits directly inside identity governance, not outside it.
• The main risk is trust manipulation through prompt injection, impersonation and automation bias, which can turn everyday workflows into attack paths.
• Security teams should map AI workflows to identities, separate recommendation from execution, and enforce auditable policy boundaries before AI systems influence action.

Key terms

• Agentic AI attack surface: The set of AI workloads, tools, prompts, and connected services that can be influenced or abused at runtime. It includes not only the model itself but also the identities and integrations that let the system act. For governance, the surface is defined by behaviour as much as by deployment.
• Automation Bias: Automation bias is the tendency to trust machine output as objective simply because it is machine-generated. In identity and governance programmes, this becomes a control problem when plausible agent decisions are accepted without questioning the embedded tradeoffs, making drift and misuse harder to detect.
• Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads — causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.
• AI-powered impersonation: The use of generative or adaptive AI techniques to mimic legitimate users, documents, or behaviours. In identity operations, this raises the quality and speed of deception, making static rules and isolated checks less reliable as primary controls.

What's in the full article

KnowBe4's full whitepaper covers the operational detail this post intentionally leaves for the source:

• Concrete examples of human-AI attack paths and the control points they exploit
• Practical guidance on reducing automation bias and improving digital mindfulness
• Operational ideas for discovery, monitoring and protection in AI-enabled environments
• Policy and governance considerations for scaling controls across human and AI collaboration

👉 KnowBe4's full whitepaper covers the attack patterns, governance model and protection approach in more operational detail.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, agentic AI identity and secrets management. It gives security and identity practitioners a common framework for governing access, scope and accountability across human and machine actors.