What breaks when an AI agent can spot phishing but still act on it?

Why detection fails when the agent still has the hands-on keyboard

The core break is not in recognition, it is in execution control. An AI agent can correctly identify a phish, a fake login page, or a suspicious form and still continue because the action path is not bound to the detection result. That means the security boundary must sit around permitted actions, not around the model’s ability to notice danger.

When teams treat “the agent knows better” as equivalent to safety, they create a false control. The agent may have awareness but no enforced refusal, no step-up check, and no hard stop before it can hand over secrets, approve consent, or submit data.

What actually breaks in delegated workflows

Delegated workflows fail when intent, context, and permission are separated. The agent may be allowed to browse, click, copy, paste, or submit on behalf of a user even after it has detected risk, so the system behaves more like an ungoverned operator than a vigilant assistant. In practice, this is an authorization and delegation problem, not a perception problem.

The dangerous pattern is broad session reuse combined with weak action scoping. If the agent operates inside a live user session, especially one with saved credentials or consent grants, the model’s internal judgment cannot reliably compensate for an overbroad permission set. The right design is to make each high-consequence action independently permissible, attributable, and revocable.

For a practical control baseline, the most useful reference point is AI Agent Authorisation Guide, because it focuses on per-action policy, task-scoped access, and approval gates rather than relying on model awareness alone.

How to govern the action boundary instead of the model’s awareness

The action boundary should define what the agent may do after it has detected risk. That means separating read-only observation from write-capable actions, constraining where the agent can authenticate, and requiring explicit policy checks before any step that can transfer value, expose secrets, or alter permissions.

Teams also need lifecycle controls around the agent itself. If the agent can persist across sessions, reuse credentials, or act through a human identity, then the trust question becomes: who owns the authority, when can it be used, and how quickly can it be withdrawn?

A useful implementation pattern is to pair delegated access with strong identity and session controls. The point is not to remove autonomy entirely, but to make autonomy conditional on bounded scope, fresh authorization, and a clear stop condition when the agent enters a suspicious flow. The Zero Trust for AI Agents guidance is a strong fit here because it frames verification, standing privilege reduction, and per-request policy enforcement as the practical controls that stop unsafe action.

What good practice looks like when the agent can see the scam

Good practice is a control stack that assumes detection is advisory, not decisive. The agent may flag a phish, but the platform should still require separate authorization for credential entry, token transfer, form submission, or consent granting. In other words, recognition should route the event, not green-light the next click.

Visibility matters as much as prevention. If the agent is going to be allowed any meaningful action at all, teams need logs that show what it detected, what it attempted next, what policy permitted or blocked, and whether a human overrode the decision. Without that trail, you cannot tell whether the control failed, the policy was too permissive, or the agent simply ignored the warning.

For operational follow-through, AI Agent Observability, Audit and Incident Response Guide is the most relevant internal navigation point because it covers attribution, logging, and kill-switch design for exactly this kind of delegated action failure.

Risk and Threat Considerations

When an agent can detect phishing but still act on it, the main risk is that adversaries only need to win the action layer once. A convincing lure may be enough if the agent is allowed to continue into credential capture, consent abuse, or destructive workflow steps after it has already recognized the page as suspicious.

Failure mechanism: The attacker exploits a gap between judgment and enforcement, so the agent’s awareness does not stop downstream execution. That creates a trust-abuse path where the model becomes a conduit for the very outcome it identified as unsafe.

Impact: The result can be token theft, credential disclosure, unauthorized consent, account compromise, or other delegated-action abuse. The security failure is magnified when the agent operates with standing privilege or a reused user session, because one mistaken continuation can have real account-level consequences.

Practitioner Guidance

What to prioritise: Treat the action boundary as the control objective. If the agent can ever move from suspicion to execution without a policy decision, a human confirmation, or a narrow allowlist, the design is still vulnerable even if detection is excellent.

What to verify: Confirm that suspicious-flow handling actually blocks the risky step, not just raises an alert. Test the exact sequence where the agent recognises a phish and then tries to click, submit, authorize, or reveal a secret.

Common mistake: Assuming “the model will know better” is a substitute for per-action authorization. For this class of problem, awareness is helpful evidence, but enforcement is the control.

Practitioner takeaway: If detection does not change what the agent is allowed to do next, then the security control has failed at the only point that matters, the moment of action.

Related resources from NHI Mgmt Group

• What is the difference between human identity governance and AI agent governance?
• When does AI agent access create more risk than it reduces?
• What is the difference between governing human access and governing AI agent access?
• When do AI agent credentials create more risk than they reduce?