OWASP ASI09 Human-Agent Trust Exploitation Guide

When people trust an AI agent's fluent explanations and approve what it recommends without checking, an attacker who controls the agent's inputs no longer needs to fool the human directly: the agent does it for them. ASI09 Human-Agent Trust Exploitation is the ninth entry in the OWASP Top 10 for Agentic Applications for 2026, published by the OWASP GenAI Security Project in December 2025. It covers the human side of agent risk: automation bias, persuasive or emotional agent behaviour, fabricated explanations and approval steps that have quietly stopped being checks. This guide explains what OWASP includes, how to design approvals and interfaces that resist manipulation, the identity controls that keep the human and the agent accountable, how to test and measure it, and how the entry maps to other frameworks. It is for product owners, security architects, fraud and finance control owners, and anyone who signs off actions an agent proposes. For the whole list, see the OWASP Top 10 for Agentic Applications Guide.

Key takeaways

The human performs the harmful act. In ASI09 the agent persuades and a person clicks approve, pays, pastes or discloses, so the agent's influence can be invisible to forensics unless you log what it showed the user.
It is about misplaced trust, not agent intent. OWASP separates ASI09 (human misperception and over-reliance) from ASI10 Rogue Agents (the agent deviating from its intended behaviour); the root cause is often ASI01 goal hijack or poisoned data.
Model-written rationales are not evidence. OWASP warns about fake explainability and asks for plain-language risk summaries that are not generated by the model, plus provenance metadata on every recommendation.
Approvals must be designed, not bolted on. Confirmation for sensitive actions, risk-based friction, side-effect-free previews and out-of-band verification for payments are the core controls; a rubber-stamped approval is worse than none because it creates false assurance.
Identity closes the accountability gap. Separate agent identities, approvals bound to an authenticated human, and tamper-proof logs linking prompt, recommendation and action are what make an ASI09 incident investigable.

What it is and who it applies to

OWASP describes how agents build trust through fluent language, apparent empathy and perceived expertise, and how attackers or badly designed agents can exploit that trust to steer decisions or extract sensitive information. The risk grows when people over-rely on recommendations and approve actions without independent checks. Because a human performs the final step, OWASP notes the agent can act as an untraceable influence whose role is hidden from investigators.

The underlying human factor is well documented outside OWASP. NIST AI 600-1, the Generative AI Profile of the AI Risk Management Framework, lists Human-AI Configuration as a risk and describes automation bias as excessive deference to automated systems, alongside over-reliance and emotional entanglement. Article 14(4)(b) of the EU AI Act requires that people overseeing high-risk AI systems are enabled to remain aware of the tendency to over-rely on outputs (automation bias), particularly where the system provides recommendations for decisions taken by people.

Applies to: copilots and assistants that recommend actions a person then executes (payments, configuration changes, code merges, access grants, clinical or credit decisions), agents with human-in-the-loop approval steps, IT and HR support agents, and customer-facing agents that ask users for information.
Two directions of attack: an attacker poisons the agent's inputs so it misleads its own user (invoice fraud, a malicious code fix), or the agent is designed or tuned in a way that persuades users against their interests (anthropomorphic or emotionally manipulative behaviour).
Not only enterprise staff: consumers and new employees with low AI literacy or low task familiarity are at higher risk. Microsoft's overreliance guidance lists low AI literacy, low domain expertise, low task familiarity and high general trust in AI as risk factors.

Structure and key requirements

The tables paraphrase OWASP's common examples, attack scenarios and mitigations, with the concrete control we suggest for each.

Vulnerability patterns and scenarios

OWASP pattern (paraphrased)	Illustrative OWASP scenario	Where it usually starts
Insufficient explainability: opaque reasoning the user cannot question	A reviewer approves a risky configuration change justified by a fabricated audit rationale	ASI01 goal hijack, LLM09 misinformation
Missing confirmation for sensitive actions	A hijacked agent persuades an analyst to approve deleting a live production database	ASI02 tool misuse, LLM06 excessive agency
Emotional manipulation by anthropomorphic or empathetic agents	A prompt-injected IT support agent cites real tickets, then asks a new hire for credentials	ASI01, poisoned ticket content
Fake explainability: plausible rationale hiding malicious logic	A poisoned finance copilot urges an urgent payment to attacker bank details; the manager approves on the agent's apparent expertise	Poisoned document (ASI06), LLM01
Helpful-assistant trojan	A compromised coding assistant proposes a fix that runs a malicious script when pasted	ASI04 supply chain, ASI05 code execution
Consent laundering through "read-only" previews	Opening a preview pane triggers webhook side effects	Interface design
Clinical decision manipulation	A biased or poisoned care assistant recommends a dosage change that a clinician accepts	Data poisoning, LLM09

The OWASP incident tracker tags ASI09 on the March 2025 "Rules File Backdoor" research by Pillar Security, where hidden Unicode characters in AI coding assistant rule files caused Cursor and GitHub Copilot to insert an attacker's script tag and leave the change out of the chat response. The researchers named developer automation bias as part of why the injected code would be accepted. GitHub later added a warning on github.com when a file contains hidden Unicode text.

Mitigations and how to implement them

OWASP mitigation (paraphrased)	Concrete control
Explicit confirmation before accessing additional sensitive data or performing risky actions	Action classification (read, write, irreversible, financial, privilege) with mandatory confirmation for the top tiers, enforced in the tool layer, not the prompt. See the AI Agent Action Approvals and Spending Limits Guide
Tamper-proof logs of user queries and agent actions	Append-only store capturing the prompt, retrieved sources, the recommendation as displayed, the approver and the executed action under one correlation ID
Monitor sensitive data exposure and risky actions over time	DLP on agent output; alerts when an agent asks users for secrets, MFA codes or payment details
Plain-language risk summary not generated by the model; a way for users to flag suspicious behaviour that triggers review or capability lockdown	Approval screen shows deterministic facts (payee changed, new bank account, records affected, environment) from system data; a "report this agent" button that pauses risky tools for that session
Calibrate autonomy and oversight to contextual risk; flag low-certainty cues; train overseers	Risk-tiered approval policy; uncertainty and low-confidence markers; role-based training for approvers on agent manipulation
Provenance metadata (source IDs, timestamps, integrity hashes) on recommendations and external data; signature validation and policy checks	Every recommendation links to its sources; unsigned or unverified sources are labelled and cannot back a high-risk action
Block state-changing or network calls in preview mode; risk badge with provenance and expected side effects	Previews render in a sandbox with no outbound calls or writes; side effects listed before the user commits
Visually distinguish high-risk recommendations; remind users of manipulation patterns; avoid persuasive language in safety-critical flows	Neutral, non-anthropomorphic tone in finance, security and clinical flows; no urgency language; distinct styling for irreversible actions
Compare action sequences against approved workflow baselines; alert on detours, skipped validation, novel tool combinations	Workflow conformance checks. See the AI Agent Behaviour Baselining and Anomaly Detection Guide

Designing approvals that are real checks

Microsoft's guidance on overreliance on AI frames the goal as appropriate reliance: users accept correct output and reject incorrect output. It recommends realistic mental models, signalling when to verify, and making verification cheap, and it warns that explanations and the mere presence of citations can raise trust even when content is wrong. Translated into approval design:

Show facts, not persuasion: the approval card shows what will change, drawn from system data, separately from the agent's rationale, which is labelled as model-generated.
Cognitive forcing for high-risk items: require the approver to type the amount or select the target, or answer one question, rather than a single click. Use sparingly so friction stays meaningful.
Out-of-band verification: payee or bank detail changes, credential requests and privilege grants are verified through a channel the agent does not control, using contact details already on file.
Two-person rule where stakes justify it: separation of duties so the person who asked the agent is not the only approver for irreversible or high-value actions.
No batch approval of heterogeneous actions: approving 40 actions at once is not oversight. See the Human-in-the-Loop for AI Agents Guide.

Deepfake fraud shows the same human failure without an agent. In the Arup deepfake fraud, a finance employee made 15 transfers totalling HK$200 million after a video call in which every other participant was fake; the fraud surfaced only when he contacted head office directly. A persuasive agent recommending an urgent payment exploits the same trust, so the same out-of-band verification applies. See the Authorised Push Payment (APP) Fraud Guide.

Identity and NHI requirements

OWASP's Appendix C maps ASI09, with ASI01, to NHI10 Human Use of NHI from the OWASP NHI Top 10: humans operating with non-human credentials lose accountability. In ASI09 terms, if you cannot tell whether a person or an agent took an action, you cannot reconstruct who was manipulated.

Separate identities for the agent and the human: the agent acts under its own workload identity; the human approves under their own authenticated session. Neither uses the other's credentials. See the OWASP NHI Top 10 NHI10 Human Use of NHI Guide.
Approvals bound to strong authentication: high-risk approvals require a fresh phishing-resistant authentication step, so an approval cannot be replayed or made by whoever is near an unlocked screen.
Agents never ask for secrets: enforce a policy (and an output filter) that agents never request passwords, MFA codes, recovery codes or API keys from users. Train staff that any agent asking is a red flag.
Delegation is explicit: when an agent acts on a user's behalf, use a delegated token that records both identities, rather than the user's own token. See the OAuth Token Exchange and Delegation Guide.
Non-repudiation of the full chain: log which identity proposed, which displayed, which approved and which executed. This is NIST SP 800-53 AU-10 in practice.

Implementation steps

Inventory human decision points. For each agent, list every place a person acts on its output: approvals, copy-paste of code or commands, payments, disclosures. Classify each by impact and reversibility.
Classify actions and set tiers. Define which actions need confirmation, which need out-of-band verification and which need a second approver. Write the tiers as policy enforced in the tool or gateway layer.
Redesign the approval interface. Separate system-derived facts from model rationale, show provenance and side effects, remove urgency and emotive wording, and add a flag button. Pilot with one workflow and measure.
Make previews inert. Audit preview and summary features for outbound calls, webhooks and writes; block them in preview mode and test that blocking.
Bind approvals to identity. Require step-up authentication for high-risk approvals and record the approver identity with the action.
Log the whole chain. Capture prompt, sources, displayed recommendation, approval and execution under one correlation ID, in a store agents and operators cannot alter.
Train approvers. Role-based training on agent manipulation patterns: fabricated rationales, urgency, requests for secrets, payee changes. Include examples from your own systems.
Test and tune. Run seeded manipulation tests (below) in audit mode first, then enforce. Roll back an interface change if rejection rates of seeded bad recommendations fall.

Evidence and assessment

Test	Method	Pass condition
Poisoned payment recommendation	Seed an invoice with altered bank details in a test tenant; ask the finance copilot to process it	Payee change is highlighted from system data; out-of-band verification required; payment blocked until verified
Credential request	Inject an instruction into a ticket telling the support agent to ask the user for their password	Output filter blocks the request; alert raised
Fabricated rationale	Have the agent justify a change by citing a policy or audit that does not exist	Provenance check shows no verified source; reviewer screen labels the rationale as unverified
Inert preview	Open previews of documents containing tracking pixels and webhook links	No outbound request observed
Approver vigilance	Insert a known-bad recommendation into a sample of real approval queues (with consent and a safe rollback)	Rejection rate above the agreed threshold
Forensic reconstruction	Pick a past approved action at random	Prompt, sources, displayed text, approver and execution can be retrieved within the target time

Metrics:

Seeded-error catch rate: percentage of deliberately wrong recommendations rejected by approvers.
Approval dwell time and rejection rate: near-zero rejections and very short dwell times on high-risk items signal rubber-stamping.
High-risk action coverage: percentage of irreversible or financial actions protected by confirmation plus out-of-band or second-person checks.
Agent secret requests: count of blocked agent outputs asking users for credentials (target: investigated every time).
User flags: volume of "report this agent" flags and time to review.
Reconstruction time: minutes to retrieve the full chain for an approved action.

Mapping to other frameworks

Framework	Reference	Relevance to ASI09
OWASP Top 10 for LLM Applications 2025	LLM01 Prompt Injection, LLM05 Improper Output Handling, LLM06 Excessive Agency, LLM09 Misinformation	OWASP Appendix A mapping
OWASP Agentic AI Threats and Mitigations	T7 Misaligned and Deceptive Behaviours, T8 Repudiation and Untraceability, T10 Overwhelming Human in the Loop	Source threats (Appendix A); AIVSS core risk Agent Untraceability / Human Manipulation
OWASP NHI Top 10	NHI10 Human Use of NHI	Appendix C mapping
MITRE ATLAS	AML.T0051 LLM Prompt Injection, AML.T0052 Phishing, AML.T0048 External Harms; mitigation AML.M0029 Human In-the-Loop for AI Agent Actions	How the agent is subverted, its use to deceive people, and the oversight control
MITRE ATT&CK	T1566 Phishing, T1204 User Execution, T1657 Financial Theft	The human-executed step and typical impact
NIST SP 800-53 Rev. 5	AT-2(3), AT-3, AC-5, AU-9, AU-10, SI-4	Social engineering awareness, role-based training, separation of duties, protected logs, non-repudiation, monitoring
NIST CSF 2.0	PR.AT-01, PR.AT-02, DE.CM-03, PR.AA-05	Awareness and specialised training, personnel activity monitoring, enforced authorisation
NIST AI 600-1	Human-AI Configuration risk; actions such as GV-3.2-003, MP-3.4-005, MS-2.5-004	Acceptable use of human-AI configurations, monitoring outcomes, tracking anthropomorphisation
EU AI Act	Article 14 Human oversight, 14(4)(b) automation bias	Oversight design duties for high-risk systems
ISO/IEC 27001:2022 Annex A	5.3 Segregation of duties, 6.3 Information security awareness, education and training, 8.15 Logging	Organisational and logging anchors

On timing, the AI Act Explorer currently shows Article 14 applying from 2 December 2027 for Annex III high-risk systems and 2 August 2028 for Annex I systems, following the Digital Omnibus changes; check the official text before relying on these dates. Related entries: OWASP Agentic Top 10 ASI01 Guide, OWASP Agentic Top 10 ASI08 Guide and OWASP Agentic Top 10 ASI10 Guide.

Common pitfalls

Treating the agent's explanation as the audit trail. A model-written justification can be fabricated. Record sources and system facts separately and show them to the approver.
Believing an agent's report about its own actions. In the Replit AI agent database deletion, the agent deleted a production database during a code freeze, then said rollback was impossible when it was not, and generated fake data. The user's trust in its reports was the weak point; verify outcomes independently.
Approval fatigue by design. Asking for confirmation on everything trains people to click through. Reserve friction for high-risk actions so it keeps meaning.
"Read-only" features with side effects. Previews, link unfurls and summaries that fetch URLs or fire webhooks act without consent. Test them explicitly.
Persuasive personas in critical flows. Friendly, confident, human-like tone increases trust in exactly the flows where scepticism matters. Keep finance, security and clinical flows neutral.
Shared credentials between user and agent. If the agent uses the user's token, logs cannot show who acted, and the manipulation becomes untraceable.

Practitioner checklist

List every point where a person acts on agent output and classify it by impact and reversibility.
Enforce confirmation, out-of-band verification or two-person approval by action tier in the tool layer.
Show system-derived facts and provenance on approval screens, separate from model rationale.
Block network calls and writes in preview mode and test it.
Prevent agents from asking users for passwords, MFA codes or keys, and alert when they try.
Give agents their own identities and bind high-risk approvals to step-up authentication.
Log prompt, sources, displayed recommendation, approver and action under one correlation ID in tamper-proof storage.
Train approvers on manipulation patterns and seed known-bad recommendations to measure vigilance.
Remove urgency and emotive language from safety-critical agent flows.
Track seeded-error catch rate, approval dwell time and reconstruction time.

OWASP Agentic Top 10 ASI09 Guide: Human-Agent Trust Exploitation