TL;DR: Attackers are now writing fake content for LLMs and AI agents, using training-data poisoning, indirect prompt injection, and poisoned retrieval content to steer AI answers, actions, and recommendations before the content ever enters an enterprise network, according to ZeroFox. The real control gap is no longer human awareness but whether organisations can govern what their AI systems read, trust, and execute.
At a glance
What this is: This is an analysis of LLM poisoning, where attackers publish fake or manipulative content to influence what AI systems ingest, repeat, or act on.
Why it matters: It matters because AI assistants and agents can now become the first consumer of public content, so governance has to cover what machines read and trust, not just what people see.
By the numbers:
- When NewsGuard tested 10 chatbots against 15 false narratives from the Pravda network, the chatbots repeated them 33% of the time.
- Anthropic, the UK AI Security Institute, and the Alan Turing Institute found that 250 malicious documents were enough to backdoor models ranging from 600 million to 13 billion parameters.
- Across the 26 models tested, four were manipulated into making a payment and two misclassified the fraudulent site as the real one.
Context
LLM poisoning is a governance problem in which external content is shaped to influence what AI systems learn, retrieve, summarise, or do. The issue matters because AI agents and copilots increasingly sit between users and information, so a malicious page can function like a control plane for the model rather than a simple misinformation source.
The identity angle is real even though the attack starts outside the perimeter. When an AI agent can browse, retrieve, or act on external content using a user’s session or credentials, poisoned content can steer machine decisions with the same operational impact that credential abuse once delivered through direct compromise.
Key questions
Q: What breaks when an AI agent acts on poisoned content?
A: The failure is not limited to a bad answer. Once the agent can call tools or use delegated credentials, poisoned context can become unauthorized execution, data exposure or policy override. The control failure is at the point where untrusted content is allowed to influence an identity-bound action.
Q: Why do poisoned web pages create business risk for AI assistants?
A: Because assistants often rank and summarise external sources as if they were authoritative, a forged page can displace the real answer and influence customer behaviour. That creates fraud, brand, and trust risk at the point of decision, not just a misinformation problem. The issue is source governance, not model accuracy alone.
Q: How should security teams handle prompt injection in AI systems?
A: Treat prompt injection as an authorisation problem, not only a content problem. Validate user input, monitor for override patterns, and block any prompt that tries to change model policy, exfiltrate data, or trigger sensitive actions. The safest design is one where untrusted prompts cannot directly reach code execution, production data, or privileged tools without policy checks.
Q: Should organisations treat AI poisoning as an IAM issue or a content-security issue?
A: Both, but the governance failure sits at the intersection. Content security matters because poisoned material must be discovered and removed, while IAM matters because the damage occurs when an agent uses real credentials or privileged sessions to act on it. If either side is weak, the attacker can still shape outcomes.
Technical breakdown
Training-data poisoning and LLM grooming
Training-data poisoning works by flooding the information ecosystem with false narratives until models absorb them as statistically plausible. LLM grooming is the scaled version of that approach, where repeated content gives a fabricated claim enough surface area to appear in model outputs. The article’s examples show that even limited malicious corpora can alter model behaviour, which means scale and repetition can matter more than sophistication. This is not a conventional perimeter problem because the content is designed to be consumed before it reaches enterprise controls.
Practical implication: treat public content exposure as an input-governance issue, not just a reputation issue.
Indirect prompt injection in retrieval and agent workflows
Indirect prompt injection hides instructions inside content that an AI system retrieves, turning data into executable intent. Unlike prompt attacks entered directly by a user, these payloads sit in web pages, schema markup, comments, or copied text and are interpreted when the agent fetches the page. That makes the retrieval layer part of the attack surface, especially when an agent can make purchases, leak data, or trigger workflows without a human in the loop. The problem is amplified when the agent has access to credentials or trusted sessions.
Practical implication: constrain tool-use permissions and validate retrieved content before any agent can act on it.
Poisoned retrieval and source-ranking abuse
The third pattern does not attack the model at all. It targets the sources that retrieval systems rank, index, and trust, using fake support numbers, lookalike domains, forum reposts, and trusted-host abuse to distort the answer layer. If the retrieval stack weights host reputation, file format, or indexing signals too heavily, a forged page can outcompete the legitimate source. That shifts the security question from model accuracy to source provenance and brand integrity across public channels.
Practical implication: monitor how your brand and contact details appear across indexed sources that AI assistants may prioritise.
Threat narrative
Attacker objective: The attacker wants to shape AI-generated answers and actions so that fraud, influence, or secret theft succeeds at machine speed.
- Entry begins when attackers publish false narratives, hidden instructions, or forged support content on public sites that AI systems are likely to crawl or retrieve.
- Credential or trust abuse follows when the poisoned content is consumed by an AI assistant or agent that already has access to a user session, API key, or enterprise tool chain.
- Impact occurs when the model repeats the falsehood, takes the wrong action, or surfaces fraudulent contact details that users treat as authoritative.
Breaches seen in the wild
- CoPhish OAuth phishing via Copilot Studio: Datadog showed Copilot Studio agents on a Microsoft domain can front OAuth consent phishing and forward stolen tokens; no victims reported.
- AI agent retail card theft campaign 2026: AI agents breached 27+ retailers for about $25 each, used cloud keys and a Secrets Manager dump, and stole 600,000+ payment cards.
Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.
NHI Mgmt Group analysis
AI poisoning is now an identity and governance problem, not just a content problem. Once an AI agent can read external content and act on it, the security boundary shifts from the website to the decision path. That means the organisation is governing machine consumption of public information, not only human exposure to misinformation. Practitioners should treat model input, retrieval provenance, and tool execution as one control chain.
Prompt injection succeeds because AI systems collapse the line between data and instruction. Traditional security models assume untrusted content stays inert until a human interprets it. In agentic workflows, that assumption breaks, because retrieved text can become an action request before review. The practical lesson is that instruction handling must be separated from content ingestion at the architecture level.
Brand abuse now creates AI answer risk as well as fraud risk. Fake support numbers, lookalike domains, and copied contact details can be indexed into assistant outputs even when the enterprise never hosted them. That expands trust and safety work into identity verification, brand protection, and AI governance. Organisations need to know where their authoritative contact data lives and which sources an assistant is likely to prefer.
Shadow AI turns poisoned content into unlogged business action. An unsanctioned agent that browses the web with real credentials can ingest and execute manipulated content without the visibility defenders rely on in email or endpoint tooling. That makes governance of AI access, not just model quality, the critical control point. The control failure is not the presence of AI, but the absence of reviewable boundaries around what it may consume and do.
From our research library:
- Only 44% of organisations have implemented any policies to manage their AI agents, despite 92% agreeing that governing AI agents is critical to enterprise security, according to the 2026 Infrastructure Identity Survey.
- 69% of organisations still authenticate machine identities with long-lived API keys, according to the 2026 State of AI Agent Identity Security Report.
- Read next: Agentic AI Identity Guide
What this signals
AI answer integrity will become a governance metric. Security teams should expect AI assistants to be judged by whether they return authoritative information, not merely by whether they remain available. That makes provenance, source allow-listing, and retrieval review part of the security operating model for any programme that exposes employees or customers to assistants.
Shadow AI multiplies the blast radius of poisoned content. If unsanctioned agents can browse the web with real access, a malicious page can influence decisions faster than traditional detection can intervene. The governance response is to inventory those agents, define their source boundaries, and make action rights proportional to trust.
AI systems are already being governed less by policy intent than by whatever they can read. That is why the first control question is not whether a model is accurate, but whether it should be allowed to consume and act on unauthorised external text at all. For practitioners, the next step is to move review from output checking to input and tool-governance design.
For practitioners
- Define AI retrieval boundaries List which copilots and agents may browse the web, which sources they may query, and which outputs they may use without human approval.
- Separate read and act permissions Prevent an agent from turning retrieved instructions into payments, commands, or credential disclosure unless a higher-trust workflow approves the action.
- Monitor public-source exposure for brand data Track lookalike domains, fake support numbers, paste sites, and reposted content that could be ingested by AI assistants as authoritative.
- Test assistant answers for poisoned loops Ask major assistants how they answer customer-service queries about your brand and compare the results against authoritative contact data.
Key takeaways
- LLM poisoning turns public content into an attack surface for AI systems that summarise, recommend, and act on external information.
- The article shows that false narratives, hidden instructions, and forged source data can influence chatbots and agents before any traditional security control sees the attack.
- Practitioners need controls over retrieval provenance, tool permissions, and authoritative source data if they want to reduce AI-driven fraud and manipulation.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10 and MITRE ATT&CK address the attack and risk surface, while NIST AI RMF and NIST SP 800-53 Rev 5 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP Agentic AI Top 10 | ASI06 — Memory & Context Poisoning | The article centres on poisoned content influencing agent context and outputs. |
| ASI09 — Human-Agent Trust Exploitation | The article shows attackers abusing the trust users and agents place in external content. | |
| Recommendation — Audit agent inputs and memory paths for poisoned or unauthorised external content. Limit agent actions when external content is the sole trigger for a decision. | ||
| NIST AI RMF | MANAGE — AI Risk Management and Governance | The article is about governing AI consumption, retrieval, and action risk. |
| Recommendation — Define AI governance controls for source trust, retrieval review, and action approval. | ||
| NIST SP 800-53 Rev 5 | IA-5 — Authenticator Management | Poisoned agents become more dangerous when they can act with live credentials. |
| Recommendation — Apply authenticator lifecycle controls to reduce the impact of agent misuse. | ||
| MITRE ATT&CK | TA0006;TA0008 — Credential Access; Lateral Movement | The attack path includes stolen credentials and agent-driven movement through trusted access. |
| Recommendation — Map poisoned-agent scenarios to credential access and lateral movement detections. | ||
Key terms
- LLM Poisoning: LLM poisoning is the deliberate corruption of a language model’s training or fine tuning data so it produces unsafe or misleading output. Attackers use this to influence code suggestions, insert backdoors, or degrade trust in generated content. In software development, it turns the model into a delivery channel for hidden risk.
- Indirect Prompt Injection: Indirect prompt injection is an attack where malicious instructions are hidden inside content that an AI system reads later. The model may treat that content as context rather than as hostile input, which can influence tool use, data access, or workflow actions if controls are weak.
- LLM Grooming: LLM grooming is the mass publication of repeated false narratives so that models absorb them as normal or credible information. The tactic works by volume and repetition, which can distort how the model answers questions even when no direct compromise of the model has occurred.
- Shadow AI: AI agents, copilots, or connected tools operating without full visibility or governance from security teams. Shadow AI becomes an identity problem when those systems authenticate with unmanaged tokens, service accounts, or OAuth apps that can reach production resources.
Deepen your knowledge
NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management. It helps security practitioners connect identity controls to the broader AI and access-governance decisions their programmes depend on.
Published by the NHIMG editorial team on October 16, 2026.
NHI Mgmt Group, the independent authority on Non-Human Identity, IAM, and Agentic AI security. nhimg.org