Users trust an assistant's answer because of the parts that look like evidence: the citation, the document title, the confident tone, the suggested next step, and an injected prompt can forge every one of them. MITRE ATLAS tracks this as AML.T0067 LLM Trusted Output Components Manipulation, in the Defense Evasion tactic (AML.TA0007), with one sub-technique, AML.T0067.000 Citations. ATLAS describes prompts that manipulate components of a response so it appears trustworthy, helping the adversary keep operating and evade detection by the users the model talks to. Output handling as an application security problem (escaping, sinks, rendering) is in the OWASP LLM Top 10 LLM05 Improper Output Handling Guide. This page is about the trust layer above that: forged citations, false provenance, suppressed disclosures and persuasive framing, and how to verify, detect and respond. It is for teams running enterprise copilots, RAG assistants and coding assistants whose users act on answers.
Key takeaways
- The target is the human. AML.T0067 does not break the application; it makes a malicious answer look like a normal, sourced one so the user acts on it.
- Citations are only as good as their binding. ATLAS lists three citation variants: a wrong citation, an invented one, or a correct citation attached to adversary-provided data.
- Silence is a component too. In ATLAS case study AML.CS0041 the hidden prompt told a coding assistant not to mention the code changes it made, removing the cue a reviewer would rely on.
- Verify citations in code, not in the prompt. Our recommendation is to bind every citation to an ID that was actually retrieved and render source metadata from the system of record, not from model text.
- Response includes the people who acted. Finding who followed a manipulated answer (paid, approved, merged) matters as much as cleaning the source.
Which components get manipulated
ATLAS created AML.T0067 in March 2025 and rates it as demonstrated. Its description lists the language and tone of the response, links, recommended follow-up actions, retrieved document metadata and citations.
| Component | Manipulation | Why users trust it | Verifying control |
|---|---|---|---|
| Citations (AML.T0067.000) | Wrong source cited, invented source, or real source cited for attacker text | A footnote looks like evidence | Citation IDs must match retrieved chunk IDs; quoted text must appear in the cited chunk |
| Document metadata | Title, author or date presented as an internal, authoritative document | Familiar internal names and recent dates | Render metadata from the source system, never from generated text |
| Links | Link text names a trusted site, the target is elsewhere | Users read the label, not the URL | Show the real domain; allowlist link targets |
| Recommended actions | "Click here to re-authenticate", "use these payment details" | It comes from the company's own assistant | Action suggestions only from templates; never credentials or payment changes from generated text |
| Tone and framing | Urgency, reassurance, claims of verification | Assistants sound confident by design | Guidelines against claims of verification; user training |
| Omissions | Instruction to hide what the model did or read | No warning means no problem | Action summaries generated by the application from tool logs, not by the model |
What the case studies show
- Payment details in an enterprise copilot (AML.CS0026): researchers sent an email to a Microsoft 365 Copilot user containing content designed to be retrieved when the user searched for banking details, carrying the attacker's details. The injected instructions told the model how to format the reference, so the answer pointed to a plausible source. ATLAS maps the reference step to AML.T0067.000 Citations. The Register reported in August 2024 that Microsoft said such methods require prior compromise or social engineering and that it was improving safeguards.
- Silent changes in AI coding assistants (AML.CS0041): a prompt hidden in a rules file with invisible Unicode characters told the assistant to insert unwanted code and not to mention those changes in its responses. ATLAS records this as AML.T0067 because the omission keeps the developer and the logs quiet.
- Agent identity as a trusted component: the SalesBleed Salesforce Agentforce disclosure described an action that posted messages as the agent without attribution to the person who triggered it, so links in those messages carried the agent's trusted identity. Salesforce fixed the reported issues before disclosure.
The common lesson is that trust signals generated by the model can be steered by any text the model reads. Signals generated by the application from its own records cannot be steered the same way.
Designing trustworthy output components
- Bind citations to retrieval: pass each retrieved chunk to the model with an opaque ID, require the model to cite IDs, and drop or flag any citation whose ID was not retrieved in that turn.
- Check quote support: where the answer quotes or states a figure, check that the text or number appears in the cited chunk; mark unsupported claims rather than presenting them as sourced.
- Render provenance from the source: the title, author, location and date shown beside a citation come from the document store, along with a trust label for its origin (internal author, external sender, public web).
- Show external origin plainly: when a cited source is an email from outside the organisation or a public web page, say so in the citation itself. This single label would have exposed the payment-details scenario.
- Generate action summaries from logs: the "what I did" summary for an agent or coding assistant is built from tool call records, so a prompt cannot suppress it.
- Template high-risk actions: re-authentication prompts, payment details and approval requests are never produced as free text; the application shows them through fixed UI with its own data.
For retrieval permissions and source trust levels, see the Permission-Aware RAG Guide.
Detection
ATLAS mitigation AML.M0024 AI Telemetry Logging is the foundation: keep prompts, retrieved chunk IDs and sources, completions and tool calls together so an answer can be reconstructed.
| Signal | What to look for | Data source |
|---|---|---|
| Citation mismatch | Citations to IDs not retrieved; quoted text absent from the cited chunk | Retrieval and completion logs |
| External source behind high-impact answer | Answers about payments, credentials or approvals whose top source is an external email or web page | Retrieval logs with source origin |
| Formatting instructions in content | Retrieved text that tells the model how to cite, how to describe itself or what not to mention | Ingestion scanning of documents and mail |
| Action without disclosure | Tool calls or file edits not reflected in the response text | Tool logs compared with responses |
| Downstream behaviour | Payment detail changes, new payees or re-authentication on unusual domains after an assistant session | Finance systems, identity provider logs |
Measuring it
- Citation validity rate: share of citations that resolve to a chunk retrieved in the same turn.
- Supported claim rate: share of quoted claims found in the cited text, from sampling or an automatic check.
- External-source answers on sensitive topics: count per week, with the share reviewed.
- Undisclosed actions: number of agent actions missing from the user-facing summary (target zero).
Prevention controls mapped to ATLAS
- AML.M0020 Generative AI Guardrails: output validators that check citations, links and claims before display.
- AML.M0021 Generative AI Guidelines: instructions never to state that a fact is verified unless a cited source supports it, and never to take formatting or citation instructions from retrieved content. Useful, but a prompt-level control.
- AML.M0025 Maintain AI Dataset Provenance: source and modification history for indexed content makes trustworthy provenance labels possible.
- AML.M0033 Input and Output Validation for AI Agent Components: schema-validated citations and action records instead of free text.
- AML.M0029 Human In-the-Loop for AI Agent Actions: approval for payments, sends and merges, with the approval screen showing source origin.
- AML.M0018 User Training: ATLAS includes educating users to spot phishing; extend finance and engineering training to cover "the assistant cited it" as a reason to verify, not to trust. See the Business Email Compromise Awareness for Finance Teams Guide.
Response runbook
- Identify the source: from logs, find the retrieved item that carried the instructions and its origin.
- Quarantine: remove or exclude the item from retrieval, and search for similar items from the same sender or domain.
- Find who acted: list users who received answers citing that item and check for payments, approvals, merges or credential entry since; treat payment changes under the Business Email Compromise Response Playbook.
- Review code paths: for coding assistants, diff changes made in affected sessions against what the responses described.
- Fix the control gap: add the missing citation check, origin label or action template, and add the sample to regression tests.
Common failure patterns
- Model-written footnotes: letting the model write source titles and links as text, so they can say anything.
- Uniform trust for all sources: displaying an external email with the same citation style as an internal policy document.
- Trusting the response summary: reviewers relying on what the coding assistant says it changed instead of the diff.
- Free-text high-risk actions: payment details or re-authentication links coming from generated text.
- No retrieval log: without chunk IDs in logs, a forged citation cannot be told apart from a real one after the fact.
Framework mapping
| Framework | Reference | Relevance |
|---|---|---|
| MITRE ATLAS | AML.T0067, AML.T0067.000, AML.TA0007, AML.CS0026, AML.CS0041 | Technique, sub-technique, tactic, case studies |
| MITRE ATLAS mitigations | AML.M0018, AML.M0020, AML.M0021, AML.M0024, AML.M0025, AML.M0029, AML.M0033 | Training, guardrails, guidelines, telemetry, provenance, approval, validation |
| OWASP Top 10 for LLM Applications 2025 | LLM01 Prompt Injection, LLM05 Improper Output Handling, LLM09 Misinformation | Entry, output handling, overreliance on unsupported output |
| NIST SP 800-53 Rev. 5 | SI-15, SI-7, SI-4 | Output filtering, information integrity, monitoring |
Practitioner checklist
- Pass retrieved chunks with opaque IDs and reject citations to IDs not retrieved.
- Render titles, authors, dates and links for citations from the document store.
- Label external emails and public web sources as external in every citation.
- Build agent action summaries from tool logs, not from model text.
- Serve payment, credential and approval prompts only from fixed UI templates.
- Scan ingested content for citation and disclosure instructions addressed to an AI.
- Log retrieved chunk IDs with every completion.
- Train finance and engineering users to verify assistant-cited facts before acting.
Standards and references
- MITRE ATLAS: LLM Trusted Output Components Manipulation (AML.T0067)
- MITRE ATLAS: Citations (AML.T0067.000)
- MITRE ATLAS: Financial Transaction Hijacking with M365 Copilot as an Insider (AML.CS0026)
- MITRE ATLAS: Rules File Backdoor: Supply Chain Attack on AI Coding Assistants (AML.CS0041)
- OWASP: Top 10 for LLM Applications 2025
- The Register: Copilot vulnerabilities presented at Black Hat 2024
- NIST: SP 800-53 Rev. 5
Related NHI Mgmt Group resources: OWASP LLM Top 10 LLM05 Improper Output Handling Guide, MITRE ATLAS AML.T0071 False RAG Entry Injection, Enterprise AI Copilot Security Guide, MITRE ATLAS Guide.