Prompt injection in court filings exposes AI review governance gaps

At a glance

What this is: NHI Mgmt Group's analysis, drawing on JetStream Security's report: Brazilian labor court sanction followed lawyers embedding hidden prompt-injection text in a filing to manipulate AI-assisted review.

Why it matters: It matters because legal, compliance, and AI governance teams now have to control not only what documents say to humans, but also what they instruct machine readers to do.

By the numbers:

• The court fined the two law firms 10% of the claim’s value.

Context

Prompt injection is not limited to chatbots and agent workflows. It can also appear inside ordinary business documents when attackers hide instructions for a machine system that will read, summarise, or act on the text.

This case shows why governance must extend to AI ingestion paths, not just model prompts or user interfaces. When a filing is designed to influence a machine reviewer, the control problem becomes document integrity, disclosure discipline, and review-layer resilience.

For legal and compliance teams, that makes AI-assisted document processing a governance boundary, not a convenience layer.

Key questions

Q: What breaks when prompt injection is hidden inside a document instead of a chat prompt?

A: The control boundary breaks because the system treats untrusted document text as if it were legitimate input. That can distort summarisation, triage, or drafting even when the visible document looks normal to a person. The fix is to screen source material, separate instructions from content, and reject hostile formatting before the AI sees it.

Q: Why do AI review systems need governance controls for uploaded documents?

A: Because the risk is not only prompt text typed into a chat box. External documents can carry instructions, hidden formatting, or extraction artefacts that influence the model or workflow. Governance controls need to cover ingestion, parsing, provenance, and human override so machine-readable content cannot silently steer a regulated decision process.

Q: What are the signs that a document may contain prompt injection?

A: Warning signs include hidden text, strange colour contrasts, unexpected font layering, unusual OCR output, and instructions that speak to the AI rather than the document’s subject. If a file tries to direct the reviewer, not just inform the reviewer, treat it as adversarial until proven otherwise.

Q: How should organisations respond when an AI-assisted filing workflow is manipulated?

A: Pause automated handling of the affected workflow, preserve the original file and processing logs, notify the responsible legal and security owners, and review whether the manipulation influenced any downstream output. The response should focus on containment, evidence preservation, and revalidation of any AI-generated result.

Technical breakdown

How hidden instructions survive normal document review

Prompt injection works when malicious instructions are embedded in content that humans are unlikely to notice but systems may still parse. In this case, the filing reportedly used white text on a white background, making the command invisible in ordinary viewing while still present in the document stream. That creates a mismatch between human interpretation and machine interpretation, which is the core risk in document-centric AI workflows. The problem is not just prompt wording. It is any content channel that allows the machine to ingest adversarial instructions as if they were legitimate input. Practical implication: treat document rendering, text extraction, and AI ingestion as separate control points. See the Prompt Injection Guide.

Practical implication: Inspect the document-rendering and extraction pipeline for hidden-text abuse before AI review sees the content.

Why AI review systems need input-boundary controls

AI review tools are vulnerable when they accept untrusted text without a strict boundary between source material and operating instructions. A model or workflow that summarises filings, drafts decisions, or flags issues can be manipulated if it treats embedded commands as higher priority than the task definition. That is why prompt injection is fundamentally an input validation and trust-separation problem, not only a model-safety problem. In regulated workflows, the system must preserve provenance, isolate instructions from content, and refuse to obey text that originates inside the reviewed document. Practical implication: require workflow-level separation between task instructions and source documents. See the Agentic AI Security Guide.

Practical implication: Separate system instructions from untrusted source text and enforce provenance checks on every ingested document.

How courts and enterprises should think about machine-readable content

The ruling treats the machine-readable layer of a filing as part of the filing itself. That is an important governance shift because compliance obligations are no longer satisfied by reviewing only the visible document presented to humans. Any content that can influence automated triage, summarisation, or case handling must be governed as operational input with integrity requirements. This applies equally to courts, legal ops, and enterprise workflows that ingest external documents into AI-assisted systems. Practical implication: extend content governance to OCR, parsing, summarisation, and review steps, not just the final artefact.

Practical implication: Apply integrity controls to OCR, parsing, and summarisation stages, not only to the final document artifact.

Threat narrative

Attacker objective: The objective was to manipulate AI-assisted judicial review so the filing would receive a weaker, less challenging machine-generated response.

1. Entry occurred when lawyers submitted a court filing that contained hidden prompt-injection text disguised within the document formatting.
2. Credentialed access was not the issue; the attack used the authority of the filing itself to reach the AI review workflow used by the court.
3. Impact was the attempted steering of the AI system toward a superficial or biased response, with legal and procedural consequences.

Breaches seen in the wild

• EchoLeak (Microsoft 365 Copilot) 2025: A crafted email could make Microsoft 365 Copilot leak data from its context with no click, a zero-click prompt injection fixed as CVE-2025-32711.
• ForcedLeak (Salesforce Agentforce) 2025: A critical Agentforce flaw let attackers steer an AI agent through a Web-to-Lead form to leak CRM data via an expired allowlisted domain.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Prompt injection now belongs in governance, not just model security. This case shows that adversarial instructions can be carried inside ordinary business artefacts and still reach an AI system with decision support value. The governance question is therefore who validates the input boundary, who owns document integrity, and who can block suspicious content before it reaches the model. Practitioners should treat untrusted documents as a first-class control surface.

Machine-readable content is now part of the regulated record. The court’s reasoning is important because it recognises that the harm sits in the text a system can read, not only in the text a human can see. That makes redaction, rendering, and extraction controls part of legal defensibility. Where AI is used in evidence, filings, or case management, the content pipeline itself becomes auditable control evidence.

AI governance debt appears when organisations assume review tools only need prompt hardening. This incident shows the failure mode is broader: hidden instructions, adversarial formatting, and trust in external documents can all bypass a simplistic prompt policy. Prompt injection guidance is relevant here because the control problem spans document handling, not just chat interfaces. Practitioners should expand review governance to ingestion, transformation, and output validation.

Legal and compliance teams now need an explicit policy for hostile submissions. The article makes clear that courts, firms, and enterprise legal operations cannot assume every inbound document is benign once it enters an AI-assisted workflow. The practical line is simple: if a document can influence an automated reviewer, it needs the same scrutiny as any other untrusted external input. Practitioners should define escalation paths, blocking rules, and human override criteria before the next submission arrives.

Prompt injection is becoming an enterprise trust-center issue, not just an AI issue. Any external-facing process that ingests documents, emails, or uploads into AI-assisted review creates a disclosure and integrity obligation to customers, counterparties, and regulators. That means the assurance story must cover both model behaviour and document hygiene. Practitioners should align AI controls with legal review, records management, and third-party intake governance.

What this signals

Prompt injection governance is becoming a document-security problem. The important shift is that AI misuse can arrive through ordinary business files, not only through chat interfaces or agent prompts. That means legal operations, compliance teams, and AI governance owners need shared controls over intake, extraction, and review boundaries.

Hostile content screening will become a standard control for AI-assisted records workflows. As more organisations let models summarise or classify external documents, the question is no longer whether the content is readable by a human. The question is whether the machine can be tricked into treating embedded instructions as trusted direction.

AI review assurance now depends on provenance and rendering integrity. If a workflow cannot prove what text was visible, extracted, and processed, it cannot reliably defend a regulated decision made with AI support. The practical direction is toward auditable ingestion pipelines and stronger content validation before model exposure.

For practitioners

• Define hostile-document screening Scan inbound filings and uploads for hidden text, colour-based concealment, OCR anomalies, and other formatting tricks before AI review processes them.
• Separate task instructions from source content Ensure the system prompt, workflow instructions, and document text are isolated so untrusted documents cannot override review behaviour.
• Add human approval for suspicious submissions Route documents with unusual formatting, embedded commands, or provenance gaps to a manual review queue before any automated summarisation or decision support.
• Instrument the ingestion pipeline Log extraction, parsing, and summarisation steps so teams can prove what the AI saw, what was blocked, and where control failed.
• Write an AI misuse response playbook Define who can suspend automated review, preserve evidence, notify counsel, and reprocess affected documents when prompt injection is detected.

Key takeaways

• The case shows that prompt injection can be embedded inside ordinary documents and still affect an AI review workflow.
• The ruling matters because it treats machine-readable content as part of the compliance and governance surface, not just the visible page.
• The most relevant controls are hostile-document screening, instruction-content separation, and auditable ingestion pipelines.

Related NHI Mgmt Group resources: Prompt Injection Guide · Agentic AI Security Guide · Red Teaming AI Agents for Identity Abuse

Key terms

• Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads, causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.
• Document Ingestion Pipeline: A document ingestion pipeline is the sequence of steps that transforms an uploaded file into text or structured data for downstream systems. It becomes a security boundary when hidden formatting, OCR errors, or embedded instructions can alter what the AI actually processes.
• Machine-Readable Content Integrity: Machine-readable content integrity is the assurance that text, formatting, and extracted data seen by automated systems accurately reflect the intended source. In AI-assisted workflows, it includes rendering, parsing, and transformation controls, not only the visible document presented to a human reviewer.
• AI Governance Debt: AI governance debt is the growing gap between how AI systems are deployed and how well they are controlled. It appears when policies, reviews, access controls, logging, and accountability lag behind AI adoption, creating unmanaged risk. In practice, it accumulates across data, identity, model, and decision workflows.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, agentic AI identity, and machine identity security. It helps practitioners connect identity controls to the AI workflows and trust boundaries their programmes now depend on.