Join our Newsletter — 33% off our NHI Course

Notifications
Clear all

Model-generated Markdown and stored XSS: are your renderers safe?


(@nhi-mgmt-group)
Member Moderator
Joined: 1 year ago
Posts: 19382
Topic starter  

TL;DR: Two customer-facing AI chat implementations exposed the same stored XSS by rendering model-produced Markdown with raw HTML enabled and no sanitizer, allowing attacker-controlled markup to execute in another user’s browser, according to Escape. The lesson is that every model output surface needs the same render-chain scrutiny as public input, because unsafe HTML handling compounds fast.

NHIMG editorial — based on content published by Escape: AI pentesting found stored XSS in customer-facing chat renderers

Questions worth separating out

Q: What breaks when AI-generated Markdown is rendered without sanitization?

A: The browser treats attacker-controlled markup as trusted HTML, which can turn a chat transcript into stored XSS.

Q: Why do model output surfaces create more risk than ordinary user comments?

A: Model output often re-enters the application through first-party APIs, making it look internal and safe even when the content is untrusted.

Q: How can security teams know whether Markdown rendering is actually safe?

A: They need to test the full render chain, not just the source library configuration.

Practitioner guidance

  • Remove raw HTML rendering from model output Disable rehype-raw or equivalent raw HTML passthrough for all AI-generated Markdown unless there is a documented business case and a post-parse sanitizer.
  • Sanitize after parsing, not before Place sanitization after Markdown and HTML parsing so the filter sees the final DOM tree, not raw tokens that can be reinterpreted later.
  • Audit every downstream render path Inventory where model output is reused, including support consoles, ticket comments, email templates, PDF generation, and analytics dashboards.

What's in the full article

Escape's full analysis covers the implementation detail this post intentionally leaves for the source:

  • The exact bundle inspection path used to trace the Markdown renderer and identify the unsafe plugin chain.
  • The step-by-step exploit reasoning that turns model-emitted Markdown into browser-executable content.
  • The renderer hardening guidance for React, Markdown, and iframe handling in the customer environment.
  • The practical payload verification approach Escape used to confirm the issue on the real origin.

👉 Read Escape's analysis of stored XSS in AI chat Markdown rendering →

Model-generated Markdown and stored XSS: are your renderers safe?

Explore further

View Full Forum →  |  NHI Foundation Course →



   
Quote
(@mr-nhi)
Member Moderator
Joined: 3 months ago
Posts: 18973
 

Model output must be treated as untrusted browser input: The security mistake here is assuming that generated text is less dangerous than user input. Once a model can emit Markdown that later becomes HTML, it belongs in the same trust category as any other untrusted content. Application security teams should align this with NIST-CSF protective controls and, where render pipelines span tools and workflows, treat it as a governance issue rather than a UI bug.

A question worth separating out:

Q: Who is accountable when an AI browser exposes sensitive data or makes a bad decision?

A: The organisation remains accountable for the access path it allowed. Security, IAM, and data-governance teams should jointly define approval boundaries, logging requirements, and content restrictions before deployment. If the browser can act across regulated systems, then its governance must be explicit before use, not after failure.

👉 Read our full editorial: Model-generated Markdown can become stored XSS in chat features



   
ReplyQuote
Share: