LLM Flowbreaking

Expanded Definition

LLM Flowbreaking describes attacks that exploit the orchestration layer around a large language model, not just the model’s text generation. The attacker aims to interrupt or reorder the sequence between prompt submission, moderation, model output, post-processing, and client rendering, so that harmful content slips through before controls can react.

This makes it different from prompt injection alone. Prompt injection focuses on changing model behaviour through crafted inputs, while flowbreaking focuses on the application pipeline that carries model responses to users, logs, agents, and downstream services. In practice, the weakness often appears where streamed tokens, partial outputs, asynchronous filtering, or delayed policy checks create a timing gap. That is why the concept fits the operational concerns described in the OWASP Agentic AI Top 10 and the NIST AI 600-1 Generative AI Profile, both of which emphasise system-level risk rather than model-only risk.

The most common misapplication is treating a downstream filter as if it were a complete control, which occurs when unsafe content is already visible, logged, or forwarded before the filter completes.

Examples and Use Cases

Implementing strong flow controls often introduces latency and engineering complexity, requiring organisations to weigh user responsiveness against the safety of the full response path.

• A streaming chatbot displays the first tokens of a response before moderation completes, allowing disallowed material to appear briefly and be captured by the user.
• An agent sends model output to a browser, ticketing system, or webhook before a policy engine inspects the content, causing unsafe instructions to propagate beyond the chat session.
• A retrieval-augmented workflow sanitises the final answer but not intermediate tool output, so harmful or sensitive material is exposed in logs and debug traces.
• A client application retries failed moderation asynchronously, meaning the model response is already rendered while the safety verdict arrives too late to matter.
• An autonomous agent chains multiple tools and stores partial outputs in memory, where a broken sequence lets one unsafe step influence the next before guardrails intervene.

These patterns align with the broader guidance in the NIST AI Risk Management Framework, which encourages organisations to manage risks across the full AI lifecycle and operating context, not only at inference time.

Why It Matters for Security Teams

LLM Flowbreaking matters because it turns a seemingly controlled AI feature into a race condition across trust boundaries. Security teams may believe guardrails are effective when they are actually seeing only the final state of a response, while users, logs, and downstream systems have already received unsafe content. That creates exposure in customer support, internal copilots, agentic workflows, and automated decision paths.

The risk becomes more serious where non-human identities, API keys, or privileged tool access are attached to the same flow. If an agent can execute actions, then a broken response pipeline is not only a content-safety issue but also a control-loss issue. That is why operational models such as the CSA MAESTRO agentic AI threat modeling framework and threat taxonomies like the MITRE ATLAS adversarial AI threat matrix are useful when assessing the surrounding system, even if they do not define the term itself.

Organisations typically encounter the consequences only after an unsafe answer, leaked token, or unauthorized tool action has already propagated, at which point flowbreaking becomes operationally unavoidable to address.

Related resources from NHI Mgmt Group

• How should security teams use LLM-based identity risk scoring in production?
• Why do LLM jailbreaks create an IAM problem?
• What is the difference between prompt injection and LLM remote code execution?
• What is the difference between vault-generated secrets and LLM-generated secrets?