Stop And Roll Attack

Expanded Definition

A Stop And Roll attack is a response-interruption tactic against streaming AI systems, especially large language model interfaces that generate output token by token. The attacker or user triggers a stop action after the model has already begun producing an answer, then retains the partial content before downstream safety checks, policy filters, or post-generation review steps finish their work.

This differs from ordinary cancellation because the security failure is not the interruption itself, but the fact that the system treats the partial answer as usable output. In well-designed systems, guardrails should be enforced before disclosure or be resilient to mid-stream termination. In practice, usage is still evolving across vendors, and implementations vary in where moderation, redaction, and policy enforcement occur. The concept is relevant to agentic AI and chat systems that expose streaming responses, tool calls, or staged validation. Readers can compare it with broader adversarial AI categories in the MITRE ATLAS adversarial AI threat matrix and with operational controls in NIST SP 800-53 Rev 5 Security and Privacy Controls.

The most common misapplication is assuming a “stop” button only halts generation, which occurs when the application returns partial text before moderation or policy enforcement has completed.

Examples and Use Cases

Implementing streaming output rigorously often introduces latency and product friction, requiring organisations to weigh conversational responsiveness against the cost of stronger pre-disclosure controls.

• A customer support chatbot streams a sensitive troubleshooting answer, then receives a stop command before a policy filter removes an unsafe instruction.
• An internal AI assistant begins drafting a phishing template, and the user halts the response after the harmful portion has already been displayed.
• A workflow agent emits a tool recommendation in stages, but a mid-stream interruption leaves a partially vetted action path visible to the operator.
• A red team tester intentionally stops output at a point where the model has started revealing restricted content, demonstrating that the interface leaks value before final review.
• A security team compares the behaviour against adversarial AI patterns documented by Anthropic — first AI-orchestrated cyber espionage campaign report and maps the interaction path to known abuse workflows in the MITRE ATT&CK Enterprise Matrix.

Why It Matters for Security Teams

Stop And Roll matters because it exposes a gap between content generation and content governance. If an AI system assumes moderation can happen after the fact, an attacker may recover disallowed material simply by interrupting the stream at the right moment. That turns a user-interface control into a policy bypass, which is especially serious for systems handling secrets, regulated advice, or operational instructions.

For security teams, the issue is less about the stop action itself and more about sequencing: guardrails need to complete before any sensitive token sequence is visible, persisted, or forwarded into another agent step. This is directly relevant to agentic AI deployments, where one partial output can become another system’s input. Controls from NIST SP 800-53 Rev 5 Security and Privacy Controls and threat intelligence from CISA cyber threat advisories help teams think about monitoring, response, and containment, but the design fix must happen in the generation pipeline itself. Organisations typically encounter the consequences only after a streamed response is partially exposed or reused downstream, at which point Stop And Roll becomes operationally unavoidable to address.

Related resources from NHI Mgmt Group

• What are the signs that a cyber defense program is failing to stop common attack paths?
• How should security teams stop exposed Kubernetes registry secrets from becoming a supply chain attack path?
• Why does Agentic AI make NHI attack surface expand so significantly?
• What is the difference between attack surface management and NHI governance?