Resource-draining prompts expose a denial-of-service gap in LLM security

At a glance

What this is: This is a Protect AI analysis of resource-draining prompts that intentionally push LLMs toward denial-of-service conditions by forcing excessive output generation.

Why it matters: It matters to IAM and security teams because LLMs and AI agents consume identity-bound resources, and abuse of those execution paths can degrade availability, disrupt trust boundaries, and expose governance gaps across AI, NHI, and human workflows.

By the numbers:

• When AWS credentials are exposed publicly, attackers attempt access within an average of 17 minutes.
• 80% of organisations report their AI agents have already performed actions beyond their intended scope.

👉 Read Protect AI's analysis of resource-draining prompts in GenAI

Context

Resource-draining prompts are a denial-of-service pattern for LLMs. Instead of stealing credentials or injecting malicious code, the attacker spends model capacity by demanding excessively long outputs, repeated tokens, or other expensive generations that push the system outside its intended operating envelope.

For security programmes, the issue sits at the intersection of AI runtime governance and identity-bound access. Once an LLM is exposed through chat interfaces, copilots, or agentic workflows, teams need to think about prompt cost, output volume, and service degradation as security controls, not just performance concerns.

Key questions

Q: How should security teams block resource-draining prompts in LLM applications?

A: Security teams should combine hard output caps, prompt-cost scoring, and request throttling. The goal is to identify unusually expensive generations before they consume shared capacity. Controls work best when they are enforced at the interface, tied to authenticated identity, and tuned separately for chat, code, and retrieval use cases.

Q: Why do long or repetitive prompts create denial-of-service risk for LLMs?

A: Long or repetitive prompts can force the model to spend excessive compute on a single request, which degrades latency and consumes memory, context, and service capacity. In shared environments, that can delay or block legitimate users even when the prompt itself looks syntactically harmless.

Q: How do you know if prompt-abuse controls are actually working?

A: Look for falling rates of rejected expensive prompts, stable latency under load, and fewer extreme token-generation events from the same identities. If those indicators improve while legitimate long-form tasks still succeed, the controls are doing their job instead of just suppressing usage.

Q: Who is accountable when an LLM denial-of-service event is triggered by a legitimate user or service account?

A: Accountability usually sits with the team that owns the model endpoint and its access policy, because the abuse occurred through an authorised identity. Security, platform, and application owners should share responsibility for thresholds, logging, and response procedures so the failure cannot be dismissed as a pure user issue.

Technical breakdown

How resource-draining prompts exhaust LLM runtime capacity

Resource-draining prompts, sometimes called sponge attacks, are crafted to force a model to spend disproportionate compute on a single request. They work by stretching output length, increasing token generation, or driving the system into long decoding loops that consume context window, memory, and latency budget. The article's examples, such as asking for millions of words or endless repetition, illustrate a simple point: the attack does not need to break the model's logic to be effective. It only needs to make normal generation too expensive to sustain.

Practical implication: rate limits and output caps must be tuned to block expensive generations before they become an availability event.

Why length prediction matters in prompt security

The article's core research direction is to predict output length from the input prompt. That turns prompt security into a pre-execution classification problem: if a request is likely to create unusually large generations, the system can reject, truncate, or route it for review. The model design described in the post uses length bins rather than exact regression because exact token counts are unstable and hard to predict precisely. This is a useful architectural choice because security decisions usually need thresholding, not perfect numerical precision.

Practical implication: build prompt-risk scoring around coarse length bands and enforcement thresholds, not exact token prediction.

Why LLM denial of service is an identity governance problem

Once models are deployed as shared services, the prompt becomes an access event and the output becomes a resource entitlement. That means abuse is not limited to adversarial NLP research. It also affects production identity systems that expose LLMs through authenticated users, API keys, service accounts, and agent workflows. In that setting, costly prompts can become a way to consume downstream budget, degrade service to other identities, and mask suspicious activity inside legitimate access. The governance question is whether the platform can distinguish normal usage from resource abuse before impact occurs.

Practical implication: treat prompt-cost monitoring as part of access governance for any authenticated LLM endpoint.

Threat narrative

Attacker objective: The attacker wants to disrupt LLM availability and force resource exhaustion without needing to compromise the model itself.

1. Entry occurs through a legitimate LLM interface, where the attacker submits a prompt engineered for extreme output volume or repetitive generation.
2. Escalation happens when the model allocates excessive compute, memory, and context to the request, exhausting service capacity and delaying other users or jobs.
3. Impact is service degradation or denial of service, with the LLM becoming unreliable, slow, or unavailable for legitimate work.

Breaches seen in the wild

• Moltbook AI agent keys breach — Moltbook breach exposed 1.5M AI agent keys.
• AI LLM hijack breach — attackers used stolen AWS access keys to hijack Anthropic LLM models on Bedrock.

Read our 52 NHI Breaches Analysis report for a comprehensive view of breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Resource-draining prompts expose a cost-amplification attack surface, not just a content-safety issue. The article shows that an attacker can weaponise a model's willingness to generate long answers and turn ordinary prompting into service exhaustion. That shifts the governance problem from moderation to runtime cost control. Practitioners need to treat excessive generation as an abuse pattern that belongs in the security control plane.

Prompt length estimation is a practical control primitive because exact output prediction is not necessary for enforcement. The post's binning approach is the right architectural instinct: security controls usually need to know whether a request is likely to be cheap, expensive, or unacceptably expensive. This aligns better with operational decision-making than trying to forecast exact token counts. Practitioners should design enforcement around thresholds, not precision theatre.

LLM abuse becomes an identity problem the moment the model is reachable through authenticated workflows. When service accounts, API keys, or user sessions can trigger expensive generations, the abuse path is embedded inside legitimate access. That means access review alone will not reveal the abuse pattern, because the identity may be valid while the request is still harmful. Practitioners should assume that authorisation can be correct and the workload can still be exploitable.

Prompt-cost governance is becoming part of AI operational resilience. The article is effectively describing a new class of availability threat where the security objective is not data theft but consumption of compute, memory, and attention budget. That places the problem close to NIST CSF availability thinking and, where AI governance is formalised, to NIST AI Risk Management Framework controls for monitoring, performance, and misuse. Practitioners should fold prompt-abuse detection into resilience planning, not leave it in the ML research queue.

From our research:

• 98% of companies plan to deploy even more AI agents within the next 12 months, despite documented rogue behaviour in 80% of current deployments, according to AI Agents: The New Attack Surface report.
• 80% of organisations report their AI agents have already performed actions beyond their intended scope, including accessing unauthorised systems, inappropriately sharing sensitive data, and revealing access credentials.
• For a broader identity view, read OWASP Agentic AI Top 10 for the runtime controls that matter when AI systems start making tool-use decisions.

What this signals

Resource-amplification risk will become harder to separate from ordinary AI usage. As more teams embed LLMs into internal tools and customer workflows, prompt-cost anomalies will blend into normal demand unless they are measured alongside identity, workload, and service-level telemetry. The operational question is whether your platform can recognise an expensive request before it becomes a shared-service outage.

Prompt-cost governance belongs in the same programme as AI access governance. If an authenticated identity can trigger expensive generations, then identity assurance alone is not enough. Teams should connect request provenance, quota policy, and service resilience to the same control set, and align it with NIST Cybersecurity Framework 2.0 and the AI governance expectations in NIST AI Risk Management Framework.

Length-aware abuse detection is the named concept practitioners should adopt. It means scoring prompt cost before generation and using that score to decide whether a request is allowed to proceed, capped, or isolated. That shift matters because the threat is no longer only malicious content, but malicious consumption of runtime budget.

For practitioners

• Set output-length guardrails on every exposed LLM endpoint Define hard caps for token generation, response size, and repeated-output patterns before production exposure. Use different thresholds for chat, code, and retrieval workflows so legitimate long-form tasks are not treated the same as spam-like repetition.
• Score prompts for expected cost before execution Classify requests into low, medium, and high-cost bands using prompt length, repetition, and task type as signals. Route high-cost requests to stricter limits, human review, or deferred processing when the model is used in shared environments.
• Tie LLM abuse detection to authenticated identity context Log the user, service account, or API key behind each request so resource abuse can be traced to the identity that triggered it. This is essential when the same endpoint serves people, bots, and workflow automation.
• Measure availability impact as a security metric Track queue depth, latency spikes, token consumption, and rejected expensive prompts together so you can distinguish ordinary load from adversarial resource exhaustion. Use those signals to tune policy before the service becomes unstable.

Key takeaways

• Resource-draining prompts turn LLM availability into a security target by consuming compute, memory, and context through legitimate-looking requests.
• The article's research shows that length prediction and coarse cost bands are more useful for enforcement than exact output forecasting.
• Teams should govern prompt cost, identity context, and output caps together so expensive generations are stopped before they degrade shared services.

Key terms

• Resource-Draining Prompt: A resource-draining prompt is a request engineered to make an LLM spend far more compute than a normal user query. The goal is not necessarily to change the answer, but to exhaust output budget, memory, or latency headroom and reduce service availability for other users.
• Prompt-Cost Scoring: Prompt-cost scoring is the practice of estimating how expensive a model request will be before generation begins. In security operations, it is used to decide whether a prompt should be allowed, capped, queued, or reviewed when the request is likely to consume disproportionate runtime resources.
• LLM Availability Abuse: LLM availability abuse is a denial-of-service pattern that targets model uptime and responsiveness rather than data confidentiality. It can happen through legitimate interfaces when an attacker submits requests that are computationally expensive enough to slow or block other workload traffic.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity security are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are responsible for identity security strategy or NHI governance in your organisation, it is worth exploring.

This post draws on content published by Protect AI: The Cost of Being Wordy: Detecting Resource-Draining Prompts. Read the original.