Prompt caching cuts agentic security LLM costs by 59% at Neo

At a glance

What this is: NHI Mgmt Group's analysis, drawing on ProjectDiscovery's blog post: Prompt caching materially lowered the cost of multi-step agentic security workflows and made repeated LLM-heavy execution economically viable.

Why it matters: IAM, NHI, and AI security teams need to understand how agent runtime design, not just model choice, changes operating cost, scale, and governance.

By the numbers:

• ProjectDiscovery reported 59% overall cost savings after prompt caching.
• ProjectDiscovery reported a cache hit rate increase from 7% to 84%.
• ProjectDiscovery said a single complex task with Opus 4.5 could consume 60 million tokens.

Context

Agentic systems reuse long prompts, tool definitions, and conversation state across many steps, so the cost problem is not model inference alone. Once a workflow chains dozens of calls together, every extra token in the stable prefix is paid for repeatedly unless the runtime can reuse it.

Prompt caching changes that economics by separating stable context from step-specific context. The governance issue is not only spend control but also whether runtime state, tool metadata, and per-user variables are structured in a way that preserves cache locality.

In AI platforms that include workload identity concerns, prompt layout, tool routing, and execution state become part of the control surface rather than mere implementation detail.

Key questions

Q: How do you keep prompt caching effective in multi-step agent workflows?

A: Keep the stable prefix byte-identical across runs, isolate volatile context at the tail, and avoid mixing per-user state into the reusable portion of the prompt. Cache locality fails when dynamic content sits inside the prefix, so the control problem is prompt composition rather than model behaviour.

Q: Why does dynamic context reduce cache reuse in LLM agents?

A: Dynamic context reduces cache reuse because the prompt prefix changes often, so the model sees a new byte sequence instead of a repeated one. In LLM agents, changing tool outputs, timestamps, retrieved documents, or conversation state breaks prefix stability and causes more cache misses. Stable system prompts and fixed tool schemas improve reuse.

Q: How can teams measure whether an agentic workflow is cache efficient?

A: Teams measure cache efficiency by tracking hit rate, prefix reuse length, cacheable token share, and the ratio of repeated prompts to unique prompts. In agentic workflows, compare runs with the same system prompt, tool schema, and retrieval pattern, then watch how often dynamic state, such as tool results or memory inserts, forces misses.

Q: What is the difference between prompt caching and automatic prompt caching?

A: Prompt caching is the general mechanism of reusing a previously processed prompt prefix to avoid recomputation. Automatic prompt caching is a platform behaviour that applies that mechanism without the developer explicitly managing cache entries, usually by detecting repeated prefixes and caching them for later requests.

Technical breakdown

Why prefix caching matters in multi-step agent workflows

Prompt caching works when the model can recognise an identical stable prefix and reuse it instead of reprocessing it on every call. In agentic systems, the stable prefix often includes system instructions, tool schemas, and shared context that can dwarf the user message itself. Because each step replays earlier context, cost grows with both depth and breadth of the workflow. That is why long-running agents become disproportionately expensive: the same prompt tokens are repeatedly sent, even when only the last block changes. See the AI Agent Memory Security Guide.

Practical implication: Treat stable prefix design as a cost control and runtime efficiency problem, not a cosmetic prompt-assembly choice.

How dynamic context breaks cache locality

Caching is prefix-based, so any change inside the cacheable region can invalidate everything that follows. If working memory, per-user variables, or runtime context sit in the middle of the prompt, they destroy reuse even when the rest of the stack is stable. The practical pattern is to isolate volatile blocks at the tail and keep the cached prefix byte-identical across runs. That is the difference between a cache that only looks efficient on paper and one that actually survives real agent behaviour.

Practical implication: Move volatile runtime state out of the stable prefix so a single changed field does not collapse reuse across the whole workflow.

Why tool ordering and provider routing affect cache performance

Tool definitions are part of the cacheable prompt surface, so their ordering and mutability matter. If dynamic tools are mixed with static ones, the cacheable chain fragments and the request stops sharing prefix state cleanly across users. Provider routing adds another layer: caches are provider-specific, so identical prompts sent to different back ends do not reuse the same cache pool. In practice, architecture decisions about tool sorting and execution routing can have as much effect on cost as model selection does.

Practical implication: Keep static tools together, route traffic consistently, and minimise provider switching on cache-sensitive paths.

NHI Mgmt Group analysis

Prompt locality has become a governance issue, not just an optimisation trick. Agentic systems now carry long-lived system prompts, tool definitions, and runtime state through every step, which means prompt structure directly affects operational cost and execution consistency. The more autonomous the workflow, the more the organisation needs to control what stays stable, what moves, and what can invalidate reuse. Prompt locality is therefore a design control for scalable AI operations, not a developer convenience.

Cache locality debt: when volatile state sits inside the reusable prefix, every added step taxes the entire workflow. That failure mode is easy to miss because the system still works, just at a much higher marginal cost. In security automation, that cost shape matters because the expensive runs are often the most valuable ones to repeat. Practitioners should design prompts so ephemeral context does not contaminate shared runtime structure.

AI infrastructure teams should treat prompt caching as part of the agent identity and workload boundary. Stable prompts, tool lists, and routing behaviour define what the platform presents to the model at runtime, which makes them part of the execution identity of the system. Where an AI workload relies on shared prefixes, the operational question becomes whether the system can preserve a consistent identity long enough to reuse it efficiently. That is a direct concern for AI infrastructure workload identity governance.

Economic viability is now a security-operations criterion for agentic tooling. If a workflow costs too much to repeat, it will not be used continuously, and security teams fall back to one-off assessments. Prompt caching changes that calculus by making high-step workloads affordable enough to run repeatedly. The practitioner conclusion is straightforward: scale economics now shape whether agentic security controls are actually operationalised.

From our research library:

• AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026.
• Read next: AI Infrastructure Workload Identity Guide

What this signals

Cache locality will become a design expectation for agentic platforms. As workflows grow from simple chat turns into long-running execution chains, the cost of misplaced dynamic context rises fast. Teams that cannot keep stable prefixes intact will see agentic security use cases priced out of continuous operation.

Runtime state handling now sits alongside model selection in AI governance decisions. Stable prompts, provider routing, and breakpoint strategy determine whether repeated execution is affordable enough to operationalise. In that sense, prompt caching is becoming a practical control for scaling AI security work, not just reducing cloud bills.

AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026. That pattern reinforces why prompt locality, secret containment, and runtime boundary design should be treated together rather than as separate hygiene tasks.

For practitioners

• Separate stable and volatile prompt layers Keep system instructions, tool schemas, and shared policy text byte-stable, and move per-run context, timestamps, and user-specific state to the tail of the request.
• Standardise tool ordering across agent types Sort static tools ahead of dynamic ones so shared cache prefixes stay identical across requests and do not fragment when new subagents are added.
• Measure cache hit rate by workflow segment Track hit rates by task type, step count, and provider path so aggregate averages do not hide low-performing execution paths.
• Limit provider switching on cache-sensitive paths Keep the same inference back end for repeated agent tasks unless an outage forces failover, because cache pools do not transfer across providers.

Key takeaways

• Agentic AI systems pay for prompt structure repeatedly, so caching decisions directly affect whether long workflows are economically sustainable.
• The biggest cache gains come from keeping stable prefixes intact and relocating volatile runtime state out of the reusable portion of the prompt.
• Prompt caching becomes more valuable as task complexity rises, which means the most expensive security workflows are often the ones that benefit most from it.

Related NHI Mgmt Group resources: AI Agent Memory Security Guide · AI Infrastructure Workload Identity Guide · LLM Provider API Key Security and LLMjacking Guide · AI Supply Chain Security and AI-BOM Guide

Key terms

• Prompt Caching: Prompt caching stores reusable parts of an AI prompt so they do not need to be resent or recomputed each time. In agentic AI and MCP-based workflows, this can reduce latency and cost, but it also creates governance risk if cached context contains sensitive data, stale instructions, or unauthorized access paths.
• Tool Definitions: Tool definitions are the structured descriptions of functions, actions, or external capabilities that an AI system can call. They usually include the tool name, parameters, expected input format, and sometimes output shape, and they help the model decide when and how to invoke a tool safely and consistently.
• Agentic workflow: An agentic workflow is a sequence of tasks executed by an AI agent with some level of tool access and decision authority. In security terms, the workflow matters because it can span multiple systems, identities, and permissions, which makes attribution and revocation harder than with ordinary automation.
• Runtime Context: Runtime context is the set of signals used to judge whether an AI agent's behaviour is appropriate while it is acting. It includes identity, data access, model behaviour, posture, and environment. In practice, it is the difference between checking permission and evaluating purpose.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, workload identity, and secrets management. It helps security practitioners connect identity controls to the operating model that AI and automation now depend on.