Topic starter
11/11/2026 5:56 pm
TL;DR: Prompt caching reduced Neo's LLM spend by 59% and raised cache hit rate from 7% to 84%, according to ProjectDiscovery. Agentic security systems need stable prefixes and controlled dynamic context to stay economically viable at multi-step scale.
Editorial analysis by NHI Mgmt Group, based on content published by ProjectDiscovery: “How We Cut LLM Costs by 59% With Prompt Caching”.
By the numbers:
- ProjectDiscovery reported 59% overall cost savings after prompt caching.
- ProjectDiscovery reported a cache hit rate increase from 7% to 84%.
- ProjectDiscovery said a single complex task with Opus 4.5 could consume 60 million tokens.
Key questions
Q: Why does dynamic context reduce cache reuse in LLM agents?
A: Dynamic context reduces cache reuse because the prompt prefix changes often, so the model sees a new byte sequence instead of a repeated one.
Q: How can teams measure whether an agentic workflow is cache efficient?
A: Teams measure cache efficiency by tracking hit rate, prefix reuse length, cacheable token share, and the ratio of repeated prompts to unique prompts.
Q: How do you keep prompt caching effective in multi-step agent workflows?
A: Keep the stable prefix byte-identical across runs, isolate volatile context at the tail, and avoid mixing per-user state into the reusable portion of the prompt.
Practitioner guidance
- Separate stable and volatile prompt layers Keep system instructions, tool schemas, and shared policy text byte-stable, and move per-run context, timestamps, and user-specific state to the tail of the request.
- Standardise tool ordering across agent types Sort static tools ahead of dynamic ones so shared cache prefixes stay identical across requests and do not fragment when new subagents are added.
- Measure cache hit rate by workflow segment Track hit rates by task type, step count, and provider path so aggregate averages do not hide low-performing execution paths.
Bottom line: Agentic AI systems pay for prompt structure repeatedly, so caching decisions directly affect whether long workflows are economically sustainable.
Glossary terms: Prompt Caching, Tool Definitions, Agentic workflow, Runtime Context
Explore further
View Full Forum → | NHI Foundation Course → | Our Services → | Read the full analysis →
11/11/2026 6:06 pm
Prompt locality has become a governance issue, not just an optimisation trick. Agentic systems now carry long-lived system prompts, tool definitions, and runtime state through every step, which means prompt structure directly affects operational cost and execution consistency. The more autonomous the workflow, the more the organisation needs to control what stays stable, what moves, and what can invalidate reuse. Prompt locality is therefore a design control for scalable AI operations, not a developer convenience.
A few things that frame the scale:
- AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026.
A question worth separating out:
Q: What is the difference between prompt caching and automatic prompt caching?
A: Prompt caching is the general mechanism of reusing a previously processed prompt prefix to avoid recomputation. Automatic prompt caching is a platform behaviour that applies that mechanism without the developer explicitly managing cache entries, usually by detecting repeated prefixes and caching them for later requests.
👉 Read our full editorial: Prompt caching cuts agentic security LLM costs by 59% at Neo