Topic starter
19/08/2026 3:42 pm
TL;DR: Prompt caching in LLMs reuses key and value tensors, not model outputs, which cuts latency and lowers token costs for long prompts, according to ngrok's explanation of OpenAI and Anthropic API behaviour. That makes caching a performance control with governance implications for AI systems, especially where prompt reuse, access patterns, and sensitive context need tighter oversight.
Editorial analysis by NHI Mgmt Group, based on content published by ngrok: “Prompt caching: 10x cheaper LLM tokens, but how?”.
By the numbers:
- Cached input tokens are 10x cheaper in dollars per token than regular input tokens for both OpenAI and Anthropic’s APIs.
- Anthropic claim that prompt caching can reduce latency by up to 85% for long prompts.
Key questions
Q: How should security teams govern prompt caching in LLM applications?
A: Treat prompt caching as part of the AI control surface, not just a cost-saving optimisation.
Q: Why can cached prompts create governance risk even when outputs are not reused?
A: Because the reused object is internal model state, not the answer text.
Q: How do you know if an LLM cache policy is actually working?
A: Look for predictable cache-hit behaviour, clear retention windows, and consistent handling of sensitive prefixes across identical workflows.
Practitioner guidance
- Map cache boundaries to sensitive prompt classes Classify which prompt prefixes may be reused, how long they remain eligible for cache hits, and which workloads must never share cached context.
- Document provider cache behaviour in AI risk reviews Record whether caching is automatic or caller-controlled, whether partial-prefix reuse occurs, and how predictable cache hits need to be for each workload.
- Align prompt handling with short-lived context Treat prompts containing regulated, confidential, or policy-bound material as ephemeral operational context and define when that context expires for governance purposes.
Bottom line: KV caching improves LLM performance by reusing internal attention state, but it also widens the governance questions around prompt scope and context persistence.
Explore further
View Full Forum → | NHI Foundation Course → | Our Services → | Read the full analysis →
19/08/2026 4:21 pm
KV caching is a governance issue as much as a performance feature. Once internal model state can be reused across requests, AI teams are no longer dealing with a single prompt-response event. They are managing a short-lived but meaningful inference context that may outlast the user interaction that created it. For practitioners, that means cache scope, visibility, and expiry deserve the same attention as prompt logging and API permissions.
A few things that frame the scale:
- AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026.
A question worth separating out:
Q: What is the difference between prompt caching and response caching in LLMs?
A: Prompt caching reuses internal computation from the input side, especially the key and value tensors used by attention. Response caching would store a finished answer and serve it again. The security and governance implications differ because prompt caching changes inference efficiency, while response caching changes what content is replayed.
👉 Read our full editorial: KV caching in LLMs: what it means for AI security teams