Notifications
Clear all
Topic starter
19/08/2026 3:42 pm
TL;DR: Prompt caching in LLMs reuses key and value tensors, not model outputs, which cuts latency and lowers token costs for long prompts, according to ngrok's explanation of OpenAI and Anthropic API behaviour. That makes caching a performance control with governance implications for AI systems, especially where prompt reuse, access patterns, and sensitive context need tighter oversight.
NHIMG editorial — based on content published by ngrok: an explainer on prompt caching, KV tensors, and LLM inference performance
By the numbers:
- 10x cheaper in dollars per token than regular, s per token than regular input tokens for both OpenAI and Anthropic APIs.
- Anthropic claim that prompt caching can reduce latency by up to 85% for long prompts.
Questions worth separating out
Q: How should security teams govern prompt caching in enterprise AI applications?
A: Treat prompt caching as a runtime governance issue, not just a performance setting.
Q: Why does KV caching matter for AI security and data risk?
A: KV caching matters because it reuses intermediate attention state derived from the prompt, which can keep sensitive context operationally relevant after the first request.
Q: What do teams get wrong about cached input tokens in LLMs?
A: Many teams assume cached input tokens mean the model is storing and replaying answers.
Practitioner guidance
- Define prompt sensitivity tiers Classify prompts by whether they may contain secrets, customer data, or internal-only context, and prohibit caching for the highest-sensitivity tiers where provider controls allow it.
- Review provider cache retention windows Document the stated cache lifetime for each model provider and compare it with your data handling policy, especially for regulated or confidential workflows.
- Separate reusable prefixes from sensitive payloads Design prompts so stable, reusable instructions are isolated from variable sensitive content, reducing the chance that private material is carried into cached state.
What's in the full article
ngrok's full article covers the architectural detail this post intentionally leaves at a governance level:
- Step-by-step walkthrough of tokenizer, embedding, attention, and feedforward stages in the LLM inference loop
- Matrix-level explanation of how key and value tensors are derived and reused across tokens
- Hands-on comparison of OpenAI and Anthropic caching behaviour, including cache hit patterns and latency effects
- Technical examples showing why temperature settings do not invalidate cached prompt reuse
👉 Read ngrok's explanation of KV caching in LLM inference →
KV caching in LLMs: are your AI controls keeping up?
Explore further
19/08/2026 4:21 pm
KV caching introduces an AI governance gap, not just a performance feature. The article shows that providers reuse intermediate attention state to reduce cost and latency, but that also means AI teams are operating with a runtime optimisation that persists beyond a single token generation step. In governance terms, this is a state-reuse problem that belongs in AI security reviews, not only engineering discussions. Practitioners should treat cache behaviour as a control consideration for sensitive prompts and regulated data.
A question worth separating out:
Q: How can organisations reduce risk when using long-context LLM APIs?
A: Separate durable instructions from sensitive content, limit what enters prompts, and set a clear rule for which workflows can tolerate provider-side reuse. Long-context systems need explicit governance because efficiency gains can hide retention and exposure assumptions that would otherwise be visible in a traditional application flow.
👉 Read our full editorial: KV caching in LLMs: what it means for AI security teams
