Prompt Caching Security: Isolation and Side Channels

Prompt caching stores the model's internal state for a prompt prefix so the next request that starts the same way runs faster and cheaper, and anything shared that way can leak to whoever shares the cache. Every major model API now caches by default or on request, self-hosted servers such as vLLM and SGLang reuse KV cache blocks across requests, and many applications add their own response or semantic caches on top. Each layer has a different isolation boundary, retention period and failure mode. This guide explains the three layers, what each major provider documents about isolation and retention, the published attacks (cache timing audits, KV cache prompt reconstruction and semantic cache key collisions), and how to configure caching so cost savings do not cross a trust boundary. It is for AI platform engineers, application security teams and architects of multi-tenant AI products.

Key takeaways

The cache boundary is the account boundary you choose. Providers isolate caches per organisation, workspace or subscription; if all your tenants share one, the provider does not separate them.
Prefix reuse is observable. A cache hit returns the first token faster, and a 2025 Stanford audit detected cross-user cache sharing at seven API providers, including OpenAI.
Self-hosted servers share by default. vLLM's automatic prefix caching reuses KV blocks across all requests unless you set a per-request cache_salt.
Semantic caches trade integrity for hit rate. Similarity-matched keys can be deliberately collided, so one user's crafted query returns, or plants, another user's answer.
Retention is a data protection decision. Extended cache retention (up to 24 hours at OpenAI and Azure OpenAI) and zero data retention settings change what is held and for how long.

The three layers of caching

Layer	What is cached	Who controls isolation	Main risk
Provider prompt cache (OpenAI, Anthropic, Azure OpenAI, Amazon Bedrock, Gemini)	Key/value attention state for an identical prompt prefix, held on inference servers	The provider, scoped to your organisation, workspace or subscription; you choose what shares that scope	Cross-user or cross-tenant leakage inside your own account; retention beyond your policy
Self-hosted KV prefix cache (vLLM, SGLang and similar)	KV cache blocks keyed by a hash of the token prefix	You, through server flags and per-request salts	Timing side channels and prompt reconstruction between users of the same server
Application response or semantic cache (gateway or app code)	Full model responses, keyed by exact prompt or by embedding similarity	You, through the cache key design	Serving one user's answer to another; collision attacks that hijack responses

The first two layers cache computation: the model still generates a fresh answer, but the prefix work is reused. The third caches answers, which is a much larger exposure because the cached output itself is returned. Tenant scoping of application caches is covered in the Multitenant AI Application Security Guide; this guide covers how each layer works and how to test it.

What the providers document

Provider documentation was checked in November 2026. Details change with new model families, so confirm the current page before relying on a value.

Provider	Isolation boundary	Retention	Controls you have
OpenAI API	Caches are not shared across organisations and cannot be reused across regional processing boundaries	In-memory entries typically 5 to 10 minutes of inactivity, up to one hour; extended retention up to 24 hours. Organisations without Zero Data Retention default to the extended setting, ZDR organisations to in-memory	`prompt_cache_key` separates cache reuse between groups of requests; OpenAI notes that separate keys help prevent cache-hit probing across users. Retention set per request
Anthropic Claude API	Isolated between organisations, and between workspaces within an organisation on the Claude API, Claude Platform on AWS and Microsoft Foundry; Google Cloud and Bedrock use organisation-level isolation only	Default 5-minute lifetime, optional 1 hour; KV representations and hashes held in memory only, not stored at rest, deleted promptly after expiry	Explicit `cache_control` breakpoints (up to 4) or automatic caching; prompt caching is ZDR eligible
Azure OpenAI (Foundry)	Caches are not shared between Azure subscriptions	Typically cleared within 5 to 10 minutes of inactivity and always within an hour of last use; extended retention up to 24 hours	`prompt_cache_key`; on newer model families caching can be switched off per request with explicit mode and no breakpoints
Amazon Bedrock	The prompt caching page states entries live on the server that processed the request; for Claude models Anthropic states organisation-level isolation	5-minute TTL for many models, 1 hour on supported models	`cachePoint` checkpoints (up to 4 for listed models)
Google Gemini API	Not stated on the caching page	Not stated on the caching page	Implicit caching is enabled by default for Gemini 2.5 and newer models; explicit caching through generateContent

Three practical points follow. First, the isolation unit is a billing or administrative boundary, so map it to your trust boundaries deliberately: one workspace, project or subscription per customer, per environment or per data classification where the risk justifies it. That makes API key and project design part of cache security; see the OpenAI API Keys Security Guide and the Anthropic API Keys Security Guide. Second, where the provider offers a cache key parameter, set it from server-side tenant or user context, never from client input. Third, record which retention mode each workload uses in your data protection records.

Published attacks and what they show

Cache timing audits across API users

Gu, Li, Kuditipudi, Liang and Hashimoto (Stanford, "Auditing Prompt Caching in Language Model APIs", arXiv 2502.07776, 2025) built statistical audits that compare response latency for cached and uncached prompts. They detected global cache sharing across users in seven API providers, including OpenAI, which could leak information about other users' prompts, and found evidence that OpenAI's embedding model is a decoder-only Transformer. The lesson for defenders is that a cache hit is an observable signal: anyone who can send requests into the same cache scope can test whether a given prefix was recently seen. Network-level timing and token-length leaks are covered in the LLM Side-Channel Attacks Guide.

Prompt reconstruction from shared KV caches

Wu and colleagues ("I Know What You Asked: Prompt Leakage via KV-Cache Sharing in Multi-Tenant LLM Serving", NDSS 2025) describe PROMPTPEEK, which abuses KV cache sharing for identical token sequences in serving systems such as vLLM and SGLang to reconstruct other users' prompts in multi-tenant deployments. Unlike the audit above, the goal is to recover content, not only to detect a hit, and it applies to anyone running a shared inference server.

Semantic cache key collisions

Zhang and colleagues ("From Similarity to Vulnerability: Key Collision Attack on LLM Semantic Caching", arXiv 2601.23088, 2026) treat semantic cache keys as fuzzy hashes and argue that the locality needed for high hit rates conflicts with collision resistance. Their CacheAttack framework reported an 86% hit rate for response hijacking and showed it could induce malicious behaviour in an LLM agent, with a financial agent case study. This is an integrity problem: an attacker who can write to a shared semantic cache can influence what other users receive.

Attack class	Attacker position	Layer	Primary mitigation
Cache hit timing probe	Another user inside the same cache scope	Provider or self-hosted prefix cache	Separate scopes or cache keys per tenant or user; salts on self-hosted servers
KV cache prompt reconstruction	Another tenant on a shared inference server	Self-hosted prefix cache	Per-tenant `cache_salt` or dedicated instances; disable prefix caching for sensitive pools
Semantic cache collision and hijacking	Any user who can query the shared cache	Application semantic cache	Scope caches per tenant and permission set, raise similarity thresholds, do not cache agent tool decisions
Over-retention	Insider, provider or legal discovery	Provider cache	Shortest retention mode that meets latency goals; ZDR where required

Securing self-hosted prefix caching

vLLM's automatic prefix caching (enabled with enable_prefix_caching or --enable-prefix-caching) reuses KV cache blocks for any request that shares a prefix. Its design documentation describes the controls that matter:

Per-request cache_salt: the salt is injected into the hash of the first block, so only requests with the same salt reuse cached blocks. vLLM states this prevents timing-based attacks that infer cached content from latency. Derive the salt server-side from the tenant (or user, for sensitive data) and never accept it from the client.
Hash algorithm: since v0.11 the default is sha256, set with --prefix-caching-hash-algo. The xxhash options are faster but not cryptographic, and the documentation warns that collisions could leak private information in multi-tenant environments. Keep sha256 or sha256_cbor on shared servers.
Separate pools by sensitivity: where salting is not possible (for example, an older server or a gateway that cannot inject the field), run separate instances for different tenants or data classes, or disable prefix caching on the sensitive pool.
Distributed KV transfer: disaggregated serving moves KV cache between nodes, and that traffic is typically unauthenticated; keep it on an isolated network. See the vLLM, Triton and NVIDIA NIM Inference Server Security Guide.

Designing application caches safely

Key on identity and permission, not just text: include tenant ID and, where answers depend on access rights, user or role in every exact or semantic cache key. A RAG answer built from documents a user can see must not be served to a user who cannot. See the Permission-Aware RAG Guide.
Do not cache what should not be shared: exclude responses that contain personal data, secrets or tool results, and exclude agent planning steps and tool-call decisions, where a hijacked cache entry turns into an action.
Treat semantic similarity as an integrity risk: set conservative similarity thresholds, separate caches per tenant, and consider caching only curated, reviewed answers (for example, FAQ content) rather than every response.
Keep secrets out of prompts: OWASP LLM07:2025 System Prompt Leakage advises that system prompts should not contain credentials or connection strings. That matters more with caching, because a long static system prompt is exactly what gets cached and probed.
Order prompts for safety and cost: put static, non-sensitive content (instructions, tool definitions, public documents) first so it can be cached broadly, and put user or tenant data after it so it falls in a narrower scope.
Expire and invalidate: set TTLs on application caches and purge on permission changes, document deletion and data subject erasure requests.

Implementation approach

Inventory caching: list every model call path and record the provider, account or workspace, cache mode, retention, cache key settings and any gateway or application cache in front of it.
Map scopes to trust boundaries: decide where a shared cache is acceptable (one tenant, one environment, one data class) and restructure workspaces, projects, subscriptions or salts to match.
Set keys and salts server-side: add prompt_cache_key or cache_salt from authenticated context in the gateway, so application teams cannot forget it.
Choose retention: pick in-memory or short TTLs for regulated data; document any extended retention and its justification.
Test before rollout: in a staging environment, send the same long prefix from two test tenants and compare time to first token and reported cached tokens; a cached-token count on the second tenant's first request means scopes are shared.
Roll out by risk: start with multi-tenant and regulated workloads; keep a feature flag to disable an application cache or salting quickly, and watch latency and cost while you do.

Small versus large organisations: a single-tenant internal assistant on a provider API usually needs only sensible retention and secrets kept out of prompts. A SaaS product serving many customers, or a shared internal inference platform, needs per-tenant scopes or salts, a tested isolation check and the application cache controls above.

Detection and measurement

Providers report cache use in the response (for example cached_tokens in OpenAI usage details and usage.total_cached_tokens for Gemini), and AI gateways can log it per tenant. Use that telemetry for both cost and security.

Cross-scope cache hits: any cached tokens on a tenant's first request for a prefix in the isolation test; target zero.
Salt and key coverage: percentage of requests to shared inference or multi-tenant workloads carrying a server-set cache key or salt.
Retention compliance: percentage of workloads whose cache retention mode matches the data classification policy.
Semantic cache precision: rate of cache hits later flagged as wrong or mismatched by users or evaluation, per tenant.
Probing signals: clients sending many near-identical long prefixes with small variations, or unusual latency measurement patterns, from a single key.

Framework mapping

Framework	Reference	Relevance to caching
NIST SP 800-53 Rev 5	SC-4 Information in Shared System Resources	Preventing information transfer through shared caches
NIST SP 800-53 Rev 5	SC-31 Covert Channel Analysis; SC-39 Process Isolation; AC-4 Information Flow Enforcement	Timing channels, separation of tenants on shared servers, flow control by tenant
NIST CSF 2.0	PR.DS Data Security; GV.SC Cybersecurity Supply Chain Risk Management	Protecting prompt data; understanding provider caching and retention terms
OWASP Top 10 for LLM Applications 2025	LLM02:2025 Sensitive Information Disclosure; LLM07:2025 System Prompt Leakage; LLM08:2025 Vector and Embedding Weaknesses	Leakage through cached content, secrets in cached system prompts, embedding-keyed semantic caches

Common failure patterns

One workspace for every customer: provider isolation is per organisation or workspace, so a SaaS product that serves all tenants from one account shares caches across them.
Client-supplied cache keys: letting the front end send prompt_cache_key or cache_salt lets a user join another tenant's cache group.
Prefix caching on a shared server without salts: the default vLLM behaviour on a multi-tenant inference platform, exposed to the attacks above.
Semantic caches keyed on text only: a permission-sensitive answer is returned to a user with different access because the question looked similar.
Extended retention by default: organisations without ZDR on OpenAI default to extended retention, which may conflict with internal data handling rules if nobody checks.
Secrets in long system prompts: credentials embedded in the most cached part of every request.

Practitioner checklist

Inventory every provider, self-hosted and application cache in your AI call paths.
Map provider workspaces, projects or subscriptions to tenant and data classification boundaries.
Set provider cache keys from server-side tenant or user context in the gateway.
Enable per-tenant cache_salt on shared vLLM servers and keep a cryptographic hash algorithm.
Include tenant and permission context in every application cache key; exclude agent tool decisions.
Choose and record cache retention per workload; use in-memory or ZDR modes where required.
Remove credentials and connection strings from system prompts.
Run a two-tenant isolation test in staging after every caching change.
Monitor cached token telemetry per tenant and alert on cross-scope hits.

Prompt Caching Security Guide: isolation boundaries, side channels and safe configuration