Cache-aware LLM routing shows why inference needs stateful control

At a glance

What this is: Equixly argues that LLM inference should be routed as a stateful workload because KV cache locality, not generic request balancing, determines whether GPU work is reused or recomputed.

Why it matters: This matters because teams operating AI services need to treat inference routing, cache awareness, and autoscaling as part of the control plane, not just a performance tweak.

Context

LLM inference is not stateless web traffic. The performance of each request depends on whether the serving node already holds the relevant KV cache for the shared prefix, which means conventional load balancing can quietly turn into a capacity and latency problem.

That creates a governance issue for AI infrastructure teams because routing decisions now influence cost, service quality, and resource efficiency. When AI systems depend on shared prompts, reusable context, or multi-turn agent workflows, state handling becomes part of operational control rather than an implementation detail.

Key questions

Q: How should teams route LLM inference when cache locality affects latency?

A: Use cache-aware routing that scores nodes by prefix overlap and current cache occupancy, then prefer the node that can reuse the most KV cache state. This reduces redundant prefill, lowers latency, and improves GPU throughput. Treat routing as part of the serving control plane, not as generic traffic distribution.

Q: Why does sticky session routing still waste GPU capacity for LLMs?

A: Sticky sessions pin users to nodes, but LLM efficiency depends on whether the node holds the relevant tokenised prefix, not only the same user. That means one user can still miss cache reuse, and different users can miss a shared prompt opportunity. The result is uneven utilisation and unnecessary recomputation.

Q: What signs show that an inference fleet is routing requests poorly?

A: Look for high prefill frequency, low cache hit ratios, rising p99 latency, and nodes that are near full while other nodes remain underused. Those signals usually mean the fleet is ignoring prefix reuse or routing too coarsely. The fix is to align placement, observability, and scaling with cache topology.

Q: Should teams scale LLM infrastructure on CPU usage or inference metrics?

A: Teams should scale on inference metrics such as queue depth, cache hit ratio, latency, and cache utilisation. CPU and memory can miss the real bottleneck in LLM serving, which is often the cost of repeated prefill and cache churn. A workload-aware scaler produces better cost and service outcomes.

Technical breakdown

Why round-robin routing fails for LLM inference

Round-robin load balancing assumes each request is independent and interchangeable. In LLM serving, that assumption breaks because the expensive part of inference is often the prefill phase, where the model computes key and value tensors for the prompt and stores them in the KV cache on a specific GPU. If the next turn lands elsewhere, the new node must repeat that work. The result is not just higher latency, but wasted compute across the fleet. For shared prompts and repeated prefixes, generic distribution logic destroys locality and undermines throughput.

Practical implication: Route requests using cache locality, not only queue length or request count.

How KV cache locality changes GPU utilisation

The KV cache is the state that makes transformer inference efficient. It stores intermediate key and value tensors so later tokens can attend to previously processed context without recomputing it. Cache hits skip the most expensive work, while cache misses force full prefill. Because the cache is tied to the node that processed the prompt, locality becomes a fleet-level property rather than a model-only property. Session affinity helps only when the same user returns to the same node, but many workloads share prefixes across users and conversations. That is why prefix overlap is a better routing signal than identity-based stickiness.

Practical implication: Measure and optimise prefix overlap at routing time, not only user affinity.

Why token-aware routing is more precise than session affinity

Cache-aware routing works at the tokenized prefix level because the KV cache is keyed on token IDs, not raw text. Two prompts that look similar to a human can tokenize differently and diverge in cache reuse, while two different users may still share a large common prefix such as a system prompt or tool instructions. A routing proxy therefore needs a prefix index, block-level hashing, and a way to score contiguous overlap across nodes. This is a control-plane function, not a simple traffic rule, because it depends on model tokenizer behaviour, cache occupancy, and eviction risk.

Practical implication: Build routing around token blocks and model-specific cache state, not raw-text similarity.

NHI Mgmt Group analysis

Stateful inference is now a governance problem, not just a performance problem. Once routing choices determine whether compute is reused or wasted, cache locality becomes part of capacity control, service reliability, and cost governance. That is especially true when shared prompts and multi-turn workflows make the inference fleet behave like a stateful system. Practitioners should treat routing policy as an operational control with direct service impact.

Cache-aware routing is the more accurate control model for shared AI services. Session affinity maps users to nodes, but inference efficiency often depends on prefix reuse across different sessions and even different users. The practical governance question is no longer whether a user keeps the same backend, but whether the fleet can preserve and exploit reusable context. Teams running internal copilots or agentic workflows should align routing policy with model tokenisation and cache occupancy.

KV cache locality: the hidden control plane for LLM serving. This article makes clear that locality, eviction risk, and prefix overlap are the variables that decide whether the fleet behaves efficiently or burns compute on avoidable prefill. That changes how AI operations teams should think about observability, autoscaling, and workload placement. The practitioner conclusion is simple: if the cache is state, then routing is state governance.

Autoscaling for inference must follow inference metrics, not generic infrastructure metrics. Queue depth, cache hit ratio, p99 latency, and cache utilisation tell you more about actual service pressure than CPU or memory alone. In AI environments, resource management can fail quietly if the scaler reacts to the wrong signals. The practical conclusion is to base scaling on workload behaviour, not generic host utilisation.

Shared-prefix workloads expose the limits of user-centric design. A fleet that only understands users and sessions cannot see the reuse opportunities created by common prompts, shared tool definitions, or repeated agent context. That gap matters wherever AI systems are deployed as operational services rather than isolated demos. The practitioner conclusion is to design routing, observability, and scaling around the real unit of reuse: the tokenised prefix.

From our research library:

• AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026.
• 92% of organisations expose NHIs to third parties, raising concerns about supply chain security, according to the Ultimate Guide to NHIs.
• Read next: AI Agent Identity Security Buyer's Guide

What this signals

Cache locality is becoming the real efficiency boundary for production AI services. Inference fleets that ignore prefix reuse will keep paying for recomputation, even when they have enough raw GPU capacity. The operational question is no longer whether a model can run, but whether the serving layer can preserve state well enough to make that capacity useful.

Prefix-aware placement should now sit alongside observability and scaling in AI operations. When a request can reuse a warm KV cache, the service behaves differently from one that lands cold, and that difference shapes user experience as well as cost. For practitioners, the control objective is to move from request balancing to state-aware workload governance.

For practitioners

• Implement cache-aware routing Score candidate nodes by contiguous tokenised prefix overlap and send requests to the node that already holds the relevant KV cache whenever possible.
• Index cache state at token level Maintain a prefix index keyed on token block hashes, not raw text, so routing decisions reflect the actual cache layout on each inference node.
• Tune autoscaling to inference signals Scale the fleet using queue depth, cache hit ratio, p99 latency, and cache utilisation rather than generic CPU or memory thresholds.
• Design for shared prefixes Treat system prompts, tool instructions, and repeated context as reusable workload components that should influence placement and reuse decisions.

Key takeaways

• LLM serving behaves like a stateful system because the KV cache determines whether the next request reuses work or recomputes it.
• Routing policies that ignore tokenised prefix reuse can create avoidable prefill cost, uneven GPU utilisation, and slower responses.
• The practical control shift is from generic load balancing to cache-aware placement, stateful observability, and inference-aware autoscaling.

Key terms

• KV Cache: A KV cache stores key and value tensors produced during transformer attention so later tokens can reuse them instead of recomputing the entire prompt. In production serving, cache locality strongly affects latency, throughput, and GPU cost because the cache lives on the node that processed the earlier tokens.
• Prefill Phase: The prefill phase is the initial part of LLM inference where the model processes the full prompt and builds the internal state used for generation. It is the most expensive stage for long contexts, so avoiding unnecessary prefills is one of the main benefits of cache reuse.
• Prefix Overlap: Prefix overlap is the amount of shared tokenised context between an incoming request and the cached state already held on a serving node. It is a practical routing signal because it measures actual reuse potential, not just whether two prompts look similar in raw text.
• Session Affinity: Session affinity is the practice of keeping related requests on the same backend path so context is preserved across multi-step interactions. In agentic environments, it matters because tool calls often depend on prior state, but that state must still remain visible and auditable.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management. It helps practitioners connect identity control to the broader operational systems their programmes depend on.