Cache-aware LLM routing shows why inference needs stateful control

At a glance

What this is: This is an analysis of why LLM inference performance depends on cache-aware routing, because the KV cache makes each GPU node stateful and expensive to ignore.

Why it matters: It matters to IAM and security teams because shared AI infrastructure increasingly behaves like a governed runtime with state, access patterns, and operational boundaries that need control.

👉 Read Equixly's analysis of cache-aware routing for LLM inference

Context

GPU scarcity changes the operational model for AI systems because compute is no longer an elastic background resource. When inference becomes expensive and capacity-constrained, the control plane around model serving matters as much as the model itself, especially where requests share prefixes, tool context, or repeated prompts.

LLM inference is stateful because the KV cache stores intermediate attention results on the serving node. That creates an intersection with identity and governance when AI systems rely on shared infrastructure, delegated tool access, or agentic workflows, because poor routing can waste resources and break the assumptions behind consistent runtime control.

Key questions

Q: How should security teams govern shared LLM inference infrastructure?

A: Treat the serving layer as a controlled runtime, not a generic compute pool. Route requests using prefix overlap and cache locality, measure cache hit ratio and p99 latency, and set scaling policy around the actual behaviour of inference workloads. That approach reduces waste, improves consistency, and gives platform owners a defensible operating model for AI services.

Q: Why do LLM workloads need different routing logic from web traffic?

A: Because each request can depend on cached state from earlier tokens, and losing that state forces expensive recomputation. Web traffic assumes requests are interchangeable, but LLM inference depends on where the prefix cache lives. Routing that ignores this creates latency spikes, unnecessary GPU spend, and avoidable cold starts.

Q: What signals show that cache-aware routing is working?

A: Look for rising cache hit ratios, lower queue depth, improved p99 latency, and fewer full prefills on cold nodes. If utilisation looks high but latency and cost remain poor, the fleet may still be routing without respecting cache topology. Good routing should make repeated prefixes cheaper and more predictable over time.

Q: When should teams prefer cache-aware routing over simple session affinity?

A: Use cache-aware routing when workloads share prompts, tool schemas, or repeated prefixes across users or sessions. Session affinity only pins a user to a node, which can create hotspots and miss shared prefix reuse. Cache-aware routing is the better choice when the cost driver is cached state, not user stickiness.

Technical breakdown

Why round robin fails for LLM inference

Round robin load balancing assumes each request is independent, but LLM inference is not. A serving node that already holds the relevant KV cache can skip the expensive prefill phase, while another node must recompute the same prefix from scratch. That means generic distribution logic increases latency, burns GPU cycles, and lowers throughput even when the fleet looks balanced on paper. The problem is not just fairness across nodes, but whether the next request lands where useful state already exists.

Practical implication: inference routing must account for cache locality, not just request count or node utilisation.

How KV cache topology changes serving economics

The KV cache stores key and value tensors produced during transformer attention, and those tensors are reused for each new token in the same sequence. Prefill is expensive because it computes attention across the whole prompt, while decode is cheaper because it reuses cached state. In production, that makes prefix reuse a first-order optimisation variable. Tokenisation also matters, because caches are keyed on token IDs, not raw text, so near-identical prompts may diverge in cacheability if routing logic ignores the tokenizer.

Practical implication: cache-aware systems must route at token level and measure overlap against the model's actual tokenizer.

Why cache-aware routing and autoscaling belong together

A cache-aware proxy does more than improve hit rates. It can maintain a prefix index, score nodes by overlap, and fall back to least-loaded routing when no useful cache exists. When paired with autoscaling, the system can expand capacity based on queue depth, cache hit ratio, and p99 latency instead of generic CPU metrics. That is important because inference bottlenecks are often memory and state related, not just compute related. The result is a serving layer that behaves like a managed control plane for AI workloads.

Practical implication: teams should tune scaling around inference metrics and cache behaviour, not infrastructure metrics alone.

NHI Mgmt Group analysis

Cache locality is now a governance variable, not just a performance optimisation. When stateful inference nodes hold unique KV cache context, the difference between a warm and cold route changes cost, latency, and fleet efficiency. That makes serving topology part of operational governance for AI systems, especially where shared prompts, tool definitions, or agent workflows are reused. Practitioners should treat cache locality as a control surface, not an implementation detail.

Token-aware routing is the named control gap this article exposes. The failure is not simply poor balancing, but routing based on request identity rather than the actual cached state that determines inference cost. This is a subtle but important distinction for AI operations because the cache is built on tokenisation, not on human-readable text or session labels. Teams that ignore that gap will keep paying for redundant prefill work.

AI serving infrastructure is converging with identity-like governance patterns. Once routing depends on prefix state, node health, cache occupancy, and workload context, the inference layer behaves less like stateless web traffic and more like a controlled runtime with policy-driven decisions. That aligns with broader AI governance thinking in frameworks such as NIST AI RMF and OWASP Agentic AI Top 10, where runtime behaviour and control boundaries matter. Practitioners should expect AI platform governance to expand beyond model selection into serving policy.

Autoscaling decisions should be driven by service quality, not abstract utilisation. Queue depth, cache hit ratio, and p99 latency are closer to the operational truth of LLM serving than generic CPU or memory thresholds. The same fleet can look healthy while silently wasting money through repeated prefills. That means finance, platform, and security stakeholders all need the same telemetry view when AI workloads become shared enterprise services.

Cache-aware inference will reward organisations that design for reuse. Shared system prompts, repeated tool schemas, and stable prefix patterns create measurable efficiency gains when routing can exploit them. That does not just lower spend. It also creates a more predictable operating model for AI services that increasingly sit inside business-critical workflows. Practitioners should expect infrastructure choices to shape how safely and consistently AI can be governed at scale.

What this signals

Shared AI infrastructure is starting to inherit the same governance tension seen in identity systems: the control that feels simplest is rarely the one that preserves state efficiently. For teams building production LLM services, routing policy will become a measurable part of service assurance, not a purely platform concern.

Token-aware control plane: the practical shift is to govern inference around token-level reuse, cache occupancy, and latency budgets rather than around coarse node counts. That is where platform engineering, AI operations, and security governance begin to overlap in a way that will shape future service design.

As AI demand continues to rise, organisations will need clearer operational boundaries for when a workload should be scaled, drained, or pinned to preserve state. The teams that build that discipline early will avoid both cost leakage and brittle serving patterns.

For practitioners

• Design routing around prefix reuse Index prompts at the token-block level so the router can choose nodes that already hold the relevant KV cache rather than sending every request to the next available GPU.
• Measure the right serving metrics Track cache hit ratio, queue depth, p99 latency, and cache occupancy together so scaling decisions reflect the real cost of recomputing prefixes.
• Treat tokenisation as part of the control plane Use the exact model tokenizer for cache matching and avoid raw-text comparisons, because whitespace and prefix boundary changes can break cache reuse.
• Align autoscaling with inference demand Scale GPU fleets based on sustained inference pressure and cache behaviour, not generic host utilisation, so cold starts and wasted prefills do not dominate service cost.

Key takeaways

• LLM inference is stateful, so generic load balancing leaves performance on the table and raises compute cost.
• Cache-aware routing works because the KV cache, not the request count, determines whether a node can skip expensive prefill.
• Teams should govern AI serving with token-level routing, cache metrics, and SLO-aware autoscaling rather than host utilisation alone.

Key terms

• KV Cache: A KV cache stores key and value tensors produced during transformer attention so later tokens can reuse them instead of recomputing the entire prompt. In production serving, cache locality strongly affects latency, throughput, and GPU cost because the cache lives on the node that processed the earlier tokens.
• Cache-Aware Routing: Cache-aware routing is a load-balancing approach that sends requests to the inference node most likely to already hold the relevant prefix state. It improves efficiency by matching on token-level overlap rather than treating all requests as interchangeable, which is critical when LLM serving is stateful.
• Prefill Phase: The prefill phase is the initial part of LLM inference where the model processes the full prompt and builds the internal state used for generation. It is the most expensive stage for long contexts, so avoiding unnecessary prefills is one of the main benefits of cache reuse.
• Session Affinity: Session affinity is the practice of keeping related requests on the same backend path so context is preserved across multi-step interactions. In agentic environments, it matters because tool calls often depend on prior state, but that state must still remain visible and auditable.

What's in the full article

Equixly's full blog covers the implementation detail this post intentionally leaves for the source:

• The exact cache-aware routing proxy flow, including prefix indexing and block-level hash matching.
• The autoscaling loop and threshold logic used to expand or drain GPU capacity based on inference metrics.
• The practical lessons the team drew from comparing round robin, sticky sessions, and cache-aware routing in production.
• The architecture choices behind token-level routing and zero-copy request handling for LLM workloads.

👉 The full Equixly blog covers the routing proxy design, scaling loop, and inference lessons in more detail.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management. It helps security and platform practitioners connect identity controls to the broader runtime dependencies their programmes rely on.