LLM inference routing and KV cache locality: are your controls keeping up?

TL;DR: LLM inference is stateful, so round robin and sticky sessions waste GPU capacity because performance is determined by KV cache locality, prefix overlap, and cache-aware routing rather than generic request balancing, according to Equixly. That makes routing logic, cache indexing, and autoscaling part of the security and governance conversation whenever AI systems depend on shared compute and predictable access patterns.

NHIMG editorial — based on content published by Equixly: How we fight GPU scarcity without compromise

Questions worth separating out

Q: How should security teams govern shared LLM inference infrastructure?

A: Treat the serving layer as a controlled runtime, not a generic compute pool.

Q: Why do LLM workloads need different routing logic from web traffic?

A: Because each request can depend on cached state from earlier tokens, and losing that state forces expensive recomputation.

Q: What signals show that cache-aware routing is working?

A: Look for rising cache hit ratios, lower queue depth, improved p99 latency, and fewer full prefills on cold nodes.

Practitioner guidance

• Design routing around prefix reuse Index prompts at the token-block level so the router can choose nodes that already hold the relevant KV cache rather than sending every request to the next available GPU.
• Measure the right serving metrics Track cache hit ratio, queue depth, p99 latency, and cache occupancy together so scaling decisions reflect the real cost of recomputing prefixes.
• Treat tokenisation as part of the control plane Use the exact model tokenizer for cache matching and avoid raw-text comparisons, because whitespace and prefix boundary changes can break cache reuse.

What's in the full article

Equixly's full blog covers the implementation detail this post intentionally leaves for the source:

• The exact cache-aware routing proxy flow, including prefix indexing and block-level hash matching.
• The autoscaling loop and threshold logic used to expand or drain GPU capacity based on inference metrics.
• The practical lessons the team drew from comparing round robin, sticky sessions, and cache-aware routing in production.
• The architecture choices behind token-level routing and zero-copy request handling for LLM workloads.

👉 Read Equixly's analysis of cache-aware routing for LLM inference →

LLM inference routing and KV cache locality: are your controls keeping up?

Explore further

View Full Forum →  |  NHI Foundation Course →