Cache-Aware Routing

How cache-aware routing works

Cache-aware routing is a request-placement strategy for stateful inference services. Instead of treating every request as equivalent, the router tries to send each prompt to the node most likely to already hold the relevant prefix state, which reduces duplicated computation and improves throughput for repeated or overlapping inputs.

This matters because LLM serving is not always stateless. When prefix state is reused effectively, the system spends less time recomputing the same context and more time serving new work. The result is lower latency, better hardware efficiency, and less waste from cache misses that force a node to rebuild state from scratch.

Why prefix overlap changes routing decisions

The key idea is that token-level overlap can be operationally meaningful. A request that shares a large prefix with prior traffic is more valuable to route toward a node that has already processed that prefix than toward an otherwise identical node that has not.

That creates a different balancing problem from classic round-robin or pure least-load routing. The “best” node is not only the one with spare capacity, but also the one with the right cached context. In practice, cache-aware routing is most useful when workloads contain repeated instructions, templated prompts, multi-turn conversations, or other recurring prefixes that create reuse opportunities.

Where cache-aware routing adds efficiency

Cache-aware routing can raise effective throughput because it preserves locality. A well-placed request may avoid recomputing prompt prefixes, reduce tail latency for repeated patterns, and improve overall cluster utilisation by making cache hits more likely.

The trade-off is that locality and balance are now competing goals. A router that over-optimises for cache reuse can concentrate traffic on a smaller subset of nodes, while a router that spreads traffic too evenly can destroy reuse. The design challenge is to preserve enough balance for resilience while still capturing the performance gains of prefix affinity.

In distributed LLM serving, this is especially important when the inference path is stateful across requests. A cache-aware strategy can be the difference between a cluster that scales efficiently under repetitive demand and one that wastes capacity rebuilding the same context on multiple nodes.

Operational limits and when the approach breaks down

Cache-aware routing depends on accurate state visibility. If the router cannot see which node has which prefix state, or if cache metadata is stale, it may make poor placement decisions and lose the performance benefit it was meant to create.

It also works best when traffic has enough repetition to justify the extra routing intelligence. If prompts are highly diverse, short-lived, or unpredictable, the overhead of maintaining cache awareness may deliver only limited gain. In those environments, simpler routing may be easier to operate and may produce more stable balancing behaviour.

Risk and Threat Considerations

Cache-aware routing introduces a concentration risk because it steers related requests toward the same inference node or node set. That improves reuse, but it can also make cache locality, node health, and routing metadata part of the service’s security and resilience assumptions.

Failure mechanism: If routing state is inaccurate, stale, or manipulated, requests may land on the wrong node, lose cache benefit, or create uneven load that amplifies latency and failure cascades. In multi-tenant or shared environments, poor routing decisions can also undermine isolation assumptions when request affinity is not designed carefully.

Impact: The practical effect is degraded performance, reduced capacity, and potentially higher blast radius if one hot node becomes overloaded or unavailable. Where cached state contains sensitive session or prefix context, bad placement can also increase exposure to cross-request leakage concerns if isolation boundaries are weak.

Practitioner Guidance

What to watch for: Treat cache-aware routing as a control-plane decision, not just a performance tweak. The routing policy, cache metadata, and isolation model need to be observable enough that operators can tell whether locality gains are coming from valid reuse or from unhealthy concentration on a small number of nodes.

Governance implication: Define who owns routing policy changes, cache invalidation behaviour, and the conditions under which locality should be sacrificed for balance or resilience. In stateful inference platforms, those choices can materially affect both service quality and trust boundaries.

Related resources from NHI Mgmt Group

• What signals show that cache-aware routing is working?
• When should teams prefer cache-aware routing over simple session affinity?
• How should security teams implement cache-aware routing for repeated LLM prompts in multi-replica inference clusters?
• What is the difference between ingress routing and identity-aware access control?