What signs show that an inference fleet is routing requests poorly?

What poor routing looks like in practice

Inference fleets usually show routing trouble before the business notices a user-facing outage. The clearest signs are uneven request placement, repeated cache misses, and latency that rises even when aggregate capacity still looks available. That pattern means the router is not matching requests to the nodes that already hold the most useful prefix state.

When routing is healthy, similar prompts tend to cluster onto the same subset of nodes long enough for prefix reuse to pay off. When it is unhealthy, traffic spreads too broadly, cache warmth decays, and the fleet behaves as if every request were cold.

Good operators treat this as a placement problem, not only a scaling problem. A fleet can be large enough on paper and still perform poorly if the scheduler or router ignores cache locality, tenant affinity, or other state that makes one node materially better than another for the next request.

Which metrics expose the mismatch

The most useful indicators are the ones that connect request shape to node behaviour. High prefill frequency suggests the system keeps rebuilding context instead of reusing it, while low cache hit ratios show that prior work is not being captured by the current placement strategy. Rising p99 latency is often the downstream symptom that makes the issue visible to users.

Node utilisation also matters, but only when you read it in context. A fleet where some nodes are near full while others remain underused often has a routing skew, not a raw capacity shortage. That is especially telling when the busy nodes are also the ones holding the hottest cache and the underused nodes are sitting on reusable capacity that the router is not exploiting.

For practical diagnosis, compare latency and cache behaviour by node, by tenant, and by prompt pattern. A single fleet-wide average can hide the fact that one route class is being over-concentrated and another is being scattered too widely.

Why cache-aware placement changes the outcome

Routing decisions in inference are not just about load balancing. They shape how often the system can reuse prefixes, how much prefill work it repeats, and how quickly hot state becomes available on the next request. If placement is too coarse, the fleet may appear balanced while silently discarding the efficiency gains of locality.

Cache topology becomes important because not all nodes are equal for every request. A router that is aware of where prefixes already live can lower compute waste, improve tail latency, and reduce the chance that a few nodes absorb disproportionate heat. In mature deployments, that often means aligning the scheduler with observability signals and autoscaling logic rather than treating them as separate layers.

At a higher level, this is the difference between throughput-oriented balancing and state-aware balancing. The former evens out traffic; the latter preserves the value of accumulated context. For inference fleets, the second approach is usually the one that produces better tail performance and more predictable cost.

Risk and Threat Considerations

Poor routing is not just an efficiency issue. When cache locality is ignored, the fleet can burn compute on repeated prefill work, hide hot-spot pressure until p99 latency degrades, and create uneven saturation that makes autoscaling react too late.

Failure mechanism: The router spreads similar requests too broadly or pins them too coarsely, so useful prefix state is lost across nodes and the system repeatedly pays the cold-start cost for requests that should have been warm.

Impact: Tail latency rises, throughput falls, and operators can misread the problem as generic capacity pressure even though the real issue is poor placement against cache topology.

Practitioner Guidance

What to verify: Check whether cache hit ratio, prefill rate, and p99 latency move together by node group. If latency spikes while some nodes remain lightly loaded, suspect routing skew before adding more capacity.

Decision rule: If similar prompts are repeatedly landing on different nodes, prioritise cache-aware routing or affinity logic over broader scaling changes. If the fleet is already saturated everywhere, treat it as a capacity problem; if saturation is uneven, treat it as a placement problem first.

What practitioners underestimate: The control loop matters as much as the router itself. Observability must show where prefix reuse is succeeding, and scaling must preserve that locality instead of destroying it with overly coarse redistribution.

Practitioner takeaway: The best signal of bad inference routing is not just slow responses, it is slow responses with broken locality, because that tells you the fleet is wasting reusable work before it runs out of raw capacity.

Related resources from NHI Mgmt Group

• What signs show that network tool access is poorly governed?
• What signs show that inference server telemetry is missing the security layer?
• Why do access requests create risk when routing is too informal?
• What signs show that DNS hygiene has drifted out of control?