Routing Decay: How Request Dispatch Goes Wrong in Multi-Instance Inference

Routing looks solved until it isn't. You deploy four replicas of a 70B model, wire them behind a load balancer, and call it horizontal scaling. Latency holds. Throughput looks fine in the daily summary. Then, six weeks later, you notice p99 has crept up by 40% and one GPU node is running at 94% memory while another sits at 61%. The balancer is healthy. No alerts fired. What you have is routing decay.

Photo by panumas nikhomkhai on Pexels.

Routing decay isn't a crash. It's a slow drift in request distribution that your monitoring doesn't surface because every individual component reports green. Understanding it requires treating your routing layer as something that degrades over time, not a static config you set once and forget.

Why Distributions Drift

Stateless round-robin works when requests are interchangeable. Inference requests are not. A 512-token prompt with a 64-token completion and a 4096-token prompt with a 2048-token completion will land on the same replica if your router has no awareness of request shape. Over time, through random variance or workload shifts, heavier requests cluster on certain replicas. KV cache fills faster on those nodes. Prefill latency climbs. The balancer sees no error signal and keeps sending traffic at equal rates.

Session affinity makes this worse. If you route by user ID for cache warming, power users with long conversation histories get pinned to specific replicas indefinitely. A replica handling three active power users alongside normal traffic will diverge from its neighbors within hours.

Version skew is the third vector. Rolling deployments, A/B weight experiments, or a replica that restarted and loaded a different quantization config can leave your fleet in a mixed state. A router that treats all backends as equivalent will distribute load across backends that are not, in fact, equivalent.

What to Measure

You need per-replica observability, not fleet aggregates. Specifically:

• Active sequence count per replica: how many requests are currently in-flight, including those in the prefill queue
• KV cache occupancy per replica: what fraction of cache capacity is consumed, polled every 5-10 seconds
• Request token length distribution per replica: a histogram, not just a mean; you want to catch skew toward long-context traffic on specific nodes
• Replica queue depth: how many requests are waiting for a slot, distinct from active sequences

If your inference server is vLLM, these metrics are available via the /metrics Prometheus endpoint. Look for vllm:gpu_cache_usage_perc, vllm:num_requests_running, and vllm:num_requests_waiting. Aggregate them per replica and compare distributions. A coefficient of variation above 0.3 across replicas on KV cache occupancy is a signal worth investigating.

graph TD
    A[Incoming Request] --> B{Router}
    B --> C[Replica 1: 61% cache]
    B --> D[Replica 2: 94% cache]
    C --> E(Prefill + Decode)
    D --> F(Prefill + Decode)
    E --> G[Response]
    F --> G

The router in this picture has no visibility into the state of C or D. It sends equal traffic to both. Replica 2 degrades while Replica 1 has headroom.

Correcting the Drift

The first correction is load-aware routing. Instead of round-robin, route based on current queue depth or available cache capacity. Both vLLM and the NVIDIA Triton Inference Server expose the signals you need; the work is building or configuring a router that consumes them. Least-connections routing is a crude but immediate improvement over strict round-robin for workloads with variable request size.

For session-pinned traffic, add a periodic rebalance pass. Track cumulative token cost per pinned session and, when a replica's session load exceeds a threshold, offer that session a migration on its next request. The user sees no interruption; you regain headroom on the hot replica.

Version skew requires discipline in your deployment process. Before a rolling update completes, your router should be segment-aware: route to the new-version pool or the old-version pool, not randomly across both. Canary routing at the fleet level sidesteps the mixed-state problem entirely.

Finally, alert on replica divergence, not just fleet aggregates. A p95 KV cache occupancy delta of more than 20 percentage points between your hottest and coolest replica is worth a page. By the time fleet-level latency degrades visibly, the imbalance has already been compounding for hours.

Routing is infrastructure that ages. The distribution you deployed into is not the distribution you're serving six weeks later. Measure the replicas individually, expose the cache and queue state to your router, and treat rebalancing as a recurring operational task rather than a one-time configuration. The machine doesn't stay calibrated on its own.