Dead Replica Detection: How Inference Fleets Carry Broken Workers Longer Than You Think

Health checks lie. Not maliciously, but structurally: the check your load balancer runs every thirty seconds is asking a different question than "is this replica serving requests correctly right now?"

Photo by panumas nikhomkhai on Pexels.

In a GPU inference fleet, that gap kills p99 latency quietly and for a long time before anyone pages.

What a Dead Replica Actually Looks Like

Fully dead replicas are easy. The process crashes, the container exits, Kubernetes marks it NotReady, the load balancer drains it. Done in under a minute.

The hard case is the replica that is alive but broken. Consider what happens when:

• A CUDA context enters a partial error state after an OOM that the process caught but did not cleanly recover from
• A model shard fails to reload after a weight refresh, leaving the replica serving from an inconsistent parameter state
• Decode throughput degrades by 4-8x because the KV cache arena is fragmented past the point of useful operation, but the process never panics
• A single tensor parallel rank loses its peer connection and starts answering requests by reconstructing output from a partial activation

Every one of these replicas will pass an HTTP /health check that returns 200. The process is running. The port is open. The endpoint responds.

Why Standard Health Checks Miss This

HTTP liveness probes answer one question: is the HTTP server accepting connections? That is a necessary condition for serving, not a sufficient one.

Readiness probes are slightly better. They can run a short forward pass. But short forward passes on tiny synthetic inputs ("The sky is") will not surface KV cache fragmentation, thermal throttle-induced decode slowdowns, or partial TP rank failures. Those pathologies only appear under load, at sequence lengths and batch sizes resembling real traffic.

The second problem is polling frequency. A thirty-second probe interval means a broken replica can absorb production traffic for up to thirty seconds before the orchestrator notices anything. At even modest QPS, that is hundreds of degraded requests.

A Layered Detection Approach

Instead of relying on a single probe, treat dead replica detection as a signal fusion problem.

graph TD
    A[Incoming Request] --> B(Load Balancer)
    B --> C{Replica Score}
    C --> D[Healthy: Route]
    C --> E[Degraded: Deprioritize]
    C --> F[Dead: Drain]
    G[Sidecar Metrics Collector] --> C
    H[Synthetic Canary Prober] --> C
    I[Per-Replica P95 Tracker] --> C

Layer 1: Per-replica latency tracking at the load balancer. Most proxy layers (Envoy, Nginx, custom dispatch) can record per-upstream response times. Track p95 decode latency per replica over a rolling sixty-second window. A replica whose p95 exceeds 2.5x the fleet median is a candidate for investigation before the health check fires.

Layer 2: Sidecar metric exporter. Run a lightweight sidecar alongside each inference worker that scrapes GPU metrics directly via nvidia-ml-py (NVML) every five seconds: SM utilization, memory used, temperature, ECC error counts. A replica with zero SM utilization while holding open request slots is not computing. A replica with correctable ECC errors climbing faster than 50/second is about to fail harder.

Layer 3: Synthetic canary prober. Run a dedicated prober process that fires real-length requests (500-2000 tokens, drawn from a fixed test set) at each replica directly, bypassing the load balancer. Measure time-to-first-token and decode throughput. This prober runs every ten to fifteen seconds and writes results into the same scoring system the load balancer consults.

Combining these three signals gives you a replica health score that updates every five to fifteen seconds and reflects actual serving behavior, not just process liveness.

Draining Without Dropping

Once a replica is marked degraded, drain it gracefully. The sequence matters:

1. Remove the replica from the load balancer's active pool so no new requests route to it.
2. Wait for in-flight requests to complete or timeout (set a hard ceiling: thirty seconds is reasonable for most workloads).
3. Emit a structured log event with the replica ID, the triggering signal, and the current metric values. This is your audit trail for the postmortem.
4. Restart the container or pod. Do not attempt in-place recovery of a CUDA context in an unknown state.
5. Hold the replica in a warm-up period after restart: run the synthetic canary prober against it for two consecutive passing cycles before re-adding it to the pool.

Skipping step five is common and painful. A replica that just restarted is cold: KV cache empty, CUDA kernels not yet compiled or cached, NCCL communicators not warmed. Its first few requests under production load will be slow. Re-adding it immediately depresses fleet p95 in a way that looks like the drain event caused the latency spike, which confuses on-call engineers and discourages future draining.

Calibrating the Thresholds

The 2.5x fleet-median threshold for p95 latency is a starting point, not a law. Fleets with highly variable request lengths will have naturally wider latency distributions; you may need to track per-bucket latency (short, medium, long prompt length bins) and score replicas within each bucket separately.

For ECC error rate thresholds: correctable errors at low rates are normal on A100 and H100 hardware under heavy load. The signal is the rate of change, not the absolute count. A replica that accumulates 200 correctable errors over four hours is fine. One that accumulates 200 in two minutes is degrading toward an uncorrectable fault.

Speculation: as inference moves toward longer context windows and more aggressive KV cache management, the failure modes for "alive but broken" replicas will multiply. Partial eviction failures, cache corruption from preemption, and rank desynchronization in speculative decoding pipelines are all surfaces that existing health check regimes will not cover. The time to instrument for them is before those workloads hit production, not after the first silent degradation event at 3am.

Profile each replica independently. Trust latency more than liveness. Drain early and warm up completely before re-admission. The fleet is only as healthy as its least observed worker.