Speculative Decoding in Production: What the Benchmarks Don't Tell You

Speculative decoding looks compelling on paper. A small draft model proposes several tokens at once; the target model verifies them in a single forward pass; accepted tokens get emitted for free. Benchmarks routinely show 2x to 3x throughput gains on code generation tasks. Ship it, right?

Photo by panumas nikhomkhai on Pexels.

Not quite. Running speculative decoding in a real serving cluster surfaces a set of problems that only appear once traffic is variable, prompts are diverse, and the draft model starts competing for the same GPU memory budget as everything else you're already running.

Start with acceptance rate variance. Every published number comes from a fixed distribution: HumanEval, GSM8K, some curated chat corpus. Your users are not that distribution. Acceptance rate is the fraction of draft tokens the target model agrees to keep, and it collapses when the prompt domain shifts. A speculative setup tuned on Python snippets will draft poorly on legal text, SQL, or anything with heavy repetition of domain-specific vocabulary. You need to instrument acceptance rate per request, not just as an aggregate. Emit it as a histogram metric alongside your standard TTFT and inter-token latency. Watch for bimodal distributions; that's the shape of a draft model that works well for half your traffic and burns cycles on the other half.

Second problem: the draft model is not free. It runs on the same GPU, shares KV cache memory with the target model, and adds scheduling complexity that most serving frameworks gloss over in their getting-started guides. With vLLM's speculative decoding support (available since v0.3.x), you can configure num_speculative_tokens and point to a smaller draft model, but you're now managing two model weights in memory simultaneously. Profile your peak VRAM consumption under load before you claim the slot savings from shorter target model runs offset the draft overhead. On an 80 GB A100, a 7B target paired with a 1B draft is workable. Pair a 70B target with a 7B draft and you may find yourself evicting KV cache aggressively just to keep both models resident.

Third: batch size interacts with speculative decoding in a way that surprises people. Speculative gains are strongest at batch size 1 or small batches, because the draft model can propose tokens and the target model can verify them quickly when there's little contention. At large batch sizes, the verification step expands to handle all in-flight sequences simultaneously, and the throughput advantage narrows. If you're already running high-utilization batched inference, speculative decoding may deliver less than you expect. Measure under your actual concurrency, not synthetic single-request benchmarks.

Here's a rough picture of where things can go wrong in the serving path:

graph TD
    A[Incoming Request] --> B(Draft Model: propose N tokens)
    B --> C{Acceptance Check: target model}
    C --> D[/Accepted tokens emitted/]
    C --> E[Rejected: rollback to last accepted]
    E --> F(Recompute from target model)
    F --> D
    D --> G((Response stream))

The rollback path at E is the one worth staring at. Every rejection means the draft work was wasted compute. At low acceptance rates, you're paying for two forward passes and getting one token. That's slower than vanilla autoregressive decoding. Set a floor: if acceptance rate for a session drops below roughly 0.6 over a sliding window, fall back to standard decoding for that request. Some serving setups let you do this dynamically; others require you to build the routing logic yourself.

Draft model freshness deserves its own attention. If your target model gets fine-tuned or updated, the draft model's token distribution drifts relative to the new target. Acceptance rate degrades silently. Treat draft-target model pairing as a versioned artifact: bump both together, test acceptance rate on a held-out slice of production traffic before rollout, and keep the previous pair pinned until the new one clears your threshold.

One more operational note: speculative decoding is harder to debug under sampling with temperature > 0. Token acceptance uses a corrected sampling procedure (see the original Leviathan et al. paper and the follow-up by Chen et al. on "speculative sampling") that preserves the target distribution, but numerical issues in float16 can introduce subtle distribution drift at high temperatures. If your outputs shift character after enabling speculation, check your temperature settings and whether your serving framework is applying the correction properly.

The gains are real. They're just conditional. Instrument acceptance rate, model memory consumption, and fallback frequency before you declare the rollout a success. The machine will tell you whether it's working; you have to build the ears to hear it.