Sampling Parameter Drift: How Temperature and Top-P Shift Output Quality at Scale

Sampling parameters are the quietest knobs on the inference stack. You set temperature and top-p once, during development, confirm the outputs feel right, and ship. Months later, something is subtly wrong with model outputs: too terse, too meandering, occasionally incoherent in ways that don't reproduce in your staging environment. Nine times out of ten, nobody has touched the model weights. What changed was the sampling configuration, and usually nobody noticed it happening.

Photo by panumas nikhomkhai on Pexels.

This is sampling parameter drift. It's worth studying carefully because it fails in slow motion.

How Parameters Reach the Sampler

In a typical inference serving stack (vLLM, TGI, Triton with a custom backend), sampling parameters travel from the client request through a request router, get merged with server-side defaults, and land at the sampler that operates on the logit tensor after the final projection layer. That merge step is where the first failure mode lives.

Most production systems maintain a layered config: operator defaults in a config file, model-card recommendations baked into a profile, and per-request overrides from the client. The merge order matters enormously. If a deployment upgrade silently changes the merge precedence, client-supplied temperature can stop being honored. The model continues serving requests and returning valid tokens. Nothing errors. Observability systems that only track latency and error rates miss this entirely.

graph TD
    A[Client Request] --> B{Router / Gateway}
    B --> C[Merge: Server Defaults]
    C --> D[Merge: Model Profile]
    D --> E[Merge: Client Overrides]
    E --> F[Sampler]
    F --> G[Token Output]

Each merge arrow is a point where the wrong value can silently win. Log the resolved parameter set at F, not the received set at A. The difference is the entire diagnostic surface.

The Temperature Collapse Pattern

Temperature values below roughly 0.3 produce near-greedy decoding. At 0.7 you get conversational fluency. Above 1.2, coherence degrades fast. These are empirical observations on most transformer families; your specific model will have its own curve.

What happens in practice: a deployment script contains a typo, or an A/B test framework applies a treatment group's config to 100% of traffic after the experiment ends. Temperature drops from 0.8 to 0.1. Outputs become clipped and repetitive. Users don't file bugs immediately because the answers are still factually correct. They just feel robotic. Customer satisfaction metrics dip two weeks later and the incident postmortem points at the model, not the sampler.

Detecting this requires actively logging the resolved temperature for every request. Compute a rolling percentile distribution hourly. A temperature P50 that moves more than 0.05 in either direction without a deliberate config change is a signal worth alerting on. The threshold is a judgment call; pick one and enforce it.

Top-P and Its Interaction with Vocabulary Size

Top-p (nucleus sampling) truncates the token probability distribution to the smallest set whose cumulative probability exceeds threshold p. At p=0.9 on a 32K vocabulary model, you're typically sampling from a few dozen tokens. On a 128K vocabulary model at the same p, that nucleus can expand dramatically depending on how flat the distribution is after the top candidates.

This creates a portability hazard when you update model weights or switch between model families without revisiting sampling config. A top_p=0.95 that worked well on Llama-3-8B may produce noticeably wilder outputs on a model with a larger vocabulary and different softmax temperature scaling during training. The right approach: profile the effective nucleus size (number of tokens in the nucleus) as a logged metric, not just top_p as a scalar. If effective nucleus size doubles after a model swap, you have a concrete, measurable signal that your sampling config needs to be recalibrated.

Min-P and Top-K: The Parameters Nobody Audits

Temperature and top-p get attention. Min-p and top-k are the parameters that accumulate neglected overrides. Top-k hard-caps the candidate set to k tokens regardless of probability mass distribution. Min-p filters any token below a minimum probability threshold relative to the top token.

In a mature inference service, these often get set once by whoever stood up the first deployment, then never revisited. When a new engineer adds per-request support for min-p without updating the merge logic, you can end up with min-p=0.05 applied on top of top-p=0.9, narrowing the nucleus further than either parameter would achieve alone. The interaction is multiplicative, and it's almost never documented.

Audit your sampler merge code quarterly. Print the full resolved parameter set in your integration tests. Make the effective sampling regime a first-class observable.

Operationalizing Sampling Observability

Concrete instrumentation steps, in priority order:

1. Log the resolved parameter vector for every request at the sampler boundary: temperature, top-p, top-k, min-p, repetition penalty, presence penalty, seed. Not what the client sent. What the sampler received.
2. Track effective nucleus size as a derived metric. This requires a small post-softmax hook in your serving code but reveals vocabulary-driven drift that raw top_p values hide.
3. Alert on parameter distribution shifts. A P5/P95 spread on temperature that changes by more than 10% in a six-hour window without a matching config deployment event is worth waking someone up for.
4. Version your sampling profiles the same way you version model weights. Tag each profile with the model checkpoint it was validated against. When you roll a new checkpoint, the sampling profile inherits a validation-required flag.
5. Canary sampling configs before full rollout. Run the new config on 5% of traffic for an hour and compare output length distributions, repetition rates, and any downstream quality signals you have. Rollback should take under two minutes.

Sampling parameters feel like tuning knobs. They are actually part of the model's operating envelope. Treat them with the same change-management rigor you'd apply to the weights themselves. The machine doesn't announce when its character changes. That's the engineer's job to notice.