Token Budget Exhaustion: What Happens When Your Context Window Fills Mid-Request

Most context-window problems show up in benchmarks, not pagers. You test with long prompts, watch perplexity hold steady, and ship. Then, three weeks into production, a user submits a document that pushes the session over the limit mid-generation. The model stops making sense. Nobody notices until a downstream consumer complains.

Photo by panumas nikhomkhai on Pexels.

Token budget exhaustion is one of those failure modes that looks different depending on where you're standing. From the model's perspective, the sequence simply gets truncated or the generation halts. From the serving layer's perspective, you might get a completed request with a 200 status and a malformed response. From the user's perspective, the output just ends awkwardly. No error. No signal. Just wrong.

Let's be precise about what "fills mid-request" actually means.

Two Distinct Failure Points

Context overflow can happen in two places: during prefill or during decode.

Prefill overflow occurs when the input token count already exceeds max_position_embeddings before a single output token is generated. Most serving systems handle this with a hard reject, returning a 400 or a server-side truncation depending on configuration. vLLM, for instance, will raise a ValueError at the engine level if the prompt exceeds the model's sequence length. TGI (Text Generation Inference) from Hugging Face has a --max-input-length flag that gates this at the API boundary. These are the easy cases. They're loud.

Decode overflow is the dangerous one. This happens when the prompt fits, but the combined prompt plus generated tokens exceeds the budget before generation finishes. The serving engine either hard-stops at max_new_tokens (truncating the output mid-sentence) or, if that parameter wasn't set conservatively, triggers an internal error only after consuming significant GPU time and KV cache space. You've already paid the cost. You just don't have a usable result.

graph TD
    A[Incoming Request] --> B{Prompt Tokens > max_position_embeddings?}
    B -- Yes --> C[/Prefill Overflow: Hard Reject/]
    B -- No --> D(Prefill Runs)
    D --> E{Prompt + Generated > Context Limit?}
    E -- Yes --> F[/Decode Overflow: Truncated or Error/]
    E -- No --> G[Complete Response]

What Actually Gets Dropped

When a system does truncate silently, where does it cut? Most inference engines truncate from the left by default: the oldest tokens go first, preserving the most recent context. For chat sessions this can be reasonable. For RAG pipelines with a system prompt that includes your retrieval results, left-truncation is catastrophic. Your retrieved facts disappear. The system prompt survives. The model answers confidently from prior weights, not from the document you just fetched.

Profile your truncation behavior explicitly. Query your serving layer with a crafted request that you know will overflow, then inspect which tokens remain. In vLLM's OpenAI-compatible API, you can compare usage.prompt_tokens in the response against your own tokenizer count. A mismatch tells you truncation happened. Log this metric in production; it surfaces problems that no P99 latency chart will catch.

Capacity Planning Around the Budget

KV cache sizing is the other side of this problem. Each active sequence occupies 2 * num_layers * num_heads * head_dim * seq_len * dtype_bytes of KV cache memory per token. At FP16 with a 70B-parameter model (80 layers, 64 heads, 128 head dim), that's roughly 2.6 MB per sequence per thousand tokens. A context window of 128K tokens holds about 333 MB of KV state for a single concurrent request.

This matters for scheduling. If your instance is configured with a KV cache that can hold, say, 40 concurrent sequences at 4K average length, a workload shift toward 32K sessions will reduce effective concurrency to around 5. Budget exhaustion at the request level turns into queue exhaustion at the system level. Watch both.

Set max_model_len in vLLM explicitly rather than letting it inherit from the model config. Pair that with a max_num_seqs guard and monitor gpu_cache_usage_perc from vLLM's /metrics endpoint. When that metric climbs above 90% consistently, you're scheduling on a thin margin.

What Good Handling Looks Like

Return structured errors when truncation occurs, not silent completions. If your serving layer doesn't support this natively, add a middleware layer that compares usage.prompt_tokens plus usage.completion_tokens against your configured limit and injects a truncation flag into the response metadata. Callers can then retry with a summarized prompt, route to a longer-context instance, or surface a meaningful error to the user.

For stateful sessions, track cumulative token counts server-side and warn callers before they hit the wall. Reserve a token margin (200 to 500 tokens is reasonable) for the model's closing turn so generation doesn't stop mid-clause.

The machine does not volunteer information about what it dropped. That's your job.