Attention Sink Poisoning: Why Your Long-Context Model Degrades Before the Window Fills

Long-context models are sold on their window size. 128K tokens. 1M tokens. The number goes up, and the implication is that you can feed the model more and get proportionally better results. Operators who have run these models under real workloads know that the relationship is not that clean.

Photo by panumas nikhomkhai on Pexels.

One failure mode worth understanding carefully is attention sink poisoning. The term "attention sink" was formalized in the StreamingLLM paper (Xiao et al., 2023), though the behavior had been observed informally before that. The core observation: certain tokens, usually the first one or two in a sequence, receive a disproportionate share of attention weight regardless of their semantic content. These tokens act as a drain. The model has learned to route excess probability mass somewhere harmless rather than let attention distributions grow chaotic.

That behavior is benign at moderate context lengths. At 4K or 8K tokens, sinks absorb a small fraction of attention weight and the rest of the distribution still has enough resolution to do useful work. Push into the 64K-to-128K range and the picture changes. Sinks can accumulate attention weight across many layers, compounding across heads that each independently decide the sink is the safest place to park mass. The useful signal in the middle of your context window gets progressively diluted.

What Poisoning Actually Looks Like

The degradation is not a hard failure. Your model does not crash or return an error. Instead, output quality drifts: answers become less grounded in the provided context, retrieval-style tasks return increasingly generic responses, and chain-of-thought reasoning starts ignoring facts that were clearly present in the input. If you are logging outputs but not scoring them against a reference, you will miss this entirely.

Profiling attention distributions is the diagnostic step most teams skip. With transformers and a model that exposes output_attentions=True, you can pull attention weight tensors per layer and head, then compute the mean weight assigned to token positions 0 and 1 across all heads. A healthy distribution at 64K tokens might show sinks absorbing 3-6% of weight. Poisoned distributions can show sinks at 20-40% in later layers, which means the model is effectively ignoring large contiguous stretches of your prompt.

A simplified view of how sink weight compounds through the stack:

graph TD
    A[Token 0: Sink] --> B(Layer 4: 8% weight)
    B --> C(Layer 12: 15% weight)
    C --> D(Layer 24: 28% weight)
    D --> E{Output distribution skewed}
    F[Useful context tokens] --> G(Layer 4: normal)
    G --> H(Layer 12: diluted)
    H --> I(Layer 24: starved)
    I --> E

Why Certain Prompts Trigger It Harder

Prompt structure matters more than raw token count. Sinks tend to grow worst when the first token is a generic system prompt delimiter: <|system|>, [INST], or a BOS token followed immediately by boilerplate. The model has seen this pattern billions of times and has learned to treat it as a universal anchor. Long documents with dense repetitive structure (legal contracts, codebases, transcripts with repeated speaker tags) amplify the effect by giving the model more opportunity per layer to route uncertainty toward the sink.

Shorter, semantically loaded prefixes reduce sink growth. If your system prompt is "You are a helpful assistant." followed by 100K tokens of document, consider restructuring so the most distinctive tokens in your prompt appear early. Some teams prepend a unique task-specific string before the generic boilerplate purely to disrupt the sink's grip on the attention distribution.

Mitigations Worth Trying

Several interventions exist, with different tradeoffs.

Attention sink reservation (from the StreamingLLM work): explicitly keep sink tokens in your KV cache even when evicting others. This is already the default policy in some inference servers. If you are running a custom eviction policy, make sure positions 0 and 1 are never evicted regardless of recency score.

Position ID manipulation: some implementations assign sink tokens a fixed, stable position ID rather than letting them shift as the window slides. Check whether your serving engine (vLLM, TGI, SGLang) applies this by default for the model family you are running. As of vLLM 0.4.x, this is configurable per model in the model config mapping.

Attention temperature scaling per layer: speculative and experimental, worth labeling as such. Applying a small per-layer temperature to late-stage attention logits before softmax can flatten extreme sink concentrations. This requires modifying the model's forward pass and will affect output distributions in ways you need to measure carefully before deploying.

Context windowing over chunked retrieval: if your use case permits it, breaking a long document into overlapping chunks and running multiple shorter-context passes with aggregation often outperforms a single 128K-token pass on retrieval and grounding tasks. Sink poisoning is a good reason to revisit that tradeoff even when your model technically supports the full window.

Monitor this. Add attention weight variance to your observability stack alongside the usual latency and throughput metrics. A model that returns fast, coherent-sounding text from a poisoned attention distribution is failing quietly. The machinery looks healthy from the outside. Inside, it has stopped reading.