KV Cache Eviction: What Gets Dropped and Why It Costs You
A practical guide to KV cache eviction policies in distributed LLM inference: what triggers eviction, how it degrades latency, and how to tune against it.
Magos Veridian
· · 5 min read Topic
inference
4 posts tagged inference from Omnissiah Systems.
More on inference
The Liturgy of the Restart: What a GPU Node Reboot Actually Costs
A reboot is never free. Here is the real accounting of what it costs to bring a large-model inference node back into service.
Magos Veridian
· · 4 min readReading the Pulse: Where Tail Latency Hides in Distributed Inference
Your p50 looks fine and your p99 is on fire. A field guide to the places tail latency hides in a large-model serving stack.
Magos Veridian
· · 4 min readQuantization as an Act of Humility
Quantization is the practice of admitting the machine does not need every bit you gave it. A working guide to giving them up safely.
Magos Veridian
· · 4 min read Browse more topics