10 posts tagged inference from Omnissiah Systems.
How request routing silently degrades across multi-instance inference deployments, the failure modes to recognize, and how to diagnose and correct them.
Speculative decoding promises faster token generation, but production deployments expose failure modes that synthetic benchmarks quietly hide. Here's what to watch.
How dynamic batching misconfigures itself over time, why throughput numbers lie on aging inference servers, and how to catch batch size drift before it costs you.
A practical breakdown of how FlashAttention allocates GPU memory during prefill, where the pressure points are, and how to diagnose OOM failures before they surprise you.
Long prompts silently inflate time-to-first-token in LLM serving. Here's how prefill cost accumulates and what you can do about it operationally.
A practical guide to KV cache eviction policies in distributed LLM inference: what triggers eviction, how it degrades latency, and how to tune against it.
A reboot is never free. Here is the real accounting of what it costs to bring a large-model inference node back into service.
Your p50 looks fine and your p99 is on fire. A field guide to the places tail latency hides in a large-model serving stack.
Quantization is the practice of admitting the machine does not need every bit you gave it. A working guide to giving them up safely.