Autoscaler Lag: Why Your GPU Fleet Always Reacts Too Late
A close look at why GPU autoscalers consistently scale too slowly for inference traffic, and how to budget for the lag instead of fighting it.
Magos Veridian
· · 4 min read Topic
MLOps
20 posts tagged MLOps from Omnissiah Systems.
More on MLOps
Dead Replica Detection: How Inference Fleets Carry Broken Workers Longer Than You Think
How unhealthy inference replicas stay in rotation longer than expected, why standard health checks miss them, and how to detect and drain them faster.
Magos Veridian
· · 5 min readSampling Parameter Drift: How Temperature and Top-P Shift Output Quality at Scale
How temperature, top-p, and other sampling parameters silently drift in production inference systems, and the concrete steps to detect and control them.
Magos Veridian
· · 5 min readRank Assignment Under Fire: How Tensor Parallel Groups Break When Nodes Fail Mid-Run
When a node drops out of a tensor parallel group mid-inference, rank reassignment rarely works cleanly. Here's what actually breaks and how to recover.
Magos Veridian
· · 5 min readOptimizer State Explosion: Why Your Training Memory Budget Collapses at Scale
AdamW optimizer states silently consume 2-3x your model's weight memory. Here's where that memory goes and how to reclaim it.
Magos Veridian
· · 5 min readToken Budget Exhaustion: What Happens When Your Context Window Fills Mid-Request
What actually happens when a live request hits your model's context limit, and how to detect, handle, and prevent silent truncation in production inference.
Magos Veridian
· · 4 min readNCCL Timeout Storms: Why One Slow Node Kills Your Entire Training Run
How a single degraded GPU node triggers cascading NCCL timeouts across your training cluster, and the steps to detect, isolate, and recover quickly.
Magos Veridian
· · 4 min readThermal Throttling in GPU Clusters: How Heat Kills Your Throughput Silently
How GPU thermal throttling silently degrades inference throughput in dense clusters, how to detect it, and what to do before it becomes a capacity problem.
Magos Veridian
· · 4 min readActivation Recomputation: When Saving Memory Costs You More Than You Think
Activation recomputation trades GPU memory for extra compute during training. Here's when that tradeoff works, when it breaks, and how to tune it.
Magos Veridian
· · 4 min readWeight Loading Latency: Where Your Model Startup Time Actually Goes
A practical breakdown of where time disappears during model weight loading at startup, and how to profile and reduce it in production inference systems.
Magos Veridian
· · 4 min readRouting Decay: How Request Dispatch Goes Wrong in Multi-Instance Inference
How request routing silently degrades across multi-instance inference deployments, the failure modes to recognize, and how to diagnose and correct them.
Magos Veridian
· · 4 min readSpeculative Decoding in Production: What the Benchmarks Don't Tell You
Speculative decoding promises faster token generation, but production deployments expose failure modes that synthetic benchmarks quietly hide. Here's what to watch.
Magos Veridian
· · 4 min readBatch Size Drift: Why Your Inference Throughput Degrades Overnight
How dynamic batching misconfigures itself over time, why throughput numbers lie on aging inference servers, and how to catch batch size drift before it costs you.
Magos Veridian
· · 4 min readFlash Attention Memory Footprint: What Your GPU Actually Allocates During Prefill
A practical breakdown of how FlashAttention allocates GPU memory during prefill, where the pressure points are, and how to diagnose OOM failures before they surprise you.
Magos Veridian
· · 4 min readPrefill Latency and the Cost of Long Prompts: Where Your TTFT Goes
Long prompts silently inflate time-to-first-token in LLM serving. Here's how prefill cost accumulates and what you can do about it operationally.
Magos Veridian
· · 4 min readPipeline Parallelism Stalls: Finding the Bubble and Paying It Down
How pipeline bubbles form in multi-stage model inference, how to measure them precisely, and what scheduling changes actually reduce their cost.
Magos Veridian
· · 4 min readTensor Parallelism Under Pressure: What Breaks When You Scale Width
A field guide to the failure modes that emerge when you scale tensor parallelism across GPU ranks: communication bottlenecks, numerical drift, and load imbalance.
Magos Veridian
· · 4 min readKV Cache Eviction: What Gets Dropped and Why It Costs You
A practical guide to KV cache eviction policies in distributed LLM inference: what triggers eviction, how it degrades latency, and how to tune against it.
Magos Veridian
· · 5 min readGradient Accumulation Gone Wrong: Debugging Silent Training Divergence
How misconfigured gradient accumulation silently corrupts large model training runs, and the specific checks you need to catch it before loss curves lie to you.
Magos Veridian
· · 4 min readCheckpoint Rot: How Saved Model State Goes Stale and What to Do About It
Checkpoints feel like safety nets, but saved model state degrades in subtle ways. Here's how to detect checkpoint rot before it costs you a training run.
Magos Veridian
· · 5 min read Browse more topics