Omnissiah Systems

Blog

Field notes on operating the machine.

MLOpsdistributed trainingdebugging

Gradient Accumulation Gone Wrong: Debugging Silent Training Divergence

How misconfigured gradient accumulation silently corrupts large model training runs, and the specific checks you need to catch it before loss curves lie to you.

Magos Veridian
· · 4 min read
All Posts
mlopstraining-infrastructure

Checkpoint Rot: How Saved Model State Goes Stale and What to Do About It

Checkpoints feel like safety nets, but saved model state degrades in subtle ways. Here's how to detect checkpoint rot before it costs you a training run.

Magos Veridian
· · 5 min read
inferencegpu

The Liturgy of the Restart: What a GPU Node Reboot Actually Costs

A reboot is never free. Here is the real accounting of what it costs to bring a large-model inference node back into service.

Magos Veridian
· · 4 min read
inferencelatency

Reading the Pulse: Where Tail Latency Hides in Distributed Inference

Your p50 looks fine and your p99 is on fire. A field guide to the places tail latency hides in a large-model serving stack.

Magos Veridian
· · 4 min read
quantizationinference

Quantization as an Act of Humility

Quantization is the practice of admitting the machine does not need every bit you gave it. A working guide to giving them up safely.

Magos Veridian
· · 4 min read