Skip to content
From below of long thin blue cables connected to row of small white connectors on system block in data center

Omnissiah Systems

Field notes on operating the machine.

Recent Writing

distributed inferencetensor parallelismfault tolerance

Rank Assignment Under Fire: How Tensor Parallel Groups Break When Nodes Fail Mid-Run

When a node drops out of a tensor parallel group mid-inference, rank reassignment rarely works cleanly. Here's what actually breaks and how to recover.

· 5 min read

Optimizer State Explosion: Why Your Training Memory Budget Collapses at Scale

Token Budget Exhaustion: What Happens When Your Context Window Fills Mid-Request

NCCL Timeout Storms: Why One Slow Node Kills Your Entire Training Run

Attention Sink Poisoning: Why Your Long-Context Model Degrades Before the Window Fills

Thermal Throttling in GPU Clusters: How Heat Kills Your Throughput Silently
All posts →

About

Omnissiah Systems explores AI systems, MLOps, and distributed inference infrastructure, written as the disciplined study of large models as living machinery. New writing published regularly.