fault tolerance

Rank Assignment Under Fire: How Tensor Parallel Groups Break When Nodes Fail Mid-Run

When a node drops out of a tensor parallel group mid-inference, rank reassignment rarely works cleanly. Here's what actually breaks and how to recover.