Topic

memory management

2 posts tagged memory management from Omnissiah Systems.

Optimizer State Explosion: Why Your Training Memory Budget Collapses at Scale

AdamW optimizer states silently consume 2-3x your model's weight memory. Here's where that memory goes and how to reclaim it.

Activation recomputation trades GPU memory for extra compute during training. Here's when that tradeoff works, when it breaks, and how to tune it.

Browse more topics