Weight Loading Latency: Where Your Model Startup Time Actually Goes

Startup time feels like a solved problem until you're staring at a 4-minute cold start on an A100 node and trying to explain to an on-call engineer why the autoscaler just made things worse. Profiling the loading path is one of those tasks that gets skipped in favor of steady-state throughput work. That is a mistake worth correcting.

Photo by Brett Sayles on Pexels.

Model weight loading decomposes into several distinct phases, and they do not all scale the same way with model size. Treating startup as a single opaque number makes it impossible to optimize. Treat each phase as a separate budget item.

The Loading Pipeline

graph TD
    A[Deserialize checkpoint] --> B(Shard weights across devices)
    B --> C[H2D transfer]
    C --> D(Cast and quantize)
    D --> E[CUDA graph capture]
    E --> F((Model ready))

Each stage has its own bottleneck surface. The wall-clock time you observe at startup is usually the sum of several stalls that happen to look like one.

Deserialization

If you're loading from a standard PyTorch .pt or .bin checkpoint, torch.load with the default pickle protocol is doing more work than you'd like. For a 70B model stored across 16 shards, deserializing metadata alone can take 20-30 seconds on a cold NFS mount. Switching to safetensors format reduces this substantially: the format encodes tensor metadata in a header that you can read without executing arbitrary Python, and memory-mapping the file sidesteps the need to materialize the full tensor buffer before you copy it to device.

Profiling tip: wrap each load_file() call with time.perf_counter() and log per-shard wall time. You will almost always find that the first shard on each node takes 3-5x longer due to filesystem cold cache. That asymmetry tells you the bottleneck is storage throughput, not CPU deserialization logic.

Host-to-Device Transfer

After the weights are on CPU, they need to move to GPU memory. This is where people underestimate pinned memory. Allocating tensors in pageable host memory forces the CUDA driver to stage each transfer through an internal pinned buffer. On a system with a PCIe 4.0 x16 link (theoretical 32 GB/s), you can easily lose 40% of that bandwidth to the staging overhead alone.

Allocate directly to pinned memory: tensor.pin_memory() before initiating the H2D copy. For large models loaded with multiple workers, use torch.multiprocessing with shared memory carefully; the default spawn context does not inherit pinned allocations cleanly.

For tensor-parallel deployments, the transfer pattern matters as much as the bandwidth. Loading full weights to rank 0 and then scattering to other ranks is common but slow. Loading each rank's shard directly from a pre-partitioned checkpoint directory eliminates the scatter entirely and lets all ranks transfer in parallel. Tools like Megatron-LM and vLLM both support pre-sharded checkpoint layouts; if yours doesn't, it is worth the one-time conversion cost.

CUDA Graph Capture

This is the phase that surprises engineers the most. After weights are resident and the model is assembled, frameworks like vLLM capture CUDA graphs for common batch sizes. Capture involves running the forward pass once per batch size under CUDA graph recording. For a model serving batch sizes 1 through 256, that is 256 recorded passes.

Capture time scales with the number of batch sizes you capture and the depth of the model. On a 34B parameter model with 48 layers, full graph capture can take 90 seconds. You have a few levers: reduce the set of captured batch sizes to powers of two only (1, 2, 4, 8, 16...), defer capture until the first request arrives for a given batch size, or disable CUDA graphs entirely and accept a small steady-state throughput penalty. Which tradeoff is right depends on your traffic profile. Predictable batch sizes favor eager capture; variable or bursty traffic favors lazy capture or no capture.

Measuring What You Have

The simplest instrumentation is a structured log emitted at each phase boundary with cumulative and delta wall time. Write it to your existing observability sink (Prometheus histogram, Grafana Loki, whatever you have) under a consistent metric name like model.startup.phase_seconds. That gives you a per-deploy breakdown you can compare across rollouts and instance types.

High startup latency is not just an annoyance. In autoscaled deployments, a 4-minute cold start means the autoscaler needs to fire 4 minutes before demand arrives to avoid a latency cliff. If you're flying blind on startup composition, your capacity planning is also blind. Profile the phases. Pay down the largest one first. The machine starts faster when you understand what it's actually doing.