Size the storage for a 2,048-GPU training cluster. What numbers actually drive it?

Capacity is the easy half and almost never the constraint. The number that sizes the system is the checkpoint write burst, which is thousands of times the steady dataset read, and the tier that serves each is different. The full derivation, and the tiering that avoids paying burst prices for archive bytes.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

more free answers with an account · no card

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Advanced

🔌 Networking & Storage🔒 Premium

Checkpoint I/OA checkpoint of a 70B model's training state is 1.13 TB, and a 405B model's is 6.5 TB. Written synchronously every half hour, it stalls thousands of GPUs for as long as the slowest rank takes to reach the disk. The design that scales writes each rank's shard in parallel, copies it off the GPU to host memory first so the run resumes in seconds, and drains it to durable storage in the background. The arithmetic is bytes per rank against the bandwidth of each hop, and the goal is a checkpoint that costs the run under 1% of its time.

Foundational

🧮 Open Weights & Serving Engines

Capacity Planning for Open-Weights FleetsPlanning a fleet for a sparse open-weights model works differently from planning one for a dense model, because memory follows total parameters and throughput follows active parameters, and those now differ by more than twenty times. The sizing goes in one direction only: from a traffic forecast to tokens per second, to replicas at a measured operating point, to GPUs, to racks and kilowatts. Doing it in the other direction, from an available GPU count, produces a fleet that fits the hardware rather than the demand.

Foundational

🖧 Hardware & Cluster Build-Out

The Bill of Materials for a Training ClusterA GPU cluster is not a pile of GPUs. A 512-GPU scalable unit built to NVIDIA's DGX SuperPOD B300 reference architecture needs 64 nodes, four separate networks, thousands of transceivers, storage that can absorb a checkpoint burst, a management plane, racks, power distribution and cooling equipment. Writing the list out in order is how a design becomes a purchase order, and the items people forget are the ones that hold up a deployment for weeks.

Foundational

🧮 Open Weights & Serving Engines

Reading config.json to Size a Model You Have Never RunEvery Hugging Face model ships a config.json, and it contains enough to compute the weight footprint, the KV cache per token, the parallel degrees that divide cleanly and the minimum GPU count, before downloading a byte. Doing that derivation is a standard whiteboard exercise in serving interviews because it is exactly what an engineer does on the morning a new model lands, and the fields that matter are the same across every recent architecture.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackDesign the out-of-band management network for a 512-GPU cluster. What connects to it?Next in this trackAir, rear-door heat exchanger or direct-to-chip? Decide for a 60 kW rack.Next in this trackBuild the schedule for a 2,048-GPU deployment. What is on the critical path?

FEDITOR'S NOTE

Scored on sizing from the checkpoint burst rather than dataset capacity, on separating burst bandwidth from steady read, and on tiering so archive capacity is not bought at burst prices.

DISCUSSION · 0

No comments yet — be the first to share your approach.