Distributed Training Interview Questions

Not the data, not the weights: the gradients, once per step, in a ring that moves almost twice the model's size through every GPU. The derivation, the per-step byte count for an 8B model, and why it still hides behind the backward pass.

Three ways to split a training job, one table, and the rule that places each of them: activations on NVLink, gradients on the fabric, stage boundaries in between. With the byte counts that justify the placement.

The one utilization number that cannot be gamed by recompute: model FLOPs over hardware peak, derived from a step time in four lines, with the band frontier labs land in and an itemized list of where the other 60% goes.

Sixteen bytes per parameter is the bill for mixed-precision Adam. ZeRO pays it in three installments: optimizer state, then gradients, then the weights themselves. The per-rank bytes for a 70B at 8, 16, 32 and 64 GPUs, and what each stage adds to the wire.

Same algorithm, two implementations, one of which now lives inside PyTorch and composes with everything else there. The differences that decide it in 2026: per-parameter sharding, DTensor, torch.compile, and the two features DeepSpeed still owns outright.

Sixteen bytes per parameter says 1.6 TB of state against a 640 GB node, so the model spans nodes before the first token. The arithmetic that sizes the fleet, the layout that puts each axis on the right link, and the two numbers that decide FSDP against pipeline across nodes.

Two phases, n minus one steps each, one slice per step: the derivation that gives 2(n−1)/n times the buffer per rank, the argument that no algorithm can beat it on bytes, and the latency term that makes NCCL switch to trees at scale.

With p stages and m micro-batches the pipeline idles (p−1)/m of its compute time, and the obvious fix, more micro-batches, buys back throughput with activation memory. 1F1B caps that memory at p micro-batches instead of m. The derivation, the numbers for PP16, and the schedules that go further.

Four all-reduces per transformer block, on activations, on the critical path. Inside the NVLink domain they cost 8% of a step; across the NIC they cost more than the step itself. The derivation for a 70B, and the second reason TP stops that has nothing to do with the network.

Mathematically identical for a mean loss and a stateless model, and different in three ways that matter to infrastructure: activation memory, communication frequency, and any layer that looks at the batch. With the byte counts and the DDP call that makes it work.

Full recomputation costs a third more compute to cut activation memory by an order of magnitude. The per-layer arithmetic that says whether you need it at all, the selective variant that costs almost nothing, and the point at which a smaller micro-batch is the cheaper way out.

At 128k tokens one layer's activations are 73 GB per sequence and attention grows with the square of the length. The arithmetic that forces the sequence onto sixteen GPUs, what a ring of KV chunks costs per layer, why the communication hides, and where Ulysses and Megatron sequence parallelism fit around it.

Every MoE layer routes each token to eight experts that live on other GPUs, twice per layer, forward and back. The bytes per rank per layer for DeepSeek-V3, why that all-to-all runs close to one-to-one against compute, and why a single hot expert stalls every rank waiting at the combine.

A router that sends 90% of tokens to 26 of 256 experts makes those ranks do nine times their share while everyone else waits. The arithmetic of the imbalance, what a capacity factor drops on the floor, why the auxiliary loss fights the model, and the bias-only balancing DeepSeek-V3 used instead.

Tensor parallel 8, pipeline 16, data parallel 128: the layout Meta trained Llama 3 405B on, rebuilt from the memory floor, the compute budget and the bandwidth of each link. Why the axes go in that order, what each one costs per step, and the bubble that the schedule has to hide.

fp16 loses gradients below 6e-8 and overflows above 65,504; bf16 has fp32's range and three digits of precision. The arithmetic that shows why an update of 3e-5 vanishes into a bf16 weight, where each of the 16 bytes per parameter comes from, and what fp8 block scaling adds on top.

Compute the step time the formula predicts, measure the one you have, and the gap is the budget to explain. The isolation order with one metric per suspect: dataloader wait, SM and tensor-core activity, HBM throughput, time in collectives, and the per-rank spread that says it is one machine.

A loss that stops moving has five different causes and one of them is not a training problem at all. The checks in order: is the step counter moving, what are the learning rate and the grad norm doing, is the data repeating, has the loss scaler collapsed, and is one rank sending zeros.

Meta's Llama 3 report counted 419 unexpected interruptions in 54 days on 16,384 GPUs, one every three hours. The failure-rate arithmetic, the loss as a function of the interval, the square-root formula that minimizes it, and why the write must be sharded and asynchronous first.

A dead node at 4 a.m. costs twenty minutes if a human restarts it and under five if the system does. The four stages of an automatic recovery, the reason a lost node takes a whole pipeline replica with it, why hot spares beat resharding, and the minutes that each stage still costs even when everything works.

Every collective ends when the last rank arrives, so a single throttled GPU taxes 16,383 others. The arithmetic of that tax, the per-rank timing that finds the rank in one step, and the ranked list of causes from a hot GPU to a NIC with symbol errors, each with the command that confirms it.

The watchdog says an all-reduce ran for ten minutes and never finished. That means one rank never entered it, entered a different one, or died. How to read the log line, the flight recorder that names the rank in seconds, the env vars that make the next one debuggable, and why the timeout is a symptom every time.

Bucketed all-reduce in backward, FSDP's one-layer-ahead prefetch and split-GEMM tensor parallelism hide most of the bytes, and none of it works below a certain number of tokens per GPU. The ratio that decides it, the SMs the NCCL kernels take, and the tail that is never hidden.

On a TPU pod you write one program, annotate how each array is sharded across a named mesh, and the compiler inserts and schedules every collective. What that removes from the engineer's job, what the torus changes about layouts, where Pathways fits, and what a compiler-owned schedule costs.

LoRA shrinks the trainable state and leaves 141 GB of frozen weights untouched. QLoRA attacks the frozen half: NF4 takes it to 35 GB, double quantization compresses the scales themselves, paged optimizers absorb the spikes. The byte accounting, what dequantization costs, and where DoRA adds a term.

Generation is a decode workload and the update is a training workload, and they want different software on different GPUs. The per-token arithmetic for each side, the 141 GB broadcast that has to happen every step, the ratio of actor to learner GPUs that follows, and the idle time that on-policy training builds in.

DPO deletes the rollout engine, the reward service and the weight broadcast, turning post-training back into one supervised job: a policy, a frozen reference and a fixed dataset. The memory, the FLOPs per token, the precompute that removes the reference from the loop, and the exploration you give up.

Sixteen samples per prompt, a verifier that runs code, no value model, responses to 16k tokens: GRPO moves the cost from the learner to rollouts and rewards. The token arithmetic for one step, the tail that holds a batch for the longest sample, and the three places a fleet idles.

Three mantissa bits and a range of 448 make fp8 unusable with one scale per tensor. The arithmetic of what an outlier channel destroys, the 128-element block scaling and fp32 promotion that the DeepSeek-V3 report used, the ops that stayed in bf16, and what the 2× peak bought in practice.

Every rank writes its own shard, a metadata file maps every chunk to a global tensor, and a loader intersects the old layout with the new. The bytes per rank for a 405B, the manifest that makes the write atomic, the metadata that makes resharding possible, and the host-memory staging that keeps the GPUs running.

Two restarts a day at fifteen minutes each is the 2%. The loss decomposed into detection, rescheduling, reload and rewound work, the lever on each, the in-memory checkpoint that makes the interval a minute, the square-root pareto of interval against write cost, and the residual that only fewer failures can remove.

A 400B dense model on 10 trillion tokens is 2.4 × 10²⁵ FLOPs, 43 days of pure compute on 16,384 H100s, and about 380 interruptions along the way. The order in which to derive every number, the layout and the data rate, the checkpoint and failure budgets, and the schedule that survives its own arithmetic.