Gradient accumulation versus a bigger per-GPU batch: same result or not, and what changes underneath?

TL;DR: For a loss that averages over tokens, running k micro-batches of size b and summing their gradients gives the same gradient as one batch of size kb, so the optimizer sees identical updates. What differs is underneath: accumulation stores activations for one micro-batch at a time (k times less memory), synchronizes gradients once per k backward passes (k times less communication per token in DDP, if no_sync() is used), and any layer that computes statistics across the batch, such as BatchNorm, sees a batch of b rather than kb. Transformers use LayerNorm and RMSNorm, so the last point rarely bites them.

How to approach it

State the equivalence with a one-line derivation so the interviewer knows you are not guessing. Then list what is different, each with a number: memory, communication, and batch-dependent layers. Say how to make the communication saving actually happen in DDP, because the default does not. Close with when to prefer each and the constraint that decides it.

A strong answer

A typical situation: a 70B fine-tuning run needs a global batch of 1 million tokens for stability, the team has 64 GPUs, and 16k tokens per GPU per step does not fit in memory. Accumulation is the tool, and the question is what it costs.

Equivalence. Let the loss over a batch B be the mean of per-token losses. Split B into k micro-batches of equal size. The gradient of the mean over B equals the mean of the k micro-batch gradients:

L(B) = (1/|B|) Σ_{x∈B} ℓ(x)
     = (1/k) Σ_{i=1..k} [ (1/b) Σ_{x∈B_i} ℓ(x) ]  = (1/k) Σ_i L(B_i)

∇L(B) = (1/k) Σ_i ∇L(B_i)

so: run k backward passes, accumulate ∇L(B_i)/k into the .grad buffers, step once.
sanity: with k = 1 this is the ordinary step; the optimizer cannot tell the difference
        because it only ever sees the summed .grad tensors.

The equivalence requires that the model's forward pass for one sample does not depend on the other samples in the micro-batch, and that the loss is a mean (a sum needs no 1/k). Both hold for transformer language models with token-mean loss, with the caveat that if micro-batches have different token counts the weighting has to be by tokens, not by micro-batch, or the "mean" is biased toward short micro-batches.

Memory. Activations are the term that scales with tokens per forward pass. For a 70B at hidden 8,192 with TP8 and no checkpointing, per-layer activations are roughly (10 + 24/t) × tokens × hidden bytes, about 13 × 8,192 × 8,192 = 0.87 GB per layer per 8k tokens, so 70 GB for 80 layers. A 16k micro-batch would need 140 GB and does not fit; two 8k micro-batches accumulated need 70 GB and do. Gradient and optimizer memory are unchanged, because the accumulated gradient buffer is the same size regardless of k.

Communication. Under Data Parallelism and DDP the gradient all-reduce fires during each backward pass by default. With k = 2 and no changes, the run all-reduces twice per optimizer step, doubling the bytes per step for no benefit. Wrapping the first k−1 backward passes in model.no_sync() suppresses the hooks so the reduction happens only on the last one:

per optimizer step, 70B gradients in bf16 (141 GB), sharded over TP8 → 17.6 GB per rank
  ring all-reduce over 8 DP nodes: 2 × 7/8 × 17.6 = 31 GB per rank per reduction
  at 50 GB/s: 0.62 s
without no_sync, k = 4:  4 × 0.62 = 2.5 s of communication per step
with no_sync,    k = 4:  1 × 0.62 = 0.62 s per step
compute per step at k = 4, 8k tokens each: 4 × 1.1 s = 4.4 s
sanity: the naive version exposes about 40% of the compute time in communication that
        overlap can only partly hide; the no_sync version is 14% and hides well.

This is also why accumulation improves the communication-to-compute ratio: k backward passes of compute against one reduction. A run that is communication-bound at k = 1 becomes compute-bound at k = 4 with no hardware change, which is one reason accumulation is used even when memory would allow the larger micro-batch. Under FSDP the picture differs, because the weight all-gathers happen per forward pass regardless; only the reduce-scatter is saved, so the benefit is about one third of DDP's.

Batch-dependent layers. BatchNorm computes mean and variance over the micro-batch it sees, so accumulation with k micro-batches of b gives statistics of b samples, not kb, and the running estimates differ. Vision models care; SyncBatchNorm fixes the cross-GPU part but not the cross-micro-batch part. Transformers normalize per token (LayerNorm, RMSNorm) and are unaffected. Dropout draws independent masks per micro-batch, which is fine. Anything that samples negatives within the batch (contrastive losses) also sees the smaller batch and is not equivalent.

Decision: at fixed global batch, use the largest micro-batch that fits with headroom, then accumulate to reach the global batch; increase k rather than shrinking the global batch when memory is short. The condition that reverses toward the bigger micro-batch is kernel efficiency: a micro-batch small enough that GEMMs run below peak (a few hundred tokens per GPU) wastes compute at every step, and at that point activation checkpointing to enable a larger micro-batch is a better trade than a larger k.

The reversal condition: a layer that looks across the batch, such as BatchNorm, where accumulation and a larger batch stop being equivalent and the substitution is a silent correctness bug rather than a performance trade. Data Parallelism and DDP is where the gradient averaging this rests on is defined. Model Memory Footprint bounds the micro-batch, and nvidia-smi --query-gpu=memory.used at peak is the check.

What interviewers probe next

• "Is the result bitwise identical?" No; floating-point accumulation order differs, so gradients agree to rounding, not bit for bit, and a test should compare with a tolerance.
• "How does this interact with mixed precision loss scaling?" The scaled loss is backpropagated per micro-batch and unscaled once before the step; overflow detection must look at the accumulated gradient, and a skipped step discards all k micro-batches.
• "Does accumulation change the learning-rate schedule?" The schedule steps per optimizer step, not per micro-batch; a run that logs per micro-batch will look k times longer than it is.
• "Why not just use k = 64 and one GPU?" Wall clock: the same tokens take 64 times longer per step; accumulation trades time for memory, and data parallelism trades GPUs for time.

Common mistakes

• Dividing the loss by k inside the loop but also averaging in the loss function, or doing neither, and shipping a learning rate that is off by k.
• Forgetting no_sync() and communicating k times per step.
• Weighting micro-batches equally when their token counts differ, which biases the gradient.
• Claiming BatchNorm is a problem for a transformer.

Key takeaways

• Gradient of a mean equals the mean of gradients: k micro-batches of b equal one batch of kb for the optimizer.
• Memory: activations scale with the micro-batch, so k buys k times less activation memory at the same global batch.
• Communication: one reduction per k backward passes, but only with no_sync(); a 4× accumulation cuts DDP bytes per token by 4×.
• BatchNorm and in-batch sampling see b, not kb; LayerNorm and RMSNorm do not care.