TL;DR: The ordinary path reads from storage into host memory, then copies host memory to the GPU: two transfers, a host buffer, and CPU cycles spent on the copy. GPUDirect Storage lets the drive or the network storage client write into GPU memory directly, so there is one transfer and no host buffer. The gain is real when the storage can deliver more than the host path can absorb: eight local NVMe drives at about 7 GB/s each is 56 GB/s aggregate, while a bounce through host memory adds a copy at roughly 25 GB/s of effective PCIe throughput plus the host memory bandwidth to perform it. Checkpoint loading is the clearest winner, since the bytes go straight into the tensors that will hold them. Training data loading usually is not, because the CPU has to decompress, tokenize, augment or collate before the GPU can use anything, and once the CPU touches the data the bounce was going to happen regardless. Ask what the CPU does to the bytes between the drive and the GPU: if the answer is nothing, this pays.
How to approach it
Describe both paths and count the transfers, because the argument is entirely about how many times the data crosses PCIe and who copies it. Then give the two cases where it pays and the one where it does not, with the test that separates them in a sentence. Close with the practical requirements, since this needs support from the filesystem as well as the driver and fails back silently when it is missing.
A strong answer
A typical situation: a team enables GPUDirect Storage for their training data loader, measures no improvement, and concludes the feature does not work. It works. Their loader reads compressed shards, decompresses them on the CPU, tokenizes and collates, so the bytes were always going to pass through host memory and the only thing removed was a copy that was never the bottleneck.
The two paths, counted:
ordinary path
drive -> host memory one PCIe crossing, plus filesystem and page-cache work
host memory -> GPU memory a second PCIe crossing, from a pinned buffer
the CPU issues and often performs part of the copy, and host memory bandwidth carries the
data twice: once in, once out
GPUDirect path
drive -> GPU memory one PCIe crossing, the storage device or client writing into a
registered GPU buffer, with the CPU issuing the request only
bandwidth available on each side of the question:
8 local NVMe drives at about 7 GB/s each = 56 GB/s aggregate
a PCIe Gen5 x16 link, realized = about 25 to 50 GB/s
host memory bandwidth spent on a bounce copy = 2 x the data volume
sanity: when the storage side can deliver more than the bounce path can absorb, the bounce is
the limit and removing it is the whole win. When the storage side is slower than the
bounce, removing the bounce changes nothing measurable
Where it pays:
| Case | Why | Rough effect |
|---|---|---|
| Checkpoint loading and restore | The bytes go straight into parameter tensors with no transformation | Loading 1.1 TB of training state at 56 GB/s rather than being capped near 25 GB/s: about 20 s instead of 45 s |
| Pre-decoded, pre-tokenized data read as raw arrays | Nothing on the CPU touches the bytes | Removes a copy and the host memory traffic that goes with it |
| Very large sequential reads by a GPU-side consumer | The GPU is the only consumer and the transform is a kernel | Same as above |
| Inference weight loading on a cold start | Same shape as checkpoint loading | Shortens the cold start, which the autoscaler is waiting on |
Where it does not:
any pipeline with a CPU-side transform between storage and the GPU:
decompression (gzip, zstd), image decode, on-the-fly tokenization, augmentation, collation
the data has to be in host memory for the CPU to touch it, so the bounce is not removable
and GPUDirect can only apply to a stage that does not exist
small random reads:
the win is per transfer and the cost is per transfer too; at 4 KB reads the file system and
request overhead dominate and the copy is noise
when the storage is the bottleneck:
a network filesystem delivering 5 GB/s to the node is nowhere near the bounce path's limit,
so the bounce is free capacity and removing it changes nothing
Checkpoint I/O covers the load path in detail; Data Loading Pipelines for Training covers the case where the CPU work is the real constraint, which is the more common situation by a wide margin.
The one-sentence test to apply in an interview: what does the CPU do to these bytes between the drive and the GPU? If the answer is nothing at all, GPUDirect Storage removes a copy that was costing real bandwidth. If the answer names any transform, the bounce is structural and the feature addresses a cost that is not the problem.
The requirements, because this is where it silently does not engage:
software a filesystem and client that implement the cuFile interface, a driver with the
peer-memory path enabled, and an application that uses the cuFile API rather than
ordinary reads. An ordinary read into a GPU pointer does not become direct by itself
hardware the drive or network adapter and the GPU should be on the same PCIe root for the
direct path to be worth taking, which is the same locality requirement GPUDirect
RDMA has
failure mode it falls back to the ordinary path with no error, so the way to confirm it engaged
is the counters the cuFile layer exposes, not the absence of a complaint
GPUDirect RDMA and GPUDirect Storage covers both members of the family and the shared locality requirement.
The reversal condition: on a node whose storage is remote and modest, or whose pipeline is CPU-transform-heavy, this is complexity without benefit and the engineering time belongs in the data pipeline instead. The reversal in the other direction is worth naming too: as data formats move toward pre-tokenized, pre-decoded shards specifically so the CPU has nothing to do, the case for the direct path strengthens, and the two decisions are usually made together rather than separately.
What interviewers probe next
- "Does this help the checkpoint write path as well?" Yes, symmetrically, and writes are the more common bottleneck since they are bursty. The same locality and filesystem requirements apply.
- "How do you verify it engaged?" Through the cuFile counters and by watching host memory traffic, which drops to near zero for the transfer when the path is direct.
- "What about page cache?" The direct path bypasses it, which is what you want for data read once and a loss for data read repeatedly. A small dataset re-read every epoch may be better served from the page cache.
- "Is this related to GPUDirect RDMA?" The same idea applied to storage rather than the network, with the same requirement that the device and GPU share a PCIe root.
Common mistakes
- Enabling it for a loader that decompresses or tokenizes on the CPU, where the bounce cannot be removed.
- Expecting ordinary file reads into a GPU buffer to take the direct path without the cuFile interface.
- Measuring no change and concluding it is broken, rather than that the bounce was not the limit.
- Using it for small random reads, where per-transfer overhead dominates the copy it saves.
Key takeaways
- Ordinary path: two PCIe crossings and a host buffer. Direct path: one crossing, no host memory involved.
- Eight NVMe drives deliver about 56 GB/s while a bounce path realizes about 25 GB/s, so the bounce is the limit exactly when the storage is fast.
- Checkpoint and weight loading are the clear wins; anything with a CPU-side transform is not.
- The test: if the CPU touches the bytes between the drive and the GPU, this addresses the wrong cost.