TL;DR: Object storage is durable, effectively unlimited, and costs on the order of $20 per terabyte-month, but its first-byte latency is tens of milliseconds and it has no POSIX semantics. A parallel filesystem gives POSIX and hundreds of gigabytes per second of aggregate throughput at roughly $100 to $200 per terabyte-month, and its practical limit is usually metadata operations rather than bandwidth: millions of small files will exhaust the metadata servers long before the data path saturates. Most clusters therefore run both. The object store is the durable source of truth holding every dataset, the parallel filesystem or local NVMe holds the working set for jobs currently running, and a copy step moves data between them. The design decisions that follow are shard size, which must be large enough to amortize object latency, and what lives in the hot tier, which is a capacity plan rather than a preference.
How to approach it
Compare on the three axes that decide it, which are cost per terabyte, latency and semantics, and give numbers on each. Then say what the parallel filesystem's real limit is, since candidates usually name bandwidth and the answer is metadata. Then describe the two-tier pattern as the conclusion rather than as a compromise, and finish with the two design decisions it forces.
A strong answer
A typical situation: a team puts a 15 trillion token dataset on a parallel filesystem as several hundred million small files, one document each. Throughput on a large sequential read is excellent. Training jobs stall anyway, because every epoch opens hundreds of millions of files and the metadata servers become the cluster's slowest component.
The comparison, on the axes that matter:
parallel filesystem object storage
cost per TB-month roughly $100 to $200 roughly $20
first-byte latency under a millisecond 20 to 100 ms
aggregate throughput 100 to 500 GB/s for a cluster scales with concurrency, effectively
unlimited for large sequential reads
semantics POSIX: open, seek, partial GET an object, PUT an object; no
write, rename, permissions partial writes, no rename semantics
durability replicated within the cluster many nines, across failure domains
practical limit metadata operations per-request latency
worked cost, 5 PB of training data:
object storage 5,000 TB x $20 = $100,000 per month
parallel filesystem 5,000 TB x $150 = $750,000 per month
difference $650,000 per month, or $7.8M a year, for data that is mostly cold
sanity: this arithmetic is why nobody keeps a full dataset on the fast tier, and it is the
entire argument for two tiers rather than a preference about interfaces
The metadata limit, which is the part that surprises people:
a parallel filesystem's metadata servers handle open, stat, create and lookup
capability: on the order of 100,000 to 500,000 metadata operations per second for a cluster
a dataset stored as one file per document, 300 million documents:
one epoch = 300 million opens, plus stats and lookups
at 200,000 ops/s that is 1,500 seconds of pure metadata work per epoch, during which the
data path is mostly idle
the same data as 30,000 shards of 10,000 documents each:
one epoch = 30,000 opens, 0.15 seconds of metadata work
sanity: a factor of 10,000 in metadata load for the same bytes. Shard size is the single
most consequential storage decision a training pipeline makes, and it is usually
made by whoever wrote the preprocessing script
Parallel Filesystems vs Object Storage covers both systems; the shard-size consequence is the same one that makes object storage usable at all, since a 20 to 100 millisecond first byte is only tolerable when amortized over a large object.
The two-tier pattern, which is where most clusters land:
tier 1, object storage every dataset ever produced, all checkpoints past the recent few,
model artifacts. Cheap, durable, slow to first byte.
tier 2, fast local the shards a running job needs, staged before or during the run.
Either a parallel filesystem shared across the cluster or NVMe on
each node with the shards it will read.
staging cost 30 TB of tokenized shards at 5 GB/s takes about 100 minutes, so it
is scheduled ahead of the job rather than at its start
what decides the tier size concurrent jobs times their working sets, plus headroom, which is
a capacity plan rather than a guess
For pre-tokenized text the working set is often small enough that node-local NVMe is the whole answer and no shared filesystem is needed: 15 trillion tokens stored as uint16 is 30 TB, which fits across a modest cluster's local drives. That is worth saying because it is a cheaper design than the shared filesystem it replaces, and it removes a component that has its own failure modes.
Watching for the metadata limit is one number rather than an investigation. Every parallel filesystem exposes metadata operations per second at its metadata servers, and that counter climbing toward its ceiling while the data path sits at a fraction of its bandwidth is the exact signature of the small-file problem. Put it on the same dashboard as throughput, because a team watching only bandwidth will see an idle filesystem and a stalled job and have no way to connect them. Data Loading Pipelines for Training covers what the loader does with the shards once the layout is right.
The reversal condition: a shared parallel filesystem earns its cost when many jobs read overlapping data, when the working set is far larger than local storage, or when POSIX semantics are required by tooling that cannot be changed. It stops earning it when the pipeline has been reorganized into large sequential shards, because then object storage plus local NVMe delivers the same read pattern at a fraction of the cost. Teams often keep the filesystem because it predates the shard format, which is a reason to revisit rather than a reason to keep.
What interviewers probe next
- "How large should a shard be?" Large enough that the object's first-byte latency is a small fraction of the time to read it: at 50 ms of latency and 1 GB/s per stream, a 1 GB shard spends 5% of its time waiting, and a 100 MB shard spends a third of it.
- "How do you shuffle if the data is in large shards?" Shuffle at two levels: the order of shards, and a buffer of samples held in memory across several shards. That gives good mixing without random reads.
- "What about checkpoints?" A different access pattern entirely, bursty writes rather than streaming reads, which is why checkpoints go to local NVMe first and then asynchronously to object storage.
- "Does a caching layer change the answer?" A read-through cache in front of object storage gives much of the two-tier benefit with less operational surface, and it is a reasonable middle path when the working set is stable.
Common mistakes
- Storing a dataset as one file per document, which turns a bandwidth problem into a metadata problem.
- Keeping a full multi-petabyte dataset on the fast tier, at several times the cost of the cold tier.
- Sizing the fast tier from the dataset rather than from the concurrent working set.
- Assuming a parallel filesystem is required, when large sequential shards make object storage plus local NVMe equivalent and cheaper.
Key takeaways
- Roughly $20 per terabyte-month against $100 to $200, which at 5 PB is $7.8M a year of difference.
- Object storage answers a first byte in 20 to 100 ms, so shards must be large enough to amortize it.
- A parallel filesystem's practical limit is metadata operations: 300 million small files is 1,500 seconds of metadata work per epoch against 0.15 seconds for 30,000 shards.
- The usual answer is two tiers, with the object store as truth and a staged working set on the fast tier.