Why does object storage behave badly for random reads, and what does a training pipeline do about it?

The bandwidth is effectively unlimited and the first byte takes fifty milliseconds, so throughput is set by how many requests you keep in flight rather than by the network. Little's law applied to a data loader, the request size where the latency stops mattering, and the two design changes that follow.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 283 remaining answers · ₹2,000 / $25

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Advanced

🔌 Networking & Storage🔒 Premium

Data Loading Pipelines for TrainingThe dataloader is the only part of a training job that runs on the CPU, the disk and the network at once, and it is the part most often found starving the GPUs. A pipeline that keeps 1,024 accelerators fed has to read sharded files sequentially, decode and tokenize in parallel workers, prefetch several batches ahead, pin memory for the PCIe copy, and do it deterministically enough to resume mid-epoch. The symptom of failure is a GPU at 30% utilization with nothing wrong on the GPU.

Core

🔌 Networking & StorageSign in

Parallel Filesystems vs Object StorageA training cluster's storage has two very different jobs: stream terabytes of training data to thousands of GPUs at a steady rate, and absorb a multi-terabyte checkpoint burst every few minutes. Parallel filesystems (Lustre, GPFS, WEKA, VAST, FSx) give POSIX semantics and hundreds of GB/s of aggregate throughput; object storage (S3 and its equivalents) gives durability and cost at a fraction of the price with high first-byte latency. Almost every real cluster uses both, and the interview question is which job goes where and how big each tier has to be.

Foundational

🧮 Napkin Math & Capacity

KV Cache SizingThe KV cache is the memory that decides how many users a serving replica can hold and how long their context can be. Its size per token comes from four numbers in the model's config file (layers, KV heads, head dimension, bytes per element) and one formula; multiplied by context and concurrency it is the number every capacity plan is built on. This page derives it, works it for four models including an MLA one, and shows the two places candidates get it wrong by a factor of eight.

Advanced

💻 Coding for Infra🔒 Premium

Producer-Consumer PipelinesA data loader, a log shipper, a batch inference job and a checkpoint writer are the same program: stages connected by bounded buffers, each running at its own pace, the slowest setting the throughput and the buffers absorbing the jitter between them. The coding screen asks you to build one (read, decode, batch, feed a consumer) and then pushes on the production questions: buffer sizes, clean stops, failure propagation, and why it runs at a third of the expected speed. This page derives throughput from stage times, implements the pipeline in threads and asyncio, and works the stop and failure semantics.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackDeduplicate and quality-filter a multi-petabyte web corpus. What does that pipeline cost and where does it bottleneck?Next in this trackA mixture-of-experts model does an all-to-all twice per layer. What does that demand of the fabric, and what changes on a rack-scale system?Next in this trackCheckpoints and datasets both live on storage. Why does one system sized for both usually serve neither well?

FEDITOR'S NOTE

Scored on applying Little's law to derive the required concurrency, on the request-size threshold where latency amortizes, and on shard layout and prefetch as the two responses.

DISCUSSION · 0

No comments yet — be the first to share your approach.