CoreWeave AI Infrastructure Engineer interview questions

A grid is a request, a block is a residency unit, a warp is what the scheduler issues. Which of those pins to an SM, why 32 matters, and how a GPU hides a 600 ns memory latency with no branch predictor and a cache that is tiny per thread.

The H200 has the same compute die as the H100 and costs more per hour. The datasheet gives two ratios, 1.43x bandwidth and 1.76x memory, and those two numbers decide exactly which workloads pay back the premium and which ones lose money on it.

Three things change at once: dense compute more than doubles with an fp4 tensor core, HBM3e reaches 8 TB/s, and NVLink stops at 72 GPUs instead of 8. Work through what each does to tensor-parallel degree, per-step weight reads and expert placement for a 671B-parameter mixture of experts.

MIG carves an H100 into up to seven hardware slices with their own SMs, memory and bandwidth; MPS lets processes share one SM pool through a single context. One gives fault and performance isolation at a fixed slice size; the other gives flexibility and a shared failure domain. The numbers that decide it.

On an HBM part ECC costs almost nothing; on a GDDR part about 6% of capacity and bandwidth. What it buys is the difference between a corrected bit and a silently flipped exponent that turns 1.0 into infinity across a 512-GPU all-reduce. The mechanisms, the signals, and the expected-loss arithmetic.

Decode throughput is bytes per step over bandwidth, price is dollars per hour, so tokens per dollar for a 70B falls out of a spec sheet in ten lines. In bf16 the three generations are within 10% per dollar; fp8 pulls the H100 ahead, and fp4 on the B200 doubles it again. The chain, the table and the caveats.

Half the expected bus bandwidth has four plausible causes needing different fixes: a link trained at reduced width, a fabric that never came up, a topology the job did not expect, or a test that measured latency. The sequence of commands that tells them apart, and what each output looks like on a healthy node.

Start from the power envelope, because it decides the GPU count more often than the budget does. The scalable unit that makes the design a copy of a known-good one, the four networks, and the three line items that gate the schedule while everyone is discussing GPUs.

Decode is bandwidth-bound, so the FP8 FLOPS number that dominates the marketing does not move it. Where the B300 pays is capacity, and capacity converts into throughput through batch size rather than directly. The arithmetic that decides it, and the case where the B200 wins.

Four separate facility limits have to be cleared and total building capacity is not one of them. Power per rack, heat rejection, floor loading and network reach each fail independently, and each has a different remedy with a different lead time. The options that exist, priced, and the one that is usually taken.

Links that never come up and links that come up and flap are different faults with different causes. The reach change between 400G and 800G explains most first-time failures, and the counters separate the rest. The order to check, and the one physical measurement that settles it.

The GPU board power is about 60 percent of the answer. The rest is CPUs, memory, NICs, fans, switches, thousands of transceivers and the cooling plant, and each is a term people leave out. The full derivation, the term that surprises everyone, and the contracted number that is larger than all of it.

Every watt in comes out as heat, so the design starts from the load and one equation. The flow rate, the CDU count and redundancy, the water quality requirements that decide whether the loop survives three years, and the automatic response that has to fire before a human can read the page.

A rail-optimized fabric makes this fault findable because the eight rails are identical by construction, so any asymmetry is a defect. Bisecting by rail, then by switch, then by node, then by port narrows 512 links to one in four steps, and the counters say which of three physical causes it is.

Each layer depends on the ones below it and a mistake at the bottom shows up six layers higher as something else entirely. The order, the dependency almost everyone gets wrong on Blackwell systems, and the validation ladder that keeps a fault attributable to one node instead of five hundred.

Acceptance is a contract document, not a checklist, so every criterion has to be a number with a tolerance that a vendor can be held to. Comparing nodes against each other rather than against a datasheet is what finds real defects, and the duration is what finds the thermal ones.

A training cluster is a synchronized load, so the peak is far above the average and the average is what everyone budgeted. The immediate mitigation takes minutes and costs a few percent of throughput, and the four durable fixes trade cost against throughput in different places.

The count follows from three inputs and one topology rule, and getting it wrong by the leaf-to-spine half is the classic error. The arithmetic, the radix check that says two tiers is enough, and the power the transceivers add that nobody put in the budget.

Single-node fine and two-node halved is a specific signature, and it points at one thing before anything else. Why the copy path costs exactly about half, the five causes that produce it, and the one log line that confirms which.

Capacity is the easy half and almost never the constraint. The number that sizes the system is the checkpoint write burst, which is thousands of times the steady dataset read, and the tier that serves each is different. The full derivation, and the tiering that avoids paying burst prices for archive bytes.

The endpoint count is larger than the node count and that surprises everyone building their first cluster. What has to be reachable, why this network must survive when every other one is down, and the security posture it needs because it can power-cycle the entire fleet.

Three approaches with overlapping ranges, and the choice is decided by the facility and the service model rather than by thermodynamics alone. Where each runs out, what the middle option buys that people underrate, and the operational cost that comes with the most capable one.

The GPUs are almost never the critical path and planning backward from their delivery date is how deployments slip. The real ordering, what can run in parallel, the two dependencies that cannot, and the cost of a month of delay expressed in the unit that gets attention.

The spares number falls out of the failure rate times the repair turnaround, and the cheap items are the ones people forget. The arithmetic for GPUs, the very different arithmetic for transceivers, and the rule that decides whether to hold a part at all.

A 44 percent failure rate is far above any plausible hardware defect rate, so the first hypothesis is that the test or the environment is wrong rather than the hardware. The three checks that separate those, the cohort analysis that localizes a real defect, and how to run the vendor conversation.

Most architecture diagrams show one network and a cluster has four. Each has a different traffic pattern, a different failure consequence and a different port count, and the one that is under-counted every time is the one you need when everything else is broken.

Same GPU count, same silicon, and a completely different machine. The domain size is the whole argument, it is worth an order of magnitude on specific traffic, and it costs a failure domain nine times larger plus a facility that most halls do not have.

The break-even is a utilization number, not a price comparison, and most teams overestimate the utilization they will actually reach. The full cost of owning, the number that decides it, and the two situations where renting wins even at high utilization.

A fixed power envelope turns hardware selection into an optimization with one objective and several overheads competing for the same watts. Where the 1 MW actually goes, why the answer is not simply the most efficient GPU, and the two overheads that can be cut to buy more compute.

It configures the NVSwitch fabric so eight GPUs present as one NVLink domain, and when it is absent nothing errors. Jobs run, results are correct, and everything multi-GPU is slower by a factor the team spends weeks not finding. The check that takes one command.

Isolation has to be physical where it matters and logical where it can be, and picking the boundary wrong is either expensive or a security problem. What partitions cleanly, what does not, and the allocation unit that decides fragmentation and margin.

The rule is one sentence and the execution is thousands of connections where a single transposition creates a fault that is invisible until a collective runs. The mapping, the labelling scheme that makes errors findable, and the verification that has to happen before anyone accepts the cluster.

Acceptance measures the hardware and production measures the whole system, so the gap lives in everything acceptance did not cover. Four candidate layers, the order to eliminate them, and the two that account for most of these cases.

A fleet-wide version change is the most likely cause of the next unexplained performance regression, so the rollout is designed to make that attributable. Cohorts, a canary that measures rather than boots, and the rollback that has to be real before the first node is touched.

Almost nothing about a generation change is the GPU. The facility usually cannot take the new part in the old positions, the fabric generation may not match, and the two fleets have to coexist for months. The sequencing that avoids a capacity trough, and the decision people get wrong.

Peak numbers are almost never the reason a part wins or loses, and the software stack decides more deployments than the silicon does. The four things to measure, the question that eliminates most challengers in one conversation, and the pilot that gives a defensible answer.

A check that is too strict removes healthy capacity and one that is too loose lets a broken node take a job down. The three tiers by cost, the fleet-median comparison that catches degradation no threshold does, and the false-positive budget that has to be set before any of it runs.

Two failures and a pass is not a passing node, it is a node with a demonstrated intermittent fault and a small sample. The arithmetic that shows how weak the evidence for acceptance is, what an intermittent fault costs once it is in a training job, and the policy that removes the argument.

Doubling per-rack density is four separate upgrades with different lead times, and the electrical one usually cannot be done while the hall runs. The phasing that keeps capacity available through it, the constraint that decides whether it is possible at all, and the number that says whether to upgrade or move.

The hardware is identical and almost everything around it moves: who can touch a node, how long a swap takes, what the contract actually guarantees, and what you can no longer measure. The six things to negotiate before signing and the two that decide whether operations work.

Modules do not fail randomly at three times a published rate, so the excess has a cause and it is usually thermal or a batch. The cohort cuts that localize it, the counters that separate a dying module from a dirty connector, and the environmental cause that looks like a hardware defect.

Firmware skew produces intermittent faults that surface layers away from their cause, so an inventory is a debugging tool rather than a compliance exercise. What to collect, how drift appears without anyone deploying anything, and the diff that turns a week-long investigation into a query.

Both run at the same port speed, so the discount is not buying slower hardware. It is buying a different operational model and a different failure mode, and whether that is worth 15 percent depends on what your team already runs and what the fabric has to carry.

Older parts are rarely worthless and often better suited to work the new fleet should not be doing. The four options ranked by what they return, the data handling that has to happen before any hardware leaves, and the reason the answer is usually not one option.

A whole rack failing at once is a shared dependency, which narrows the causes to four before anyone touches anything. What the out-of-band network tells you in the first minute, why the order of restoration matters, and the decision about liquid cooling that has to be made before power returns.

The watchdog says an all-reduce ran for ten minutes and never finished. That means one rank never entered it, entered a different one, or died. How to read the log line, the flight recorder that names the rank in seconds, the env vars that make the next one debuggable, and why the timeout is a symptom every time.

Weights take a fixed slice, the KV cache takes the rest, and the rest divided by context is your user count. Work it in bf16 and fp8, then find out why the answer also depends on how you split the model.

The answer comes from the model card and config.json in five minutes, before downloading anything. The weight footprint, the KV per token that is far smaller than the parameter count suggests, and the parallel degree that has to divide the head count.

Two point eight trillion parameters fits in one node, because the experts ship in a four-bit format. The footprint arithmetic that reproduces the project's own published hardware minimum, the hybrid cache that needs specific engine support, and why batch-one throughput is a tenth of the bound.

Forming and then hanging is a different fault from failing to form, and it points at a small set of causes. The rank field that is a cumulative count rather than an index, the collective that never completes, and the three checks that separate configuration from fabric.

Six steps forward from traffic, never backward from an available GPU count. The one input that has to be measured rather than derived, the headroom that is not optional, and the utilization term that moves cost per token more than any tuning flag.

Identical replicas serving identical traffic should produce identical numbers, so a difference is a defect and the fleet median is the tool that finds it. Five causes, the three that are about the replica and the two that are about what it was sent.

Twenty models of different sizes is a packing problem with a cold-start penalty, and the two pull in opposite directions. What stays resident, what loads on demand, the arithmetic that decides which is which, and the interface that stops every team from asking for a dedicated replica.

700 W per GPU is the number everyone knows and it is only 40% of the answer. The chain from GPU TDP to node, to IT load, to facility power with PUE: about 1.8 MW, and what that means in racks and dollars.

Capex plus two years of power and operations, minus what the cards are worth at the end, against a rental rate at a utilization. The chain that produces a break-even utilization, and the two inputs that swing it from 60% to over 100%.

8 TB/s over the bytes per step: 57 tokens per second single-stream in bf16, 113 in fp8, 227 in fp4, and the batch curve on a single card that now holds the whole model. Where the curve bends and what caps it.

Decode is bandwidth-bound, so cost per token tracks dollars per terabyte per second, and on spec the MI300X wins by 2x. The table, the memory-capacity term that changes the replica shape, and the software-efficiency discount that decides whether the spec ranking survives a benchmark.

Wire GPU number three of every node to the same switch and the collective that matters most never leaves it. The rank mapping that makes the layout pay, the switch and cable count for a thousand GPUs worked out, and what happens to a job whose ranks are assigned in the wrong order.

Find the scale at which it breaks before looking at any node, because the scale tells you which layer to open. The commands in order, the number each should return on a healthy fabric, and the four causes that this measurement sequence separates without guessing.

Four of them diagnose, four configure the machine correctly, and several popular ones override tuning that was already right. Which is which, the symptom each addresses, and why a cluster whose job scripts carry a dozen of these has a configuration problem rather than a tuning one.

Three networks, not one, and the mistake that hurts most is putting two of them on the same wires. The port and switch counts for each, the cable plan, the split of the bill between switches and optics, and the two decisions that are hard to change once the cluster is racked.

A collective stops when one participant stops, and a job of a thousand ranks has no way to distinguish a link that will come back in two seconds from a rank that has died. What the job actually does during those seconds, the cost per flap, and why the fault passes every test you would run.

Every tenant bursts independently, so the shared tier sees the sum of uncorrelated peaks rather than the sum of averages. The provisioning arithmetic that follows, the three isolation boundaries, and why per-tenant local storage solves most of it before any policy is written.

A uniform loss across every node rules out hardware and points at something that changed everywhere at once. The differential procedure that isolates a reset default from a real regression, the settings firmware resets on its own, and the change process that catches it in an hour.

The four layers between `nvidia.com/gpu: 4` and a container that can run CUDA, the one failure each layer produces, the stranded-GPU arithmetic the integer model causes, and why Dynamic Resource Allocation replaces the count with a claim.

Hardware partitions, shared SMs, and context switching are three different promises about isolation and waste. The slice arithmetic for an H100, what each mode gives up, and the one rule about tenants that decides most of it.

Slurm gives you gang scheduling, topology and MPI launch for free and nothing for services; Kubernetes gives you services, elasticity and the ecosystem and makes you bolt on the batch semantics. The hybrid that most large clusters land on, with the numbers that push them there.

A distributed training job is 64 pods that start together or not at all. The deadlock two partially placed jobs produce, counted out on a 64-GPU cluster, and how Kueue and Volcano make the job the unit of admission.

Fractions are either hardware slices with fixed shapes or soft shares with no isolation, and the scheduler has to know which. The representation, the packing score, the fragmentation arithmetic that spread placement produces, and the defragmentation step that fixes it.

Same job, same GPU count, half the speed: the slow one was split 4 and 4 across two nodes and its tensor-parallel all-reduces run at fabric speed instead of NVLink speed. The bandwidth arithmetic, the per-parallelism placement rule, and the scheduler policy that enforces it.

A node with eight healthy H100s that Kubernetes thinks has none. The layered check from kernel driver to container toolkit to device plugin to taints and allocatable, in the order that isolates the fault fastest, with the log lines each layer prints when it is the one that broke.

A GPU node needs a driver, a container toolkit, a device plugin, a metrics exporter and a validator, in that order, and each needs to know which nodes it belongs on. The operator's component chain, the labels NFD writes to trigger it, and the bring-up arithmetic that rules out doing it by hand.

A shared pool recovers the GPUs inference holds for its peaks, but a preempted training gang takes minutes to give them back and an SLO breaks in seconds. The utilization of each design, the reclaim-time arithmetic against the traffic ramp, and the floor-plus-borrow split most fleets land on.

Committed capacity is cheaper per hour and cannot shrink, so the decision is a utilization break-even, not a discount. The arithmetic for where the reserved baseline should sit, why the p50 of steady demand is the target, and how on-demand, spot and capacity blocks absorb everything above it.

Allocated is not usable: a cluster full of half-empty nodes has hundreds of idle GPUs that no gang can take. The arithmetic of how mixed job sizes fragment a fleet, the packing policies that prevent it, the defragmentation moves that repair it, and what each costs the jobs already running.

A pre-flight suite is a list of tests with a number on each: DCGM health, a GEMM near the fleet median, NCCL at rated bandwidth, NICs at line rate, links active, storage reachable. What goes in the two-minute gate, what waits for the long diagnostic, and why the gate pays for itself.

You cannot remove one rank from a synchronous job; you replace the node and restart from checkpoint, and the whole craft is making that take three minutes instead of thirty. The detection signals, the drain-restart-quarantine sequence, the spare-pool arithmetic, and where elastic training changes the answer.

A pod's default network is an overlay that no RDMA NIC will speak. The three ways to give a pod the fabric (host networking, SR-IOV virtual functions via a NIC device plugin, or a macvlan secondary interface), what each costs in isolation and speed, how NCCL finds the right NIC, and the failure that halves bandwidth.

A driver upgrade touches every layer at once: kernel module, CUDA runtime, container toolkit, NCCL, fabric driver, and every job's image. The compatibility matrix that decides whether a job can run on the new node, the canary that proves it, the wave arithmetic for 2,000 nodes, and the rehearsed rollback.

Free GPUs and a pending gang are not a contradiction: the gang needs 64 whole nodes at once and the free GPUs sit on 300 half-used nodes. The isolation order (quota, fit, per-node reasons, one node's device plugin), the command at each step and what it means, and the fixes ranked by displacement.

A published run gives the numbers directly: 419 unexpected interruptions in 54 days on 16,384 GPUs, and about three quarters of them hardware. What that implies per GPU-hour, which components dominate, and why a fleet above a few thousand GPUs must treat failure as routine rather than exceptional.

The driver logs a numbered code when something goes wrong on a GPU, and the number tells you whether to retry the job, drain the node or file a hardware return. The codes worth memorizing in three groups, the action each implies, and the automation that turns a log line into a drained node.

The exporter offers hundreds of fields and about a dozen of them change decisions. The health fields that predict failure, the performance fields that tell you whether work is happening, the one everybody alerts on that means nothing, and the diagnostic levels with the time each takes.

Correctable errors are corrected, so nothing is wrong yet, and that is exactly why they are the most useful signal a fleet has. What rising rates predict, the remapping budget that decides whether a device can heal itself, and the policy that converts a trend into a scheduled drain instead of a failed training run.

The device has stopped answering on PCIe, so the driver cannot reset it and no software fix exists: the host has to re-enumerate the bus. What causes it, why the node is out until it reboots, the three-strike policy that separates a transient from a dying card, and the automation that handles it without a human.

Infant mortality is real and diagnostics that run for a minute do not find it. The five-stage acceptance suite with what each stage catches and how long it takes, the pass criteria as numbers rather than judgments, and the sustained-load stage that finds the faults everything else misses.

Throttling is the hardware protecting itself, so it produces no error and no failure, only a job that is quietly slower. The two telemetry fields that name it, the arithmetic linking clock to throughput, and why the response differs completely depending on whether the cause is one node or the room.

A fleet of ten thousand GPUs generates enough hardware events to page someone hourly, and a rota that receives them stops reading them within a month. The three tests an alert must pass to page, the arithmetic of a sustainable rota, and the automation that has to exist first.

The controller's real design problem is not detecting faults but deciding when a signal is strong enough to remove a node from a job. The six states and their transitions, the cost asymmetry that sets every threshold, and the arithmetic showing what a one-percent false-positive rate costs at fleet scale.

Four mechanics do most of the work: symptom-based rules rather than cause-based, burn rate rather than thresholds, deduplication so one event is one page, and suppression during known windows. The arithmetic for each, and the rule that keeps the set from growing forever.

A cold plate holds a few tens of grams of water against seven hundred watts, so temperature rises about two degrees a second and the limit arrives in under twenty. What that timescale rules out, the protection layers that act inside it, and the choice that decides whether you lose a job or the hardware.

Every replacement costs a spare, a maintenance window and a return process; every device kept costs the risk of a job it will fail. Four triggers that decide it without a case-by-case argument, the arithmetic that sets the repeat threshold, and why the device's history beats any diagnostic result.

Published data gives an interruption rate, and replacements are a fraction of interruptions, so the comparison needs care before it means anything. The derivation that turns one into the other, the answer for a healthy fleet, and the four cuts through the data that turn an anomalous number into a cause.

A GPU cloud sells whole nodes and fabric slices to tenants who expect them in minutes and expect them to work. The node state machine, the reconciler that keeps thousands converging, the isolation lines on shared fabric, the health loop that pulls a bad node first, and billing that survives a control-plane outage.

The union is a sort and a scan. The intersection is the same sweep with a counter, and it is the one that goes wrong, because a single GPU reporting overlapping intervals will be counted twice and produce an answer that looks plausible. Both implementations, the trap, and the tests that catch it.

Three one-line policy differences produce different fragmentation, and the metric that separates them is not utilization. The measured outcome on the same job sequence, why the largest free block is what matters on a training cluster, and the case where the intuitive policy is exactly wrong.

Unbounded goroutines are the default failure: ten thousand shards means ten thousand concurrent connections and an immediate rate limit. The semaphore pattern, the context that makes cancellation propagate, and the two mistakes that produce a leak nothing reports.

The arithmetic is one multiplication and the difficulty is entirely in the events that do not pair cleanly. Clipping to the window, an allocation still running when the period ends, a stop with no start, and why every anomaly must be reported rather than dropped.

Four resources, one state machine and three semantics that separate a working API from one that corrupts state under retries. The idempotency key that makes a duplicate POST safe, why cancel returns 202 rather than 200, and the cursor that survives concurrent inserts.

Metrics arrive in time order and are queried by time window, which is the pair of facts the whole design turns on. Sealed segments with a sparse index, pruning that skipped 82 of 84 segments in the executed test, and TTL compaction that drops whole files without rewriting a byte.

A war story with no learning in it scores zero, however dramatic. The five beats of an incident narrative, the decision under uncertainty that is the actual subject of the question, and how to tell it when the incident was somebody else's fault.

The answer that fails is a compliment to the company. The answer that lands is a thesis about where the constraint sits, tied to something you have actually done, and it works whether you are coming from distributed systems, kernels, hardware or SRE.

Measure the hours before you rank anything, because the task that feels worst is usually not the one that costs most. The ordering rule that keeps automation from causing the outage it was meant to prevent, and why detection always ships before remediation.

Everyone has this story and most candidates tell a flattering version of it. The bias that produces almost all of these mistakes, the cost you have to be willing to name, and the process change that is the difference between an anecdote and a lesson.

Naming three newsletters answers nothing. The answer that works has a filter, a source of ground truth that is not a blog post, and one thing you reproduced yourself, because that is the only evidence that you learn rather than accumulate.

Any engineer can cut cost. The question is whether you measured what it cost you elsewhere, and whether you knew that before you shipped it. The three mechanisms that produce most of the savings, and the metric you have to watch on the way down.

Most postmortem documents are written, filed and never read again. The three parts that decide whether one changes anything, why action items without a named owner and a date are decoration, and the meeting rule that keeps blamelessness from becoming vagueness.

You cannot win this with an opinion, and a flat objection to a purchase reads as an engineer who does not understand the business. The utilization evidence that carries it, the smaller counter-proposal that usually wins, and what to do once the decision goes against you.

Arguing is the losing move and so is capitulating. What moves a vendor case is a reproducer small enough for their engineer to run, counters that isolate their layer from yours, and a bisection that removes your configuration as a variable entirely.

The first move is not technical. It is deciding who runs what, because one person serially debugging three incidents is slower than three people in parallel and much slower than one person coordinating. The ordering rule, and the two things that outrank everything else.

The interviewer is checking whether you know what you are trading, because both directions are reasonable and both have a version that ends badly. What each environment actually gives you, and the answer that works when the real reason is compensation.

A confident plan built before you know anything is the failure mode, and so is refusing to answer until you have looked. The structure that handles both: a dated shape with the decision points named, one committed win, and the conditions that would change everything after day 30.

How you reported it matters more than how you found it. The internal disclosure that gets a fix instead of a defensive reaction, the four places isolation gaps hide in GPU infrastructure, and the test that keeps the fix from regressing.

Before the first byte moves, the library has already discovered the machine's topology, built rings and trees over it, split the work into parallel channels and chosen a wire protocol by message size. What each of those four decisions is for, and the environment variables that let you see and override them.

Four operations cover almost everything a distributed training step sends. What each one does to the data, the bytes each rank moves for a message of size S across N ranks, and which parallelism strategy generates which, worked through for a 70B model so the numbers are concrete rather than symbolic.

Both carry RDMA at the same line rate. One arrives lossless because the transport was designed that way, the other becomes lossless only if a set of switch settings is correct on every port. What that difference costs in operations, what it saves in money and hiring, and the fleet size where the answer flips.

A single core moves a few gigabits per second of TCP once copies and interrupts are counted, so filling one 400 gigabit link would take most of a server's cores doing nothing but networking. What RDMA removes, what a queue pair actually is, and where the GPU fits when the data never belongs to the host at all.

Port counts decide everything: a two-tier Clos of 64-port switches tops out at exactly 4,096 endpoints at full bisection, which is why that number appears in so many cluster designs. The switch and optics count both ways, what oversubscription actually slows down, and where the money goes.

Thirty-two senders aimed at one port fill a switch buffer in microseconds, and what happens next depends on which of two mechanisms fires first. The buffer arithmetic, why the slow mechanism is the safe one and the fast one is the dangerous one, and the counters that tell you which is happening.

Reading straight from NVMe into GPU memory skips a copy through host memory, which is worth a lot for one workload and nothing for another. The bandwidth arithmetic for both paths, the two cases where the bounce is the bottleneck, and the far more common case where the CPU has to touch the data anyway.

One gives you POSIX semantics and hundreds of gigabytes per second at a price per terabyte that hurts at petabyte scale. The other is cheap and durable and answers a first byte in tens of milliseconds. The cost arithmetic that makes most clusters run both, and the metadata limit that decides how you shard.