Microsoft AI Infrastructure Engineer interview questions

Three things change at once: dense compute more than doubles with an fp4 tensor core, HBM3e reaches 8 TB/s, and NVLink stops at 72 GPUs instead of 8. Work through what each does to tensor-parallel degree, per-step weight reads and expert placement for a 671B-parameter mixture of experts.

192 GB and 5.3 TB/s per card beat the H100 on both memory numbers at a lower hourly price. Whether that wins in production is a question about achieved bandwidth and ROCm kernel coverage, and there is a short list of numbers to run before you sign the order.

On an HBM part ECC costs almost nothing; on a GDDR part about 6% of capacity and bandwidth. What it buys is the difference between a corrected bit and a silently flipped exponent that turns 1.0 into infinity across a 512-GPU all-reduce. The mechanisms, the signals, and the expected-loss arithmetic.

Four separate facility limits have to be cleared and total building capacity is not one of them. Power per rack, heat rejection, floor loading and network reach each fail independently, and each has a different remedy with a different lead time. The options that exist, priced, and the one that is usually taken.

The GPU board power is about 60 percent of the answer. The rest is CPUs, memory, NICs, fans, switches, thousands of transceivers and the cooling plant, and each is a term people leave out. The full derivation, the term that surprises everyone, and the contracted number that is larger than all of it.

Every watt in comes out as heat, so the design starts from the load and one equation. The flow rate, the CDU count and redundancy, the water quality requirements that decide whether the loop survives three years, and the automatic response that has to fire before a human can read the page.

Acceptance is a contract document, not a checklist, so every criterion has to be a number with a tolerance that a vendor can be held to. Comparing nodes against each other rather than against a datasheet is what finds real defects, and the duration is what finds the thermal ones.

A training cluster is a synchronized load, so the peak is far above the average and the average is what everyone budgeted. The immediate mitigation takes minutes and costs a few percent of throughput, and the four durable fixes trade cost against throughput in different places.

The endpoint count is larger than the node count and that surprises everyone building their first cluster. What has to be reachable, why this network must survive when every other one is down, and the security posture it needs because it can power-cycle the entire fleet.

Three approaches with overlapping ranges, and the choice is decided by the facility and the service model rather than by thermodynamics alone. Where each runs out, what the middle option buys that people underrate, and the operational cost that comes with the most capable one.

The GPUs are almost never the critical path and planning backward from their delivery date is how deployments slip. The real ordering, what can run in parallel, the two dependencies that cannot, and the cost of a month of delay expressed in the unit that gets attention.

A 44 percent failure rate is far above any plausible hardware defect rate, so the first hypothesis is that the test or the environment is wrong rather than the hardware. The three checks that separate those, the cohort analysis that localizes a real defect, and how to run the vendor conversation.

Switch radix decides it, and at 10,000 endpoints the answer sits right at the boundary for current high-radix switches. The arithmetic, what a third tier costs in switches, optics and power, and the two designs that avoid it entirely.

Same GPU count, same silicon, and a completely different machine. The domain size is the whole argument, it is worth an order of magnitude on specific traffic, and it costs a failure domain nine times larger plus a facility that most halls do not have.

A fixed power envelope turns hardware selection into an optimization with one objective and several overheads competing for the same watts. Where the 1 MW actually goes, why the answer is not simply the most efficient GPU, and the two overheads that can be cut to buy more compute.

Acceptance measures the hardware and production measures the whole system, so the gap lives in everything acceptance did not cover. Four candidate layers, the order to eliminate them, and the two that account for most of these cases.

A fleet-wide version change is the most likely cause of the next unexplained performance regression, so the rollout is designed to make that attributable. Cohorts, a canary that measures rather than boots, and the rollback that has to be real before the first node is touched.

Almost nothing about a generation change is the GPU. The facility usually cannot take the new part in the old positions, the fabric generation may not match, and the two fleets have to coexist for months. The sequencing that avoids a capacity trough, and the decision people get wrong.

A check that is too strict removes healthy capacity and one that is too loose lets a broken node take a job down. The three tiers by cost, the fleet-median comparison that catches degradation no threshold does, and the false-positive budget that has to be set before any of it runs.

Two failures and a pass is not a passing node, it is a node with a demonstrated intermittent fault and a small sample. The arithmetic that shows how weak the evidence for acceptance is, what an intermittent fault costs once it is in a training job, and the policy that removes the argument.

Doubling per-rack density is four separate upgrades with different lead times, and the electrical one usually cannot be done while the hall runs. The phasing that keeps capacity available through it, the constraint that decides whether it is possible at all, and the number that says whether to upgrade or move.

Firmware skew produces intermittent faults that surface layers away from their cause, so an inventory is a debugging tool rather than a compliance exercise. What to collect, how drift appears without anyone deploying anything, and the diff that turns a week-long investigation into a query.

Both run at the same port speed, so the discount is not buying slower hardware. It is buying a different operational model and a different failure mode, and whether that is worth 15 percent depends on what your team already runs and what the fabric has to carry.

A whole rack failing at once is a shared dependency, which narrows the causes to four before anyone touches anything. What the out-of-band network tells you in the first minute, why the order of restoration matters, and the decision about liquid cooling that has to be made before power returns.

Sixteen bytes per parameter is the bill for mixed-precision Adam. ZeRO pays it in three installments: optimizer state, then gradients, then the weights themselves. The per-rank bytes for a 70B at 8, 16, 32 and 64 GPUs, and what each stage adds to the wire.

Same algorithm, two implementations, one of which now lives inside PyTorch and composes with everything else there. The differences that decide it in 2026: per-parameter sharding, DTensor, torch.compile, and the two features DeepSpeed still owns outright.

Meta's Llama 3 report counted 419 unexpected interruptions in 54 days on 16,384 GPUs, one every three hours. The failure-rate arithmetic, the loss as a function of the interval, the square-root formula that minimizes it, and why the write must be sharded and asynchronous first.

Every rank writes its own shard, a metadata file maps every chunk to a global tensor, and a loader intersects the old layout with the new. The bytes per rank for a 405B, the manifest that makes the write atomic, the metadata that makes resharding possible, and the host-memory staging that keeps the GPUs running.

Sixteen bytes per parameter is 6.5 TB per checkpoint; over a 100 GB/s filesystem that is about a minute, which at a 30-minute cadence is 3% of the run. The chain, the storage tiers that meet it, and why the write is made asynchronous.

700 W per GPU is the number everyone knows and it is only 40% of the answer. The chain from GPU TDP to node, to IT load, to facility power with PUE: about 1.8 MW, and what that means in racks and dollars.

Port counts decide everything: a two-tier Clos of 64-port switches tops out at exactly 4,096 endpoints at full bisection, which is why that number appears in so many cluster designs. The switch and optics count both ways, what oversubscription actually slows down, and where the money goes.

Six and a half terabytes leaving sixteen thousand GPUs at once is a burst no shared filesystem absorbs, and the fix is to stop trying. What the barrier actually has to wait for, the per-node arithmetic that makes it seconds, and the manifest rule that decides whether a checkpoint is usable at all.

Past the two-tier limit there are two ways forward and they are not close. One triples the switch count and adds fifty percent more optics for two extra hops of latency; the other is the same shape with bigger switches. The counts for both, and the reason the industry follows switch radix.

The exporter offers hundreds of fields and about a dozen of them change decisions. The health fields that predict failure, the performance fields that tell you whether work is happening, the one everybody alerts on that means nothing, and the diagnostic levels with the time each takes.

Three quantities decide it: how often the job is interrupted, how long a checkpoint takes, and how long a restart takes. The formula that turns them into an optimal interval, the goodput it leaves, and the measured result showing that making checkpoints faster is worth more than checkpointing more often.

Power rises faster than clock speed does, so giving up thirty percent of the power costs closer to eleven percent of the throughput. The relationship behind that, the uniformity requirement that matters more than the level, and why an uncoordinated cap is worse than a deeper coordinated one.

A cold plate holds a few tens of grams of water against seven hundred watts, so temperature rises about two degrees a second and the limit arrives in under twenty. What that timescale rules out, the protection layers that act inside it, and the choice that decides whether you lose a job or the hardware.

Three separate reserves get merged into one number and then argued about. The repair pipeline from Little's law, the restart pool that has to be instantly available, and the correlated-failure buffer sized by the largest thing that can fail at once, each derived and then added.

Published data gives an interruption rate, and replacements are a fraction of interruptions, so the comparison needs care before it means anything. The derivation that turns one into the other, the answer for a healthy fleet, and the four cuts through the data that turn an anomalous number into a cause.

The answer that fails proposes changes in week one. The answer that lands reads the postmortems first, draws the map from the job's point of view, ships one small fix, and ends the month with a list of questions nobody on the team could answer.

The interesting content is the order things broke in, because that order is a property of the system and it repeats. What breaks first at each scale step, why the control plane usually goes before the data plane, and the thing you would skip if you did it again.

Not depth, which is where most people push and where the returns flatten. What changes is the size of the problem you are handed, how ambiguous it is when you get it, and whether things happen because you asked rather than because you built them.

Bad news delivered as a problem gets you a meeting. Delivered as a forecast with options, costs and a decision date, it gets a decision. The four-part structure, the projection that has to be defensible, and the mistake of waiting until you are certain.

Not the data, not the weights: the gradients, once per step, in a ring that moves almost twice the model's size through every GPU. The derivation, the per-step byte count for an 8B model, and why it still hides behind the backward pass.

Three ways to split a training job, one table, and the rule that places each of them: activations on NVLink, gradients on the fabric, stage boundaries in between. With the byte counts that justify the placement.

The one utilization number that cannot be gamed by recompute: model FLOPs over hardware peak, derived from a step time in four lines, with the band frontier labs land in and an itemized list of where the other 60% goes.

Sixteen bytes per parameter says 1.6 TB of state against a 640 GB node, so the model spans nodes before the first token. The arithmetic that sizes the fleet, the layout that puts each axis on the right link, and the two numbers that decide FSDP against pipeline across nodes.

Two phases, n minus one steps each, one slice per step: the derivation that gives 2(n−1)/n times the buffer per rank, the argument that no algorithm can beat it on bytes, and the latency term that makes NCCL switch to trees at scale.

With p stages and m micro-batches the pipeline idles (p−1)/m of its compute time, and the obvious fix, more micro-batches, buys back throughput with activation memory. 1F1B caps that memory at p micro-batches instead of m. The derivation, the numbers for PP16, and the schedules that go further.

Four all-reduces per transformer block, on activations, on the critical path. Inside the NVLink domain they cost 8% of a step; across the NIC they cost more than the step itself. The derivation for a 70B, and the second reason TP stops that has nothing to do with the network.

Mathematically identical for a mean loss and a stateless model, and different in three ways that matter to infrastructure: activation memory, communication frequency, and any layer that looks at the batch. With the byte counts and the DDP call that makes it work.