Lambda AI Infrastructure Engineer interview questions

The H200 has the same compute die as the H100 and costs more per hour. The datasheet gives two ratios, 1.43x bandwidth and 1.76x memory, and those two numbers decide exactly which workloads pay back the premium and which ones lose money on it.

An H100 SXM moves 900 GB/s over NVLink; a PCIe card moves about 64 GB/s per direction over its slot. Whether that 7x to 14x gap costs you anything depends on how many bytes your parallelism strategy sends per step, and the answer is very different for tensor and data parallelism.

512 GPUs at 400 Gb/s each is 205 Tb/s of injection bandwidth a two-tier fabric must carry with no oversubscribed link. The port arithmetic that gives 16 leaves and 8 spines, the rail wiring that keeps NCCL traffic one hop away, and the line where 900 GB/s of NVLink becomes 50 GB/s of InfiniBand.

On an HBM part ECC costs almost nothing; on a GDDR part about 6% of capacity and bandwidth. What it buys is the difference between a corrected bit and a silently flipped exponent that turns 1.0 into infinity across a 512-GPU all-reduce. The mechanisms, the signals, and the expected-loss arithmetic.

A two-socket GPU node has two PCIe trees, two memory pools and one inter-socket link that every wrong-sided transfer must cross. A dataloader on the far socket shares that link with seven other GPUs' traffic, and its copies run at half speed. The topology map, the arithmetic and the pinning.

Start from the power envelope, because it decides the GPU count more often than the budget does. The scalable unit that makes the design a copy of a known-good one, the four networks, and the three line items that gate the schedule while everyone is discussing GPUs.

Links that never come up and links that come up and flap are different faults with different causes. The reach change between 400G and 800G explains most first-time failures, and the counters separate the rest. The order to check, and the one physical measurement that settles it.

The GPU board power is about 60 percent of the answer. The rest is CPUs, memory, NICs, fans, switches, thousands of transceivers and the cooling plant, and each is a term people leave out. The full derivation, the term that surprises everyone, and the contracted number that is larger than all of it.

Each layer depends on the ones below it and a mistake at the bottom shows up six layers higher as something else entirely. The order, the dependency almost everyone gets wrong on Blackwell systems, and the validation ladder that keeps a fault attributable to one node instead of five hundred.

The part with no NVLink can be the correct choice, and the reason is the workload shape rather than the specification. What fits on one card, what MIG partitioning buys a multi-tenant service, and the exact point where the missing scale-up link makes the decision flip.

A training cluster is a synchronized load, so the peak is far above the average and the average is what everyone budgeted. The immediate mitigation takes minutes and costs a few percent of throughput, and the four durable fixes trade cost against throughput in different places.

Single-node fine and two-node halved is a specific signature, and it points at one thing before anything else. Why the copy path costs exactly about half, the five causes that produce it, and the one log line that confirms which.

The endpoint count is larger than the node count and that surprises everyone building their first cluster. What has to be reachable, why this network must survive when every other one is down, and the security posture it needs because it can power-cycle the entire fleet.

Three approaches with overlapping ranges, and the choice is decided by the facility and the service model rather than by thermodynamics alone. Where each runs out, what the middle option buys that people underrate, and the operational cost that comes with the most capable one.

The spares number falls out of the failure rate times the repair turnaround, and the cheap items are the ones people forget. The arithmetic for GPUs, the very different arithmetic for transceivers, and the rule that decides whether to hold a part at all.

Most architecture diagrams show one network and a cluster has four. Each has a different traffic pattern, a different failure consequence and a different port count, and the one that is under-counted every time is the one you need when everything else is broken.

The break-even is a utilization number, not a price comparison, and most teams overestimate the utilization they will actually reach. The full cost of owning, the number that decides it, and the two situations where renting wins even at high utilization.

It configures the NVSwitch fabric so eight GPUs present as one NVLink domain, and when it is absent nothing errors. Jobs run, results are correct, and everything multi-GPU is slower by a factor the team spends weeks not finding. The check that takes one command.

Isolation has to be physical where it matters and logical where it can be, and picking the boundary wrong is either expensive or a security problem. What partitions cleanly, what does not, and the allocation unit that decides fragmentation and margin.

A fleet-wide version change is the most likely cause of the next unexplained performance regression, so the rollout is designed to make that attributable. Cohorts, a canary that measures rather than boots, and the rollback that has to be real before the first node is touched.

Peak numbers are almost never the reason a part wins or loses, and the software stack decides more deployments than the silicon does. The four things to measure, the question that eliminates most challengers in one conversation, and the pilot that gives a defensible answer.

Two failures and a pass is not a passing node, it is a node with a demonstrated intermittent fault and a small sample. The arithmetic that shows how weak the evidence for acceptance is, what an intermittent fault costs once it is in a training job, and the policy that removes the argument.

The hardware is identical and almost everything around it moves: who can touch a node, how long a swap takes, what the contract actually guarantees, and what you can no longer measure. The six things to negotiate before signing and the two that decide whether operations work.

Firmware skew produces intermittent faults that surface layers away from their cause, so an inventory is a debugging tool rather than a compliance exercise. What to collect, how drift appears without anyone deploying anything, and the diff that turns a week-long investigation into a query.

Older parts are rarely worthless and often better suited to work the new fleet should not be doing. The four options ranked by what they return, the data handling that has to happen before any hardware leaves, and the reason the answer is usually not one option.

Chunk the work, put copies and kernels on different streams, and the total stops being copy time plus compute time. The pipeline arithmetic that says how many chunks are enough, the three conditions without which the calls run in sequence anyway, and the two levels of asynchrony that are often confused.

Capex plus two years of power and operations, minus what the cards are worth at the end, against a rental rate at a utilization. The chain that produces a break-even utilization, and the two inputs that swing it from 60% to over 100%.

6ND to a compute budget, the fleet equation to a GPU count, the rental rate to a bill: 64 H100s, about 19 days, roughly $75k plus a buffer. The chain, the reasons the answer is 64 and not 59, and the memory check that says the fleet is compute-sized, not memory-sized.

Find the scale at which it breaks before looking at any node, because the scale tells you which layer to open. The commands in order, the number each should return on a healthy fabric, and the four causes that this measurement sequence separates without guessing.

Port counts decide everything: a two-tier Clos of 64-port switches tops out at exactly 4,096 endpoints at full bisection, which is why that number appears in so many cluster designs. The switch and optics count both ways, what oversubscription actually slows down, and where the money goes.

Three tools measure three different things, and a link can pass one while failing another. The message-size sweep that separates a latency problem from a bandwidth problem, the number each test should return on a healthy 400 gigabit port, and the single test that catches the failure the others miss.

It fails open: the data still moves, the job still runs, and everything is about half as fast. The one log line that settles it, the four reasons the library declines to use it, and why the bandwidth signature is close enough to exactly half to be diagnostic on its own.

Spot is 60% cheaper until the preemptions eat the discount. The expected-loss formula (waste per preemption = half the checkpoint interval plus restart), the break-even worked at three cadences, the checkpoint cost that pushes back the other way, and the gang arithmetic that says which jobs must never be on spot.

Committed capacity is cheaper per hour and cannot shrink, so the decision is a utilization break-even, not a discount. The arithmetic for where the reserved baseline should sit, why the p50 of steady demand is the target, and how on-demand, spot and capacity blocks absorb everything above it.

A pre-flight suite is a list of tests with a number on each: DCGM health, a GEMM near the fleet median, NCCL at rated bandwidth, NICs at line rate, links active, storage reachable. What goes in the two-minute gate, what waits for the long diagnostic, and why the gate pays for itself.

Slurm's GPU path is four pieces: GRES declares the devices, a partition groups nodes with limits, the batch script asks for a shape, and cgroups confine the job to its GPUs. The script that gets eight GPUs on one node, the flags that scatter them across eight nodes, and how ranks find their GPU.

A driver upgrade touches every layer at once: kernel module, CUDA runtime, container toolkit, NCCL, fabric driver, and every job's image. The compatibility matrix that decides whether a job can run on the new node, the canary that proves it, the wave arithmetic for 2,000 nodes, and the rehearsed rollback.

A published run gives the numbers directly: 419 unexpected interruptions in 54 days on 16,384 GPUs, and about three quarters of them hardware. What that implies per GPU-hour, which components dominate, and why a fleet above a few thousand GPUs must treat failure as routine rather than exceptional.

The driver logs a numbered code when something goes wrong on a GPU, and the number tells you whether to retry the job, drain the node or file a hardware return. The codes worth memorizing in three groups, the action each implies, and the automation that turns a log line into a drained node.

The device has stopped answering on PCIe, so the driver cannot reset it and no software fix exists: the host has to re-enumerate the bus. What causes it, why the node is out until it reboots, the three-strike policy that separates a transient from a dying card, and the automation that handles it without a human.

Five causes produce the same symptom and each has a check that takes under a minute. The order to run them in, what each one eliminates, and the arithmetic showing that one slow node in a thousand costs the whole job its speed rather than a thousandth of it.

Infant mortality is real and diagnostics that run for a minute do not find it. The five-stage acceptance suite with what each stage catches and how long it takes, the pass criteria as numbers rather than judgments, and the sustained-load stage that finds the faults everything else misses.

Every replacement costs a spare, a maintenance window and a return process; every device kept costs the risk of a job it will fail. Four triggers that decide it without a case-by-case argument, the arithmetic that sets the repeat threshold, and why the device's history beats any diagnostic result.

A GPU cloud sells whole nodes and fabric slices to tenants who expect them in minutes and expect them to work. The node state machine, the reconciler that keeps thousands converging, the isolation lines on shared fabric, the health loop that pulls a bad node first, and billing that survives a control-plane outage.

You cannot win this with an opinion, and a flat objection to a purchase reads as an engineer who does not understand the business. The utilization evidence that carries it, the smaller counter-proposal that usually wins, and what to do once the decision goes against you.

Arguing is the losing move and so is capitulating. What moves a vendor case is a reproducer small enough for their engineer to run, counters that isolate their layer from yours, and a bisection that removes your configuration as a variable entirely.

The interviewer is checking whether you know what you are trading, because both directions are reasonable and both have a version that ends badly. What each environment actually gives you, and the answer that works when the real reason is compensation.

The first number every serving interview asks for, derived from parameter count and bytes per parameter, in three precisions, with the part people forget to add.

Four numbers from the config file, one formula, and a per-sequence result that is a third of the model's own weights. Plus the mistake that makes the answer eight times too big.

The 6ND rule, where the 6 comes from, and the 6.3e24 that every training-time and GPU-count question in the loop is built on.

The fleet equation applied to 6.35e24 FLOPs: 11 days, and how the answer swings from 9 to 15 with the one parameter the interviewer wants you to state.

The fleet equation solved for GPU count: about 6,200 H100s at 40% MFU, why it rounds up to a power of two, and the reasons a good answer adds 15% before naming a number.

Why decode speed is a division of bandwidth by weight bytes, why the answer is about 24 tokens a second regardless of how fast the tensor cores are, and the two ways to double it.

Fleet dollars per second over tokens per second, times a million: a worked $2.78, the utilization term that turns it into $6.94, and why batch is the lever that moves it by 50x.

Sixteen bytes per parameter is 1.1 TB of static state before a single activation; LoRA keeps the 141 GB of frozen weights and shrinks the rest to a few gigabytes. The derivation, the GPU counts, and the activations that both still pay.