Meta AI Infrastructure Engineer interview questions

Every format is a trade between how big a number can be and how finely it is spaced. The bit layouts, the largest and smallest values derived from them, why fp16 needed loss scaling and bf16 did not, and why fp8 comes in two flavors with a scale factor attached.

Start from the power envelope, because it decides the GPU count more often than the budget does. The scalable unit that makes the design a copy of a known-good one, the four networks, and the three line items that gate the schedule while everyone is discussing GPUs.

Four separate facility limits have to be cleared and total building capacity is not one of them. Power per rack, heat rejection, floor loading and network reach each fail independently, and each has a different remedy with a different lead time. The options that exist, priced, and the one that is usually taken.

Links that never come up and links that come up and flap are different faults with different causes. The reach change between 400G and 800G explains most first-time failures, and the counters separate the rest. The order to check, and the one physical measurement that settles it.

Every watt in comes out as heat, so the design starts from the load and one equation. The flow rate, the CDU count and redundancy, the water quality requirements that decide whether the loop survives three years, and the automatic response that has to fire before a human can read the page.

A rail-optimized fabric makes this fault findable because the eight rails are identical by construction, so any asymmetry is a defect. Bisecting by rail, then by switch, then by node, then by port narrows 512 links to one in four steps, and the counters say which of three physical causes it is.

Each layer depends on the ones below it and a mistake at the bottom shows up six layers higher as something else entirely. The order, the dependency almost everyone gets wrong on Blackwell systems, and the validation ladder that keeps a fault attributable to one node instead of five hundred.

Acceptance is a contract document, not a checklist, so every criterion has to be a number with a tolerance that a vendor can be held to. Comparing nodes against each other rather than against a datasheet is what finds real defects, and the duration is what finds the thermal ones.

The count follows from three inputs and one topology rule, and getting it wrong by the leaf-to-spine half is the classic error. The arithmetic, the radix check that says two tiers is enough, and the power the transceivers add that nobody put in the budget.

Single-node fine and two-node halved is a specific signature, and it points at one thing before anything else. Why the copy path costs exactly about half, the five causes that produce it, and the one log line that confirms which.

Capacity is the easy half and almost never the constraint. The number that sizes the system is the checkpoint write burst, which is thousands of times the steady dataset read, and the tier that serves each is different. The full derivation, and the tiering that avoids paying burst prices for archive bytes.

The GPUs are almost never the critical path and planning backward from their delivery date is how deployments slip. The real ordering, what can run in parallel, the two dependencies that cannot, and the cost of a month of delay expressed in the unit that gets attention.

The spares number falls out of the failure rate times the repair turnaround, and the cheap items are the ones people forget. The arithmetic for GPUs, the very different arithmetic for transceivers, and the rule that decides whether to hold a part at all.

A 44 percent failure rate is far above any plausible hardware defect rate, so the first hypothesis is that the test or the environment is wrong rather than the hardware. The three checks that separate those, the cohort analysis that localizes a real defect, and how to run the vendor conversation.

Switch radix decides it, and at 10,000 endpoints the answer sits right at the boundary for current high-radix switches. The arithmetic, what a third tier costs in switches, optics and power, and the two designs that avoid it entirely.

Same GPU count, same silicon, and a completely different machine. The domain size is the whole argument, it is worth an order of magnitude on specific traffic, and it costs a failure domain nine times larger plus a facility that most halls do not have.

A fixed power envelope turns hardware selection into an optimization with one objective and several overheads competing for the same watts. Where the 1 MW actually goes, why the answer is not simply the most efficient GPU, and the two overheads that can be cut to buy more compute.

It configures the NVSwitch fabric so eight GPUs present as one NVLink domain, and when it is absent nothing errors. Jobs run, results are correct, and everything multi-GPU is slower by a factor the team spends weeks not finding. The check that takes one command.

The rule is one sentence and the execution is thousands of connections where a single transposition creates a fault that is invisible until a collective runs. The mapping, the labelling scheme that makes errors findable, and the verification that has to happen before anyone accepts the cluster.

Acceptance measures the hardware and production measures the whole system, so the gap lives in everything acceptance did not cover. Four candidate layers, the order to eliminate them, and the two that account for most of these cases.

A fleet-wide version change is the most likely cause of the next unexplained performance regression, so the rollout is designed to make that attributable. Cohorts, a canary that measures rather than boots, and the rollback that has to be real before the first node is touched.

Almost nothing about a generation change is the GPU. The facility usually cannot take the new part in the old positions, the fabric generation may not match, and the two fleets have to coexist for months. The sequencing that avoids a capacity trough, and the decision people get wrong.

Peak numbers are almost never the reason a part wins or loses, and the software stack decides more deployments than the silicon does. The four things to measure, the question that eliminates most challengers in one conversation, and the pilot that gives a defensible answer.

A check that is too strict removes healthy capacity and one that is too loose lets a broken node take a job down. The three tiers by cost, the fleet-median comparison that catches degradation no threshold does, and the false-positive budget that has to be set before any of it runs.

Two failures and a pass is not a passing node, it is a node with a demonstrated intermittent fault and a small sample. The arithmetic that shows how weak the evidence for acceptance is, what an intermittent fault costs once it is in a training job, and the policy that removes the argument.

Doubling per-rack density is four separate upgrades with different lead times, and the electrical one usually cannot be done while the hall runs. The phasing that keeps capacity available through it, the constraint that decides whether it is possible at all, and the number that says whether to upgrade or move.

Modules do not fail randomly at three times a published rate, so the excess has a cause and it is usually thermal or a batch. The cohort cuts that localize it, the counters that separate a dying module from a dirty connector, and the environmental cause that looks like a hardware defect.

Firmware skew produces intermittent faults that surface layers away from their cause, so an inventory is a debugging tool rather than a compliance exercise. What to collect, how drift appears without anyone deploying anything, and the diff that turns a week-long investigation into a query.

Both run at the same port speed, so the discount is not buying slower hardware. It is buying a different operational model and a different failure mode, and whether that is worth 15 percent depends on what your team already runs and what the fabric has to carry.

Older parts are rarely worthless and often better suited to work the new fleet should not be doing. The four options ranked by what they return, the data handling that has to happen before any hardware leaves, and the reason the answer is usually not one option.

A whole rack failing at once is a shared dependency, which narrows the causes to four before anyone touches anything. What the out-of-band network tells you in the first minute, why the order of restoration matters, and the decision about liquid cooling that has to be made before power returns.

Four stages: a bytecode interpreter captures a graph with guards, an autograd pass splits forward from backward, a compiler emits Triton for everything but the matmuls, and a mode that replays the step as one launch. What each buys, and the two failures that silently give it back.

A float format's resolution is relative, so the damage depends on how large a running total grows against the terms being added. The measured error for a 4,096-term dot product in each direction, the weight update that vanishes entirely, and why bf16 removed loss scaling but not master weights.

Compute the utilization first so you know how much time is unaccounted for, then read one timeline in a fixed order: GPU idle, then what the host was doing in the gaps, then whether the gradient all-reduce overlapped the backward pass, then the optimizer. Four causes, the evidence for each, and what each is worth.

Not the data, not the weights: the gradients, once per step, in a ring that moves almost twice the model's size through every GPU. The derivation, the per-step byte count for an 8B model, and why it still hides behind the backward pass.

Three ways to split a training job, one table, and the rule that places each of them: activations on NVLink, gradients on the fabric, stage boundaries in between. With the byte counts that justify the placement.

The one utilization number that cannot be gamed by recompute: model FLOPs over hardware peak, derived from a step time in four lines, with the band frontier labs land in and an itemized list of where the other 60% goes.

Sixteen bytes per parameter is the bill for mixed-precision Adam. ZeRO pays it in three installments: optimizer state, then gradients, then the weights themselves. The per-rank bytes for a 70B at 8, 16, 32 and 64 GPUs, and what each stage adds to the wire.

Same algorithm, two implementations, one of which now lives inside PyTorch and composes with everything else there. The differences that decide it in 2026: per-parameter sharding, DTensor, torch.compile, and the two features DeepSpeed still owns outright.

With p stages and m micro-batches the pipeline idles (p−1)/m of its compute time, and the obvious fix, more micro-batches, buys back throughput with activation memory. 1F1B caps that memory at p micro-batches instead of m. The derivation, the numbers for PP16, and the schedules that go further.

Mathematically identical for a mean loss and a stateless model, and different in three ways that matter to infrastructure: activation memory, communication frequency, and any layer that looks at the batch. With the byte counts and the DDP call that makes it work.

Full recomputation costs a third more compute to cut activation memory by an order of magnitude. The per-layer arithmetic that says whether you need it at all, the selective variant that costs almost nothing, and the point at which a smaller micro-batch is the cheaper way out.

At 128k tokens one layer's activations are 73 GB per sequence and attention grows with the square of the length. The arithmetic that forces the sequence onto sixteen GPUs, what a ring of KV chunks costs per layer, why the communication hides, and where Ulysses and Megatron sequence parallelism fit around it.

Every MoE layer routes each token to eight experts that live on other GPUs, twice per layer, forward and back. The bytes per rank per layer for DeepSeek-V3, why that all-to-all runs close to one-to-one against compute, and why a single hot expert stalls every rank waiting at the combine.

Tensor parallel 8, pipeline 16, data parallel 128: the layout Meta trained Llama 3 405B on, rebuilt from the memory floor, the compute budget and the bandwidth of each link. Why the axes go in that order, what each one costs per step, and the bubble that the schedule has to hide.

fp16 loses gradients below 6e-8 and overflows above 65,504; bf16 has fp32's range and three digits of precision. The arithmetic that shows why an update of 3e-5 vanishes into a bf16 weight, where each of the 16 bytes per parameter comes from, and what fp8 block scaling adds on top.

Meta's Llama 3 report counted 419 unexpected interruptions in 54 days on 16,384 GPUs, one every three hours. The failure-rate arithmetic, the loss as a function of the interval, the square-root formula that minimizes it, and why the write must be sharded and asynchronous first.

A dead node at 4 a.m. costs twenty minutes if a human restarts it and under five if the system does. The four stages of an automatic recovery, the reason a lost node takes a whole pipeline replica with it, why hot spares beat resharding, and the minutes that each stage still costs even when everything works.

Every collective ends when the last rank arrives, so a single throttled GPU taxes 16,383 others. The arithmetic of that tax, the per-rank timing that finds the rank in one step, and the ranked list of causes from a hot GPU to a NIC with symbol errors, each with the command that confirms it.

The watchdog says an all-reduce ran for ten minutes and never finished. That means one rank never entered it, entered a different one, or died. How to read the log line, the flight recorder that names the rank in seconds, the env vars that make the next one debuggable, and why the timeout is a symptom every time.

Bucketed all-reduce in backward, FSDP's one-layer-ahead prefetch and split-GEMM tensor parallelism hide most of the bytes, and none of it works below a certain number of tokens per GPU. The ratio that decides it, the SMs the NCCL kernels take, and the tail that is never hidden.

Every rank writes its own shard, a metadata file maps every chunk to a global tensor, and a loader intersects the old layout with the new. The bytes per rank for a 405B, the manifest that makes the write atomic, the metadata that makes resharding possible, and the host-memory staging that keeps the GPUs running.

Two restarts a day at fifteen minutes each is the 2%. The loss decomposed into detection, rescheduling, reload and rewound work, the lever on each, the in-memory checkpoint that makes the interval a minute, the square-root pareto of interval against write cost, and the residual that only fewer failures can remove.

A 400B dense model on 10 trillion tokens is 2.4 × 10²⁵ FLOPs, 43 days of pure compute on 16,384 H100s, and about 380 interruptions along the way. The order in which to derive every number, the layout and the data rate, the checkpoint and failure budgets, and the schedule that survives its own arithmetic.

Same model size, three attention layouts, a 30x spread in cache per token. The formulas are short, and they explain why DeepSeek can hold four times the context of a dense 70B on the same cards.

The 6ND rule, where the 6 comes from, and the 6.3e24 that every training-time and GPU-count question in the loop is built on.

The fleet equation applied to 6.35e24 FLOPs: 11 days, and how the answer swings from 9 to 15 with the one parameter the interviewer wants you to state.

Gradient bytes per step over step time, with the ring all-reduce factor that nearly doubles it: the arithmetic for a 7B and a 70B, the link speeds that cover it, and the step-time threshold where overlap stops hiding it.

6ND on 15.6 trillion tokens gives 3.8e25 FLOPs; converting to GPU-hours needs an MFU, and Meta's published 30.8 million hours lets you solve for it. The chain, the check, and what the implied MFU tells you.

Sixteen bytes per parameter is 6.5 TB per checkpoint; over a 100 GB/s filesystem that is about a minute, which at a 30-minute cadence is 3% of the run. The chain, the storage tiers that meet it, and why the write is made asynchronous.

The ring all-reduce formula, 2(n−1)/n times the buffer over the link bandwidth, applied to 141 GB of bf16 gradients on an NVLink node: 0.27 s by the formula, and the two reasons the measured number is different.

The token stream is tens of megabytes a second and the checkpoints are tens of gigabytes a second; the two workloads differ by a thousand and want different tiers. The chain from tokens per second to bytes, the checkpoint burst, and the restore that sets the real spec.

Dollars per FLOP from the GPU price and the MFU, times 6N: a 70B training token costs about three quarters of a microdollar, and the whole 15T-token run follows in one multiplication. The chain, the comparison to an inference token, and why the training token is cheaper.

Activations are the term the 16-bytes-per-parameter rule leaves out, and at 8k context they are bigger than the weights: about 34 × sequence × hidden bytes per layer, 2.3 GB, 182 GB per micro-batch across 80 layers. The chain, what FlashAttention removes, and what checkpointing does to the number and the compute.

Before the first byte moves, the library has already discovered the machine's topology, built rings and trees over it, split the work into parallel channels and chosen a wire protocol by message size. What each of those four decisions is for, and the environment variables that let you see and override them.

Both carry RDMA at the same line rate. One arrives lossless because the transport was designed that way, the other becomes lossless only if a set of switch settings is correct on every port. What that difference costs in operations, what it saves in money and hiring, and the fleet size where the answer flips.

Wire GPU number three of every node to the same switch and the collective that matters most never leaves it. The rank mapping that makes the layout pay, the switch and cable count for a thousand GPUs worked out, and what happens to a job whose ranks are assigned in the wrong order.

Thirty-two senders aimed at one port fill a switch buffer in microseconds, and what happens next depends on which of two mechanisms fires first. The buffer arithmetic, why the slow mechanism is the safe one and the fast one is the dangerous one, and the counters that tell you which is happening.

One gives you POSIX semantics and hundreds of gigabytes per second at a price per terabyte that hurts at petabyte scale. The other is cheap and durable and answers a first byte in tens of milliseconds. The cost arithmetic that makes most clusters run both, and the metadata limit that decides how you shard.

For text training the bytes are trivial and the bottleneck is never bandwidth, so the usual advice about faster storage misses. What the loader actually has to deliver per second, the four things that consume the time instead, and the order to fix them with the measurement that proves each.

A tokenizer moves about a megabyte of text per second per core, which makes this a seven-hundred-core-day batch job rather than something to run during training. The throughput arithmetic per stage, the shard format the loader needs, and the determinism requirements that let you resume without corrupting a run.

Six and a half terabytes leaving sixteen thousand GPUs at once is a burst no shared filesystem absorbs, and the fix is to stop trying. What the barrier actually has to wait for, the per-node arithmetic that makes it seconds, and the manifest rule that decides whether a checkpoint is usable at all.

Three parallelism dimensions, two link speeds eighteen times apart, and one mapping that makes the job fast. The traffic each dimension generates per step, why the loudest one must stay inside the node, and the arithmetic showing what a wrong rank order costs before anyone notices.

The number comes from one requirement: the gradient reduction has to finish inside the backward pass that produces it. Working that requirement backward gives a bandwidth per GPU, and the answer lands near the port speed the industry ships, which is not a coincidence.

RoCE inherited a transport that assumes one ordered path per connection, which is why it needs a lossless network built by hand underneath it. What a redesigned transport changes about that, which of today's operational burdens it removes, and why the honest answer to a purchase decision is still about what ships.

A collective stops when one participant stops, and a job of a thousand ranks has no way to distinguish a link that will come back in two seconds from a rank that has died. What the job actually does during those seconds, the cost per flap, and why the fault passes every test you would run.

Exact duplicates are a hash and a group-by. Near-duplicates are a similarity search over billions of documents, which becomes a shuffle rather than a computation. The signature arithmetic, why the shuffle is the expensive stage, and where a GPU classifier fits in a pipeline that is otherwise all CPU.

The signature is unusual: many jobs degrade together, throughput collapses, and the drop counters stay at zero. Reading pause counters backward along the paths to find the origin, the three conditions that produce one, and the configuration change that stops it happening again.

Past the two-tier limit there are two ways forward and they are not close. One triples the switch count and adds fifty percent more optics for two extra hops of latency; the other is the same shape with bigger switches. The counts for both, and the reason the industry follows switch radix.

A uniform loss across every node rules out hardware and points at something that changed everywhere at once. The differential procedure that isolates a reset default from a real regression, the settings firmware resets on its own, and the change process that catches it in an hour.

Static quotas waste half the fleet and a free-for-all starves the small teams. The four-layer policy (guaranteed quota, borrowing, fair-share ordering, preemption) with the pooling arithmetic that justifies it, the fair-share ratio worked by hand, and the five ways teams game it.

You cannot remove one rank from a synchronous job; you replace the node and restart from checkpoint, and the whole craft is making that take three minutes instead of thirty. The detection signals, the drain-restart-quarantine sequence, the spare-pool arithmetic, and where elastic training changes the answer.

Allocated means a scheduler handed the GPU out; utilized means it did work. A 60-point gap is jobs holding GPUs they do not use: idle notebooks, placeholder jobs, a loader-bound run at 20% tensor-active. The three metrics that separate the causes, the per-tenant table that names them, and the policies that close it.

A published run gives the numbers directly: 419 unexpected interruptions in 54 days on 16,384 GPUs, and about three quarters of them hardware. What that implies per GPU-hour, which components dominate, and why a fleet above a few thousand GPUs must treat failure as routine rather than exceptional.

A training run has one number that matters and a handful that explain it. Goodput as the top-level metric, the per-rank timing that finds a straggler among a thousand, the hardware layer beneath it, and the three dashboards that serve three different people asking three different questions.

Correctable errors are corrected, so nothing is wrong yet, and that is exactly why they are the most useful signal a fleet has. What rising rates predict, the remapping budget that decides whether a device can heal itself, and the policy that converts a trend into a scheduled drain instead of a failed training run.

A hang is a collective that one rank never entered, and finding it means asking which rank is missing rather than what is broken. The flight recorder that answers that in seconds, what to do when it is not enabled, and the four causes that a missing rank turns out to be.

Three quantities decide it: how often the job is interrupted, how long a checkpoint takes, and how long a restart takes. The formula that turns them into an optimal interval, the goodput it leaves, and the measured result showing that making checkpoints faster is worth more than checkpointing more often.

Availability is the wrong promise for a platform whose jobs run for weeks. Four objectives that a researcher can actually plan against, how each is measured from data you already collect, the error budget that makes them enforceable, and the one number that should never be an objective.

A link that corrupts occasionally does not fail; it retransmits, and each retransmission costs bandwidth that shows up as a slow job with no error anywhere. The counters that reveal it, the arithmetic turning a replay rate into lost bandwidth, and the threshold where watching becomes draining.

The timeline is the easy part and the detection gap is the valuable part: not what broke, but how long it was broken before anyone knew and why. A worked example with its five sections, the distinction between the trigger and the cause, and the property that separates action items that ship from ones that do not.

The controller's real design problem is not detecting faults but deciding when a signal is strong enough to remove a node from a job. The six states and their transitions, the cost asymmetry that sets every threshold, and the arithmetic showing what a one-percent false-positive rate costs at fleet scale.

The number is easy to state and hard to make honest, because every minute has to be classified and the classification is where the value is. The definition that survives scrutiny, the five categories the missing time falls into, and the property that turns a metric into a work list.

Each rank keeps a ring buffer of its recent collectives, and on a timeout every rank dumps it. Comparing sequence numbers across a thousand of those buffers finds the missing participant in one pass, and the three patterns those comparisons produce point at three different causes.

Error correction catches memory faults and reports them. Nothing catches an arithmetic unit that occasionally returns a wrong product, which is why this class is found by comparing results rather than by reading counters. Three detection strategies with their costs, and the signature in a loss curve.

Every replacement costs a spare, a maintenance window and a return process; every device kept costs the risk of a job it will fail. Four triggers that decide it without a case-by-case argument, the arithmetic that sets the repeat threshold, and why the device's history beats any diagnostic result.

Three separate reserves get merged into one number and then argued about. The repair pipeline from Little's law, the restart pool that has to be instantly available, and the correlated-failure buffer sized by the largest thing that can fail at once, each derived and then added.

Published data gives an interruption rate, and replacements are a fraction of interruptions, so the comparison needs care before it means anything. The derivation that turns one into the other, the answer for a healthy fleet, and the four cuts through the data that turn an anomalous number into a cause.

Ten thousand GPUs is 1,280 nodes, ten pods, about 700 switches, a filesystem that swallows a 5 TB checkpoint in under a minute and a 14 MW feed. The bill of materials with a reason on every line, from the FLOPs the model needs to the spare pool the failure rate demands.

Fifteen trillion tokens starts as a few petabytes of raw text and ends as 30 TB of shards a training job reads at 1.7 million tokens per second. The stages, the bytes at each boundary, the throughput each one needs to finish in two weeks, and the two stages where the pipeline actually spends its time.

Every 30 minutes a thousand GPUs write a terabyte in a burst that must finish in a minute, then nothing until the next. The burst arithmetic that sizes the write tier, the two-stage path to durable storage, retention that keeps the right checkpoints, resharding so a 512-GPU checkpoint resumes on 256, and lineage.

A research cluster is shared by teams whose managers bought it and by researchers who want it now. A quota tree with guarantees and borrowing, a fair-share formula that decays so last week's usage does not cost you today, preemption that is cheap because checkpoints are, and the incentives that make hoarding lose.

A three-month run on 8,000 GPUs is a large bet against everything that can happen to one building. The recovery point and the recovery time as numbers, the checkpoint replication that sets the first, the second cluster and its warm state that set the second, and the drills that make the numbers true.

A few lines of comparison, and every difficulty is in the threshold and the clock. Why the collector's receive time rather than the sender's, the boundary case that decides whether an exactly-late rank is reported, and the arithmetic that turns a false-positive rate into a threshold.

Fair sharing of one resource is obvious and fair sharing of three is not, because a GPU-heavy job and a memory-heavy job are not comparable. The rule that makes them comparable, the greedy loop that implements it, and the tie-breaking bug that gave one user everything.

Full GPU utilization during a hang is the clue, because a spinning collective looks identical to real work. The isolation order that finds the mismatched rank in minutes, a runnable reproducer that hangs on demand, and the fix that gates the logging rather than the collective.

The interviewer is not checking whether you can say no. They are checking whether your no came with a number, an alternative and a date. The four-part shape that makes this story land, a worked example with the arithmetic a real one carries, and the version that fails.

A war story with no learning in it scores zero, however dramatic. The five beats of an incident narrative, the decision under uncertainty that is the actual subject of the question, and how to tell it when the incident was somebody else's fault.

Saying no is easy and it costs you the relationship. The move that works is to make the queue visible, give the researcher something they control, and offer a smaller thing today. The three mechanisms that turn this from a recurring argument into a system.

Every migration story has the same technical answer and a different social one, and the social one is what is being scored. The sequencing that makes a migration reversible, the rollback that has to be real, and why the status page matters more than the tooling.

Everyone has this story and most candidates tell a flattering version of it. The bias that produces almost all of these mistakes, the cost you have to be willing to name, and the process change that is the difference between an anecdote and a lesson.

The answer that fails proposes changes in week one. The answer that lands reads the postmortems first, draws the map from the job's point of view, ships one small fix, and ends the month with a list of questions nobody on the team could answer.

A capacity fight is almost never about capacity. It is two teams with different information and no shared number. The move that ends it is publishing utilization both teams can see, and the escalation that works is a written proposal with options rather than a request for a ruling.

The interesting content is the order things broke in, because that order is a property of the system and it repeats. What breaks first at each scale step, why the control plane usually goes before the data plane, and the thing you would skip if you did it again.

You cannot win this with an opinion, and a flat objection to a purchase reads as an engineer who does not understand the business. The utilization evidence that carries it, the smaller counter-proposal that usually wins, and what to do once the decision goes against you.

Taking the keyboard teaches nothing and is the reflex under pressure. The handoff that works, the four questions that make someone's reasoning visible, and the point at which you stop asking questions and take over.

This is a test of whether you have a position you can defend, not of which problem you pick. What makes a thesis defensible, two worked examples with the arithmetic behind them, and the counterargument you have to be able to state before the interviewer does.

The question is about how you handle uncertainty, so a story where you gathered more data until you were sure answers a different question. Reversibility as the thing that sets the bar, the check-in you schedule when you decide, and what to say about the ones that went badly.

Being right is a small part of this. The move that works is converting an opinion into a document with the tradeoff written out, and then committing fully once the decision is made. The two cases where escalating is correct, and how to do it without a fight.

Not depth, which is where most people push and where the returns flatten. What changes is the size of the problem you are handed, how ambiguous it is when you get it, and whether things happen because you asked rather than because you built them.

Most platform tools are built for the platform team's model of the work rather than the work. What makes a tool get adopted, the measurement that proves it helped, and why the fast path for small jobs beats almost anything else you could build.

A confident plan built before you know anything is the failure mode, and so is refusing to answer until you have looked. The structure that handles both: a dated shape with the decision points named, one committed win, and the conditions that would change everything after day 30.

Bad news delivered as a problem gets you a meeting. Delivered as a forecast with options, costs and a decision date, it gets a decision. The four-part structure, the projection that has to be defensible, and the mistake of waiting until you are certain.

Four claims that are defensible from arithmetic available today, each with the counterargument that could sink it, and what each one implies about the work. Dated to 2026, because a thesis with no date is not a prediction.

Sixteen bytes per parameter says 1.6 TB of state against a 640 GB node, so the model spans nodes before the first token. The arithmetic that sizes the fleet, the layout that puts each axis on the right link, and the two numbers that decide FSDP against pipeline across nodes.

Two phases, n minus one steps each, one slice per step: the derivation that gives 2(n−1)/n times the buffer per rank, the argument that no algorithm can beat it on bytes, and the latency term that makes NCCL switch to trees at scale.

Four all-reduces per transformer block, on activations, on the critical path. Inside the NVLink domain they cost 8% of a step; across the NIC they cost more than the step itself. The derivation for a 70B, and the second reason TP stops that has nothing to do with the network.

A router that sends 90% of tokens to 26 of 256 experts makes those ranks do nine times their share while everyone else waits. The arithmetic of the imbalance, what a capacity factor drops on the floor, why the auxiliary loss fights the model, and the bias-only balancing DeepSeek-V3 used instead.

Compute the step time the formula predicts, measure the one you have, and the gap is the budget to explain. The isolation order with one metric per suspect: dataloader wait, SM and tensor-core activity, HBM throughput, time in collectives, and the per-rank spread that says it is one machine.

A loss that stops moving has five different causes and one of them is not a training problem at all. The checks in order: is the step counter moving, what are the learning rate and the grad norm doing, is the data repeating, has the loss scaler collapsed, and is one rank sending zeros.

On a TPU pod you write one program, annotate how each array is sharded across a named mesh, and the compiler inserts and schedules every collective. What that removes from the engineer's job, what the torus changes about layouts, where Pathways fits, and what a compiler-owned schedule costs.

LoRA shrinks the trainable state and leaves 141 GB of frozen weights untouched. QLoRA attacks the frozen half: NF4 takes it to 35 GB, double quantization compresses the scales themselves, paged optimizers absorb the spikes. The byte accounting, what dequantization costs, and where DoRA adds a term.