AI Infra Interviews logo
413 QUESTIONS · 150 CONCEPTS · 41 COMPANY LOOPS

Interview preparation forAI infrastructure engineers

The questions under the GPUs, with the numbers worked out. Kernels, distributed training, inference and serving, and the clusters that run them: 413 questions with worked answers, hand-built diagrams and the follow-ups interviewers hold back, from loops at NVIDIA, OpenAI, Anthropic, Google, CoreWeave, Together AI and Fireworks.

Start practicing free →See every question

no card · free answers in every topic · 41 company loops · 413 added or revised in the last 30 days

Live · will it fit?
full calculator →
KV per token
328 KB
Per sequence
10.7 GB
8 × H100 80GB SXM · 640 GB
40 concurrent sequences at 32k, after 141 GB of weights and 10% reserved

KV/token = 2 × 80 layers × 8 KV heads × 128 × 2 B

Which loop are you preparing for?

one title, four different interviews
KernelsKernel & performanceLive CUDA or Triton, a kernel take-home, the roofline, FlashAttention internals.NVIDIA CUDA teams, Fireworks, Together, lab performance teamsDistributedDistributed trainingParallelism strategy, collectives and fabrics, MFU, checkpointing, what breaks at 10k GPUs.Frontier labs, xAI, Crusoe, hyperscaler training teamsDesignInference & servingThe KV cache, batching, speculation, disaggregation, and the design round with TTFT on the board.OpenAI, Anthropic, Baseten, Together, Fireworks, Red HatPlatformPlatform & fleetKubernetes GPU scheduling, MIG, Slurm, DCGM and XID codes, the on-call reality.CoreWeave, Modal, Nebius, Lambda, ML platform teams

Not sure which? The guide sorts the tracks out, or take the 15-minute readiness check.

SIX FORMULAS EVERY LOOP EXPECTS COLDthe full sheet, with worked examples →
Training FLOPs6 × N × DKV cache / token2 × L × H_kv × d × bytesRidge pointpeak FLOPS ÷ bandwidthMFUachieved ÷ peakDecode tok/sbatch × BW ÷ bytes/step$ / M tokens$/s ÷ tok/s × 10⁶

Better than the answer a model gives you. That is the whole point.

You can get a definition from any chatbot. What you cannot get is the number worked through with its assumptions listed, the signal an on-call engineer actually reads, the decision table with the condition that flips it, and the follow-up the interviewer is holding in reserve. Every answer here carries at least three of those.

  • Worked numbers with stated assumptions, checked against the calculators.
  • Hand-built diagrams that show a mechanism: the ring, the bubble, the block table.
  • Real failure signals: the XID code, the DCGM field, the NCCL log line.
  • Decision tables with the reversal condition, not a menu of options.
  • The accelerator canon beyond NVIDIA: TPU, Trainium, MI300-class.
  • Every code snippet executed before it ships. No em dashes, no filler.

Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

Napkin math · what an answer looks like

Will Llama 3.1 70B serve 64 concurrent users at 32k context on 8 × H100? Show your work.

TL;DR: Not in bf16 for the cache. Weights take 141 GB, the 64 sequences need 687 GB of KV, and the node has 640 GB. Quantize the cache to fp8 or cap concurrency at about 40, and say which one the latency SLO prefers.

KV/token  = 2 × 80 layers × 8 KV heads × 128 × 2 B = 327,680 B
per seq   = 327,680 B × 32,768 tokens                = 10.7 GB
64 seqs   = 687 GB          weights (bf16) = 141 GB
budget    = 8 × 80 GB × 0.9 = 576 GB  →  576 − 141 = 435 GB free
max seqs  = 435 / 10.7 ≈ 40 in bf16, ≈ 81 with fp8 KV

The interviewer is waiting for two things: that you reached for the KV cache before the weights, and that you said “8 KV heads” rather than 64. Using the query-head count makes the cache eight times bigger and the design wrong from the first line.

check it in the calculator →

The hardware, dated, and not only NVIDIA.

Interviewers ask about the part your target company actually runs. The concept curriculum maps TPU, Trainium, MI300-class and Blackwell onto the same ideas, so the roofline you learn on an H100 still works when the interviewer says “Trillium”. Specs here are dense peaks, because the marketing numbers include sparsity and no dense GEMM reaches them.

GPU & accelerator architecture →Roofline calculator
Hardware at a glance · dense peaks · September 2026
full spec table →
MEMORY BANDWIDTH (TB/s)BF16 DENSE (TFLOPS)A100 80GB SXM2.04312H100 80GB SXM3.35989H200 141GB SXM4.8989B200 (HGX)82,250MI300X5.31,307TPU v6e (Trillium)1.64918Trainium22.9667ridge point = TFLOPS ÷ TB/s · H100 ≈ 295 · B200 ≈ 281 · MI300X ≈ 247

41 company loops, tagged by what they weight.

all companies →

Each company page carries the loop round by round with a confidence level, the signature questions that are publicly reported, and the tracks to drill for it. Never an invented round.

CompanyKernelsPlatformDistributedDesignCostLoop
OpenAIdocumented
Anthropicdocumented
Google DeepMindpartial
Google Cloud and TPUdocumented
xAIpartial
Metadocumented
Mistral AIlimited
Coherelimited
Amazon Web Servicespartial
Microsoftlimited
Oracle Cloud Infrastructurelimited
NVIDIAdocumented
AMDpartial
Cerebraslimited
Groqlimited
SambaNovalimited
Tenstorrentlimited
Intellimited
Kernels: cuda & kernel heavyPlatform: kubernetes & platform heavyDistributed: distributed systems & networkingDesign: ml systems designCost: cost & capacity
LOOPS THIS BANK IS MODELLED ON · NOT ENDORSEMENTS

Start with the must-know set. Free.

the whole map →
GPU & Accelerator Architecture · easyWalk me through the CUDA execution model: what are grids, blocks and warps, and what does the hardware actually schedule?A grid is a request, a block is a residency unit, a warp is what the scheduler issues. Which of those pins to an SM, why 32 matters, and how a GPU hides a 600 ns memory latency with no branch predictor and a cache that is tiny per thread.Hardware, Cabling & Cluster Build-Out · mediumYou have budget for roughly 1,000 GPUs and a greenfield site. Specify the cluster.Start from the power envelope, because it decides the GPU count more often than the budget does. The scalable unit that makes the design a copy of a known-good one, the four networks, and the three line items that gate the schedule while everyone is discussing GPUs.Distributed Training & Parallelism · easyIn data-parallel training, what actually gets communicated between GPUs, and how much is it per step?Not the data, not the weights: the gradients, once per step, in a ring that moves almost twice the model's size through every GPU. The derivation, the per-step byte count for an 8B model, and why it still hides behind the backward pass.LLM Inference & Serving · easyWhy do prefill and decode behave so differently, and why does that matter for the hardware you serve on?One forward pass reads every weight. Whether that read is the bottleneck depends on how many tokens ride along with it, and the answer is different for the two halves of a request.Open-Weights Models & Serving Engines · mediumGLM-5.3 was released this morning. How many GPUs do you need to serve it, and of what kind?The answer comes from the model card and config.json in five minutes, before downloading anything. The weight footprint, the KV per token that is far smaller than the parameter count suggests, and the parallel degree that has to divide the head count.Napkin Math, Cost & Capacity · easyHow much GPU memory does it take to run Llama 3.1 70B?The first number every serving interview asks for, derived from parameter count and bytes per parameter, in three precisions, with the part people forget to add.

13 topics, each a journey from easy to expert.

150 concepts · 400 practice items · 4 courses
🧩GPU & Accelerator Architecture30
SMs, warps and the memory hierarchy, tensor cores, the roofline model, FP8 and FP4 numerics, NVLink and HBM generations, and the non-NVIDIA canon: TPU, Trainium, MI300-class, Cerebras and Groq. The hardware physics every other round assumes you know cold.
🖧Hardware, Cabling & Cluster Build-Out40
Choosing between H100, H200, B200, B300 and RTX PRO 6000; NVLink domains and rack-scale systems; InfiniBand and Ethernet fabrics; the cables, transceivers and optics power nobody budgets; rack power, busways and liquid cooling; bring-up, burn-in and acceptance. The physical layer every cluster rests on, dated to 2026.
⚡CUDA, Triton & Kernel Engineering30
Coalescing, shared memory and bank conflicts, occupancy, fusion, tiled GEMM, FlashAttention internals, Triton, CUTLASS, Nsight profiling and torch.compile: the live-coding and take-home round at NVIDIA, Fireworks, Together and the labs' performance teams.
🕸️Distributed Training & Parallelism32
DDP, ZeRO and FSDP, tensor, pipeline, context and expert parallelism, collectives and their cost, MFU, activation checkpointing, elastic and fault-tolerant training, checkpoint economics and the RL post-training stack. Owning the training run at cluster scale.
🚀LLM Inference & Serving30
Prefill versus decode, the KV cache, PagedAttention and continuous batching, chunked prefill, speculative decoding, disaggregated serving, quantization, vLLM, SGLang and TensorRT-LLM, multi-LoRA and routing: hosting open-weight models at a latency SLO and a cost you can defend.
🧮Open-Weights Models & Serving Engines40
Running the 2026 open-weights frontier: GLM-5.3, Kimi K3 and DeepSeek V4. Reading config.json to size a model you have never run, latent attention and sparse indexers, vLLM and SGLang configuration, expert parallelism and all-to-all backends, weight formats, and the benchmarks that do not lie.
🧮Napkin Math, Cost & Capacity30
Memory footprints, 6ND, arithmetic intensity and the ridge point, bandwidth-bound decode, communication volume, GPU counts and time to train, cost per million tokens, TCO and buy versus rent. The estimation round almost every AI infra loop includes, with every assumption stated.
🔌Networking, Interconnects & Storage30
NCCL and the collective algorithms, RDMA, InfiniBand versus RoCE, rail-optimized and fat-tree fabrics, congestion control, GPUDirect, parallel filesystems versus object storage, data loading and checkpoint I/O: the fabric and the disks that decide whether ten thousand GPUs act like one.
🗂️Kubernetes, Slurm & GPU Scheduling30
Device plugins and dynamic resource allocation, MIG, MPS and time-slicing, gang scheduling with Kueue and Volcano, topology-aware placement, multi-tenancy and quotas, Slurm versus Kubernetes, containers and cold starts: the platform round at CoreWeave, Modal, Nebius and every GPU cloud.
🩺GPU Fleet Reliability & Observability30
DCGM, the XID taxonomy, ECC and row remapping, NVLink faults, stragglers and hangs, thermal and power events, node health checks, SLOs for training and serving, incident response and postmortems at fleet scale. The on-call reality most prep sites skip.
📐AI Infrastructure System Design31
Design an inference platform at 10k requests per second, a 10,000-GPU training cluster, a job scheduler with preemption and checkpointing, a serverless GPU runtime with sub-second cold starts, a multi-tenant fine-tuning service, an eval pipeline. The whiteboard round at OpenAI, Anthropic, Baseten and Together.
💻Coding for Infra30
Practical builds in Python, Go and C++: a GPU credit scheduler, a rate limiter, a versioned KV store, merging GPU idle intervals, a batching queue, retry with backoff, concurrency under load, parsing a kernel trace. The screens that test whether you can ship infra code in 45 minutes.
🧭Behavioral & Ownership30
Pushing back on a launch for reliability, the on-call story, the migration nobody wanted, working with researchers, the safety and mission rounds at the labs, and your view on where AI infrastructure is going. The rounds that decide between two technically equal candidates.

One payment unlocks all of it

Every answer across 13 topics, the full concept curriculum and all 4 courses, for 6 months. One payment, no auto-renewal. Start free, no card needed.

Unlock everything · ₹2,000 / $25Start with the free questions
ONE QUESTION A WEEK, FREE

A worked AI infra interview question in your inbox each week. No spam, one click to leave.