AI Infrastructure Engineer Interview Questions
A curated, continuously updated set of 413 AI infrastructure interview questions with worked answers, organized by topic and by company. These loops do not stop at “can you build the system.” They ask whether you know what the hardware is doing underneath it: why decode is memory-bound, what a slow all-reduce looks like in the logs, how many GPUs a training run needs and why the answer moves with MFU. The questions are modeled on loops at NVIDIA, OpenAI, Anthropic, Google, CoreWeave, Together AI, Fireworks, Modal and more, across the eleven areas that decide AI infra offers.
Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.
What AI infrastructure interviews test
Across companies, the loop probes five dimensions. Which one dominates depends on the track: kernel and performance roles weight the first, platform roles the fourth, and every loop has some of the third.
- Hardware numeracy and kernels
The memory hierarchy, the roofline, coalescing and occupancy, FlashAttention internals. Kernel roles write CUDA or Triton live; everyone else is expected to read a profile and say why a kernel is slow.
- Distributed training at cluster scale
Data, tensor, pipeline, context and expert parallelism, the collectives under them, MFU, checkpointing, and what a run does at 3 a.m. when a GPU falls off the bus.
- Napkin math and systems design
How many GPUs, how much KV cache, how many tokens per second, what a million tokens costs; then a design that uses those numbers: an inference platform, a training cluster, a scheduler with preemption.
- Cluster, network and fleet operations
Kubernetes device plugins and DRA, MIG and sharing, gang scheduling, InfiniBand versus RoCE, DCGM, the XID taxonomy, stragglers, and the incident craft of keeping ten thousand GPUs acting like one.
- Practical coding and judgment
Build a GPU credit scheduler or a batching queue in 45 minutes, then the behavioral round: the launch you pushed back on, the migration nobody wanted, and at the labs, the safety and mission conversation.
GPU Architecture Interview Questions
SMs, warps and the memory hierarchy, tensor cores, the roofline model, FP8 and FP4 numerics, NVLink and HBM generations, and the non-NVIDIA canon: TPU, Trainium, MI300-class, Cerebras and Groq. The hardware physics every other round assumes you know cold.
- Walk me through the CUDA execution model: what are grids, blocks and warps, and what does the hardware actually schedule?NVIDIA · Google · CoreWeave
- Describe the GPU memory hierarchy. Where can a byte live on an H100, and what does each level cost?NVIDIA · AMD
- Why are GPUs so much faster than CPUs for deep learning? Be specific about what the silicon is doing differently.NVIDIA · Google
- What is a tensor core, and what does a kernel have to do to actually use one?NVIDIA
- Explain arithmetic intensity and the roofline model. Where is the ridge point on an H100, and what does it tell you about a kernel?NVIDIA · Fireworks · Together AI
- Is LLM decode memory-bound or compute-bound? Show me the arithmetic that decides it.OpenAI · Anthropic · Together AI
- bf16, fp16, fp8: what is the difference at the bit level, and where does each one belong in training and serving?NVIDIA · Meta
- How much faster is an H200 than an H100, really? Which workloads see the gain and which do not?NVIDIA · CoreWeave · Lambda
- NVLink versus PCIe: when does the interconnect actually matter for an LLM workload, and when can you get away with PCIe?NVIDIA · Lambda · Crusoe
- What is warp divergence, why does it cost you, and how would you find and fix it in a kernel?NVIDIA
- A kernel runs at 25% occupancy. Is that a problem? Walk me through what occupancy buys and when you would leave it low on purpose.NVIDIA · Fireworks
- Explain what HBM is and why memory bandwidth, not compute, is the wall for LLM inference.NVIDIA · AMD · Micron
- What actually changes with Blackwell and the NVL72 rack, and what does it do to how you would serve a large MoE model?NVIDIA · CoreWeave · Microsoft
- You have trained on GPUs. What transfers to training on TPUs, and what do you have to relearn?Google · Google DeepMind
- Explain how a systolic array computes a matrix multiply. Why is it efficient, and why does it want big square tiles?Google · Groq · Cerebras
- Would you move a cost-sensitive training and serving fleet from H100 to Trainium2? What do you gain, and what do you have to plan for?AWS
- The MI300X has 192 GB per card. When does AMD actually win against an H100 for LLM serving, and what would you check before betting on it?AMD · Microsoft · Oracle
- Why are Groq and Cerebras so fast at batch 1, and what does that speed cost at scale?Groq · Cerebras
- MIG versus MPS: what isolation does each give you when sharing a GPU, and which would you pick for a multi-tenant inference node?NVIDIA · CoreWeave · Nebius
- You are moving a model to fp4 inference on Blackwell. What breaks first, and how would you measure whether the result is acceptable?NVIDIA · Fireworks
GPU Cluster Hardware and Deployment Interview Questions
Choosing between H100, H200, B200, B300 and RTX PRO 6000; NVLink domains and rack-scale systems; InfiniBand and Ethernet fabrics; the cables, transceivers and optics power nobody budgets; rack power, busways and liquid cooling; bring-up, burn-in and acceptance. The physical layer every cluster rests on, dated to 2026.
- You have budget for roughly 1,000 GPUs and a greenfield site. Specify the cluster.CoreWeave · Crusoe · Meta
- Your fleet is decode-heavy. Is a B300 worth 1.4 times a B200's power for 1.6 times the memory?NVIDIA · Baseten · Together AI
- Leadership bought GB300 NVL72 racks. Your datacenter is built for 15 kW cabinets. What do you tell them?Microsoft · CoreWeave · Crusoe
- After recabling a row to 800G, a third of the links will not come up. Find the cause.NVIDIA · CoreWeave · Meta
- How much power does a 512-GPU cluster actually draw, end to end?Crusoe · CoreWeave · Microsoft
- Design the cooling for a 40-rack hall of liquid-cooled GPU racks.Microsoft · CoreWeave · Crusoe
- During acceptance, seven rails hit expected bandwidth and one is 15 percent low. Find it.Meta · CoreWeave · NVIDIA
- New nodes arrive with no software. Walk me through bring-up, and say why the order matters.NVIDIA · CoreWeave · Meta
- 512 GPUs are installed and the vendor wants sign-off. Design the acceptance test.CoreWeave · Meta · Crusoe
- You are building a fine-tuning service for customer models under 30B. RTX PRO 6000 or H100?Lambda Labs · Modal · Baseten
- Your new rack is tripping the contracted power limit during training. What do you do?CoreWeave · Crusoe · Microsoft
- Count the switches, cables and transceivers for a 2,048-GPU rail-optimized fabric.Crusoe · CoreWeave · NVIDIA
- After a maintenance window, two-node bandwidth halved. Single-node is unchanged. Diagnose it.NVIDIA · CoreWeave · Meta
- Size the storage for a 2,048-GPU training cluster. What numbers actually drive it?WEKA · VAST Data · CoreWeave
- Design the out-of-band management network for a 512-GPU cluster. What connects to it?Crusoe · CoreWeave · Lambda Labs
- Air, rear-door heat exchanger or direct-to-chip? Decide for a 60 kW rack.Microsoft · CoreWeave · Crusoe
- Build the schedule for a 2,048-GPU deployment. What is on the critical path?Crusoe · CoreWeave · Microsoft
- How many spare GPUs, nodes, cables and transceivers do you hold for a 2,048-GPU fleet?CoreWeave · Meta · Lambda Labs
- Twenty-eight of sixty-four new nodes fail acceptance. What do you do first?CoreWeave · Meta · Crusoe
- A GPU cluster has four separate networks. Name them and say what each one carries.NVIDIA · CoreWeave · Crusoe
CUDA & Triton Kernel Interview Questions
Coalescing, shared memory and bank conflicts, occupancy, fusion, tiled GEMM, FlashAttention internals, Triton, CUTLASS, Nsight profiling and torch.compile: the live-coding and take-home round at NVIDIA, Fireworks, Together and the labs' performance teams.
- Write a CUDA vector add and explain the launch: grid math, the bounds check, and why the copies dominate.NVIDIA
- What is memory coalescing, why does a strided access pattern hurt, and how do you see it in a profiler?NVIDIA · Fireworks
- Given the addresses each thread in a warp touched, classify the access pattern: coalesced, strided or random. Write the classifier.NVIDIA
- Explain shared memory bank conflicts with the bank arithmetic, show a kernel that has them, and fix it with padding.NVIDIA · Together AI
- Why fuse kernels, how much does it save, and what can fusion not fix?Fireworks · Together AI · OpenAI
- Write a fused row softmax in Triton, explain why it is one HBM pass, and say where it stops scaling.OpenAI · Fireworks · Together AI
- Take a GEMM from naive to 70% of peak: the steps, the speedup at each, and the arithmetic that says why.NVIDIA · Fireworks
- Explain occupancy and register pressure: launch bounds, spills, the calculator, and why 50% occupancy can beat 100%.NVIDIA
- Explain FlashAttention. Why is it called IO-aware, and what does it actually save?Fireworks · Together AI · Anthropic
- What changed between FlashAttention 1, 2 and 3, and why did each change buy what it did?Fireworks · Together AI · NVIDIA
- When do you write a kernel in Triton, and when do you have to drop down to CUDA?OpenAI · Together AI
- Here is an Nsight Compute report for a slow kernel. Read it, name the bottleneck, and tell me what you would change.NVIDIA · Fireworks
- Sum 100 million floats on the GPU as fast as the hardware allows. Write the kernel and justify each step.NVIDIA
- Transpose a large matrix at close to copy bandwidth. Why is the naive version slow, and what does each fix buy?NVIDIA
- What does torch.compile actually do to your model, and when does it fail to help?OpenAI · Anthropic · Meta
- Why do serving engines capture decode steps into CUDA graphs, and what does capture require of the rest of the code?Fireworks · Together AI · NVIDIA
- Sketch a paged attention kernel. What changes from FlashAttention once the KV cache is not contiguous?Together AI · vLLM
- cuBLAS already gives you a fast GEMM. What does CUTLASS give you that it does not, and when is that worth it?NVIDIA · Fireworks
- A take-home gives you a working layernorm kernel at a tenth of memory bandwidth. Make it fast and justify every change.Anthropic
- Implement the verification step of speculative decoding, including the rollback of the KV cache after a rejection.Together AI
Distributed Training Interview Questions
DDP, ZeRO and FSDP, tensor, pipeline, context and expert parallelism, collectives and their cost, MFU, activation checkpointing, elastic and fault-tolerant training, checkpoint economics and the RL post-training stack. Owning the training run at cluster scale.
- In data-parallel training, what actually gets communicated between GPUs, and how much is it per step?Meta · Google · OpenAI
- Compare data, tensor and pipeline parallelism. What does each one shard, what does each one communicate, and where does each one live?NVIDIA · Meta · OpenAI
- What is MFU, how do you compute it from a running job, and what counts as a good number?Meta · Google · Anthropic
- Explain ZeRO stages 1, 2 and 3. How much memory does each stage leave per GPU for a 70B model?Microsoft · Meta
- FSDP or DeepSpeed ZeRO-3: which would you pick for a new training codebase today, and why?Meta · Microsoft · Databricks
- You need to train a 100B dense model and it does not fit on one node. Walk me through how you would lay it out.OpenAI · Anthropic · xAI
- Derive the cost of a ring all-reduce. Why is it bandwidth-optimal, and where does it stop scaling?NVIDIA · Google
- Do the pipeline bubble math for me. How do micro-batches shrink it, and what does 1F1B change?NVIDIA · Meta
- Why does tensor parallelism stop at 8? Show me the numbers.NVIDIA · Anthropic
- Gradient accumulation versus a bigger per-GPU batch: same result or not, and what changes underneath?Meta · Databricks
- When does activation checkpointing pay for itself, and what would you checkpoint first?Meta · OpenAI
- How do sequence parallelism and context parallelism make 128k-context training of a 405B possible, and what do they cost?Meta · Anthropic · Google
- Explain expert parallelism for an MoE model: what the all-to-all is, how much it moves, and why load balance is the whole problem.DeepSeek · Mistral · Meta
- Your MoE router sends 90% of tokens to 10% of the experts. What happens to the step, and how do you fix it without hurting the model?DeepSeek · Mistral
- You have 16,384 H100s and a 405B dense model. Choose the parallelism layout and justify the order of the axes.Meta · NVIDIA
- Explain mixed-precision training. Why does fp16 need loss scaling and bf16 not, why keep fp32 master weights, and what changes with fp8?NVIDIA · Meta
- Your training run is at 60% of the step time you projected. How do you find out whether it is compute, memory, network or I/O?Anthropic · OpenAI · Crusoe
- Training loss went flat at step 40k after descending normally. Walk me through how you debug it.Anthropic · OpenAI · xAI
- At 16,000 GPUs something fails every few hours. How do you choose the checkpoint interval, and what does the write have to look like?Meta · Anthropic · Microsoft
- Design a training system that survives losing a node without a human in the loop. What does elasticity cost you?Anyscale · Meta · Crusoe
LLM Inference & Serving Interview Questions
Prefill versus decode, the KV cache, PagedAttention and continuous batching, chunked prefill, speculative decoding, disaggregated serving, quantization, vLLM, SGLang and TensorRT-LLM, multi-LoRA and routing: hosting open-weight models at a latency SLO and a cost you can defend.
- Why do prefill and decode behave so differently, and why does that matter for the hardware you serve on?OpenAI · Anthropic · Baseten
- What is the KV cache, and why does it keep growing while a request is being served?OpenAI · Baseten · Fireworks
- What is the difference between static and continuous batching, and why did it change LLM serving?Baseten · Together AI · Anyscale
- How does PagedAttention work, and what problem was it solving?vLLM · Together AI · Baseten
- Define TTFT, TPOT and goodput, and tell me how you would measure each one in production.Baseten · Anyscale · OpenAI
- You have eight H100s and Llama 3.1 70B. How many concurrent users can you serve, and what changes the number?Together AI · Baseten · CoreWeave
- A long prompt arrives while sixty users are mid-generation. What happens, and how does chunked prefill fix it?vLLM · Anthropic · Baseten
- When does speculative decoding speed up serving, and when does it break even or hurt?Together AI · Fireworks · Anthropic
- Compare the KV cache footprint of multi-head, grouped-query and multi-head latent attention with numbers.DeepSeek · Meta · Fireworks
- You need to quantize a model for serving. Which method do you pick, and what do you measure before shipping it?Fireworks · Together AI · Red Hat
- When does splitting prefill and decode onto separate GPU pools pay for itself, and what does the KV transfer cost?Anthropic · Fireworks · NVIDIA
- Every request shares a 2,000-token system prompt. How does prefix caching exploit that, and how does the radix tree work?SGLang · Anthropic · OpenAI
- You have one GPU and a synchronous API that receives 100 documents at once. Design the batching, and show the latency math.Anthropic
- Your p99 TTFT tripled last night and p50 did not move. Walk me through how you find the cause.Baseten · Anthropic · OpenAI
- vLLM, SGLang or TensorRT-LLM: which engine do you pick for a new deployment, and what would change your mind?Baseten · Together AI · NVIDIA
- How would you serve hundreds of LoRA adapters on one base model, and what does it cost in throughput?Anyscale · Fireworks · Databricks
- Design an autoscaler for GPU inference replicas that reacts to load without thrashing.Baseten · Modal · Anyscale
- A new replica has to load a 70B model and serve traffic in under a minute. Where do the seconds go, and how do you get there?Modal · Baseten · RunPod
- What changes when you serve DeepSeek-V3 compared with a dense 70B model?Fireworks · Together AI · DeepSeek
- How do you route requests across replicas to maximize prefix-cache hits without unbalancing the fleet?SGLang · Anthropic · Perplexity
Open-Weights Model Serving Interview Questions
Running the 2026 open-weights frontier: GLM-5.3, Kimi K3 and DeepSeek V4. Reading config.json to size a model you have never run, latent attention and sparse indexers, vLLM and SGLang configuration, expert parallelism and all-to-all backends, weight formats, and the benchmarks that do not lie.
- GLM-5.3 was released this morning. How many GPUs do you need to serve it, and of what kind?Together AI · Fireworks AI · Baseten
- vLLM crashes with out of memory during startup on a model that should fit. Debug it.Fireworks AI · Baseten · Together AI
- A model you have never heard of trends on Hugging Face. Estimate the deployment before downloading it.Baseten · Together AI · Fireworks AI
- Your expert-parallel deployment is slower than the tensor-parallel one. Find out why.Fireworks AI · Together AI · DeepSeek
- You are serving an agent product with long shared prompts. vLLM or SGLang?Baseten · Together AI · Modal
- Serve Kimi K3, a 2.8 trillion parameter model. What is the minimum viable configuration?Together AI · Fireworks AI · Baseten
- Your model decodes at a tenth of its bandwidth bound at batch one. Explain the gap.Fireworks AI · Together AI · NVIDIA
- Set max-model-len and max-num-seqs for a chat product from first principles.Baseten · Modal · Together AI
- Your new model deployment produces fluent answers that score badly on evaluations. Where do you look?Baseten · Together AI · Modal
- Design the benchmark you would run before putting a new open-weights model into production.Together AI · Fireworks AI · Baseten
- The model ships in FP8. Should you requantize to four bits to fit more of it on fewer GPUs?Fireworks AI · Together AI · Baseten
- Prefix caching cut your benchmark's latency in half. Why might production see none of that?Baseten · Together AI · Modal
- Would you split prefill and decode onto separate pools for a 753B model? Work the decision.Fireworks AI · Together AI · DeepSeek
- A multi-node vLLM deployment starts, forms a cluster, and hangs on the first request. Debug it.Together AI · Fireworks AI · CoreWeave
- Your product forecasts 12,800 output tokens per second at peak. Size the fleet.Baseten · Together AI · Modal
- Is speculative decoding worth enabling on a trillion-parameter mixture-of-experts model?Together AI · Fireworks AI · Baseten
- Your deployment is limited by KV cache rather than by compute. What are the options?Baseten · Together AI · Fireworks AI
- Your p99 time to first token is four times p50. Find out why.Baseten · Together AI · Modal
- You have one week to get a newly released trillion-parameter model serving production traffic. Plan it.Together AI · Fireworks AI · Baseten
- Self-host a 753B open-weights model or call a hosted API? Work the crossover.Baseten · Together AI · Modal
Napkin Math & Estimation Interview Questions
Memory footprints, 6ND, arithmetic intensity and the ridge point, bandwidth-bound decode, communication volume, GPU counts and time to train, cost per million tokens, TCO and buy versus rent. The estimation round almost every AI infra loop includes, with every assumption stated.
- How much GPU memory does it take to run Llama 3.1 70B?
- How big is the KV cache for Llama 3.1 70B at a 128k context?OpenAI · Anthropic · Baseten
- How many FLOPs does it take to train a 70B model on 15 trillion tokens?Meta · OpenAI
- How long does that 70B run take on 16,384 H100s at 40% MFU?Meta · Anthropic
- How many H100s do you need to train a 70B model on 15 trillion tokens in 30 days?OpenAI · Anthropic · xAI
- How many tokens per second can a 70B model generate for a single user on H100s?Together AI · Baseten
- What does it cost per million output tokens to serve a 70B model on eight H100s?Baseten · Together AI · Fireworks
- How much memory does it take to fine-tune a 70B model, full fine-tuning versus LoRA?Databricks · Anyscale
- A kernel does 4 TFLOP and moves 40 GB in one call. On an H100, is it memory-bound or compute-bound?NVIDIA · Fireworks
- How much network bandwidth does data-parallel training need?Meta · Crusoe
- How long does prefill take for an 8k-token prompt on a 70B model?Anthropic · Fireworks
- Estimate how many H100-hours it took to train Llama 3.1 405B, then check it against the paperMeta
- Size an inference fleet for a 70B model serving 1,000 concurrent usersOpenAI · Baseten · Together AI
- A MoE model and a dense model reach the same quality. How do their GPU counts compare for serving and for training?DeepSeek · Mistral · Google
- Estimate how long it takes to write a checkpoint for a 405B training runMeta · Microsoft
- What is the power budget for a cluster of 1,024 H100s?Crusoe · CoreWeave · Microsoft
- Should we buy or rent 1,000 H100s for the next two years?CoreWeave · Lambda · OpenAI
- How much does moving from bf16 to fp8 save in serving cost?Fireworks · Together AI
- How long does an all-reduce of a 70B model's gradients take on eight GPUs?NVIDIA · Meta
- How many tokens per second can one B200 decode for a 70B model?NVIDIA · CoreWeave
Networking, Interconnect & Storage Interview Questions
NCCL and the collective algorithms, RDMA, InfiniBand versus RoCE, rail-optimized and fat-tree fabrics, congestion control, GPUDirect, parallel filesystems versus object storage, data loading and checkpoint I/O: the fabric and the disks that decide whether ten thousand GPUs act like one.
- What does NCCL actually do when you call all-reduce, and how does it decide which algorithm to use?NVIDIA · Meta
- Name the four collectives a training job uses, say what each moves, and match them to the parallelism that needs them.NVIDIA · Google
- InfiniBand or RoCE version 2 for a new GPU training cluster. Make the call and say what would reverse it.NVIDIA · Crusoe · xAI
- What is RDMA, and why can a training cluster not just use TCP at 400 gigabits per second?NVIDIA · Crusoe
- Explain a rail-optimized fabric. What does it buy over a plain fat tree, and how many switches does one need?NVIDIA · Meta · CoreWeave
- A 64-node job's all-reduce runs at a third of the expected bandwidth. Walk me through finding the cause.Crusoe · Lambda · CoreWeave
- Design the fabric for 4,096 GPUs with 64-port switches. What does 1:1 cost, and what does 2:1 give up?Crusoe · Lambda · Microsoft
- What is incast, why does it break an RDMA fabric, and what do ECN and priority flow control each do about it?xAI · NVIDIA · Meta
- When does GPUDirect Storage actually help, and when is it just a more complicated read?NVIDIA · WEKA
- Parallel filesystem or object storage for training data? Make the call and explain the two-tier pattern most clusters end up with.Crusoe · Meta · Databricks
- The GPUs are idle between steps and the profiler says the data loader. Find the actual constraint and fix it.Meta · Anthropic · Databricks
- You have 60 terabytes of filtered text and need 15 trillion training tokens. Design the tokenization and sharding stage.Meta · Databricks · Anthropic
- Design the checkpoint write path for a 405B model on 16,384 GPUs. What has to be true for the pause to stay under a minute?Meta · Microsoft
- Map a 1,024-GPU job's parallelism onto the hardware. Which dimension goes on NVLink, which on the fabric, and what does a wrong order cost?NVIDIA · Meta
- Which NCCL environment variables are worth knowing, what does each change, and which should you almost never set?NVIDIA · CoreWeave
- How do you establish that the link between two GPU nodes is healthy, and what numbers should each test return?Lambda · Crusoe
- Why do large training clusters provision roughly 400 gigabits per second per GPU rather than more or less?NVIDIA · Meta · xAI
- What is Ultra Ethernet trying to fix, and does it change how you would choose a fabric today?Broadcom · Meta · AMD
- Design the complete network for a 1,024-GPU cluster: compute fabric, storage, management. What are the counts and where does the money go?Crusoe · CoreWeave · Nebius
- Does multi-node inference need the same fabric as training, or can it run on ordinary networking?Together AI · Baseten
Kubernetes GPU Scheduling & Slurm Interview Questions
Device plugins and dynamic resource allocation, MIG, MPS and time-slicing, gang scheduling with Kueue and Volcano, topology-aware placement, multi-tenancy and quotas, Slurm versus Kubernetes, containers and cold starts: the platform round at CoreWeave, Modal, Nebius and every GPU cloud.
- Walk me through what happens when a pod asks Kubernetes for four GPUs, from the manifest to the container seeing them.CoreWeave · Nebius · Modal
- MIG, time-slicing and MPS all let several jobs share one GPU. What is the difference, and when would you pick each?CoreWeave · Nebius · NVIDIA
- Slurm or Kubernetes for a 2,000-GPU training cluster? Make the case, and tell me what you lose either way.CoreWeave · Crusoe · Nebius
- What is gang scheduling, and what goes wrong on a Kubernetes cluster that does not have it?CoreWeave · Anyscale
- Design a GPU-aware scheduler that supports fractional GPUs: what isolation does each fraction get, and where does it break?Together AI · CoreWeave
- Two identical 8-GPU jobs get 8 GPUs each. One runs at half the speed of the other. What did the scheduler do, and how do you stop it?NVIDIA · CoreWeave
- Eight research teams share 1,024 GPUs. Design the quota and fairness policy, and tell me how they will game it.OpenAI · Anthropic · Meta
- Here is a Kubernetes GPU node. Pods requesting nvidia.com/gpu stay Pending. Fix it in front of me and narrate what you check.Nebius · CoreWeave
- What does the NVIDIA GPU Operator actually install on a node, and what is Node Feature Discovery doing underneath it?NVIDIA · CoreWeave
- Design a serverless GPU platform where a function that loads a 7B model cold-starts in under a second. Where does every second go today?Modal · Baseten · RunPod
- Run untrusted user code at 50,000 concurrent sessions, some on GPUs. Pick the isolation boundary and defend the density you lose.Modal · Anthropic · OpenAI
- Your fine-tunes run on spot GPUs preempted about once every four hours. How often should they checkpoint, and when does spot stop paying?Anyscale · Lambda · RunPod
- One fleet: training that wants every idle GPU, and inference with a p99 SLO. Separate pools, or one pool with preemption? Show the numbers.Anthropic · Nebius · CoreWeave
- Design a job queue for 100k GPU jobs with preemption: what state, what ordering, and what happens when a quota owner returns?OpenAI · Anyscale
- What is Ray on Kubernetes good for, and where do its scheduler and the Kubernetes scheduler fight each other?Anyscale · Databricks
- Our GPU pods take four minutes to start on a fresh node and sometimes fail with 'no CUDA-capable device'. Walk me through both.Modal · Baseten
- We rent GPUs. When should we buy committed capacity instead of paying on demand, and what do we do with the rest of the demand?CoreWeave · Lambda · AWS
- Our cluster is 85% allocated but 64-GPU jobs wait for hours. Explain the fragmentation and what a scheduler should do about it.CoreWeave · Together AI
- Kubernetes device plugins versus Dynamic Resource Allocation: what changes for GPU scheduling, and what would you adopt in 2026?NVIDIA · Google
- What do you run on a GPU node before you let a job land on it, how long does it take, and what happens on failure?CoreWeave · Crusoe · Lambda
GPU Fleet Reliability & Observability Interview Questions
DCGM, the XID taxonomy, ECC and row remapping, NVLink faults, stragglers and hangs, thermal and power events, node health checks, SLOs for training and serving, incident response and postmortems at fleet scale. The on-call reality most prep sites skip.
- How do GPUs actually fail at fleet scale, how often, and which failures should the platform expect to handle every day?Meta · CoreWeave · Lambda
- What is an XID error, which ones mean the hardware is bad, and which ones mean somebody's kernel has a bug?NVIDIA · CoreWeave · Lambda
- With DCGM available on every node, what do you actually collect, what do you alert on, and what do you deliberately ignore?NVIDIA · CoreWeave · Microsoft
- Design observability for a large training cluster. What do you collect, what does each signal answer, and what pages someone?Anthropic · OpenAI · Meta
- One GPU's correctable memory error rate has been climbing for a week. What does that predict and what do you do about it?Meta · CoreWeave
- A node reports a GPU has fallen off the bus. What happened, what can software do, and what should the platform do automatically?Lambda · CoreWeave
- A training run hangs every few hours with no error, and the GPUs sit idle until the timeout fires. Find the cause.Meta · Anthropic · OpenAI
- One node in a job runs at half the speed of its peers. What do you check, in what order, and what does each answer rule out?Crusoe · Lambda
- At 16,384 GPUs something fails every three hours. How often should you checkpoint, and what goodput does that leave?Meta · Microsoft
- What service level objectives would you promise researchers on a training platform, and how would you measure each?Google · Meta · OpenAI
- Define the service level objectives for an LLM serving fleet, and the alerting that tells you when one is about to be missed.OpenAI · Anthropic · Baseten
- A new GPU node arrives. What do you run before it serves a customer, and what does each test actually catch?CoreWeave · Lambda · Crusoe
- How would you detect that GPUs are thermally throttling, and what is the right response when they are?CoreWeave · Crusoe
- What do NVLink errors look like in telemetry, when is a link degrading rather than broken, and when do you drain the node?NVIDIA · Meta
- Write the postmortem for a training run that lost twelve hours. What goes in it, and what makes the action items stick?Anthropic · OpenAI · Meta
- You own the on-call rota for a GPU fleet. What is allowed to wake someone at 3am, and what must not?CoreWeave · Modal
- Design the controller that decides whether a GPU node is healthy. What are its states, and how do you tune its false-positive rate?Meta · CoreWeave · Google
- Error rate on an inference fleet tripled ten minutes after a deploy. What do you do first, and what should have caught it?Baseten · OpenAI
- How do you measure effective training time, and where does the missing ten percent of a well-run cluster actually go?Meta · Anthropic
- Walk me through reading a collective flight-recorder dump. What is in it, and how do you find the rank that caused a hang?Meta · NVIDIA
AI Infrastructure System Design Interview Questions
Design an inference platform at 10k requests per second, a 10,000-GPU training cluster, a job scheduler with preemption and checkpointing, a serverless GPU runtime with sub-second cold starts, a multi-tenant fine-tuning service, an eval pipeline. The whiteboard round at OpenAI, Anthropic, Baseten and Together.
- Walk me through an inference platform for a hosted LLM. What are the pieces, and what does each one do?OpenAI · Anthropic · Baseten
- You have 45 minutes and the prompt is 'design our serving platform'. How do you run the round?
- Design an LLM batching system end to end: the queue, the batch, the KV cache and streaming. Give me numbers.Anthropic
- Design a GPU credit system: accounts, spending rates, priorities and fairness. Start with the data model.OpenAI
- Design a distributed search system with an LLM answer layer at 10,000 queries per second. Size both tiers and name the SLOs.Anthropic
- Design a training cluster for 10,000 GPUs: pods, fabric, storage, failure domains and the scheduler. Justify every number.Meta · xAI · Crusoe
- Design a job scheduler for 100,000 jobs on a shared GPU cluster, with preemption and checkpointing. Show me the state machine.OpenAI · Anyscale
- Design a serverless GPU platform: a customer deploys a function with a model and pays per second. Where does the cold start budget go?Modal · RunPod · Baseten
- Design the eval pipeline for a frontier model: thousands of evals per checkpoint, sharded inference, caching, reproducible results.OpenAI · Anthropic
- Design the pipeline that produces 15 trillion training tokens: ingest, dedup, tokenize, shard, serve. Throughput per stage.Meta · Anthropic · Databricks
- Design serving for 100 fine-tuned variants of one 70B base. Multi-LoRA on shared replicas or a replica per variant?Anyscale · Fireworks · Databricks
- Design autoscaling for an LLM service whose traffic triples within a minute. What signal, what cold-start budget, what warm pool?Baseten · Modal
- Design a KV cache tier across GPU memory, host memory and NVMe so prefixes survive across sessions. Bandwidths, and when it pays.NVIDIA · Perplexity
- Design the registry and deploy pipeline for model weights: 141 GB artifacts, versioning, distribution to a thousand nodes, integrity.Databricks · Hugging Face
- Design rate limiting for an LLM API. Why tokens instead of requests, and how does a bucket work when the cost is unknown until the end?OpenAI · Anthropic
- Design a multi-region inference deployment: capacity per region, routing, failover, and getting the weights everywhere.OpenAI · Anthropic · Google
- Design the playground behind an LLM API: streaming, sessions, quotas and abuse, for a million free-tier users.OpenAI
- Design the checkpoint store for a lab running several large training jobs: write bursts, retention, resharding and lineage.Meta · Anthropic
- Design observability for an inference fleet: per-request spans, GPU metrics, and cost attribution per tenant.Baseten · OpenAI
- Design the control plane for a GPU cloud: tenancy, provisioning, health, billing, and the API a customer sees.CoreWeave · Nebius · Lambda
Coding Interview Questions for Infra Engineers
Practical builds in Python, Go and C++: a GPU credit scheduler, a rate limiter, a versioned KV store, merging GPU idle intervals, a batching queue, retry with backoff, concurrency under load, parsing a kernel trace. The screens that test whether you can ship infra code in 45 minutes.
- Given usage records of GPU allocations, compute what each tenant owes. Write it, and say what you would test.OpenAI
- Implement a token bucket rate limiter. Make it thread-safe, and explain what the two parameters actually control.OpenAI · Anthropic
- Given per-GPU idle intervals, compute when any GPU was idle and when every GPU was idle. Write both.OpenAI · CoreWeave
- Implement retry with exponential backoff and jitter. Why is the jitter the part that matters, and what must never be retried?OpenAI · Baseten
- Implement a key-value store where a read can ask for the value as of an earlier version. What is the data structure?OpenAI
- Implement a batcher that flushes when the batch is full or when a timeout expires. What breaks in the timer path?Baseten · Together AI · Anthropic
- Implement a scheduler that admits jobs by priority and preempts lower-priority work when it must. What are the rules?OpenAI · Anyscale
- You are handed a 40 GB kernel trace. Write the parser that turns it into per-instruction access statistics without running out of memory.NVIDIA
- Implement a cache for model weights on a serving node. What makes it different from a normal LRU?Modal · Baseten
- Write a producer-consumer pipeline with a bounded queue. What are the three bugs that show up in every first attempt?Anthropic · OpenAI
- Place GPU jobs onto nodes. Compare first fit, best fit and worst fit, and say which one a training cluster wants.CoreWeave · Together AI
- Tokens arrive as byte fragments and a character can span several. Write the decoder that streams text correctly.OpenAI
- Simulate a ring all-reduce on arrays. Verify both the result and the bytes each rank sends.NVIDIA · Google
- Implement the block allocator behind a paged KV cache, including prefix sharing and copy-on-write.vLLM · Together AI
- Implement the scheduler loop of a continuous-batching engine: admit, step, preempt. Where does it livelock?Baseten · Anyscale
- In Go, download many model shards concurrently with a concurrency limit, a timeout and clean cancellation. Write it.CoreWeave · Crusoe
- Route requests to replicas by prefix using a consistent hash ring. Why virtual nodes, and how many?SGLang · Anthropic
- Implement sampling from a model's logits with temperature, top-k and top-p. What are the numerical traps?OpenAI · Anthropic
- Compute per-tenant GPU-hours for a billing period from a log of allocation start and stop events.CoreWeave · Modal
- Implement a circuit breaker in front of a model endpoint. What are the states, and what does the half-open state buy?Baseten · OpenAI
Behavioral Interview Questions for AI Infra Engineers
Pushing back on a launch for reliability, the on-call story, the migration nobody wanted, working with researchers, the safety and mission rounds at the labs, and your view on where AI infrastructure is going. The rounds that decide between two technically equal candidates.
- Tell me about a time you pushed back on a launch because of a reliability concern.Google · Meta · OpenAI
- Walk me through the worst on-call incident you have handled.CoreWeave · Modal · Meta
- Why do you want to work in AI infrastructure, and why now?OpenAI · Anthropic · NVIDIA
- A researcher needs 256 GPUs today and the cluster is full. How do you handle it?OpenAI · Anthropic · Google DeepMind
- Describe a migration nobody wanted. How did you get it done without losing the team's trust?Meta · Databricks · Google
- What is your view on AI safety, and what does it actually mean for infrastructure work?Anthropic · OpenAI · Google DeepMind
- Tell me about a performance win you are proud of. How did you measure it?NVIDIA · Fireworks AI · Together AI
- Your team is drowning in manual work. How do you decide what to automate first?CoreWeave · Google · Modal
- Describe a time you were confidently wrong about a root cause. What did it cost?Meta · Anthropic · CoreWeave
- You are joining a team running a 10,000 GPU cluster. What do you do in your first month?Meta · xAI · Anthropic
- Two teams both need the same capacity and both have escalated. How do you resolve it?OpenAI · Google · Meta
- A researcher asks you to bypass a required check to hit a deadline. What do you do?Anthropic · OpenAI · Google DeepMind
- How do you keep current in a field that changes every few months?NVIDIA · Anthropic · CoreWeave
- Tell me about a cost reduction you led. How did you prove reliability did not suffer?Baseten · CoreWeave · Modal
- How do you run a postmortem that actually changes something?Google · Anthropic · CoreWeave
- Tell me about scaling a system by ten times. What broke, and in what order?Meta · OpenAI · Microsoft
- Leadership wants to buy another 2,000 GPUs. You think it is the wrong call. What do you do?Crusoe · Lambda Labs · CoreWeave
- A vendor says your configuration is the problem. You think it is their firmware. How do you proceed?CoreWeave · Lambda Labs · NVIDIA
- How do you bring a junior engineer through their first serious incident?Google · Meta · Anthropic
- What do you think is the most underrated problem in AI infrastructure right now?Anthropic · OpenAI · NVIDIA
AI infra interview questions by company
Each company runs a different loop and weights a different axis: CUDA-heavy, Kubernetes-heavy, distributed systems, ML systems design, or cost and capacity. These pages break down the process and what each round is really testing, compiled from public reports with a confidence level.
Before the interview: the role, the numbers, the tools
AI infra interview questions: FAQ
What does an AI infrastructure engineer interview test?
Whether you can build and run the compute under large models: GPU and accelerator internals, kernel performance, distributed training across thousands of chips, inference serving under latency and cost targets, cluster scheduling, the network and storage fabric, and keeping a fleet healthy. Loops mix practical coding, a systems-design round with real numbers, a napkin-math or debugging round, and a behavioral or mission round. Frontier labs, chip makers, GPU clouds and ML platform teams all run some version of it.
How is it different from a normal software engineer or SRE interview?
A standard loop asks whether you can build a distributed system. An AI infra loop adds hardware physics: is this kernel memory-bound, how many bytes does decode move per token, why did MFU drop when you doubled the cluster, which XID code means a GPU has to leave the fleet. SRE loops share the reliability half but not the accelerator half. Grinding LeetCode alone does not carry you; the differentiators are numeracy about hardware and experience with what breaks at scale.
What rounds does an AI infra interview usually have?
A common loop is a recruiter screen, a practical coding screen (build a scheduler, a rate limiter, a batching queue, not puzzles), a systems-design round (an inference platform, a training cluster, a job scheduler), a performance or debugging round (a slow all-reduce, a p99 spike, a kernel to optimize), and a behavioral round. Kernel and performance roles add a take-home or live CUDA or Triton exercise. Rounds vary by company and team, so confirm yours with the recruiter; the company pages below carry what is publicly reported with a confidence level.
Do I need to know CUDA?
It depends on the track, and the site tags every company by which axis it weights. Kernel and performance roles at NVIDIA, Fireworks, Together and the labs' performance teams test live CUDA or Triton. Platform and cluster roles at GPU clouds test Kubernetes, Linux and distributed systems and rarely ask you to write a kernel, though they expect you to know why one is slow. Inference and training roles sit in between: read kernels, reason about them, rarely write them in the room.
What is napkin math in an AI infra interview?
Estimation from first principles: how many GPUs to train a 70B model on 15 trillion tokens in a month, how big the KV cache is at 128k context, whether decode is memory-bound, what a million output tokens costs. The interviewer wants the formula, the assumptions stated, and the right order of magnitude. The formula sheet and calculators on this site cover every one of these.
Which companies hire AI infrastructure engineers?
Frontier labs (OpenAI, Anthropic, Google DeepMind, xAI, Meta), hyperscalers (Google Cloud, AWS, Azure, Oracle), chip makers (NVIDIA, AMD, Cerebras, Groq, SambaNova), GPU clouds and serving platforms (CoreWeave, Together AI, Fireworks, Baseten, Modal, Lambda, Nebius, Crusoe, Anyscale), and ML platform teams at Databricks, Netflix, Uber, LinkedIn and Pinterest. India has a growing set of roles at NVIDIA, AMD, Microsoft and Google engineering centers plus domestic AI clouds. See the company list below.
Is there an AI infra interview questions PDF or sheet with answers?
This page is a continuously updated, structured set of AI infrastructure interview questions with worked answers, organized by topic and by company, with the hardware numbers refreshed on each GPU generation. A static PDF goes stale the day a new part ships.
Prep every AI infra interview question, with answers that hold up in the room
413 questions across 13 topics, ordered easy to expert, each with a worked answer, a hand-built diagram where one earns its place, and the follow-ups interviewers hold back. The first questions in every track are free.
Interview loops change and vary by team, level, and region. Treat this as directional preparation and confirm the exact rounds with your recruiter.