AI Infrastructure Engineer Skills: What the Loops Test

Every loop in this field, from a new-grad NVIDIA screen to an Anthropic performance loop, tests whether you can reason from first principles about memory, compute, bandwidth and cost. The questions are the same everywhere: how much memory does a 70B model need to serve, how big is the KV cache at 32k context, how many GPU-hours does a run take, why is decode memory-bound, what does a million tokens cost, how long does an all-reduce take. The formulas fit on one page and the formula sheet carries all twelve with worked examples; the calculators let you check your arithmetic; the napkin math track teaches each derivation step by step.

The interviewer is not checking whether you memorized the number. They are checking whether you can produce it from inputs you state, sanity-check it against a physical bound (does it fit in 80 GB, is that above peak FLOPS), and say what would change it. A candidate who says 141 GB is right; a candidate who says 70.6 billion parameters times two bytes is 141 GB, which does not fit one 80 GB card and needs at least two, and here is what quantizing to fp8 does, has passed the question.

The GPU: the execution model (SMs, warps, blocks), the memory hierarchy (registers, shared memory, L2, HBM), tensor cores and the numeric formats they support, the roofline and the ridge point, NVLink and NVSwitch, and the generations from A100 through Hopper and Blackwell with their headline numbers. The alternatives: TPUs and their systolic arrays, Trainium, AMD Instinct and ROCm, and the dataflow parts (Groq, Cerebras). The GPU architecture track covers this, and every kernel, training and inference question stands on it.

Distributed systems: collectives and their costs, the parallelism strategies and when each applies, checkpointing at scale, fault tolerance, and the failure statistics that make reliability an engineering problem. Networking and storage: RDMA over InfiniBand and RoCE, rail-optimized fabrics, congestion control, parallel filesystems against object storage, data loading and checkpoint I/O. Scheduling: Kubernetes with the GPU operator and a batch layer, Slurm, gang and topology-aware placement, quotas and fairness, MIG and other sharing modes. Reliability: XIDs, DCGM, ECC and row remapping, stragglers and hangs, node health, SLOs and error budgets, incident response. Each has a track on this site.

Kernel and performance: CUDA and Triton fluency, coalescing, shared memory and bank conflicts, occupancy, fusion, tiled GEMM, FlashAttention internals, CUTLASS, profiling with Nsight, torch.compile and CUDA graphs. The interview is a kernel problem and a performance investigation, and the topics are in CUDA and kernels.

Training infrastructure: DDP, ZeRO and FSDP, tensor, pipeline, context and expert parallelism, MFU and its measurement, activation checkpointing, checkpointing at scale, fault-tolerant training, RL post-training infrastructure. The interview is a parallelism plan for a model on a cluster with the traffic on each link derived, plus a slow-step investigation. The topics are in distributed training.

Inference and serving: prefill and decode, the KV cache, continuous batching, paged attention, chunked prefill, speculative decoding, disaggregated serving, prefix caching, attention variants, quantization, the serving engines, latency metrics and goodput, multi-LoRA, autoscaling. The interview is a platform design with a sizing chain and a latency budget. The topics are in inference and serving and AI systems design.

Fleet and platform: everything in scheduling and reliability above, plus the operational judgment (drain first, investigate second; canary and rollback; quota arithmetic against a researcher's request). The interview is a scheduler design, an incident story and a migration story, and the topics are in scheduling, reliability and behavioral.

The coding rounds in this field are systems problems, not puzzles: a credit-based job scheduler with accounts and priorities (the widely reported OpenAI screen), a token-bucket rate limiter, a request batcher with timeouts and bounded queues, the global-idle-intervals problem over per-GPU busy logs, a thread-safe LRU cache, a producer-consumer pipeline with backpressure, a streaming parser that classifies kernel trace records. The scored behaviours are working code early, tests named for invariants, complexity stated unasked, and calm follow-ups. Some companies allow an AI assistant in the round and some ban it. The coding for infra track works each of these with code and tests.

Languages: Python everywhere, Go at the fleet and platform teams (Kubernetes operators, controllers), C++ and CUDA in the kernel and runtime teams, Rust at a few (Modal, OpenAI's runtime postings). The postings usually list two or three and accept any; the loop tests fluency in the one you choose.

From the postings' skills lists: Kubernetes with operators and controllers, Slurm, PyTorch DDP and FSDP, NeMo and Megatron, JAX for TPU teams, NCCL and nccl-tests, InfiniBand and RoCE tooling, Lustre or GPFS, Enroot and Pyxis, DCGM, Nsight, vLLM, SGLang or TensorRT-LLM, Ray, Terraform and Ansible, Prometheus and Grafana, and cloud GPU instances on at least one provider. You do not need production experience with all of them; you need to have run a multi-GPU training job, served a model with a real engine, read a profiler trace, and debugged one thing that was slow for a non-obvious reason. The study paths sequence the concept tracks for each background.

Good candidates know the mechanisms and can explain them. Great candidates derive the numbers, name the failure modes before being asked, state the reversal condition on every trade-off, and tell incident stories that end with the page never firing again. The difference is audible in the first five minutes of a design round and the last five of a behavioral one, and it is the difference between a senior and a staff offer; the leveling page lists the signals. The other marker of great is currency: knowing what changed in the last year (the hardware lineup, the serving engines' features, the scheduling stack, the failure statistics) rather than what was true in 2024.