NVIDIA LLM Inference & Serving interview questions

Q: How should I prepare for the NVIDIA LLM Inference & Serving round?

Work the llm inference & serving questions on this page in order, study the underlying concepts linked below, then review NVIDIA's full interview process for how this round fits the loop. For NVIDIA specifically, interviewers weight this: Expertise in the team's domain, reported as the dominant filter.

Q: Does NVIDIA hire AI infrastructure engineers?

Yes, in many titles: AI and ML Infrastructure Software Engineer, GPU Clusters (new-grad 2026); DGX Cloud Performance Engineer; Senior GPU and HPC Infrastructure Engineer; Senior HPC AI Cluster Engineer; Deep Learning Software Engineers for LLM performance, TensorRT and TensorRT-LLM; cuDNN and CUTLASS library engineers; Developer Technology engineers; and Triton Inference Server systems engineers. India postings include a DGX Cloud Performance Engineer in Pune and Site Reliability Engineers in Bengaluru.

Q: What does the NVIDIA AI infrastructure interview test?

Reported loop: a recruiter call; one or two phone screens of 45 to 60 minutes mixing C++ trivia and fundamentals with graph and array problems; a team-dependent take-home in some loops (one 2022 report); then four to six onsite rounds of about an hour with domain-specific content (systems architecture, performance, team-specific coding) and a behavioral round, sometimes a hiring-manager or director round. Candidates report that expertise in the team's domain matters more than LeetCode.

LLM Inference & Serving is a core part of the NVIDIA AI Infrastructure Engineer loop. Prefill versus decode, the KV cache, PagedAttention and continuous batching, chunked prefill, speculative decoding, disaggregated serving, quantization, vLLM, SGLang and TensorRT-LLM, multi-LoRA and routing: hosting open-weight models at a latency SLO and a cost you can defend. Below are the llm inference & serving questions to prepare, the ones tagged to NVIDIA first, then the highest-signal questions from our LLM Inference & Serving track, each with an answer written to a senior-engineer bar.

WHAT NVIDIA LOOKS FOR HERE · Expertise in the team's domain, reported as the dominant filter. See the full NVIDIA interview process →

LLM Inference & Serving questions tagged to NVIDIA

3 questions · 0 unlocked for you

Open full answer →

More LLM Inference & Serving questions for NVIDIA's loop

The highest-signal llm inference & serving questions candidates rate most useful, modeled on what NVIDIA's AI Infrastructure Engineer loop tests.

15 questions · 10 unlocked for you

Open full answer →

Concepts behind NVIDIA's LLM Inference & Serving round

The vocabulary and mental models these questions assume. Start with the foundations free; the deeper, interview-defining ideas are part of premium.

Foundational

Prefill vs DecodeAn LLM request runs in two phases with opposite hardware profiles: prefill reads the whole prompt in one compute-bound pass and decides time to first token, decode emits one token per forward pass and is bound by memory bandwidth. Every serving decision, from batch size to which GPU to buy to whether to split the two phases across machines, follows from that split.

Foundational

The KV CacheThe KV cache stores each token's attention keys and values so decode never recomputes them, turning a quadratic cost into a linear one at the price of memory that grows with every token in every concurrent sequence. Its size, 128 KB per token for Llama 3.1 8B and 320 KB for 70B in bf16, is what caps concurrency and context on a given GPU, so it decides batch size, replica count and whether a model fits at all.

Core

Continuous BatchingContinuous batching schedules at the granularity of a single decode step instead of a whole request, so a finished sequence's slot is refilled on the next iteration rather than when the longest request in the batch ends. It is the scheduling idea that turned LLM serving from a padded, half-idle GPU into one that stays full, and it decides how the engine's scheduler, memory manager and latency SLOs interact.

Advanced

🔒 Premium

PagedAttentionPagedAttention stores the KV cache in fixed-size blocks scattered across HBM and maps each sequence's logical positions to physical blocks through a block table, the same trick an operating system uses for virtual memory. It removes the reservation and fragmentation waste of contiguous allocation, lets blocks be shared between sequences, and is why an engine can decide admission by counting free blocks.

Advanced

🔒 Premium

Chunked PrefillA long prompt's prefill can occupy a GPU for hundreds of milliseconds, and every sequence mid-decode on that GPU waits for it. Chunked prefill splits the prompt into fixed token budgets and interleaves each chunk with a decode step, so decode latency stays flat at the cost of a slower first token for the long prompt. The chunk budget is a knob between TTFT and TPOT, and the interview question is how you would set it.

Advanced

🔒 Premium

Speculative DecodingDecode is memory-bound: each step reads every weight to produce one token. Speculative decoding has a cheap draft propose several tokens, then verifies them all in one forward pass of the big model, so one weight read yields several tokens with output distribution unchanged. It wins 2x to 3x at small batch, breaks even near the ridge point where the GPU is already compute-bound, and lives or dies on the acceptance rate, which is what interviewers ask you to reason about.

Advanced

🔒 Premium

Disaggregated Prefill and DecodePrefill is compute-bound and decode is memory-bound, so running both on the same GPUs means each phase interferes with the other and neither runs on the hardware it wants. Disaggregation puts them on separate pools and ships the KV cache from prefill nodes to decode nodes over the fabric. It lets TTFT and TPOT scale independently and puts high-bandwidth parts where they pay, at the price of a KV transfer per request and a control plane. It pays at scale with long prompts; it does not pay for a small fleet.

Advanced

🔒 Premium

Prefix Caching and KV ReuseMost requests to a production LLM share a prefix: the same system prompt, the same few-shot examples, the same conversation up to the latest turn. Prefix caching keeps the KV blocks for those tokens resident and skips their prefill, so a 4,000-token system prompt costs compute once instead of once per request. Radix trees make the lookup cheap, block-aligned hashing makes it safe, and the hit rate is what decides whether it is a 2x or a 10x win. The interview question is how you would route to make it hit.

Explore the full curriculum →

NVIDIA LLM INFERENCE & SERVING FAQ

What LLM Inference & Serving questions does NVIDIA ask in interviews?▲

NVIDIA's AI Infrastructure Engineer loop draws llm inference & serving questions such as "When does splitting prefill and decode onto separate GPU pools pay for itself, and what does the KV transfer cost?", "vLLM, SGLang or TensorRT-LLM: which engine do you pick for a new deployment, and what would change your mind?", "When does offloading the KV cache to CPU memory or NVMe beat recomputing it?". Prefill versus decode, the KV cache, PagedAttention and continuous batching, chunked prefill, speculative decoding, disaggregated serving, quantization, vLLM, SGLang and TensorRT-LLM, multi-LoRA and routing: hosting open-weight models at a latency SLO and a cost you can defend. The full set, ordered easy to hard with expert answers, is below.

How should I prepare for the NVIDIA LLM Inference & Serving round?▼

Does NVIDIA hire AI infrastructure engineers?▼

What does the NVIDIA AI infrastructure interview test?▼

Other NVIDIA interview rounds

The other tracks NVIDIA's AI Infrastructure Engineer loop tests.

NVIDIA cuda kernels→NVIDIA gpu architecture→NVIDIA coding systems→NVIDIA networking storage→

Prep the whole NVIDIA AI Infrastructure Engineer loop

LLM Inference & Serving is one round. Unlock every answer across NVIDIA's full loop, plus the concept curriculum, for 6 months. One payment, no auto-renewal. Free questions in every track to start.

Unlock all answers · ₹2,000 / $25 Full NVIDIA interview guide

Independent and not affiliated with NVIDIA. All trademarks belong to their owners.

NVIDIA LLM Inference & Serving interview questions

LLM Inference & Serving questions tagged to NVIDIA

When does splitting prefill and decode onto separate GPU pools pay for itself, and what does the KV transfer cost?

vLLM, SGLang or TensorRT-LLM: which engine do you pick for a new deployment, and what would change your mind?

When does offloading the KV cache to CPU memory or NVMe beat recomputing it?

More LLM Inference & Serving questions for NVIDIA's loop

Why do prefill and decode behave so differently, and why does that matter for the hardware you serve on?

What is the KV cache, and why does it keep growing while a request is being served?

What is the difference between static and continuous batching, and why did it change LLM serving?

How does PagedAttention work, and what problem was it solving?

Define TTFT, TPOT and goodput, and tell me how you would measure each one in production.

You have eight H100s and Llama 3.1 70B. How many concurrent users can you serve, and what changes the number?

A long prompt arrives while sixty users are mid-generation. What happens, and how does chunked prefill fix it?

When does speculative decoding speed up serving, and when does it break even or hurt?

Compare the KV cache footprint of multi-head, grouped-query and multi-head latent attention with numbers.

You need to quantize a model for serving. Which method do you pick, and what do you measure before shipping it?

Every request shares a 2,000-token system prompt. How does prefix caching exploit that, and how does the radix tree work?

You have one GPU and a synchronous API that receives 100 documents at once. Design the batching, and show the latency math.

Your p99 TTFT tripled last night and p50 did not move. Walk me through how you find the cause.

How would you serve hundreds of LoRA adapters on one base model, and what does it cost in throughput?

Design an autoscaler for GPU inference replicas that reacts to load without thrashing.

Concepts behind NVIDIA's LLM Inference & Serving round

Other NVIDIA interview rounds

Prep the whole NVIDIA AI Infrastructure Engineer loop