Red Hat LLM Inference & Serving interview questions

Q: How should I prepare for the Red Hat LLM Inference & Serving round?

Work the llm inference & serving questions on this page in order, study the underlying concepts linked below, then review Red Hat's full interview process for how this round fits the loop. For Red Hat specifically, interviewers weight this: Serving engines and their Kubernetes deployment: routing, disaggregated serving, autoscaling.

Q: Does Red Hat hire AI infrastructure engineers?

Yes, for OpenShift AI and RHEL AI and for upstream work on vLLM, llm-d, KServe and the Kubernetes GPU ecosystem; titles are software engineering titles on those teams.

Q: What does the Red Hat AI infrastructure interview test?

No reliable public breakdown of an AI-infrastructure-specific loop was found, so we do not list rounds. Prepare serving-engine internals and their Kubernetes deployment (routing, disaggregated prefill and decode, autoscaling, GPU scheduling), and expect open-source contribution history to matter.

LLM Inference & Serving is a core part of the Red Hat AI Infrastructure Engineer loop. Prefill versus decode, the KV cache, PagedAttention and continuous batching, chunked prefill, speculative decoding, disaggregated serving, quantization, vLLM, SGLang and TensorRT-LLM, multi-LoRA and routing: hosting open-weight models at a latency SLO and a cost you can defend. Below are the llm inference & serving questions to prepare, the ones tagged to Red Hat first, then the highest-signal questions from our LLM Inference & Serving track, each with an answer written to a senior-engineer bar.

WHAT RED HAT LOOKS FOR HERE · Serving engines and their Kubernetes deployment: routing, disaggregated serving, autoscaling. See the full Red Hat interview process →

LLM Inference & Serving questions tagged to Red Hat

2 questions · 1 unlocked for you

Open full answer →

More LLM Inference & Serving questions for Red Hat's loop

The highest-signal llm inference & serving questions candidates rate most useful, modeled on what Red Hat's AI Infrastructure Engineer loop tests.

15 questions · 9 unlocked for you

Open full answer →

Concepts behind Red Hat's LLM Inference & Serving round

The vocabulary and mental models these questions assume. Start with the foundations free; the deeper, interview-defining ideas are part of premium.

Foundational

Prefill vs DecodeAn LLM request runs in two phases with opposite hardware profiles: prefill reads the whole prompt in one compute-bound pass and decides time to first token, decode emits one token per forward pass and is bound by memory bandwidth. Every serving decision, from batch size to which GPU to buy to whether to split the two phases across machines, follows from that split.

Foundational

The KV CacheThe KV cache stores each token's attention keys and values so decode never recomputes them, turning a quadratic cost into a linear one at the price of memory that grows with every token in every concurrent sequence. Its size, 128 KB per token for Llama 3.1 8B and 320 KB for 70B in bf16, is what caps concurrency and context on a given GPU, so it decides batch size, replica count and whether a model fits at all.

Core

Continuous BatchingContinuous batching schedules at the granularity of a single decode step instead of a whole request, so a finished sequence's slot is refilled on the next iteration rather than when the longest request in the batch ends. It is the scheduling idea that turned LLM serving from a padded, half-idle GPU into one that stays full, and it decides how the engine's scheduler, memory manager and latency SLOs interact.

Advanced

🔒 Premium

PagedAttentionPagedAttention stores the KV cache in fixed-size blocks scattered across HBM and maps each sequence's logical positions to physical blocks through a block table, the same trick an operating system uses for virtual memory. It removes the reservation and fragmentation waste of contiguous allocation, lets blocks be shared between sequences, and is why an engine can decide admission by counting free blocks.

Advanced

🔒 Premium

Chunked PrefillA long prompt's prefill can occupy a GPU for hundreds of milliseconds, and every sequence mid-decode on that GPU waits for it. Chunked prefill splits the prompt into fixed token budgets and interleaves each chunk with a decode step, so decode latency stays flat at the cost of a slower first token for the long prompt. The chunk budget is a knob between TTFT and TPOT, and the interview question is how you would set it.

Advanced

🔒 Premium

Speculative DecodingDecode is memory-bound: each step reads every weight to produce one token. Speculative decoding has a cheap draft propose several tokens, then verifies them all in one forward pass of the big model, so one weight read yields several tokens with output distribution unchanged. It wins 2x to 3x at small batch, breaks even near the ridge point where the GPU is already compute-bound, and lives or dies on the acceptance rate, which is what interviewers ask you to reason about.

Advanced

🔒 Premium

Disaggregated Prefill and DecodePrefill is compute-bound and decode is memory-bound, so running both on the same GPUs means each phase interferes with the other and neither runs on the hardware it wants. Disaggregation puts them on separate pools and ships the KV cache from prefill nodes to decode nodes over the fabric. It lets TTFT and TPOT scale independently and puts high-bandwidth parts where they pay, at the price of a KV transfer per request and a control plane. It pays at scale with long prompts; it does not pay for a small fleet.

Advanced

🔒 Premium

Prefix Caching and KV ReuseMost requests to a production LLM share a prefix: the same system prompt, the same few-shot examples, the same conversation up to the latest turn. Prefix caching keeps the KV blocks for those tokens resident and skips their prefill, so a 4,000-token system prompt costs compute once instead of once per request. Radix trees make the lookup cheap, block-aligned hashing makes it safe, and the hit rate is what decides whether it is a 2x or a 10x win. The interview question is how you would route to make it hit.

Explore the full curriculum →

RED HAT LLM INFERENCE & SERVING FAQ

What LLM Inference & Serving questions does Red Hat ask in interviews?▲

Red Hat's AI Infrastructure Engineer loop draws llm inference & serving questions such as "You need to quantize a model for serving. Which method do you pick, and what do you measure before shipping it?", "vLLM, SGLang or TensorRT-LLM: which engine do you pick for a new deployment, and what would change your mind?", "Why do prefill and decode behave so differently, and why does that matter for the hardware you serve on?". Prefill versus decode, the KV cache, PagedAttention and continuous batching, chunked prefill, speculative decoding, disaggregated serving, quantization, vLLM, SGLang and TensorRT-LLM, multi-LoRA and routing: hosting open-weight models at a latency SLO and a cost you can defend. The full set, ordered easy to hard with expert answers, is below.

How should I prepare for the Red Hat LLM Inference & Serving round?▼

Does Red Hat hire AI infrastructure engineers?▼

What does the Red Hat AI infrastructure interview test?▼

Other Red Hat interview rounds

The other tracks Red Hat's AI Infrastructure Engineer loop tests.

Red Hat cluster orchestration→

Prep the whole Red Hat AI Infrastructure Engineer loop

LLM Inference & Serving is one round. Unlock every answer across Red Hat's full loop, plus the concept curriculum, for 6 months. One payment, no auto-renewal. Free questions in every track to start.

Unlock all answers · ₹2,000 / $25 Full Red Hat interview guide

Independent and not affiliated with Red Hat. All trademarks belong to their owners.

Red Hat LLM Inference & Serving interview questions

LLM Inference & Serving questions tagged to Red Hat

You need to quantize a model for serving. Which method do you pick, and what do you measure before shipping it?

vLLM, SGLang or TensorRT-LLM: which engine do you pick for a new deployment, and what would change your mind?

More LLM Inference & Serving questions for Red Hat's loop

Why do prefill and decode behave so differently, and why does that matter for the hardware you serve on?

What is the KV cache, and why does it keep growing while a request is being served?

What is the difference between static and continuous batching, and why did it change LLM serving?

How does PagedAttention work, and what problem was it solving?

Define TTFT, TPOT and goodput, and tell me how you would measure each one in production.

You have eight H100s and Llama 3.1 70B. How many concurrent users can you serve, and what changes the number?

A long prompt arrives while sixty users are mid-generation. What happens, and how does chunked prefill fix it?

When does speculative decoding speed up serving, and when does it break even or hurt?

Compare the KV cache footprint of multi-head, grouped-query and multi-head latent attention with numbers.

When does splitting prefill and decode onto separate GPU pools pay for itself, and what does the KV transfer cost?

Every request shares a 2,000-token system prompt. How does prefix caching exploit that, and how does the radix tree work?

You have one GPU and a synchronous API that receives 100 documents at once. Design the batching, and show the latency math.

Your p99 TTFT tripled last night and p50 did not move. Walk me through how you find the cause.

How would you serve hundreds of LoRA adapters on one base model, and what does it cost in throughput?

Design an autoscaler for GPU inference replicas that reacts to load without thrashing.

Concepts behind Red Hat's LLM Inference & Serving round

Other Red Hat interview rounds

Prep the whole Red Hat AI Infrastructure Engineer loop