TL;DR: Node Feature Discovery runs on every node, reads the PCI bus and the kernel, and writes labels such as
feature.node.kubernetes.io/pci-10de.present=trueand the kernel version. The GPU Operator watches those labels and, on each matching node, runs DaemonSets in dependency order: the kernel driver (as a container, or it detects a host-installed one), the container toolkit, the device plugin that advertisesnvidia.com/gpu, DCGM and its exporter for metrics, GPU Feature Discovery for finer labels (product, MIG mode, driver version), MIG manager where configured, and a validator that gates the node until every step passes.
How to approach it
Start from the pod's point of view: a pod asking for nvidia.com/gpu needs a scheduler that sees the resource, a runtime that injects the driver, and a driver that is loaded. Name the component responsible for each, bottom-up, then say how each DaemonSet knows which nodes to land on, which is NFD's job. Close with why an operator instead of a golden image: the number of nodes times the number of versions that must agree.
A strong answer
A typical situation: a platform engineer joins a team running 200 GPU nodes and asks why there are eleven DaemonSets in the gpu-operator namespace. Each one exists because a layer below the pod has to be installed, kept in version agreement, and re-checked when the node changes.
Node Feature Discovery comes first and is not NVIDIA-specific. Its worker pod runs on every node, probes hardware and kernel, and publishes labels: PCI vendor 10de present (NVIDIA), the kernel version, CPU features, the OS release. Nothing in NFD knows what a GPU is. It knows that a device with vendor ID 10de sits on the bus. That label is the selector the operator's DaemonSets use, so a node where NFD did not run, or where the label was overwritten, is a node the operator ignores. Kubernetes GPU Scheduling describes the device plugin's side; here is what stands under it.
The operator's chain, in the order it must succeed:
| Component | What it does | What breaks if it is missing |
|---|---|---|
| driver DaemonSet | builds or loads the kernel module matching the node's kernel; skipped if the host already has a driver | nvidia-smi fails; nothing above works |
| container toolkit | installs the runtime hook that injects libcuda.so and device nodes into containers; registers the nvidia runtime with containerd | every GPU container fails at start with a library load error |
| device plugin | enumerates GPUs through NVML and registers nvidia.com/gpu: N with the kubelet | Allocatable stays at zero; pods Pending |
| GPU Feature Discovery | writes finer labels: nvidia.com/gpu.product, gpu.count, gpu.memory, mig.strategy, driver and CUDA version | scheduling by product or driver version is impossible |
| DCGM plus DCGM exporter | runs the datacenter GPU manager and exposes SM activity, memory, XID counts, power and thermals on a Prometheus endpoint | no per-GPU metrics; utilization questions cannot be answered |
| MIG manager | applies a MIG profile per node label and restarts the plugin so slices are advertised | MIG nodes advertise whole GPUs or nothing |
| validator | runs a CUDA workload and checks each layer, removing the operator's taint when everything passes | nodes with a half-installed stack accept pods and fail them |
Why an operator rather than baking it all into the node image comes down to the versions that must agree and the count of nodes:
inputs: 200 nodes; per node the stack has 5 versions that must agree
(kernel, driver, toolkit, device plugin, DCGM)
one driver upgrade per quarter, one kernel patch per month
manual bring-up per node: driver install and reboot ~15 min, toolkit ~5 min, plugin and
exporter ~5 min, a validation run ~5 min = 30 min
fleet: 200 × 30 min = 6,000 node-minutes = 100 engineer-hours per full rollout
per year: 4 driver rollouts + 12 kernel patches, each touching every node
≈ 16 × 100 h = 1,600 engineer-hours, before any failure handling
operator: the same rollout is a version bump in a ClusterPolicy, applied node by node
with the validator gating each one; engineer time is the canary and the watch
sanity: 1,600 hours is most of an engineer's year spent typing the same commands;
the operator exists because the stack has to be re-done every time any layer moves
The operator's other job is convergence: when a node reboots on a new kernel, NFD updates the kernel label, the driver DaemonSet pod restarts and rebuilds, the validator re-runs, and the node is gated until it passes. A golden image gives the same first-boot result and none of the re-convergence.
The condition that reverses the choice: a small fleet with one node pool, one image, and a team that reimages rather than upgrades in place can run a host-installed driver and only the device plugin, and skip the operator's moving parts. CoreWeave-style fleets with many node types and MIG profiles run the full operator because the label-driven per-node configuration is the point.
The reversal condition: a small fleet with one node pool, one image and a team that upgrades by hand. There the operator's reconciliation is more machinery than the problem needs, and a pinned DaemonSet is easier to reason about. Cluster Bring-Up: Firmware, Drivers and the Stack is the order the layers have to come up in.
What interviewers probe next
- "A node's NFD label is missing. What happens?" Nothing, which is the failure: no DaemonSet selects the node, it advertises no GPUs, and it looks like a CPU node until someone checks.
- "Where do
nvidia.com/gpu.productlabels come from?" GPU Feature Discovery, which runs after the plugin and queries NVML; NFD only knows the PCI vendor. - "What does the validator test?" Driver loaded, toolkit injecting, plugin registered, and a CUDA sample running on a real GPU; it removes the operator's taint on success and leaves it on failure.
- "Where do you read GPU utilization for a pod?" DCGM exporter fields labeled with pod and namespace,
DCGM_FI_DEV_GPU_UTILand the SM activity fieldDCGM_FI_PROF_SM_ACTIVE, scraped by Prometheus.
Common mistakes
- Saying NFD "detects GPUs"; it detects a PCI vendor ID, and everything NVIDIA-specific comes from GPU Feature Discovery.
- Listing the components without their order, then being unable to say which one a given failure comes from.
- Forgetting the validator and the taint it manages, which is the usual reason a healthy-looking node refuses pods.
- Treating the operator as optional decoration rather than the thing that keeps five versions in agreement across the fleet.
Key takeaways
- NFD labels the bus and kernel; the operator's DaemonSets select on those labels, in order: driver, toolkit, device plugin, GFD, DCGM exporter, MIG manager, validator.
- Allocatable
nvidia.com/gpucomes from the device plugin; product and driver labels come from GPU Feature Discovery; metrics come from the DCGM exporter. - 200 nodes × 30 minutes × 16 rollouts a year is about 1,600 engineer-hours done by hand; the operator turns it into a version bump gated per node.
- A missing NFD label makes a GPU node invisible; a failed validator leaves a taint behind.