Node Lifecycle: Drain, Upgrade and Return
A node moves through a fixed cycle between provisioning and decommissioning, and most fleet operations are one lap around it: cordon so nothing new lands, drain so running work finishes or moves, act, validate, then return to the pool. The wall-clock cost of a fleet-wide change is dominated by draining rather than by the change itself, which makes the plan a scheduling document rather than a technical one.
TL;DR: Five states and the transitions between them: schedulable, cordoned, draining, out of service, and returning. Cordoning is instant and cheap, because it only stops new work from landing. Draining is where the time goes, since a running job either finishes or is preempted, and a fleet-wide operation's wall-clock is set by the longest running job rather than by the work being done. A node coming back must pass the same health checks a new node passes, or a repaired node reintroduces the fault it was repaired for. Two policies decide whether this is smooth: a drain deadline after which work is preempted rather than waited for, and a limit on how many nodes may be out at once, which is what stops a rolling operation from removing more capacity than the fleet can spare.
The cycle
Cordoning and draining are separate on purpose. Cordon is reversible and free, so it is the correct first action for a suspected problem: the node stops taking new work while you decide whether it needs to come out at all. Draining commits to losing the node's current capacity.
Draining is where the wall-clock goes
a fleet-wide operation on 256 nodes
the work itself: reimage and reboot, about 20 minutes per node
total node time = 256 x 20 min = 85 hours
but nodes must drain first
mean remaining runtime when a node is cordoned, for jobs averaging 12 hours: about 6 hours
with 16 nodes drained and worked at a time: 256 / 16 = 16 batches
wall-clock = 16 x (6 h drain + 0.33 h work) = 101 hours = about 4.2 days
what changes it
upgrading at checkpoint boundaries agreed with job owners cuts the drain term sharply
more nodes per batch, bounded by how much capacity can be offline at once
a drain deadline, after which work is preempted rather than waited for
sanity: 85 hours of work becomes 4.2 days of calendar, and the whole difference is drain, so
the plan is a scheduling document and the estimate belongs in job boundaries rather
than node counts
Returning a node, which is where repairs regress
A repaired node is a new node as far as trust is concerned. Returning it without the same checks a fresh node passes is how a fault comes back, and the fault will look new because nobody associates it with the repair.
| Check | What it catches |
|---|---|
| Firmware inventory against the fleet standard | A repair that reintroduced a version skew |
| Every GPU present at the expected clocks and power limit | A card reseated wrongly or a different part fitted |
| NVLink present between all GPUs | A fabric manager not started, or an NVSwitch fault |
| Fabric ports Active at the expected rate | A cable disturbed during the physical work |
| A single-node collective benchmark within tolerance of the fleet median | Anything the individual checks miss |
| A short burn-in under load | A part that passes idle and fails hot |
Node Health Checks and Burn-In covers what belongs in each of those. The lifecycle point is narrower: the checks run on the return transition and not only at provisioning, and a node that fails them goes back out rather than into the pool with a note.
The two policies
- A drain deadline. Without one, a node waits on the longest job indefinitely and a rolling operation stalls. With one, work is preempted at the deadline and the fleet's checkpoint discipline is what makes that survivable, which is a good reason to have both.
- A concurrent-out limit. Cap how many nodes may be out of service at once, expressed as a fraction of the fleet or as whole racks, so a rolling operation cannot remove more capacity than the queue can absorb. This is the policy that stops an automated repair loop from draining the cluster when a bad health signal fires fleet-wide.
That second policy connects to automation directly: any system that cordons nodes on a signal needs a rate limit and a concurrency cap, or one wrong signal drains a large fraction of the fleet in minutes.
What interviewers are listening for
That drain dominates. Candidates estimate fleet operations from the work per node and are usually out by a factor of several, and saying that the wall-clock is set by the longest running job is the observation that separates a plan from an estimate. The second signal is running full health checks on the return transition rather than trusting a repair. The third is the concurrent-out limit, because it is the difference between an automation that helps and one that causes the outage it was built to prevent.
Key takeaways
- Five states: schedulable, cordoned, draining, out of service, validating. Cordon is free and reversible; drain commits.
- A 256-node operation is 85 hours of work and about 4.2 days of calendar, and the gap is entirely drain time.
- Return a node through the same checks a new node passes, including a collective benchmark against the fleet median.
- Set a drain deadline, after which work is preempted, or a rolling operation stalls on the longest job.
- Cap how many nodes may be out at once, which is what stops an automated cordon loop from draining the fleet on a bad signal.