Fleet and Reliability
Running the machine that other people's jobs run on: how a GPU gets scheduled and why fragmentation is the real cost, the four networks and what each one breaks, how devices fail and what catches it, and how to share a cluster between tenants without lying to any of them.
The three research briefs that shaped this curriculum were written by people thinking about inference. All three of them left this course out, and one said so directly: a platform or SRE-flavoured reader was told to stop after the serving course, because the kernel material would not repay them.
That advice leaves the largest part of the field unserved. Running the cluster is its own job, it is where a great many of the open roles are, and its interviews test things no serving course covers: a broken node to diagnose, a scheduling design under multiple tenants, a fabric that is slow for reasons the application cannot see.
Four modules. The scheduler first, because a GPU is not a resource your orchestrator was designed for and almost every capacity surprise starts there. Then the fabric, because a cluster has four networks and confusing them costs a week. Then failure, which at fleet scale is a schedule item rather than an event. Then tenancy, which is the part that is technically simple and politically hard, and the part most likely to be what you are actually hired to fix.
The capstone produces the acceptance runbook for a new cluster: the document that decides whether hardware you have paid for is allowed to run anything.
When you finish, you can
- Explain why a fleet can be ninety percent allocated and thirty percent utilised, and what recovers the difference
- Name the deadlock gang scheduling prevents and the cost of preventing it
- Place a distributed job with the topology in mind, and say what a bad placement costs
- Distinguish the four networks in a GPU cluster and diagnose which one a symptom belongs to
- Recognise the common device failure signatures and the telemetry that catches each before users do
- Design a health gate that stops a bad node from eating a good job
- Set quota and fair-share policy that survives contact with the researchers it constrains
- Produce an acceptance runbook for a new cluster, in the capstone
BEFORE YOU START · Foundations of AI Infrastructure, or the equivalent. Kubernetes familiarity helps and is not assumed; this course explains what is different about scheduling accelerators rather than teaching container orchestration from scratch.
Syllabus
The scheduler
How a GPU job gets a GPU, and why that is harder than it sounds: an indivisible resource makes fragmentation the dominant cost, distributed jobs need all-or-nothing placement, and where the devices sit matters more than which scheduler you chose.
The fabric
What connects the devices and what feeds them: the four separate networks in a GPU cluster and which symptom belongs to which, why storage stops being about throughput once the files get small, and the physical constraints that decide what any of it can promise.
The fleet
Failure as a scheduled item rather than an event: the small number of ways devices actually fail and how each announces itself, the health gate that keeps a bad node away from a good job, and on-call designed as a queue instead of a rota.
The tenancy
Sharing a cluster without lying to anyone: quota as the policy engineers get handed rather than given, what isolation actually guarantees at each level, and a capstone that produces the acceptance runbook deciding whether new hardware may run anything.