Every performance question is bound by something before it is anything else

TL;DR: There are five things a workload can be waiting on: memory bandwidth, arithmetic throughput, latency it cannot hide, communication between devices, or the host that feeds it. Name which one binds before proposing anything, because every optimisation only moves one of them and the other four are unaffected.

Where you are. Last lesson of the first module. You have the imbalance between arithmetic and bandwidth, and you have the two hierarchies. This lesson turns them into the procedure you will use for the rest of the course.

Five candidates, and no others

Performance conversations go wrong when they open with a fix. "Increase the batch size", "use a faster GPU", "turn on the compiler" are all answers to questions nobody established. The discipline is to name the binding resource first, and the list is short enough to hold in your head.

The right-hand column is the one that earns its keep, because it tells you what not to try. An engineer who knows a workload is memory-bound and still proposes a chip with more arithmetic throughput has not used the diagnosis.

The procedure, without a profiler

Interviews rarely hand you a profiler, and you are expected to reason anyway. Four steps.

One: compute the floor. What must this work move, and what must it compute, at minimum? Bytes divided by bandwidth gives a memory time. Operations divided by achievable arithmetic rate gives a compute time. The larger is your floor.

Two: compare the floor with the observation. If the measured time is close to the floor, there is no bug and the answer is that the work is that expensive. Say so. A meaningful share of real "performance problems" are physics, and recognising that is worth more than another week of investigation.

Three: if it is far above the floor, find which of the five. Ask what grows when you change one thing. Does the time scale with device count? Communication. With batch size sublinearly? You were latency-bound and now you are not. Not at all when you halve the model precision? Not memory-bound.

Four: propose the fix that moves the binding term, and say what you expect it to buy. A prediction is what separates a diagnosis from a guess, and it is checkable afterwards.

The two errors worth naming

Skipping to the fix. This is the common one, and it is what the procedure is for.

Never checking the floor. This is the expensive one. A team investigates a collective that takes eight seconds, spends a week on the network, and eventually discovers the operation moves enough bytes that eight seconds is close to the best the fabric can do. The floor calculation takes two minutes and would have redirected the week. Compute it first, every time, even when you are confident.

There is a third that only shows up under pressure: diagnosing from the dashboard rather than the arithmetic. Utilisation, as the first lesson said, reports residency. Memory-used reports allocation, not traffic. Both are compatible with a machine doing almost nothing useful, so neither one names a binding resource on its own.

Do this before moving on

Take a workload you understand and write the five-row table for it, filling in what you would expect to observe if each of the five were the binding resource. Then pick the one you believe it is and write down a prediction: "if I halve the precision, time should fall by about X." Being wrong is useful here. Being unable to make a prediction at all means the diagnosis was not real.

Go deeper

• Memory-Bound vs Compute-Bound Kernels takes the first two of the five candidates down to kernel level, where the classification gets sharper.
• Roofline Model is the tool that separates the first two, and the notation the rest of the course uses.
• Arithmetic Intensity by Operation tabulates where common operations land, which is the shortcut once you trust the method.
• Your training run is slow: compute, memory, network or IO? is this procedure applied to the hardest version of the question.
• Read this Nsight summary: what is the bottleneck? is what the same reasoning looks like when you do have a profiler in front of you.

Key takeaways

• Five candidates bind a workload: memory bandwidth, arithmetic, latency, communication and the host. Name one before proposing anything.
• Compute the floor first. If the measurement is near it, the work is simply that expensive, and saying so is the right answer.
• Each optimisation moves exactly one term, so a diagnosis tells you what not to try as much as what to try.
• Attach a prediction to the fix; that is what makes it a diagnosis rather than a guess.
• Dashboards report residency and allocation, not traffic or efficiency, so they cannot name a binding resource on their own.

Check yourself

Answer before you look. Recalling it is what makes it stick; recognising it does not.

1. 1A distributed job's step time grows noticeably as devices are added, while the work assigned to each device stays constant. Which resource binds?

   Arithmetic throughputMemory bandwidthThe hostCommunication

2. 2A collective operation takes eight seconds and the team suspects a network fault. What is the first thing to compute, and why does it usually change the investigation?

   Answer it, then reveal

3. 3The device shows regular short bursts of activity separated by consistent gaps. Which two candidates does that pattern point at, and how do you separate them?

   Communication and the host; separate them by device countMemory and arithmetic; separate them by comparing achieved bandwidth with achieved FLOPsLatency and the host; separate them by asking whether the gaps shrink when you enlarge the unit of work or when you feed the device fasterArithmetic and communication; separate them by lowering precision