TL;DR: Decide by whether any single job needs more than one card. NVIDIA publishes the RTX PRO 6000 Blackwell Server Edition at 96 GB of GDDR7, 1,597 GB/s, 2 PFLOPS of FP8, up to 600 W, PCIe Gen5, and MIG partitioning into up to four instances, with no NVLink. An H100 SXM is 80 GB at 3.35 TB/s, 1,979 TFLOPS of FP8, 700 W, and 900 GB/s of NVLink. For a fine-tuning service on models up to 30B using parameter-efficient methods, one card holds the model and its optimizer state, jobs never cross a card, and the RTX PRO 6000's larger memory and four-way MIG partitioning make it the better multi-tenant unit. The decision flips the moment a job needs two cards, because tensor parallelism over PCIe Gen5 at about 64 GB/s each way against 900 GB/s of NVLink is a fourteen-times penalty on the traffic that happens twice per layer. Full-parameter fine-tuning of anything above roughly 13B crosses that line.
How to approach it
Establish the largest job the service must support, in bytes, because that decides whether multi-card is required at all. Compute what fits on each card for the training method offered. Then the multi-tenancy story, which is where MIG matters. Then the case that flips it, stated as a model size. Close with what you would do if the product later needs larger models, since that is the real risk in this choice.
A strong answer
A typical situation: a platform team is sizing hardware for a service that fine-tunes customer models up to 30B with low-rank adaptation, with occasional full-parameter runs on models up to 8B. Tenants are isolated, jobs are small and numerous, and utilization matters more than peak speed.
What fits, which is the whole first half:
low-rank adaptation on a 30B model, bf16 base with fp32 adapter state
base weights 30e9 x 2 B = 60 GB
adapter parameters at ~0.5% of the base, with optimizer state at 16 B per parameter:
0.15e9 x 16 = 2.4 GB
activations and workspace for a modest batch and sequence length: say 15 GB
total = 77.4 GB
on RTX PRO 6000 (96 GB): fits, with 18.6 GB spare
on H100 SXM (80 GB): does not fit comfortably; needs the base in FP8 or a second card
full-parameter fine-tuning, at the corpus figure of 16 bytes per parameter of training state
8B model: 8e9 x 16 = 128 GB -> two cards minimum on either part
13B model: 13e9 x 16 = 208 GB -> three cards on RTX PRO 6000, three on H100
30B model: 30e9 x 16 = 480 GB -> five or more cards, definitely multi-card
sanity: the low-rank case fits on one RTX PRO 6000 and not on one H100, and the
full-parameter case fits on neither, so the product's method mix decides the part
Model Memory Footprint covers the 16 bytes per parameter figure and where it comes from. Accelerator Selection: H100 to B300 and RTX PRO 6000 covers the three gates in general.
The multi-tenancy argument, which is where the RTX PRO 6000 earns it:
MIG on the RTX PRO 6000: up to four fully isolated instances per card
a 96 GB card becomes four instances of roughly 24 GB each
each instance has its own memory and a partition of the compute, with hardware isolation
so one card serves four small tenants with a guaranteed share rather than a scheduler's
promise
what that does to the economics of a fine-tuning service
many jobs are small: a 7B low-rank fine-tune fits in a 24 GB instance
without partitioning, that job holds a whole card and wastes most of it
with four-way partitioning, the same card serves four such jobs
effective capacity for the small-job tail rises up to four times, and the utilization
number is what decides the service's margin
sanity: this is the same argument as any GPU-sharing platform, and the reason it lands
harder here is that fine-tuning jobs are bursty and mostly small, so the tail of
small jobs is a large fraction of the workload
MIG, MPS and Time-Slicing covers the isolation mechanisms and what each guarantees. The numbers that confirm the choice on real hardware are ordinary ones:
what to measure before committing to either part
nvidia-smi mig -lgip available MIG profiles and how many of each fit
DCGM_FI_DEV_FB_USED per instance whether a 24 GB instance actually holds the job
DCGM_FI_PROF_SM_ACTIVE whether small jobs leave a whole card idle today
lspci -vv on the NIC and GPU the PCIe generation and width each card negotiated,
since a card at x8 halves the already-slow path
nccl-tests all_reduce_perf, 2 GPUs the PCIe number, measured rather than assumed, against
the same test on an NVLink pair
sanity: the two-GPU nccl-tests result on each part is the single measurement that settles the
argument, because it turns a specification difference into a number for your workload
The point where the decision flips, stated as arithmetic:
tensor-parallel traffic across two cards, per token, for a 13B model
hidden size about 5,120, 40 layers
two all-reduces per layer per token, bf16, TP=2 (factor 2(N-1)/N = 1.0):
40 x 2 x 5,120 x 2 B x 1.0 = 819 KB per token
time for that traffic
PCIe Gen5 x16 at 64 GB/s: 819e3 / 64e9 = 12.8 microseconds
NVLink at 900 GB/s: 819e3 / 900e9 = 0.9 microseconds
ratio 14x
for training rather than decode the relevant traffic is larger still
gradient all-reduce per step for a 13B model in bf16: 26 GB
over PCIe at 64 GB/s: 0.41 s per step of pure communication
over NVLink at 900 GB/s: 0.03 s
sanity: 0.41 seconds of communication against a step time measured in hundreds of
milliseconds means the PCIe part spends more time communicating than computing, and
that is the sentence that decides the question the moment a job needs two cards
The reversal condition: if the roadmap includes full-parameter fine-tuning above roughly 13B, or serving the resulting models at tensor-parallel degrees above one, the RTX PRO 6000 is the wrong purchase and the analysis above becomes an argument against it. The risk in this decision is not the current product but the next one, so the useful move is to ask what the largest supported model will be in eighteen months and size to that. A mixed fleet is also legitimate and often correct: RTX PRO 6000 cards for the many small partitioned jobs, and a smaller number of NVLink-connected nodes for the jobs that need them, with the scheduler routing by job size.
What interviewers probe next
- "What about inference on the same fleet?" Serving a model that fits on one card is fine and often excellent there, since replicas do not communicate. Serving one model across cards has the same PCIe problem.
- "Does MIG hurt performance?" Each instance gets a fraction of the compute, so a single large job is slower than on the whole card. For many small jobs the aggregate is much better.
- "What about power?" 600 W against 700 W per card, and the RTX PRO 6000 is PCIe in standard chassis, so rack density and cooling are easier, which is a real operational advantage.
- "How would you hedge?" Buy a small number of NVLink nodes alongside, and route by job size. The hedge costs little and removes the roadmap risk.
Common mistakes
- Comparing FP8 FLOPS between the parts and ignoring that the workload never uses more than one card.
- Missing that the absence of NVLink is a hard constraint rather than a performance difference.
- Ignoring MIG partitioning, which is where the multi-tenant economics live.
- Sizing for today's largest model rather than the roadmap's.
- Treating the choice as exclusive when a mixed fleet routed by job size is usually better.
Key takeaways
- One card per job is the criterion: a 30B low-rank fine-tune needs about 77 GB and fits on 96 GB but not on 80 GB.
- Full-parameter fine-tuning at 16 bytes per parameter needs 128 GB for 8B and 480 GB for 30B, which is multi-card on any part.
- MIG into four instances per card turns the tail of small jobs into up to four times the effective capacity.
- Two-card tensor parallelism over PCIe Gen5 is about fourteen times slower than NVLink on the same traffic, and a 13B gradient all-reduce takes 0.41 s over PCIe against 0.03 s over NVLink.
- Size to the eighteen-month roadmap, and consider a mixed fleet routed by job size rather than one part for everything.