Expert Parallel and All-to-All Backends
A mixture-of-experts model can be split two ways and the choice changes everything. Tensor parallelism shards each expert across GPUs, which keeps every GPU busy and reads every expert's shard on every token. Expert parallelism gives whole experts to whole GPUs, which reads only the selected experts but requires an all-to-all to route tokens to them and back. The all-to-all is the cost, its backend is a configuration choice matched to the interconnect, and expert load imbalance is what actually limits the result.
TL;DR: Expert parallelism trades communication for bandwidth. With tensor parallelism over a mixture-of-experts layer, every GPU holds a slice of every expert and reads its slice of all of them, so the bytes read per token do not fall with sparsity. With expert parallelism, each GPU holds whole experts, reads only the ones its tokens selected, and pays an all-to-all in each direction to get the tokens there and the results back. For a model with 256 experts and 8 active, the read saving is large and the all-to-all is the price. In vLLM the switch is
--enable-expert-parallel, the effective width isEP_SIZE = TP_SIZE x DP_SIZE, and--all2all-backendselects the implementation:allgather_reducescatterworks anywhere,deepep_high_throughputanddeepep_low_latencytarget multi-node prefill and decode over RDMA, and the FlashInfer NVLink backends target systems where the domain is large enough to keep the all-to-all inside it. The thing that limits the result in practice is not the backend; it is that experts are not selected evenly, which--enable-eplbaddresses by replicating hot experts.
Why expert parallelism exists
a mixture-of-experts layer with E experts, k active per token
tensor parallelism across N GPUs
each GPU holds 1/N of every expert
per token, each GPU reads its shard of the k selected experts... but the shards are
interleaved, so in practice the layer's weights are read as a sharded whole
bytes read scale with E's active portion but the parallelism does not exploit locality
communication: two all-reduces of the hidden vector, the same as a dense layer
expert parallelism across N GPUs
each GPU holds E/N whole experts
per token, only the GPUs holding the k selected experts do any work
bytes read per token across the system = k experts' full weights, no more
communication: an all-to-all to send each token to the GPUs holding its experts, then a
second all-to-all to bring the results back
worked, GLM-5.3's shape: E = 256, k = 8, one expert is 3 x 6,144 x 2,048 = 37.7M parameters
in FP8, one expert = 37.7 MB
per token, expert weights read = 8 x 37.7 MB = 302 MB (plus the shared expert)
spread over EP=64: each GPU reads only what it holds and was selected for
all-to-all volume per token = hidden x 2 B x k, each direction
= 6,144 x 2 B x 8 = 98 KB per token per direction
sanity: 302 MB of weight reads against 98 KB of communication is a ratio of about 3,000 to 1,
which is why expert parallelism wins whenever the interconnect is not the bottleneck,
and why it collapses when the all-to-all crosses a slow fabric
The backends, and matching them to the interconnect
| Backend | Where it belongs |
|---|---|
allgather_reducescatter | The default; correct anywhere, and the fallback when nothing else applies |
deepep_high_throughput | Multi-node prefill over RDMA, where the batch is large and bandwidth matters more than latency |
deepep_low_latency | Multi-node decode over RDMA, where each step is small and latency dominates; works with CUDA graphs |
flashinfer_nvlink_one_sided | Systems where the expert-parallel group sits inside one NVLink domain |
flashinfer_nvlink_two_sided | The same, with a two-sided exchange strategy |
The reason a rack-scale NVLink domain matters so much for these models is exactly this all-to-all. NVLink Domains and the NVL72 Rack gives the roughly eighteen-times cliff between inside and outside the domain, and the all-to-all is the traffic that pays it twice per MoE layer per token.
Load imbalance, which is what actually limits it
the problem
routing is learned, not uniform. Some experts are selected far more often than others
with EP, a GPU's work is proportional to how often its experts are chosen
a decode step finishes when the SLOWEST rank finishes, so the hottest expert sets the pace
what that costs, illustratively
suppose the busiest rank receives twice the mean token count
the step takes twice as long as a perfectly balanced one
the other ranks idle for half the step
so the fleet delivers half the throughput its bandwidth would allow, and no backend change
fixes it
the mechanism vLLM provides
--enable-eplb turns on the expert parallel load balancer
--eplb-config controls it: window_size 1000 engine steps of history by default,
step_interval 3000 steps between rebalances, num_redundant_experts 0 by default,
use_async true so transfers do not block
redundant experts replicate hot experts onto additional ranks so their load splits
the cost is memory: the documentation gives roughly 2.4 GB per redundant expert for
DeepSeek-V3, so redundancy is bought in units of gigabytes per rank
sanity: measure the imbalance before enabling anything, because a balanced workload gains
nothing from EPLB and pays the memory anyway
The decision, in order
- Is the model a mixture of experts? If not, none of this applies and tensor parallelism is the whole story.
- Does the expert-parallel group fit inside one NVLink domain? If yes, use an NVLink backend and the all-to-all is nearly free. If not, choose a DeepEP backend by phase.
- Set the width. In vLLM,
EP_SIZE = TP_SIZE x DP_SIZE, so expert parallelism is configured by choosing tensor and data parallel degrees rather than by naming EP directly. - Measure imbalance with the balancedness logging before enabling the balancer.
- Enable EPLB with a redundancy budget you have the memory for, then measure again.
What interviewers are listening for
That expert parallelism is a trade rather than an upgrade. The strong answer names what it buys (only the selected experts are read) and what it costs (two all-to-alls per MoE layer per token), then says the backend follows the interconnect, then adds that load imbalance is the real limit. That last point is the one that separates people who have run these deployments, because every published all-to-all benchmark looks fine and every real MoE deployment has a hot-expert problem.
Key takeaways
- Expert parallelism reads only the selected experts and pays two all-to-alls per MoE layer per token; tensor parallelism reads everything and pays two all-reduces.
- For a 256-expert model with 8 active, per-token expert reads are around 302 MB against roughly 98 KB of all-to-all each way, a ratio near 3,000 to 1.
- In vLLM the width is
EP_SIZE = TP_SIZE x DP_SIZE, enabled with--enable-expert-parallel. - Match
--all2all-backendto the interconnect: NVLink backends inside a domain,deepep_high_throughputfor multi-node prefill,deepep_low_latencyfor multi-node decode. - Expert load imbalance sets the step time because the slowest rank finishes last;
--enable-eplbwith redundant experts fixes it at roughly 2.4 GB per redundant expert.