TEMPO: Makespan-Aware Expert-Parallel Load Balancing Across Memory- and Compute-Bound Regimes

Abstract: In expert-parallel (EP) MoE serving, every layer synchronizes at the slowest GPU. Dispatchers balance token counts (EPLB, LPLB, UltraEP) or activated-expert counts (METRO), assuming expert time is linear in one. Measurements on two datacenter GPU generations show it is neither: below $\nstar!\approx!156$--$168$ tokens, HBM weight streaming dominates---cost attaches to \emph{activated replicas}, not tokens; above it, grouped GEMM rounds tokens to 128-tile $M$-tiles, so \emph{splitting} an expert adds padded compute. A max-affine profile $t=\max(a+bG,\,c+βN)$ captures both regimes. Realistic decode batches hold hot experts in the linear regime and cold in the flat \emph{simultaneously}; recorded batches show proxy dispatches differ by $1.4$--$1.6\times$ in modeled block time (p95 up to $1.7\times$), and \emph{which} proxy wins flips with the regime. We formalize per-batch dispatch as a fixed-charge makespan problem---NP-hard on two fully replicated GPUs, polynomial in degenerate limits---and present \sys{}, a makespan-aware dispatcher solving it in milliseconds off the critical path; its SGLang integration runs out-of-process and fuses dispatch with count collection into one in-graph kernel. Anchored by an 8-GPU Testbed~A microbenchmark, \sys{} stays within 1\% of the best fixed baseline everywhere and wins by up to $15.5\%$ where regimes mix. End-to-end on Testbed~B, Qwen3-235B (inside the win region) gains $4$--$6\%$ throughput and cuts p99 latency by ${\sim}15.6\%$; DeepSeek-V3 (outside, communication-dominated) shows only mechanism cost. A phase diagram, not a universal win, is the claim: it predicts both outcomes before deployment.

• The paper introduces TEMPO, a millisecond-scale dispatcher that minimizes a calibrated makespan combining activated-expert costs, token traffic, and 128-token GEMM tile padding across memory- and compute-bound regimes.
• TEMPO tracks the best fixed dispatch policy across tested phase-diagram cells, delivering up to 8.5–11.8% modeled gains and improving Qwen3-235B throughput by 5.0% while reducing p99 TPOT by 15.6%.
• The paper shows that no single token- or activation-balancing proxy is reliable because 92–100% of real decode batches mix both cost regimes, while DeepSeek-V3 results demonstrate that adaptive dispatch provides little benefit when statistical averaging removes exploitable imbalance.

Motivation: the dispatch layer is where imbalance lives

In expert-parallel (EP) Mixture-of-Experts serving, both the dispatch and combine all-to-all collectives synchronize the EP group, so each MoE layer costs exactly the makespan—the maximum per-GPU time—regardless of how idle other GPUs are. Production systems split the balancing problem into a placement layer (e.g., EPLB), which replicates hot experts on a minutes-scale horizon using long-run averages, and a per-batch dispatch layer that decides how each expert's tokens are split across its replicas. Because batch-to-batch routing fluctuates far more than the average—the bottleneck GPU of a given batch routinely runs 1.5–2× the mean—the dispatch layer retains most of the recoverable time.

Existing dispatchers optimize proxies for time rather than time itself. Token-balancing methods (LPLB's linear program, UltraEP's exact quotas, EPLB's uniform replica split) minimize maximum tokens per GPU; activation-balancing methods such as METRO minimize maximum activated experts per GPU. Each proxy encodes an implicit linearity assumption—time proportional to tokens or to expert count—and this paper's central contribution is to show, measure, and exploit why neither assumption holds.

A measured two-regime cost model

Microbenchmarking DeepGEMM fp8 masked grouped GEMM across activation-count and token-count grids on two GPU generations reveals a two-regime structure well captured by a max-affine model:

$t_g = \max\bigl(a + b\,G_g,\; c + \beta\,N_g\bigr)$

where $G_g$ counts activated (expert, replica) pairs and $N_g$ token count on GPU $g$. Below an inflection of roughly 156–168 tokens per expert, cost is flat in tokens: streaming the expert's weights from HBM dominates, so cost attaches to activated replicas, not tokens. Above it, cost is linear in tokens—but not exactly linearly, because grouped GEMM rounds every expert's tokens up to 128-token $M$-tiles, meaning fragmentation manufactures padded compute. The fitted activation floor $b$ spans $1.74\,\mu$s (Qwen3-30B shape) to $14.8\,\mu$s (DeepSeek-V3 shape)—an 8.5× spread—with mean fit error of 4–8%. A roofline check confirms the mechanism: fitted floors sit at 1.13–1.24× the HBM-bandwidth bound, i.e., weight streaming at 80–88% of peak bandwidth.

A tile-aware extension adds one parameter ($b_2 \approx b/3$) to price the sub-linear later steps of the staircase, cutting multi-tile-region model error from ~10% (worst 25%) to 2.0%. This term matters at prefill scale—tile-aware search changes 97–99% of batches rescaled to 4096–8192 tokens/GPU and cuts modeled block time by a median 4.5–6.0%—but is inert at decode, where the floor is the only active step.

Two empirical facts make this model operationally decisive. First, on recorded batches with fixed placement, the dispatches produced by deployed proxy policies differ by 1.4–1.6× in modeled block time (p95 up to 1.7×), and which proxy pays flips with regime: at batch size 128 the uniform replica split pays 51% over the modeled optimum while METRO pays least; at 512–1024 static pays 47–56% while token-LP is optimal. Second, the regimes are not separable operating points: 92–100% of real decode batches contain both regimes simultaneously, with hot experts deep in the linear region carrying ~91% of tokens while roughly half of activated experts remain flat-region. No fixed proxy can be tuned into safety; the transition zone is the decode operating range itself.

Problem formulation and theory

The paper formalizes per-batch dispatch as a fixed-charge makespan problem: minimize the maximum over GPUs of $$\max(a + b\sum_e z_{e,g},\; c + \beta \sum_e x_{e,g})$$, where $G_g$0 are activation indicators and $G_g$1 token shares supported on each expert's replica set. Three results frame its difficulty:

• NP-hardness: deciding feasibility of makespan $G_g$2 is NP-complete already with two fully replicated GPUs and $G_g$3, via reduction from Balanced PARTITION. The max form turns one budget into two simultaneous knapsack constraints—an activation-cardinality cap and a token-capacity cap.
• Polynomial degenerate limits: with $G_g$4 the problem reduces exactly to the token LP; with $G_g$5 it becomes an optimal semi-matching solvable by augmenting paths. Hardness, like the systems failure, lives precisely in the regime interaction.
• Additive approximation: under full replication, descending-token round-robin achieves makespan $G_g$6, via row-wise domination showing every monotone load measure is minimized on the same GPU. Empirically the constant appears tight (worst observed ratio to $G_g$7 of 0.998 over 1296 instances including adversarial constructions), though no matching lower-bound proof is given. Under restricted replica sets no analogous guarantee exists—an explicitly open problem.

Sweeping the calibrated model over Zipf skew, batch size, replication, and EP degree yields a phase diagram whose winning fixed policy flips along an analytically predictable boundary $G_g$8 (driven by either average-cost crossing or hot-expert saturation; the predicted boundary lands inside the observed flip band in 12/12 grid columns). TEMPO tracks the per-cell best everywhere (min gain ≥ −0.3%) and wins by up to 8.5–11.8% in the mixed zone, persisting to EP32–64 extrapolations with mean gains of 4.7–6.5% and a maximum of 15.5%.

The tempo_fast solver

The solver runs in about 2 ms off the critical path and reaches within 1.005 mean (1.033 worst) of a 10-second HiGHS MILP. Its four stages each target a distinct failure mode:

1. Cost-aware greedy seeding places experts whole by descending token count, avoiding the token-LP mistake of fragmenting cold experts.
2. Augmenting-chain activation rebalancing handles the flat regime, where single moves stall because direct moves re-raise the destination; 1- and 2-step chains truncate the semi-matching algorithm optimal in the $G_g$9 limit.
3. Bottleneck local search with partial migrations moves token mass off the bottleneck in the linear regime, using ternary search over split sizes (the restricted objective is piecewise-linear unimodal).
4. Ensemble with a 1% switching tolerance scores token-LP and the round-robin certificate under the same model, guaranteeing "never worse than token-LP beyond 1%" by construction and transferring the additive bound under full replication.

Ablations show every stage binds somewhere: removing partial migrations costs up to 16.3% in the EP32 transition zone; removing augmenting chains costs the entire −9.8% win at $N_g$0 but actively damages the compute-regime solution if left alone; removing the ensemble costs up to 4.5% at EP64 deep-compute. Term ablations confirm the objective's structure is load-bearing: dropping the traffic term forfeits more than half the win at $N_g$1–1024, and dropping the floor term collapses the objective to bare token balancing, since the floor is the only term making the objective non-token-reducible.

Systems integration

The SGLang integration achieves zero marginal in-graph cost: probabilistic dispatch and count collection fuse into one CUDA-graph-resident kernel (with cumulative counting eliminating zeroing kernels and masking against padding rows—a phantom-hot-expert bug also present in shipped LPLB), while the solver runs out-of-process, publishing tables race-safely with a tear-fallback to uniform-over-valid-replicas. In-thread solving costs 8–10% end-to-end from GIL contention; out-of-process solving eliminates it. On multi-node EP, a two-stage topology-aware split solves per-GPU loads first, then pairs sources to replicas same-node-first (provably preserving loads, optimally minimizing inter-node traffic among splits realizing the solved shares).

Evaluation: a phase diagram, not a universal win

The evaluation is structured as three layers—calibrated simulation anchored by wall clock, then end-to-end serving—and is notably candid about what transfers. An 8-GPU microbenchmark validates simulator transfer at 93% pairwise ranking agreement and 2.2–5.5 pp gain-transfer error after recalibration, and confirms both single-proxy failure directions in wall clock: TEMPO stays within 5% (~run noise) of the per-batch-size best fixed policy while every fixed policy has a ≥7% failure region.

End-to-end results on Testbed B bracket the predicted win region with two flagship models:

Model / setting	Result
Qwen3-235B (inside win region)	GovReport throughput +5.0%; p99 TPOT −15.6% under Poisson load; 2-node EP16 +4–7% with topology-aware split
DeepSeek-V3 (outside)	−2 to −3% on all workloads, indistinguishable from noop control
Token-LP dispatcher	Collapses on both models (−10% to −56%), dominated by its in-graph per-layer collective

The DSv3 negative result is diagnostic rather than incidental: SGLang's own dynamic rebalancer also gains nothing there, confirming that at 32 experts per GPU statistical averaging leaves no exploitable imbalance. A like-for-like port of token-LP into TEMPO's worker honestly attributes most of the headline gap versus SGLang's shipped LP dispatcher to integration architecture; the objective's residual edge is stability at the compute-bound point (window spread 4.4 vs. 20 pp). Additional findings include: staleness costs 5–7 pp of a 22 pp exact-solve headroom; faster EPLB refresh is composable with dispatch but counterproductive alone in the floor regime (fresher uniform splits fragment more experts); MoonEP-style weight-moving prefetch cannot amortize at decode scale (prefetch alone exceeds the whole MoE-layer budget) yet owns the high-skew prefill corner; and a measured crossover $N_g$2 correctly brackets where adaptive dispatch stops paying end-to-end.

Limitations and open questions

The paper is explicit that headline phase/scale numbers are model-space, with mid-range transfer error of 2–6 pp; full DSv3 scale (EP32+) remains extrapolation from per-layer measurements. The win region is conditional and now mapped: small-expert bf16 shapes show no gain, fresh placement with ample replication yields parity-at-zero-tax, and multi-node adaptivity requires sufficient expert-compute share of step time. The $N_g$3 model has a known floor—DeepGEMM prefers uniform per-slot loads at equal $N_g$4, a third cost dimension that decides near-ties and explains the degenerate Qwen3 refit; a slot-level term is left as future work. Restricted replica sets lack any approximation guarantee, and the additive bound lacks a lower-bound proof. Staleness, whole-slot granularity, inherited placement, and a depth-scaled remap tax (9–20% on v1, 2–11% on v2) are deliberate trades the solver quality does not remove.

Conclusion

TEMPO replaces proxy objectives with the makespan of a calibrated two-regime cost function, prices what token and activation counts cannot see—weight-streaming floors and $N_g$5-tile padding—and solves the resulting fixed-charge makespan problem in milliseconds with guarantees against both classical proxies. The durable contributions are the cost model, the hardness characterization locating difficulty in the regime interaction, and the empirically validated phase diagram predicting when time-based dispatch pays and when it returns only its mechanism cost.