ELDR: Expert-Locality-Aware Decode Routing for PD-Disaggregated MoE Serving

Abstract: In prefill-decode (PD) disaggregated LLM serving, each request is assigned to a decode worker after prefill. Existing decode routers balance only load; for mixture-of-experts (MoE) models this is incomplete: equally loaded workers can differ in latency, since each decode step loads the weights of every distinct expert its batch activates. We present ELDR, an expert-locality-aware decode router for PD-disaggregated MoE serving. From a request's prefill expert activations, ELDR builds an expert signature predicting the experts it will activate during generation. Offline, balanced K-means partitions signature space across decode workers; online, locality-band routing sends each request to the least-loaded worker among those best matching its signature. A signature cache, co-indexed with the KV cache at KV-block granularity, keeps signatures exact under prefix caching. Implemented in vLLM and evaluated on deployments of up to 40 GPUs, ELDR reduces median TPOT by 5.9-13.9% over the strongest of four load-balancing baselines across three MoE models and two workloads, with model outputs unchanged.

• The paper introduces ELDR, a framework that leverages prefill-derived expert signatures to optimize decode routing in PD-disaggregated MoE systems.
• It employs balanced K-means clustering with IDF-weighted discrete counts to reduce median TPOT by up to 13.9% and tail TPOT by up to 6.0%.
• The approach integrates seamlessly with prefix caching and scales to massive MoE models, ensuring consistent latency improvements under high utilization.

ELDR: Expert-Locality-Aware Decode Routing for PD-Disaggregated MoE Serving

Motivation and Problem Formulation

Large-scale LLM serving systems increasingly utilize Prefill-Decode (PD) disaggregation, where prompt computation (prefill) and token generation (decode) are handled by independent worker pools. This separation is central for high-throughput and efficient resource utilization in practical deployments. However, when serving Mixture-of-Experts (MoE) models, decode-phase routing becomes fundamentally distinct from dense models. Under MoE, each decode batch routes each token to a subset of experts, creating a critical performance bottleneck: decode latency scales with the number of distinct experts loaded per step, not batch size. The memory-bandwidth saturation induced by loading numerous expert weights dominates overall time-per-output-token (TPOT), unlike dense models where compute is the primary constraint.

Figure 1: MoE layer latency scales with the number of active experts, not batch size, highlighting the centrality of expert selection for decode throughput.

Empirically, expert activation is structured by request domain (e.g., code, math, medical), and highly correlated between prefill and decode phases. That is, requests from the same domain consistently activate overlapping subsets of experts both during prompt processing and generation.

Figure 2: Decode-phase per-expert activation normalized relative to the cross-domain mean for three MoE models by task (top) and language (bottom); demonstrating strong domain-dependent expert specialization.

This domain-expert affinity enables a second routing axis: expert-locality. By batching requests with similar expert usage profiles, one can reduce the union of loaded experts per step, achieving significant TPOT reductions. Critically, since prefill expert activations are highly predictive of those during decode (correlations $\rho=0.70$ to $0.92$ across models), the necessary routing signal is available at the prefill$\to$decode handoff.

Figure 3: Prefill expert activation predicts decode activation, validating prefill signatures as a routing signal.

The ELDR Mechanism

Expert Signature Construction

ELDR encapsulates a request’s expert-locality profile via a prefill-extracted expert signature—a compact vector summarizing per-layer, per-expert top-$k$ activation counts, enhanced by information-centric weighting. Three design principles drive the signature:

1. Discrete Activity Counting: Only top-$k$ selections trigger expert parameter loads; thus discrete counts yield a signature precisely matched to bandwidth usage.
2. Inverse Document Frequency (IDF) Weighting: Counts are reweighted using IDF across calibration data, attenuating the influence of ubiquitous “generalist” experts and accentuating discriminative “specialist” activations.
3. Layer Selection: The signature utilizes only a subset of model layers most predictive of decode-phase behavior, maximizing rank correlation $\rho$ with actual expert overlap while minimizing dimension.

   Figure 4: Signature quality $\rho$ for discrete vs. continuous and other candidate transformations across models and workloads, demonstrating the superiority of IDF-weighted discrete counts.

Offline Clustering and Online Routing

Routing unifies the dual objectives of expert-locality and load balance through a two-stage process:

• Offline: Balanced $K$-means (Hungarian-balanced) partitions calibration signatures into $K$ balanced clusters—one per decode worker—ensuring domain coverage and equitable request distribution.
• Online: At the prefill$\to$decode transition, the router matches the request’s signature to nearest centroids (those within a locality band, $0.92$0) and dispatches to the least-loaded among them. Parameter $0.92$1 controls the balance-locality trade-off, set empirically at $0.92$2 for optimal median and tail TPOT.

  Figure 5: ELDR architecture schematic; offline clustering and online routing by signature similarity and live load.

  

  Figure 6: PCA visualization of calibration signatures and centroids, showing nontrivial distribution and cluster boundary alignment with semantic domains in expert-locality space.

Prefix-Cache Coherent Signatures

For maximal practical value, ELDR composes with prefix caching by maintaining a block-granular signature cache co-indexed with the KV cache. Each KV block carries the expert activation footprint of its tokens, enabling exact recomputation of a request’s full signature across cache hits, partial hits, and evictions without signature recomputation or cache disablement.

Figure 7: ELDR’s signature cache design, tightly integrated with the KV block manager for cache coherence and efficiency.

Numerical Results

ELDR’s efficacy is substantiated across three public MoE models (Qwen3-30B-A3B, GPT-OSS-120B, and Gemma-4-26B-A4B) and heterogeneous workloads (task benchmarks, large-scale multilingual queries). The key results:

• On task workloads: Median TPOT is reduced by $0.92$3–$0.92$4 and tail TPOT by $0.92$5–$0.92$6 compared to the strongest load-balancing baseline. Against an oracle Domain router, ELDR achieves up to $0.92$7 further gain due to finer intra-domain clustering. Model outputs are kept identical (no top-$0.92$8 gating modification).
• On language workloads: Median TPOT is reduced by $0.92$9–$\to$0, outperforming both domain-based and load-based strategies. Where domain labels are coarse, ELDR captures intra-language expert locality, exceeding static or load-based policies.
• High utilization (100 qps): The improvement is robust under high offered rates and larger decoder pool sizes; median TPOT reduction scales with decode pool cardinality.

Figure 8: TPOT (median, p99) and median TTFT vs request rate for the task workload at 8P16D, evidencing ELDR’s consistent superiority over all baselines.

Figure 9: Mean active experts per decode step with Qwen3-30B-A3B, task domain, 8P16D: ELDR reduces distinct experts per batch step by 22\% compared to round-robin batching.

ELDR generalizes to massive settings: For Qwen3-235B-A22B ($\to$1-B param, 40 GPUs under $\to$2, $\to$3), median TPOT reduction remains $\to$4–$\to$5 (tail up to $\to$6), demonstrating scalability with expert parallelism.

Figure 10: TPOT (median, p99) for Qwen3-235B-A22B on language workload at 2P8D/$\to$7/ $\to$8 (40 GPUs), establishing generality under large cluster conditions.

Design and Ablation Studies

Figure 11: Signature transform ablation reveals that discrete, IDF-weighted counts drive the TPOT advantage; continuous softmax alternatives degrade performance.

Figure 12: Balanced $\to$9-means clustering is required for tail latency robustness; vanilla $k$0-means yields tail regressions due to imbalanced cluster loads.

Figure 13: Locality-band parameter $k$1 ablation; $k$2 optimally balances locality and live-load adaptation for TPOT at both median and tail percentiles.

Prefix Cache Compatibility

ELDR composes seamlessly with prefix caching. Activating the cache reduces TTFT for both ELDR and baselines, with TPOT reductions preserved, confirming that the signature cache design introduces no regressions under realistic LLM workload patterns.

Figure 14: ELDR and prefix caching are additive—cache activation collapses TTFT without affecting TPOT improvements (GPT-OSS-120B, task, 8P16D).

Implications and Future Directions

ELDR demonstrates that MoE-specific bottlenecks in distributed LLM serving can be losslessly mitigated by exploiting structured expert-locality signals available at the prefill$k$3decode handoff. The signature-based routing protocol is orthogonal to intra-worker expert-parallel load balancing and composes with existing affinity-aware prefill routers and prefix caches, integrating as a thin middleware layer atop established stacks (e.g., vLLM).

Future extensions may investigate:

• Adaptive cluster tracking to handle workload drift or dynamic model updates.
• Hierarchical or multi-level clustering to further disaggregate expert-locality within very large decoder pools.
• Integration with advanced expert-activation pattern mining and semantic parallelism (Li et al., 6 Mar 2025) for hybrid scheduling of compute and communication.
• Application to other sparsely activated LLM architectures, beyond current top-$k$4 MoE designs.

Conclusion

ELDR introduces a second routing axis—expert-locality—into decode worker assignment for PD-disaggregated MoE serving, reducing not only median but also tail TPOT while being fully compositional with existing system optimizations such as prefix caching. Numerical evidence across scales and models confirms that routing by expert-locality, as predicted by judiciously constructed prefill signatures and balanced clustering, is essential for efficient large-scale MoE LLM inference and deployment (2607.00466).