Communication-Aware Placement and Pruning for Efficient Mixture-of-Experts Inference

Abstract: As MoE models scale to hundreds of experts, placement and pruning decisions increasingly dictate communication volume, affecting the performance of distributed inference across GPUs and nodes. We propose CAP (Communication-Aware Assignment and Pruning), a framework that considers computation, communication and accuracy together for efficient MoE inference through expert placement and pruning. It consists of three components: (1) Co-activation driven expert placement, which groups frequently co-activated experts to reduce inter-device and inter-node communication; (2) Communicationcomputation trade-off adjustment, which generates placements with different computational load and communication volume; and (3) Communication-aware expert pruning, which selectively removes routing destinations to reduce communication with limited accuracy degradation. By combining these components, CAP selects an efficient operating strategy for different hardware configurations. Across our single-node and multi-node experiments, it achieves 1.23x - 1.86 x throughput improvement over DeepSeek EPLB and sequential placement in vLLM, and preserves better model accuracy at the same target speedup under lossy acceleration.

• The paper introduces CAP, a framework that optimizes MoE inference by integrating communication-aware expert placement and pruning.
• It employs a two-phase weighted graph partitioning and swap-based local search to reduce inter-device communication and balance computation.
• Empirical results show a 1.23x–1.86x throughput boost across varied hardware setups while maintaining high model accuracy.

Communication-Aware Placement and Pruning for Mixture-of-Experts Inference

Introduction and Background

As Mixture-of-Experts (MoE) LLM architectures scale to hundreds of experts per layer, the associated optimization challenges shift. Traditional MoE systems emphasize computation load balancing and compute efficiency, generally neglecting the increasingly significant impact of intra- and inter-device communication during distributed inference. In modern settings—e.g., Qwen3-30B-A3B with 128 experts per layer—expert placement and pruning impact the communication volume, which can be the primary bottleneck in multi-GPU and multi-node inference. This paper introduces CAP (Communication-Aware Assignment and Pruning), an integrated framework that simultaneously optimizes for communication cost, computation load balancing, and model accuracy via expert placement and pruning decisions.

Figure 1: Expert parallelism combined with data parallelism.

CAP targets modern inference workflows using both expert parallelism (partitioning MoE experts across devices) and data parallelism (replicating dense layers). Tokens are routed post-attention to their selected experts, inducing all-to-all communication that scales with the number—and, critically, the distribution—of active experts.

Figure 2: Zero-redundancy Token Transfer.

Classic systems like vLLM or DeepSeek-EPLB ignore communication locality in their placement and pruning, often yielding suboptimal throughput. CAP, in contrast, jointly leverages expert co-activation patterns, communication topology, and layer-wise routing statistics.

CAP Framework: Components and Pipeline

Communication-Aware Expert Placement

The first component identifies frequently co-activated expert pairs and co-locates them using a two-phase weighted graph partitioning heuristic. Co-activation matrices demonstrate that expert activation is highly structured, enabling reduced communication by grouping these experts tightly.

Figure 3: Qwen3-30B-A3B Layer 47 – Expert co-activation matrix.

Figure 4: Expert placement impacts the communication.

The assignment minimizes the weighted sum of device-pair communication penalties over all expert pairs, accounting for intra-device, intra-node, and inter-node costs. The algorithm produces an expert-to-device mapping per layer using a greedy seeded growth strategy, followed by hierarchical node grouping.

Figure 5: Grouping frequently co-activated experts onto the same devices and nodes using co-activation graph.

Communication-Computation Trade-off Adjustment

Optimizing only for communication locality often exacerbates load imbalance, clustering hot experts together and stalling parallel execution. CAP’s second module adjusts placements via swap-based local search, trading off communication cost and computation balance. This yields a Pareto spectrum of placements parameterized by the trade-off coefficient $\lambda$.

Figure 6: Comm. {additional_guidance} Comp. Trade-off Adjustment.

Profiling across hardware allows selecting the optimal configuration. Comm-constrained nodes (e.g., those lacking NVLink) benefit from communication-minimal placements; comm-rich nodes (e.g., H100 clusters) can prioritize balance.

Communication-Aware Expert Pruning

Classic pruning (e.g., by cumulative routing score per token) is computation-centric and often communication-oblivious. CAP’s pruning objective incorporates device-level contribution and communication cost explicitly, preferentially pruning entire routing destinations that cause expensive transfers. This device-level pruning greatly amplifies communication volume reduction achievable under a fixed accuracy degradation.

Figure 7: Communication-aware pruning with a 0.3 threshold.

The pruning policy greedily selects devices by the utility-to-cost ratio until the minimum cumulative routing mass is met, supporting fast GPU execution.

Pipeline Integration

The three components integrate into a staged pipeline:

Figure 8: The 4-step Pipeline of CAP.

1. Pruning is applied first, producing pruned routing traces.
2. Placement is performed based on the pruned traces, capturing the true prevalent co-activation under pruning.
3. Adjustment iteratively refines load balance and communication trade-offs from the initial placement.
4. Profiling across a spectrum of placements on target hardware selects the optimal deployment.

Empirically, pruning increases load imbalance, as device-hot experts with high routing scores are preserved, driving the necessity for secondary adjustment.

Figure 9: Load balance under different pruning thresholds (left); Hot experts tend to get higher routing scores (right).

Experimental Evaluation

End-to-End Performance

Evaluation encompasses a range of MoE models (notably Qwen3-30B-A3B and DeepSeek-V2-Lite), datasets (LMSYS-Chat, Arxiv Abstracts), and hardware: from comm-constrained (8x RTX 3090, PCIe-only) to comm-rich (8x H100, NVLink). Baselines include vLLM-style sequential placement and DeepSeek-EPLB’s load-balance-optimized assignments.

Figure 10: End-to-end Performance.

CAP yields $1.23\times$–$1.86\times$ higher throughput, with the largest gains on PCIe-only (comm-constrained) systems. Notably, under lossy acceleration (e.g., $p=0.3$ pruning threshold), CAP maintains stronger accuracy than classic computation-centric pruning, as its device-level pruning targets communication bottlenecks more effectively.

Placement Spectrum and Pruning Analysis

Detailed results show the optimal placement moves along the communication-load imbalance frontier as the hardware shifts from bandwidth-limited to compute-limited. The spectrum generated by CAP invariably dominates the default sequential placement.

Figure 11: Placement Spectrum Performance on 3 Nodes.

Figure 12: Performance and Communication-Computation Characteristics of Placement Spectrum.

Device-aware pruning drastically outperforms expert-level approaches, especially in balancing speedup and accuracy. Under fixed latency budgets, CAP achieves the same throughput boost at lower accuracy loss.

Figure 13: In-workload stability of expert grouping on LMSYS Chat. Groupings generated from three different windows (w0, w2000, and w6000) are evaluated on all 16 windows of all 8000 requests.

Analysis: Stability and Optimization Overhead

The CAP pipeline demonstrates strong stability and transferability: expert placements derived from small routing traces over one domain generalize to other datasets and remain robust across inference windows, negating the need for frequent regrouping. The offline optimization phase is lightweight (≈6 minutes end-to-end for a full Qwen3-30B-A3B deployment on A100s), and operational overhead is minimal due to vectorized GPU implementations.

Implications and Future Directions

CAP’s results contradict the previously dominant assumption that load balancing is the key system bottleneck for MoE inference. Instead, the communication cost induced by expert routing dominates on most modern, bandwidth-asymmetric clusters.

Pragmatically, this research provides a template for future MoE deployment frameworks: hierarchical, communication-aware assignment must be standard for high-expert-count models. Future work should focus on:

• Online adaptation to dynamic routing distribution shifts, including cross-workload and cross-session adjustments.
• Further reduction of optimization overhead for even larger MoEs by exploiting more scalable approximate graph partitioning techniques.
• Extending communication-optimized pruning to integrate with routing policy designs and further compression/quantization techniques.

With MoE architectures scaling well past contemporary LLMs, communication-aware systems-level optimization will remain an essential axis for both theoretical analysis and practical deployment in high-throughput, cost-sensitive production inference.

Conclusion

CAP demonstrates that expert placement and pruning in large-scale MoE inference must be communication-aware to achieve optimal efficiency. By exploiting co-activation locality, hardware-adaptive trade-off searching, and device-level communication modeling, CAP outperforms previous computation- and load-balance-centric paradigms under both lossless and lossy settings, and does so with stable, transferable assignments and minimal operational overhead.