MoE-Hub: Sparse MoE on Multi-GPU Systems

• MoE-Hub is a hardware-software co-design for sparse Mixture-of-Experts that decouples logical routing from address allocation, enabling immediate data communication.
• It introduces a destination-agnostic communication interface and runtime APIs to overlap data transmission with computation, significantly reducing software mediation overhead.
• Evaluation shows per-layer speedups of 1.40×–3.08× and near-ideal multi-GPU scaling, highlighting its efficiency over traditional systems.

MoE-Hub is a hardware–software co-design for sparse Mixture-of-Experts execution on multi-GPU systems. It addresses a systems bottleneck that arises when MoE routing yields only logical destinations—expert ID and GPU ID—while modern GPU interconnects remain strictly address-centric and require concrete virtual addresses for remote writes. The design introduces a destination-agnostic communication paradigm in which data transmission is decoupled from address management: producers can issue communication immediately after routing through a logical destination, while address allocation and data-flow orchestration are handled by lightweight hardware in the GPU hub. In the reported evaluation, this enables seamless overlap of communication with computation and yields $1.40\times\!-\!3.08\times$ per-layer and $1.21\times\!-\!1.98\times$ end-to-end speedup over state-of-the-art systems (Zhou et al., 7 May 2026).

1. Problem setting and the scalability bottleneck

In a sparse MoE layer with $E$ experts and $G$ GPUs, each token is routed at runtime to $K\!\ll\!E$ experts. Parallel execution of those experts requires two All-to-All collectives per layer: a dispatch phase that sends each token’s embedding to its $K$ destination experts, and a combine phase that gathers the $K$ expert outputs back to each token. The paper reports real measurements showing that up to $47\%$ of forward-pass time can be spent in these collectives (Zhou et al., 7 May 2026).

MoE-Hub attributes this cost to an abstraction mismatch rather than to communication volume alone. MoE routing produces a logical expert ID and GPU ID per token, but NVLink- and NVSwitch-class GPU interconnects are address-centric: every remote write must provide a concrete virtual address. Existing software systems therefore need a mediation phase that synchronizes token counts across GPUs, builds producer-side and consumer-side token-count tables, allocates and scatter/gathers expert buffers, and only then issues address-to-address remote writes. According to the paper, this mediation adds tens of microseconds per layer and prevents communication from beginning until after routing, which blocks fine-grained overlap (Zhou et al., 7 May 2026).

This framing is significant because it shifts the optimization target. The limiting factor is not only collective bandwidth, but also the control plane required to translate dynamic, irregular token-to-expert mappings into address-resolved remote writes.

2. Destination-agnostic communication interface

MoE-Hub breaks the link between “when I know where to send” and “where in memory to put it” by introducing a destination-agnostic communication interface (Zhou et al., 7 May 2026). Its ISA extension is the PTX-style instruction

$K\!\ll\!E$3

Here, MallocID is a small integer encoding (target GPU, region ID), RowID is a unique tag per token row, and RowOffset is a byte offset within that row for splitting transfers larger than 128 B. The semantics are to issue a remote write using only logical destination information; the store is sent immediately to the hub without further software coordination. The optional .nop suffix marks non-critical data such as combine metadata, allowing lower-priority scheduling (Zhou et al., 7 May 2026).

The runtime complement is a CUDA-style API:

$K\!\ll\!E$4

This call allocates a contiguous device-memory region of size $\mathrm{rowSize}\times\mathrm{maxRows}$ on the target GPU, registers $(\mathrm{BaseAddr}, \mathrm{RowSize})$ in that GPU’s hub through an MMIO command, and returns a 16-bit MallocID to be used by st.rowsp (Zhou et al., 7 May 2026).

The end-to-end programming model is correspondingly compact. After routing, a kernel identifies the destination GPU, derives a unique row tag for the token, and emits st.rowsp directly. No host-side synchronization is needed. This is central to the paper’s programmability claim: communication can start immediately after routing, without waiting for address-resolution software to complete.

3. Hub-resident microarchitecture

All new hardware in MoE-Hub resides in the GPU hub, defined in the paper as the switch that bridges the on-chip crossbar and NVLink. The design comprises three main units: the Address Allocation Unit, the Runtime Packet Manager, and the Data Availability Manager (Zhou et al., 7 May 2026).

Unit	Purpose	Key structures
AAU	On-demand mapping from logical to physical address	RAT, APT
RPM	Coalescing and scheduling of rowsp writes	Remote Write Buffer Pool
DAM	Event-driven wake-up of dependent thread blocks	Dependency Table, per-TB counters

The Address Allocation Unit (AAU) maps $1.21\times\!-\!1.98\times$0 to a physical address on demand. It uses a Row Allocation Table (RAT), an associative cache from $1.21\times\!-\!1.98\times$1 to LocalRowID, and an Allocation Pointer Table (APT), which tracks the next free LocalRowID for each MallocID. On each incoming st.rowsp packet, the AAU performs a RAT lookup; on a miss it allocates a new LocalRowID = \mathrm{APT}[\mathrm{MallocID}]++, inserts the mapping, computes the write address as BaseAddr + LocalRowID × RowSize + RowOffset, and forwards the request to L2 or memory. RAT entries are evicted in FIFO order once a row is fully received, with spill to DRAM if reused later (Zhou et al., 7 May 2026).

The Runtime Packet Manager (RPM) reshapes and schedules the stream of small, out-of-order rowsp writes to maximize link utilization. It employs a banked Remote Write Buffer Pool, with one bank per peer GPU; within each bank, entries are indexed by (RowID, RowOffset) and carry a validity mask so writes can be coalesced into full 128 B flits. Its policy is round-robin across banks to avoid burst congestion, preference for fully coalesced entries, respect for .nop versus default requests, and—within default traffic—smallest-RowID-first scheduling so consumers can start on full rows as early as possible (Zhou et al., 7 May 2026).

The Data Availability Manager (DAM) replaces software polling on the consumer side. It maintains a CAM-based Dependency Table from address ranges to dependent thread blocks, per-thread-block counters for required rows, and a global counter for total expected writes. When a write acknowledgment returns, the DAM performs a CAM lookup, increments the counters of matching thread blocks, emits a READY signal when a block’s required rows have arrived, and issues ALL_READY when the global counter reaches the expected total (Zhou et al., 7 May 2026).

4. Overlap model and execution semantics

The paper’s central execution claim is that MoE-Hub hardware-accelerates the entire communication control plane, making overlap seamless and transparent (Zhou et al., 7 May 2026). Because producers transmit data immediately after routing using only logical destination information, and because address allocation is deferred to the receiving hub, the system removes the serialized dependence between routing completion and address-ready communication launch.

This mechanism is reflected in the per-layer decomposition

$1.21\times\!-\!1.98\times$2

The reported result is that MoE-Hub drives $1.21\times\!-\!1.98\times$3 and $1.21\times\!-\!1.98\times$4 completely under $1.21\times\!-\!1.98\times$5 for most layers, whereas software solutions still expose $1.21\times\!-\!1.98\times$6 of collective time (Zhou et al., 7 May 2026).

The execution model also changes the role of the consumer. Instead of warp-level polling and host-mediated progress tracking, expert kernels are awakened by DAM-generated events once the necessary rows have arrived. This event-driven readiness model is important both for latency hiding and for software simplification. A plausible implication is that MoE-Hub treats the control plane as a first-class architectural target, rather than as auxiliary runtime machinery.

5. Evaluation methodology and reported performance

The reported evaluation uses a BookSim2-based simulator configured as $1.21\times\!-\!1.98\times$7 simulated H800 GPUs, each with a 400 GB/s link and $1.21\times\!-\!1.98\times$8 round-trip NVLink latency. The single-layer focus covers Mixtral-8×7 B with $1.21\times\!-\!1.98\times$9, $E$0, $E$1; Qwen2-2.7 B with $E$2, $E$3, $E$4; and Phi-3.5 B with $E$5, $E$6, $E$7. Expert GEMMs are implemented through CUTLASS. Baselines are Megatron-TE, FasterMoE, Tutel, Comet, CCFuser, and an ideal MoE-layer lower bound consisting of local routing, experts, and scaling only (Zhou et al., 7 May 2026).

Under 8 GPUs and sequence length 2048, the reported figures are Megatron-TE at $E$8, FasterMoE at $E$9, Tutel at $G$0, Comet at $G$1, CCFuser at $G$2, and MoE-Hub at $G$3 per-layer and $G$4 end-to-end. The paper further states that MoE-Hub comes close to $G$5 of the ideal MoE-layer performance (Zhou et al., 7 May 2026).

Sensitivity studies are reported along three axes. First, when sequence length varies from 128 to 32768, speedups range from $G$6 to $G$7, with the largest gains at small batch because control-plane overhead is removed. Second, under token imbalance with standard deviation up to 0.05, MoE-Hub remains approximately $G$8 faster than Comet even under heavy skew. Third, in strong scaling from 2 to 16 GPUs, it achieves near-ideal FLOPS scaling within $G$9 and outperforms baselines by $K\!\ll\!E$0 (Zhou et al., 7 May 2026).

These results suggest that the benefit is not confined to a single model shape or routing regime. The gains are linked to elimination of software mediation overhead and to sustained overlap under irregular traffic.

6. Software complexity, applicability, and limitations

MoE-Hub is also presented as a response to software complexity. The paper states that baseline systems require hundreds to thousands of lines of host and device code for address resolution, synchronization, and warp-level polling, and gives Comet as an example with more than 6k lines for address resolution alone. By contrast, MoE-Hub requires zero host scheduling code and fewer than 10 instructions in the device kernel, essentially one st.rowsp per token (Zhou et al., 7 May 2026).

Integration is described as a wrapper around rowspMalloc and st.rowsp. Existing frameworks such as Megatron, Fairseq, and DeepSpeed can adopt it by replacing All-to-All calls with two calls to rowspMalloc plus tagging stores. The paper emphasizes that this introduces no new kernel launches, no manual SM partitioning, and no warp-poll loops, so fine-grained overlap becomes “free” in the programming model (Zhou et al., 7 May 2026).

The hardware overhead is reported as modest: the full hub extensions occupy $K\!\ll\!E$1, less than $K\!\ll\!E$2 of the H800 die, and add negligible pipeline critical path. At the same time, several limitations are explicit. The current rowspMalloc model is static; dynamic paging and KV-cache schemes can be layered on top only if the AAU tables can track splits. For hybrid TP+EP, each TP shard is treated as a separate consumer endpoint, with one MallocID per GPU in the TP group. Dynamic expert placement and load-balancing systems may require extending APT and RAT to many-to-one or many-to-many mappings. The same destination-agnostic philosophy is also stated to apply to other sparse or DAG workloads, including sparse KV-cache exchanges and block-sparse attention (Zhou et al., 7 May 2026).

Future work proposed by the paper includes unified memory integration such as GPU-side paging for larger models, in-network support such as NVSwitch microcode that would move AAU and RPM logic off-chip, and more advanced RAT eviction or compression schemes for massive row counts (Zhou et al., 7 May 2026). Collectively, these directions position MoE-Hub as a systems abstraction for dynamic sparse communication, not only as an optimization for a single MoE kernel.