DySHARP: Dynamic In-Switch MoE Accelerator

• DySHARP is a dynamic in-switch computing framework accelerating Mixture-of-Experts execution on multi-GPU systems using dynamic multimem addressing and token-centric kernel fusion.
• It minimizes redundant communication in MoE by reducing up to 50% traffic and overcoming static collective limitations with dynamic multicast and reduction.
• The framework achieves up to 1.79× end-to-end training speedup over COMET, highlighting its efficiency in expert parallelism scenarios.

Searching arXiv for “DySHARP” and closely related entries to ground the article in the cited literature. DySHARP is a dynamic in-switch computing framework for accelerating Mixture-of-Experts (MoE) execution on multi-GPU systems under expert parallelism (EP). It extends the NVLink SHARP (NVLS) style of in-switch multicast and reduction to support the dynamic, irregular communication patterns induced by token routing in MoE layers, and it couples this communication substrate with a communication-aware execution schedule. In the formulation introduced in "Accelerating MoE with Dynamic In-Switch Computing on Multi-GPUs" (Zhang et al., 7 May 2026), DySHARP consists of two tightly coupled components: dynamic multimem addressing, which supports dynamic target sets and asymmetric per-GPU memory layouts, and token-centric kernel fusion, which fuses the Dispatch–computation–Combine pipeline so that traffic reduction yields end-to-end speedup. The reported headline result is up to $1.79\times$ speedup over COMET in end-to-end training (Zhang et al., 7 May 2026).

1. Definition and execution context

DySHARP is defined in the literature as a dynamic in-switch computing framework for accelerating MoE on multi-GPU systems (Zhang et al., 7 May 2026). Its target workload is MoE execution with expert parallelism, in which experts are distributed across GPUs and each token activates only a small subset of them, typically the top-$k$ experts selected by a gating network. Under this regime, each MoE layer decomposes into three stages: Token dispatch, Expert computation, and Token combine (Zhang et al., 7 May 2026).

The motivation for DySHARP is the observation that, in MoE EP, the two communication operators—Dispatch and Combine—are dominant costs. Prior work and the paper’s profiling indicate that communication can take 50–80% of MoE layer execution time, and on DeepSeek-V3 running on a simulated GH200 NVL32-like system the reported figure is 70.4% (Zhang et al., 7 May 2026). This makes MoE communication not merely an implementation detail but the principal systems bottleneck.

The framework is positioned as an integral solution to two coupled problems. First, existing MoE optimizations improve kernels, overlap, and communication libraries, but do not eliminate redundant transfers of identical or aggregatable data. Second, existing in-switch computing mechanisms such as NVLS can remove redundancy only for static collectives with regular patterns, not for the dynamic communication with irregular patterns characteristic of MoE (Zhang et al., 7 May 2026). DySHARP addresses both the missing communication primitive and the scheduling reorganization needed to convert traffic reduction into realized acceleration.

2. Redundant communication in MoE and the limits of NVLS

The central empirical observation behind DySHARP is that MoE communication contains substantial structural redundancy. In Dispatch, if a token on one GPU must be sent to multiple destination GPUs because its activated experts are distributed, the same token payload is conventionally transmitted multiple times over the source GPU-to-switch link. In Combine, outputs from different experts for the same token are often aggregatable, but in the baseline design they are still returned separately from the switch to the source GPU, where aggregation occurs later (Zhang et al., 7 May 2026).

The paper quantifies this redundancy. On DeepSeek-V3 over a 32-GPU GH200 NVL32-like system, nearly 50% of total communication traffic is redundant when the number of activated experts is $\ge 8$ (Zhang et al., 7 May 2026). This is the main systems-level justification for introducing in-switch multicast and in-switch reduction into the MoE execution path.

DySHARP is also motivated by a precise critique of NVLink SHARP / NVLS. NVLS provides in-switch computing primitives such as multimem.st for multicast and multimem.ld_reduce for reduction, but these are tailored to static collectives like AllGather and Reduce-Scatter. Such collectives assume two regularities: fixed target sets and symmetric addressing. MoE communication violates both assumptions. Different tokens are sent to different subsets of experts and GPUs, and each destination GPU stores received tokens in its own compacted local layout, so a token may land at different memory offsets on different GPUs (Zhang et al., 7 May 2026).

The paper therefore argues that existing NVLS is fundamentally insufficient for MoE. It also evaluates a software workaround that reinterprets Dispatch as AllGather and Combine as Reduce-Scatter. That workaround is reported to generate 340% useless traffic because all GPUs participate even when they do not need the data (Zhang et al., 7 May 2026). In this framing, DySHARP is not simply a faster implementation of NVLS; it is a dynamic extension designed for communication patterns that NVLS cannot natively express.

3. Dynamic multimem addressing

The first core component of DySHARP is dynamic multimem addressing, described as a co-design across ISA, microarchitecture, and runtime (Zhang et al., 7 May 2026). Its purpose is to generalize NVLS multimem so that a single request can represent a multi-destination operation even when the target set varies per token and the destination GPUs use asymmetric local memory layouts.

A straightforward explicit-addressing design would attach a full destination address for every target GPU, but the paper rejects this on two grounds. First, it reduces payload efficiency: for eight targets, payload efficiency is reported to drop from 80% to 69%. Second, it imposes sender-side software overhead: the sender must track remote memory state, maintain counters such as per-expert token positions, precompute remote addresses, and synchronize metadata, which the paper associates with >5% performance loss from synchronization and 10–20% of GPU compute resources consumed by software memory management in prior practice (Zhang et al., 7 May 2026).

DySHARP instead adopts an algebraic index plus local layout mapping abstraction. The packet carries one multimem address whose offset is an algebraic index together with a target expert list. Each destination GPU then performs algebraic-layout mapping locally to derive the actual virtual address (Zhang et al., 7 May 2026). This preserves the compactness of multimem while supporting both dynamic target sets and asymmetric compaction.

The ISA extensions are dymultimem.st for Dispatch multicast and dymultimem.ld_reduce for Combine reduction (Zhang et al., 7 May 2026). Their operands include r2 as multimem address, r1 as data operand or receive register, r3 as target count, and r4 as base address of the contiguous target list. For MoE Combine, DySHARP preserves unweighted reduction in .ld_reduce; weighted aggregation is implemented by applying gating weights in the epilogue of GEMM-2, so Combine reduces $\sum_i (w_i \cdot o_i)$ (Zhang et al., 7 May 2026).

The packet format is extended so that the original 64-bit address field in flit0 becomes a 48-bit multimem address, a 1-bit stage field, and a 15-bit target count. Target extension flits then store 16-bit expert IDs, with eight expert IDs per flit (Zhang et al., 7 May 2026). This packetization is central to the claim that DySHARP retains high payload efficiency while supporting dynamic addressing.

At the destination side, DySHARP introduces a hardware memory manager that performs multimem-virtual (MV) translation before ordinary Link MMU translation. The manager maintains an AL Table in GPU DRAM. Each entry stores Valid and LIdx, occupies 4 B, and the total size is given as

$4 \times \text{nToken} \text{ bytes}.$

For 1M tokens, this consumes 4 MB per layer (Zhang et al., 7 May 2026). Address translation is specified as

$AIdx = \frac{MAddr - MBase}{bsize}$

and

$$VAddr = VBase + LIdx \cdot bsize + MAddr \bmod bsize.$$

These formulas formalize the conversion from a carried algebraic index to a destination-specific compacted layout (Zhang et al., 7 May 2026).

To reduce lookup overhead, DySHARP adds an AL TLB whose tag is the concatenation of Expert ID and AIdx. The evaluation chooses a 512-entry AL TLB as a sweet spot (Zhang et al., 7 May 2026). This subsystem underpins the claim that dynamic multimem addressing can support MoE communication with compact packets and local address resolution rather than sender-managed explicit addresses.

4. Token-centric kernel fusion and readiness-driven scheduling

The second core component of DySHARP is token-centric kernel fusion, which addresses a subtle asymmetry in traffic reduction. The paper argues that dynamic in-switch computing reduces communication volume asymmetrically across directions: in Dispatch, in-switch multicast reduces GPU $\rightarrow$ switch traffic but not switch $\rightarrow$ GPU traffic; in Combine, in-switch reduction reduces switch $\rightarrow$ GPU traffic but not GPU $k$0 switch traffic (Zhang et al., 7 May 2026). If Dispatch and Combine are executed as separate phases, each phase still retains one unreduced bottleneck direction, so traffic reduction alone does not fully translate into speedup.

DySHARP’s response is to reorganize the MoE layer as a token-paced pipeline rather than a sequence of four isolated operators: Dispatch, GEMM-1, GEMM-2, and Combine. The key claim is that downstream work becomes issuable at token or tile-of-tokens granularity. GEMM-1 can start when a tile has been dispatched, GEMM-2 can start when the corresponding GEMM-1 row finishes, and Combine can start when all $k$1 expert outputs for a token are ready (Zhang et al., 7 May 2026). This pipelines Dispatch and Combine so that their opposite directional communication patterns overlap and complement one another.

Readiness tracking is implemented through a token tracker that monitors three dependency chains: Dispatch $k$2 GEMM-1, GEMM-1 $k$3 GEMM-2, and GEMM-2 $k$4 Combine (Zhang et al., 7 May 2026). It uses three tables:

• TS Table: tracks per-tile status, including Valid, ExpID, Row, DAcc, TBCnt1, and TBCnt2.
• TID Table: stores token IDs belonging to a token tile.
• OR Table: tracks per-token output readiness via nReady until $k$5.

The scheduler is implemented as a megakernel with persistent thread blocks, so original thread blocks are transformed into tasks fetched from a task list (Zhang et al., 7 May 2026). SMs are partitioned into four groups dedicated to Dispatch, GEMM-1, GEMM-2, and Combine; GEMM-1 and GEMM-2 can share SMs if one has no ready work. Issuance is then gated by both resource availability and readiness predicates read from the tracker.

The paper distinguishes this design from prior overlap schemes by emphasizing that DySHARP fuses the entire Dispatch $k$6 GEMM-1 $k$7 GEMM-2 $k$8 Combine chain and schedules according to token/tile readiness boundaries rather than operator completion (Zhang et al., 7 May 2026). This is the basis for the claim that dynamic multimem addressing and token-centric kernel fusion are integral, not separable, components.

5. Hardware support, runtime integration, and quantitative results

DySHARP is evaluated on a simulated NVIDIA GH200 NVL32 system with 32 GPUs, 9 NVSwitches, and a fully connected fat-tree topology, using BookSim2 and customized Accel-Sim. The simulated interconnect uses NVLink 4.0 parameters, including 900 GB/s bidirectional bandwidth, 250 ns single-link latency, 1 $k$9 round-trip latency, and 16 B flits (Zhang et al., 7 May 2026). Validation against DGX-H100 measurements reports average errors within 6% for GEMM and DeepEP communication operators (Zhang et al., 7 May 2026).

The target models are DeepSeek-V3-style MoE configurations with three sizes: Small, Medium, and Large, and activated experts $\ge 8$0, where the official DeepSeek-V3 configuration is Large-8 (Zhang et al., 7 May 2026). Baselines include DeepEP, NVLS, FasterMoE, Tutel, CCFuser, COMET, and DualPipe (Zhang et al., 7 May 2026).

The main quantitative claim is that DySHARP achieves up to:

• $\ge 8$1 over DeepEP,
• $\ge 8$2 over NVLS,
• $\ge 8$3 over FasterMoE,
• $\ge 8$4 over Tutel,
• $\ge 8$5 over CCFuser,
• $\ge 8$6 over COMET,
• $\ge 8$7 over DualPipe,

in end-to-end training (Zhang et al., 7 May 2026). For the MoE layer alone, the reported peak speedups are even larger, reaching $\ge 8$8 over COMET and $\ge 8$9 over NVLS (Zhang et al., 7 May 2026).

The paper attributes these gains to two sources. First, dynamic multimem addressing reduces communication traffic by nearly 50% relative to DeepEP and avoids the 340% useless traffic associated with static-NVLS emulation (Zhang et al., 7 May 2026). Second, token-centric kernel fusion improves bandwidth utilization by merging complementary asymmetric traffic directions. A communication-only evaluation reports that DySHARP achieves over 90% of the ideal predicted from reduced traffic volume and available bandwidth (Zhang et al., 7 May 2026).

Scaling studies further report that DySHARP consistently outperforms DeepEP and COMET from 4 to 64 GPUs, and that its advantage widens as GPU count increases (Zhang et al., 7 May 2026). It also remains fastest across sequence lengths from 1024 to 16384, across training token-distribution standard deviations from 0.01 to 0.05, and across inference power-law imbalance parameters $\sum_i (w_i \cdot o_i)$0 from 0.5 to 2.5 (Zhang et al., 7 May 2026). The paper additionally reports gains in end-to-end inference, including prefill and decode, and evaluates GPT-OSS-120B and Qwen3-235B as further model cases (Zhang et al., 7 May 2026).

6. Architectural cost, limitations, and nomenclatural disambiguation

DySHARP is best understood as an architectural extension of NVLS-like systems rather than a software-only optimization. It requires switch support beyond current NVLS, packet-format extensions, new ISA instructions, GPU Hub memory-manager changes, and runtime integration (Zhang et al., 7 May 2026). The switch modifications are characterized as lightweight, adding one cycle to the datapath with area overhead < 0.01 mm$\sum_i (w_i \cdot o_i)$1 and less than 0.1% of NVSwitch die area. On the GPU side, the added structures include MultimemQ with 32 entries, an AL TLB with 512 entries, and tracker tables with 1024 entries each for TS Table and OR Table, for a reported total area overhead of 0.198 mm$\sum_i (w_i \cdot o_i)$2, or about 0.024% of H100 die area (Zhang et al., 7 May 2026).

The paper also makes clear that DySHARP’s benefits are largest in communication-heavy MoE settings: moderate or large $\sum_i (w_i \cdot o_i)$3, substantial cross-GPU routing, and regimes where communication dominates compute (Zhang et al., 7 May 2026). It does not fundamentally solve expert load imbalance, and although it improves performance under imbalance, stragglers can still dominate latency. The design is also tailored to NVLink/NVSwitch-like systems with shared-memory semantics and in-switch computing support, so it is not directly portable to arbitrary interconnects (Zhang et al., 7 May 2026).

A common source of confusion is the similarity of the name DySHARP to other arXiv systems with partially overlapping terminology. SHARP, in "Spectrum-aware Highly-dynamic Adaptation for Resolution Promotion in Remote Sensing Synthesis" (Zhao et al., 23 Mar 2026), is a training-free, timestep-aware RoPE adaptation method for remote sensing text-to-image synthesis and is not the same system. DynaShard, in "Secure and Adaptive Blockchain Sharding Protocol with Hybrid Consensus and Dynamic Shard Management" (Liu et al., 2024), is a blockchain sharding protocol and likewise is not DySHARP. The MoE systems paper explicitly introduces DySHARP as a dynamic in-switch computing solution built around NVLS-style multicast and reduction for multi-GPU expert parallelism (Zhang et al., 7 May 2026).

In that literature, DySHARP’s distinguishing claim is not merely that it accelerates MoE communication, but that it does so by combining a dynamic extension of multimem addressing with a token-centric fused schedule. This suggests that its contribution lies as much in the interaction between communication primitive and execution model as in either mechanism alone, a point reinforced by the ablation results reported in the paper (Zhang et al., 7 May 2026).