---
title: 'DySHARP: Dynamic In-Switch MoE Accelerator'
url: https://www.emergentmind.com/topics/dysharp
type: topic
---

# DySHARP: Dynamic In-Switch MoE Accelerator

Searching arXiv for “DySHARP” and closely related entries to ground the article in the cited literature.
DySHARP is a dynamic in-switch computing framework for accelerating Mixture-of-Experts (MoE) execution on multi-GPU systems under expert parallelism (EP). It extends the NVLink SHARP (NVLS) style of in-switch multicast and reduction to support the dynamic, irregular communication patterns induced by token routing in MoE layers, and it couples this communication substrate with a communication-aware execution schedule. In the formulation introduced in "Accelerating MoE with Dynamic In-Switch Computing on Multi-GPUs" [2605.05607], DySHARP consists of two tightly coupled components: **dynamic multimem addressing**, which supports dynamic target sets and asymmetric per-GPU memory layouts, and **token-centric kernel fusion**, which fuses the Dispatch–computation–Combine pipeline so that traffic reduction yields end-to-end speedup. The reported headline result is up to \(1.79\times\) speedup over COMET in end-to-end training [2605.05607].

## 1. Definition and execution context

DySHARP is defined in the literature as a **dynamic in-switch computing framework for accelerating MoE on multi-GPU systems** [2605.05607]. Its target workload is MoE execution with **expert parallelism**, in which experts are distributed across GPUs and each token activates only a small subset of them, typically the top-\(k\) experts selected by a gating network. Under this regime, each MoE layer decomposes into three stages: **Token dispatch**, **Expert computation**, and **Token combine** [2605.05607].

The motivation for DySHARP is the observation that, in MoE EP, the two communication operators—**Dispatch** and **Combine**—are dominant costs. Prior work and the paper’s profiling indicate that communication can take **50–80%** of MoE layer execution time, and on DeepSeek-V3 running on a simulated GH200 NVL32-like system the reported figure is **70.4%** [2605.05607]. This makes MoE communication not merely an implementation detail but the principal systems bottleneck.

The framework is positioned as an **integral solution** to two coupled problems. First, existing MoE optimizations improve kernels, overlap, and communication libraries, but do not eliminate redundant transfers of identical or aggregatable data. Second, existing in-switch computing mechanisms such as NVLS can remove redundancy only for **static collectives with regular patterns**, not for the **dynamic communication with irregular patterns** characteristic of MoE [2605.05607]. DySHARP addresses both the missing communication primitive and the scheduling reorganization needed to convert traffic reduction into realized acceleration.

## 2. Redundant communication in MoE and the limits of NVLS

The central empirical observation behind DySHARP is that MoE communication contains substantial structural redundancy. In **Dispatch**, if a token on one GPU must be sent to multiple destination GPUs because its activated experts are distributed, the same token payload is conventionally transmitted multiple times over the source GPU-to-switch link. In **Combine**, outputs from different experts for the same token are often aggregatable, but in the baseline design they are still returned separately from the switch to the source GPU, where aggregation occurs later [2605.05607].

The paper quantifies this redundancy. On DeepSeek-V3 over a 32-GPU GH200 NVL32-like system, **nearly 50% of total communication traffic is redundant** when the number of activated experts is \(\ge 8\) [2605.05607]. This is the main systems-level justification for introducing in-switch multicast and in-switch reduction into the MoE execution path.

DySHARP is also motivated by a precise critique of **NVLink SHARP / NVLS**. NVLS provides in-switch computing primitives such as **`multimem.st`** for multicast and **`multimem.ld_reduce`** for reduction, but these are tailored to static collectives like AllGather and Reduce-Scatter. Such collectives assume two regularities: **fixed target sets** and **symmetric addressing**. MoE communication violates both assumptions. Different tokens are sent to different subsets of experts and GPUs, and each destination GPU stores received tokens in its own compacted local layout, so a token may land at different memory offsets on different GPUs [2605.05607].

The paper therefore argues that existing NVLS is fundamentally insufficient for MoE. It also evaluates a software workaround that reinterprets Dispatch as AllGather and Combine as Reduce-Scatter. That workaround is reported to generate **340% useless traffic** because all GPUs participate even when they do not need the data [2605.05607]. In this framing, DySHARP is not simply a faster implementation of NVLS; it is a dynamic extension designed for communication patterns that NVLS cannot natively express.

## 3. Dynamic multimem addressing

The first core component of DySHARP is **dynamic multimem addressing**, described as a co-design across **ISA**, **microarchitecture**, and **runtime** [2605.05607]. Its purpose is to generalize NVLS multimem so that a single request can represent a multi-destination operation even when the target set varies per token and the destination GPUs use asymmetric local memory layouts.

A straightforward explicit-addressing design would attach a full destination address for every target GPU, but the paper rejects this on two grounds. First, it reduces payload efficiency: for eight targets, payload efficiency is reported to drop from **80% to 69%**. Second, it imposes sender-side software overhead: the sender must track remote memory state, maintain counters such as per-expert token positions, precompute remote addresses, and synchronize metadata, which the paper associates with **>5% performance loss** from synchronization and **10–20% of GPU compute resources** consumed by software memory management in prior practice [2605.05607].

DySHARP instead adopts an **algebraic index plus local layout mapping** abstraction. The packet carries **one multimem address whose offset is an algebraic index** together with a **target expert list**. Each destination GPU then performs **algebraic-layout mapping** locally to derive the actual virtual address [2605.05607]. This preserves the compactness of multimem while supporting both dynamic target sets and asymmetric compaction.

The ISA extensions are **`dymultimem.st`** for Dispatch multicast and **`dymultimem.ld_reduce`** for Combine reduction [2605.05607]. Their operands include **`r2`** as multimem address, **`r1`** as data operand or receive register, **`r3`** as target count, and **`r4`** as base address of the contiguous target list. For MoE Combine, DySHARP preserves unweighted reduction in `.ld_reduce`; weighted aggregation is implemented by applying gating weights in the **epilogue of GEMM-2**, so Combine reduces \(\sum_i (w_i \cdot o_i)\) [2605.05607].

The packet format is extended so that the original 64-bit address field in flit0 becomes a **48-bit multimem address**, a **1-bit stage** field, and a **15-bit target count**. Target extension flits then store **16-bit expert IDs**, with eight expert IDs per flit [2605.05607]. This packetization is central to the claim that DySHARP retains high payload efficiency while supporting dynamic addressing.

At the destination side, DySHARP introduces a hardware memory manager that performs **multimem-virtual (MV) translation** before ordinary Link MMU translation. The manager maintains an **AL Table** in GPU DRAM. Each entry stores **`Valid`** and **`LIdx`**, occupies **4 B**, and the total size is given as
\[
4 \times \text{nToken} \text{ bytes}.
\]
For **1M tokens**, this consumes **4 MB per layer** [2605.05607]. Address translation is specified as
\[
AIdx = \frac{MAddr - MBase}{bsize}
\]
and
\[
VAddr = VBase + LIdx \cdot bsize + MAddr \bmod bsize.
\]
These formulas formalize the conversion from a carried algebraic index to a destination-specific compacted layout [2605.05607].

To reduce lookup overhead, DySHARP adds an **AL TLB** whose tag is the concatenation of **Expert ID** and **AIdx**. The evaluation chooses a **512-entry AL TLB** as a sweet spot [2605.05607]. This subsystem underpins the claim that dynamic multimem addressing can support MoE communication with compact packets and local address resolution rather than sender-managed explicit addresses.

## 4. Token-centric kernel fusion and readiness-driven scheduling

The second core component of DySHARP is **token-centric kernel fusion**, which addresses a subtle asymmetry in traffic reduction. The paper argues that dynamic in-switch computing reduces communication volume **asymmetrically across directions**: in Dispatch, in-switch multicast reduces **GPU \(\rightarrow\) switch** traffic but not **switch \(\rightarrow\) GPU** traffic; in Combine, in-switch reduction reduces **switch \(\rightarrow\) GPU** traffic but not **GPU \(\rightarrow\) switch** traffic [2605.05607]. If Dispatch and Combine are executed as separate phases, each phase still retains one unreduced bottleneck direction, so traffic reduction alone does not fully translate into speedup.

DySHARP’s response is to reorganize the MoE layer as a **token-paced pipeline** rather than a sequence of four isolated operators: Dispatch, GEMM-1, GEMM-2, and Combine. The key claim is that downstream work becomes issuable at **token** or **tile-of-tokens** granularity. GEMM-1 can start when a tile has been dispatched, GEMM-2 can start when the corresponding GEMM-1 row finishes, and Combine can start when all \(topk\) expert outputs for a token are ready [2605.05607]. This pipelines Dispatch and Combine so that their opposite directional communication patterns overlap and complement one another.

Readiness tracking is implemented through a **token tracker** that monitors three dependency chains: **Dispatch \(\Rightarrow\) GEMM-1**, **GEMM-1 \(\Rightarrow\) GEMM-2**, and **GEMM-2 \(\Rightarrow\) Combine** [2605.05607]. It uses three tables:

- **TS Table**: tracks per-tile status, including `Valid`, `ExpID`, `Row`, `DAcc`, `TBCnt1`, and `TBCnt2`.
- **TID Table**: stores token IDs belonging to a token tile.
- **OR Table**: tracks per-token output readiness via `nReady` until \(nReady = topk\).

The scheduler is implemented as a **megakernel** with **persistent thread blocks**, so original thread blocks are transformed into tasks fetched from a task list [2605.05607]. SMs are partitioned into four groups dedicated to Dispatch, GEMM-1, GEMM-2, and Combine; GEMM-1 and GEMM-2 can share SMs if one has no ready work. Issuance is then gated by both resource availability and readiness predicates read from the tracker.

The paper distinguishes this design from prior overlap schemes by emphasizing that DySHARP fuses the entire **Dispatch \(\rightarrow\) GEMM-1 \(\rightarrow\) GEMM-2 \(\rightarrow\) Combine** chain and schedules according to **token/tile readiness boundaries** rather than operator completion [2605.05607]. This is the basis for the claim that dynamic multimem addressing and token-centric kernel fusion are **integral**, not separable, components.

## 5. Hardware support, runtime integration, and quantitative results

DySHARP is evaluated on a simulated **NVIDIA GH200 NVL32** system with **32 GPUs**, **9 NVSwitches**, and a fully connected fat-tree topology, using **BookSim2** and customized **Accel-Sim**. The simulated interconnect uses **NVLink 4.0** parameters, including **900 GB/s bidirectional bandwidth**, **250 ns** single-link latency, **1 \(\mu s\)** round-trip latency, and **16 B** flits [2605.05607]. Validation against DGX-H100 measurements reports average errors within **6%** for GEMM and DeepEP communication operators [2605.05607].

The target models are DeepSeek-V3-style MoE configurations with three sizes: **Small**, **Medium**, and **Large**, and activated experts \(topk \in \{8,16,32\}\), where the official DeepSeek-V3 configuration is **Large-8** [2605.05607]. Baselines include **DeepEP**, **NVLS**, **FasterMoE**, **Tutel**, **CCFuser**, **COMET**, and **DualPipe** [2605.05607].

The main quantitative claim is that DySHARP achieves up to:

- **\(2.31\times\)** over DeepEP,
- **\(5.12\times\)** over NVLS,
- **\(2.11\times\)** over FasterMoE,
- **\(1.98\times\)** over Tutel,
- **\(1.85\times\)** over CCFuser,
- **\(1.79\times\)** over COMET,
- **\(1.88\times\)** over DualPipe,

in end-to-end training [2605.05607]. For the MoE layer alone, the reported peak speedups are even larger, reaching **\(1.94\times\)** over COMET and **\(6.93\times\)** over NVLS [2605.05607].

The paper attributes these gains to two sources. First, **dynamic multimem addressing** reduces communication traffic by **nearly 50% relative to DeepEP** and avoids the **340% useless traffic** associated with static-NVLS emulation [2605.05607]. Second, **token-centric kernel fusion** improves bandwidth utilization by merging complementary asymmetric traffic directions. A communication-only evaluation reports that DySHARP achieves **over 90% of the ideal** predicted from reduced traffic volume and available bandwidth [2605.05607].

Scaling studies further report that DySHARP consistently outperforms DeepEP and COMET from **4 to 64 GPUs**, and that its advantage widens as GPU count increases [2605.05607]. It also remains fastest across sequence lengths from **1024 to 16384**, across training token-distribution standard deviations from **0.01 to 0.05**, and across inference power-law imbalance parameters \(\alpha\) from **0.5 to 2.5** [2605.05607]. The paper additionally reports gains in end-to-end inference, including prefill and decode, and evaluates GPT-OSS-120B and Qwen3-235B as further model cases [2605.05607].

## 6. Architectural cost, limitations, and nomenclatural disambiguation

DySHARP is best understood as an architectural extension of NVLS-like systems rather than a software-only optimization. It requires switch support beyond current NVLS, packet-format extensions, new ISA instructions, GPU Hub memory-manager changes, and runtime integration [2605.05607]. The switch modifications are characterized as lightweight, adding **one cycle** to the datapath with area overhead **< 0.01 mm\(^2\)** and **less than 0.1%** of NVSwitch die area. On the GPU side, the added structures include **MultimemQ** with **32 entries**, an **AL TLB** with **512 entries**, and tracker tables with **1024 entries** each for TS Table and OR Table, for a reported total area overhead of **0.198 mm\(^2\)**, or about **0.024%** of H100 die area [2605.05607].

The paper also makes clear that DySHARP’s benefits are largest in **communication-heavy MoE** settings: moderate or large \(topk\), substantial cross-GPU routing, and regimes where communication dominates compute [2605.05607]. It does not fundamentally solve expert load imbalance, and although it improves performance under imbalance, stragglers can still dominate latency. The design is also tailored to **NVLink/NVSwitch-like systems with shared-memory semantics and in-switch computing support**, so it is not directly portable to arbitrary interconnects [2605.05607].

A common source of confusion is the similarity of the name **DySHARP** to other arXiv systems with partially overlapping terminology. **SHARP**, in "Spectrum-aware Highly-dynamic Adaptation for Resolution Promotion in Remote Sensing Synthesis" [2603.21783], is a training-free, timestep-aware RoPE adaptation method for remote sensing text-to-image synthesis and is not the same system. **DynaShard**, in "Secure and Adaptive Blockchain Sharding Protocol with Hybrid Consensus and Dynamic Shard Management" [2411.06895], is a blockchain sharding protocol and likewise is not DySHARP. The MoE systems paper explicitly introduces **DySHARP** as a dynamic in-switch computing solution built around NVLS-style multicast and reduction for multi-GPU expert parallelism [2605.05607].

In that literature, DySHARP’s distinguishing claim is not merely that it accelerates MoE communication, but that it does so by combining a **dynamic extension of multimem addressing** with a **token-centric fused schedule**. This suggests that its contribution lies as much in the interaction between communication primitive and execution model as in either mechanism alone, a point reinforced by the ablation results reported in the paper [2605.05607].

Source: https://www.emergentmind.com/topics/dysharp