Dynamic Multimem Addressing for MoE

• Dynamic multimem addressing is a mechanism that extends NVLink SHARP’s static multimem model to support irregular, per-token communication in Mixture-of-Experts execution.
• It reduces redundant inter-GPU traffic in Dispatch and Combine phases by carrying a single algebraic multimem address alongside a compact target list and using local algebraic-layout mapping.
• The design introduces new instructions and an AL TLB to enable efficient token mapping and in-switch multicasting, resulting in up to 1.94× speedup over traditional methods.

Dynamic multimem addressing is a communication mechanism for Mixture-of-Experts (MoE) expert parallelism that extends NVLink SHARP (NVLS) from static collectives to dynamic, irregular, per-token communication. In DySHARP, it is presented as a dynamic extension of NVLS’s multimem addressing and is designed to reduce redundant inter-GPU transfers in the Dispatch and Combine phases of MoE execution. Its defining idea is to carry one multimem address together with a lightweight target list, while deferring final per-GPU placement through local algebraic-layout mapping. This separates a shared algebraic index from asymmetric GPU-local memory layouts and makes in-switch multicast and reduction applicable to MoE patterns that violate the fixed target sets and symmetric addressing assumed by NVLS (Zhang et al., 7 May 2026).

1. Problem setting in MoE expert parallelism

In MoE expert parallelism, each token is routed to a topk subset of experts, and those experts may live on different GPUs. This induces two communication phases. Dispatch sends token data from the source GPU to destination GPUs hosting activated experts. Combine sends expert outputs back and aggregates them at the token’s source GPU. The central inefficiency identified in DySHARP is that the same token data is often transferred multiple times, and likewise aggregatable outputs are transferred multiple times, even though the data is identical or reducible (Zhang et al., 7 May 2026).

The paper gives two concrete examples. In Dispatch, a single token destined for multiple GPUs is repeatedly sent from the source GPU to the switch, generating redundant GPU→switch traffic. In Combine, partial outputs for the same token are separately sent back to the source GPU, generating redundant switch→GPU traffic. Profiling on DeepSeek-V3 on a simulated GH200 NVL32 reports that nearly 50% of total traffic is redundant when the activated expert count is large ($\ge 8$). The same study also states that naïvely emulating MoE communication with static collectives can generate 340% useless traffic, eliminating any benefit from in-switch computing.

These observations motivate a communication primitive that preserves the compactness of multimem-style communication while admitting per-request variation in the destination set. This suggests that the principal challenge is not only bandwidth consumption, but also representational mismatch between MoE communication patterns and the assumptions built into static collective hardware.

2. Why static NVLS is insufficient

NVLink SHARP supports in-switch computing via multimem instructions, but it is designed for static collectives such as AllGather and Reduce-Scatter. Its operating assumptions are twofold: every request in the collective talks to the same GPU group, and the same logical element is at the same memory offset across GPUs. MoE violates both assumptions because each token may be routed to a different subset of experts or GPU destinations, and because tokens are compacted per GPU, so the layout offset of a given token differs across GPUs (Zhang et al., 7 May 2026).

A static NVLS packet carries only one address and relies on preconfigured destinations. Consequently, it cannot represent MoE’s per-token, per-request dynamic target set. If Dispatch and Combine are forced into static collectives, large amounts of useless traffic are introduced. Dynamic multimem addressing is therefore framed not as a general replacement for multimem, but as a specific generalization from static, symmetric, fixed-target collectives to dynamic, irregular, per-token MoE communication.

The key conceptual claim is that Dispatch and Combine remain algebraically related to AllGather and Reduce-Scatter, but their memory layouts are asymmetric because each GPU compacts only the tokens it actually receives. DySHARP resolves this by separating the algebraic index from the layout index. The former is represented by the multimem address; the latter is computed locally on each destination GPU.

3. ISA, packet, and runtime design

DySHARP introduces two new instructions: dymultimem.st for multicast in Dispatch and dymultimem.ld_reduce for reduction in Combine. These extend existing multimem instructions with target information. The instruction fields are r2 for the multimem address, whose offset is the algebraic index; r1 for the data operand for .st or the destination for the reduced value for .ld_reduce; r3 for the target count; and r4 for the base address of a contiguous target list. The instruction first fetches the target list from memory and then issues the complete request (Zhang et al., 7 May 2026).

The NVLink packet format is correspondingly extended. In flit0, the original 64-bit address is replaced by a 48-bit multimem address, a 1-bit stage field indicating Dispatch or Combine, and a 15-bit target count. This is followed by target extension flits encoding destination expert IDs, 16 bits each, eight per flit. Payload and byte-enable flits remain unchanged. Relative to explicit addressing, the point is to avoid listing full destination addresses and instead send only one algebraic address plus a compact target list.

A concise summary of the modified components is given below.

Component	Extension
Source GPU	LSU fetches target list; MultimemQ holds instructions awaiting target fetch completion
Switch	Route logic becomes target-aware; packet is replicated per output port and trimmed per port
Destination GPU	Hub memory manager performs algebraic-layout (AL) index mapping
Runtime	Adds CUDymulticastObjectProp, cuDyMulticastCreate, cuDyMulticastBindAddr

The runtime extensions specify block size bsize, stage, size of multimem region sets, and multimem and virtual address spaces. The runtime also obtains nactive[expert], the number of tokens routed to a given expert, determined by gating before Dispatch. An additional design constraint is that weighted reduction is not supported directly in dymultimem.ld_reduce because that would increase hardware complexity. Instead, weights are applied in the epilogue of GEMM-2, so the switch performs an unweighted reduction over values already scaled by gating weights: $\sum_i (w_i \cdot o_i).$

4. Algebraic-layout mapping and dynamic redirection

The heart of dynamic multimem addressing is the rule that the carried multimem address encodes the algebraic index, while each destination GPU uses an AL Table to map that algebraic index to a layout index. The request therefore does not contain the final memory offset on each GPU. It contains a logical index into the expert’s token tensor, and the GPU-local memory manager decides where the token is physically stored (Zhang et al., 7 May 2026).

During Dispatch, when a token arrives with an unseen algebraic index ($\text{AIdx}$), the memory manager allocates the next available layout block and records the mapping in the AL Table, including Valid and LIdx = layout block index. A counter tracks the next free layout block. The paper describes this as an accumulative allocation scheme that compacts a fragmented algebraic tensor into a dense layout tensor. During Combine, the expert computation does not change token order, so the same algebraic-layout mapping built during Dispatch is reused. When a dymultimem.ld_reduce request arrives, the memory manager looks up the AL Table and retrieves the corresponding LIdx.

The address translation is given explicitly: $\text{AIdx} = (MAddr - MBase) / bsize$

$$VAddr = VBase + LIdx \cdot bsize + MAddr \bmod bsize$$

where MAddr is the multimem address, MBase and VBase are base addresses, bsize is the block size, and LIdx is the layout index from AL management. This makes the dynamic multimem address a logical index rather than the final physical address.

To accelerate lookup, DySHARP introduces an AL TLB. Its tag is the concatenation of Expert ID and AIdx. Entries are stored using CAM for tag matching and SRAM for data. On a hit, LIdx is returned directly; on a miss, the AL Table is consulted and the entry is brought into the AL TLB. The paper notes that accesses within a token vector tend to be contiguous, giving strong temporal locality and good hit rate.

5. Execution path across source GPU, switch, and destination GPU

The source GPU, switch, and destination GPU all participate in the dynamic addressing semantics. At the source GPU, the SM’s LSU is extended to fetch the target list before issuing a dymultimem request, and the separate MultimemQ stores instructions waiting for target fetch completion. Once the target list is ready, the LSU sends the complete request packet onto the network (Zhang et al., 7 May 2026).

At the switch, the routing logic becomes target-aware. For each target $i$, the output port is computed as

$$\text{OutPort}_i = \text{Target}_i / \#\text{expert\_per\_GPU}.$$

The switch replicates the packet per output port and trims each replica so that it contains only the targets relevant to that port. For dymultimem.ld_reduce, the switch’s reduction logic tracks the number of targets and decrements a counter as partial responses arrive; when the counter reaches zero, the reduced result is returned.

At the destination GPU, a hardware memory manager in the Hub performs algebraic-layout index mapping. It translates the multimem address to a local virtual address before virtual-to-physical translation. The paper summarizes the request path in eight stages: the dymultimem instruction enters LSQ; the LSU fetches the target list; the source issues the request; the switch computes output ports and forwards replicas; the destination queries the AL TLB; on a hit, MV translation directly yields the virtual address; on a miss, the AL Table is accessed and new layout blocks are allocated on first touch in Dispatch; finally, the translated virtual address is used for memory access.

A compact addressing rule follows from this workflow: one packet equals one multimem address plus a target list. This preserves multimem-style payload compactness while making the destination set dynamic on a per-request basis.

6. Role in Dispatch and Combine, and the asymmetry problem

Dynamic multimem addressing is the communication primitive that enables the full MoE pipeline. In Dispatch, the source GPU issues dymultimem.st; the packet carries the source token’s algebraic index and target expert list; the switch multicasts only to destination GPUs that host activated experts; and destination GPUs dynamically allocate local layout blocks for arriving tokens. During expert computation, the expert GEMMs execute locally, and token order does not change. That invariant is what allows the same AL mapping to be reused in Combine. For weighted combine semantics, the expert’s output is multiplied by the gate weight in the GEMM-2 epilogue. In Combine, once all topk outputs for a token are ready, the source GPU issues dymultimem.ld_reduce; the packet carries the same multimem address or algebraic index and the target list; the switch performs in-switch reduction across partial outputs; and the result is returned only once to the source GPU (Zhang et al., 7 May 2026).

The design therefore provides two capabilities simultaneously: multicast without explicit full destination addresses and reduction with local per-GPU address asymmetry hidden behind AL mapping. The paper also emphasizes that traffic reduction alone does not directly translate into speedup because the reduction is directionally asymmetric. Multicast reduces GPU→switch traffic, whereas reduction reduces switch→GPU traffic. If Dispatch and Combine run in isolation, the under-reduced direction becomes the bottleneck.

This asymmetry is why DySHARP couples dynamic multimem addressing with token-centric kernel fusion. The ablation study distinguishes DySHARP-Basic, defined as dynamic multimem addressing without overlap; DySHARP-COMET, defined as dynamic multimem addressing plus basic overlap; and Kernel fusion only, defined as overlap without dynamic multimem addressing. The stated conclusion is that dynamic multimem addressing is responsible for the traffic reduction, but kernel fusion is needed to expose that benefit as runtime improvement.

7. Performance characteristics and terminological scope

The evaluation isolates several effects attributable to dynamic multimem addressing. Compared with DeepEP, it cuts traffic by nearly 50%. Compared with NVLS used as a static-collective workaround, DySHARP avoids the large amount of useless traffic; NVLS can even increase traffic relative to DeepEP. When Dispatch and Combine are run without computation to measure pure communication, DySHARP achieves, on average, over 90% of the ideal performance based on traffic volume and bandwidth. The paper also reports that dynamic multimem addressing has consistently higher payload efficiency than explicit addressing; for 8 destinations, explicit addressing lowers payload efficiency from an ideal 80% to 69%, whereas dynamic multimem addressing retains near-ideal payload efficiency. For the full DySHARP system, end-to-end speedup is up to 1.79$\times$ over COMET; in MoE-layer execution, speedup is up to 1.94$\times$ over COMET and 6.93$\times$ over NVLS (Zhang et al., 7 May 2026).

Within the DySHARP framework, these numbers establish a specific division of labor. Dynamic multimem addressing removes redundant traffic and makes dynamic in-switch multicast and reduction viable for MoE. Token-centric kernel fusion converts the resulting communication savings into realized runtime speedup by addressing directional asymmetry.

The phrase “dynamic addressing” also appears in unrelated literatures. In the Dynamic Neural Turing Machine, a trainable memory addressing scheme gives each memory cell an address vector and a content vector, enabling continuous and discrete read/write mechanisms (Gulcehre et al., 2016). In route-restricted networks, balanced dynamic content addressing in trees combines greedy embedding with interval-based coordinates and dynamic stabilization under churn (Roos et al., 2017). In MANETs, a dynamic distributed address allocation algorithm seeks an “IP resembles topology” state through hierarchical clusters and local address tables (Khatri et al., 2016). This suggests that “dynamic addressing” is a cross-domain term, but in DySHARP it has a precise hardware-communication meaning: one multimem address plus a target list, interpreted through local algebraic-layout mapping, for dynamic MoE Dispatch and Combine.