---
title: 'DeepEP: GPU-Initiated Comm for MoE'
url: https://www.emergentmind.com/topics/deepep
type: topic
---

# DeepEP: GPU-Initiated Comm for MoE

DeepEP is a specialized communication system for Mixture-of-Experts (MoE) models that implements **GPU-initiated, token-level RDMA** for expert parallelism. In the systems literature, it functions as the expert-parallel communication substrate for sparse all-to-all token dispatch and combine, and is repeatedly used as the performance baseline for MoE communication on tightly integrated GPU–NIC stacks. It is also described as an open-source expert-parallel communication library from DeepSeek, originally built around **NVSHMEM + IBGDA**, with distinct **Low-Latency (LL)** and **High-Throughput (HT)** kernels for decode versus prefill/training workloads [2512.19849][2606.05951].

## 1. Communication problem and MoE setting

DeepEP is defined by the communication pattern induced by expert parallelism. In an MoE layer, a router selects a small subset of experts for each token; experts are sharded across GPUs; and each layer therefore performs three phases: **dispatch** from the token’s original GPU to the GPUs hosting its selected experts, **expert compute** on those destination GPUs, and **combine** to return expert outputs to the original token owner for merging. The resulting pattern is a **sparse all-to-all** whose connectivity and volume are determined at runtime by routing decisions on the GPU [2512.19849].

Several properties make this pattern difficult for conventional collective libraries. Tokens are small—described in one study as approximately **7 KB per activation**, with the example of **FP8** and hidden size **7168**—yet each token may be sent to multiple experts, with an example of **top-\(K = 8\)**. The communication is therefore **fine-grained**, **irregular**, and **frequent**, because every MoE layer incurs both dispatch and combine. Standard host-driven collectives such as NCCL AllToAll are characterized as poorly matched to this regime because they assume dense, bulk-synchronous traffic and impose host-side launch and control overheads that are difficult to overlap with GPU-resident routing and expert execution [2512.19849][2606.05951].

DeepEP emerged as a response to this exact workload structure. Its purpose is not generic collectives, but MoE-specific token movement under expert parallelism. Later work on DeepSeek-V3 makes this context concrete: tokens are routed to **1 shared expert** and **8 routed experts** from a pool of **256 routed experts**, so expert-parallel all-to-all becomes a first-order systems concern at cluster scale [2505.09343].

## 2. Architectural model and communication substrate

The central architectural idea of DeepEP is to let the GPU, rather than the CPU, decide and initiate network communication. DeepEP is characterized as a **GPU-initiated, token-level RDMA communication system** in which GPU threads directly post RDMA work queue entries to the NIC via **IBGDA (InfiniBand GPUDirect Async)**, with no CPU in the critical path. On NVIDIA+Mellanox-style platforms, the GPU can write directly to NIC MMIO doorbells, fill work queue entries from GPU memory, and use GPUDirect RDMA to access remote GPU memory. This enables token transfers to begin as soon as a token or token chunk is ready inside the kernel, rather than after a host round-trip [2512.19849].

DeepEP’s original implementation, referred to as **DeepEP V1** in the NVSHMEM analysis, uses **NVSHMEM’s device-side symmetric-memory programming model** as a low-level substrate rather than as a high-level collective layer. Symmetric GPU memory is allocated identically across processing elements; DeepEP stores per-peer RDMA ring buffers, head/tail counters, credits, and metadata in that symmetric heap; and the kernels rely on device-side one-sided operations and atomics such as `nvshmemi_ibgda_put_nbi_warp` and `nvshmemi_ibgda_amo_nonfetch_add` to implement their own transport protocol [2606.05951].

A defining optimization is that DeepEP does not merely ship bytes between ranks. It embeds MoE-specific data movement logic directly into GPU kernels. On dispatch, **token deduplication** ensures that if several experts for the same token reside on GPUs inside the same node, the token is sent only once over the network and then duplicated or forwarded within the node via **NVLink**. On combine, DeepEP performs **hierarchical reduce**, first reducing expert outputs intra-node and only then sending the aggregated per-token result across nodes. This structure minimizes redundant inter-node traffic and exploits the large bandwidth asymmetry between NVLink and network RDMA [2512.19849].

The system has consequently been integrated into frameworks such as **Megatron-LM**, **vLLM**, and **SGLang**, and later papers consistently treat it as the high-performance reference design for expert-parallel MoE communication [2512.19849][2605.10670].

## 3. Low-latency and high-throughput execution modes

DeepEP exposes two communication modes because decode and training/prefill place different demands on the dispatch/combine path. The **HT path** targets large batches, where bandwidth is the priority. The **LL path** targets small batches, where end-to-end layer latency dominates [2512.19849][2606.05951].

In the **HT path**, DeepEP uses a **two-stage RDMA + NVLink pipeline**. Cross-node communication occurs only between GPUs with the same local index across nodes, implemented in the NVSHMEM-based design through **eight parallel NVSHMEM “world” teams**, one per GPU slot, assuming **8 P2P-accessible GPUs per node**. Once tokens reach the correct node via RDMA, they are redistributed inside the node through NVLink using direct GPU loads and stores into symmetric-memory buffers. The HT kernel is heavily **warp-specialized**: sender warps pack token messages, sender-coordinator warps aggregate them into larger RDMA bursts, RDMA-to-NVLink forwarder warps move data from inter-node buffers into intra-node ring buffers, and receiver warps place forwarded tokens into their final expert input tensors. A preliminary **notify_dispatch** phase exchanges token counts and prefix information so that the hot path can stream directly without dynamic routing decisions left unresolved [2606.05951].

In the **LL path**, DeepEP adopts a simpler, RDMA-centric structure to minimize latency. It removes the intra-node forwarding stage and instead uses a **full all-to-all RDMA mesh** in which each GPU can send directly to any expert’s GPU. A single global NVSHMEM world team is used, together with a strided-team overlay for slot-matched GPUs across nodes. Within an SM, warps are grouped by local experts; each warp iterates over tokens, handles one top-\(k\) destination, optionally converts activations to FP8, reserves remote receive-buffer slots, and either performs an IBGDA put or a direct NVLink copy when the target expert is intra-node and P2P-accessible. A remote atomic then publishes the final received-token count for that expert, and receiver warp groups poll the count until expert-local work is ready [2606.05951].

These two modes express the same underlying design principle: communication is driven from inside the GPU kernel at token or token-chunk granularity, and synchronization is performed with lightweight one-sided primitives rather than host-mediated collectives. Later work on NCCL GIN interprets DeepEP precisely in these terms, as a specialized MoE library whose dispatch and combine are fundamentally **device-initiated, one-sided, fine-grained**, and fused with layout conversion, quantization, and reduction logic [2511.15076].

## 4. Performance characteristics and deployment

DeepEP’s performance is discussed across multiple deployment settings, both as a native library and as a reference baseline. In the DeepSeek-V3 hardware analysis, DeepEP is the all-to-all expert-parallel implementation used in a system with **eight 400 Gbps InfiniBand NICs per node**. On the **Multi-Plane Fat-Tree (MPFT)** deployment, the reported EP dispatch and combine kernels communicate across **16 to 128 GPUs**, with **4096 tokens per GPU**, and achieve **bandwidth exceeding 40 GB/s per GPU**, described as nearly saturating the **400 Gbps** NIC bandwidth [2505.09343].

The same DeepSeek-V3 analysis ties DeepEP directly to the system’s overlap strategy. DeepSeek-V3 is described as architected to leverage **dual micro-batch overlap**, intentionally overlapping communication latency with computation, and DeepEP is cited in that context. The paper also attributes substantial gains to **IBGDA**, emphasizing that GPU control of the NIC eliminates the latency overhead of GPU–CPU coordination and allows many GPU threads to distribute control-plane work across work queues [2505.09343].

A separate line of work integrates DeepEP with **NCCL 2.28** through **GPU-Initiated Networking (GIN)** and reports that the DeepEP communication pattern can be expressed inside NCCL’s Device API with essentially the same performance envelope. For the NVSHMEM-to-GIN comparison, HT dispatch and combine bandwidths are reported to be typically **within 1–2%** between DeepEP’s NVSHMEM backend and a GIN backend, while some LL kernels show slightly lower latency with GIN when NVLink is enabled [2511.15076]. In a later NCCL EP study, DeepEP remains the reference point: NCCL EP’s LL dispatch is reported as matching or exceeding DeepEP in some multi-node cases, while end-to-end vLLM inference remains approximately **7–10%** behind DeepEP in the tested configuration [2603.13606].

DeepEP also serves as the stable communication layer on which other optimization work is built. **FEPLB**, for example, keeps DeepEP as the standard expert-parallel backend and adds a second, intra-node load-balancing phase using the NVLink Copy Engine and CPU scheduling. On GLM-5’s MoE layers with up to **16 H100 GPUs**, FEPLB reports **51–70%** reduction in token straggler, **50–68%** reduction in GEMM straggler, and **no measurable EP communication overhead**, with **8–15%** MoE layer speedup, all while leaving DeepEP’s inter-node dispatch/combine unchanged [2604.19654].

## 5. Portability limits, ordering assumptions, and failure semantics

The strongest criticism of DeepEP in later literature concerns portability. UCCL-EP argues that DeepEP’s architecture is **non-portable** because it assumes a hard coupling between GPU and NIC. DeepEP requires a NIC that supports GPUDirect RDMA and **IBGDA-like GPU-initiated operations**, a GPU driver stack that allows kernels to write into NIC MMIO doorbells or driver-defined registers, and strong NIC ordering guarantees. In the formulation used there, the engineering cost of reproducing an IBGDA-style stack across \(m\) GPU vendors and \(n\) NIC vendors scales as \(O(m \times n)\) [2512.19849].

The ordering assumptions are equally specific. DeepEP kernels are written under the expectation of **write-then-atomic ordering**, so that token payload writes become visible before a completion atomic or flag, and of **per-queue in-order delivery** akin to reliable connected InfiniBand semantics. Those assumptions are natural on Mellanox ConnectX RC transport, but they do not hold on fabrics such as **AWS EFA + SRD**, which is described as reliable but unordered, lacks hardware RDMA atomics, and provides no IBGDA interface for GPU-issued writes and atomics [2512.19849].

These architectural assumptions affect not only portability but also service semantics. In the fault-tolerance study **EEP**, DeepEP appears as a **fixed-membership** expert-parallel backend. There, DeepEP is characterized as executing dispatch and combine over a **preconfigured EP group** whose communicator, expert placement, routing metadata, and CUDA graph capture are all fixed at initialization. As a result, a single rank failure invalidates peer reachability, expert coverage, and graph-visible routing simultaneously, forcing a full restart in the evaluated serving setup. That paper reports a **348 s** unavailability period for the fixed-membership full-restart baseline, whereas EEP preserves steady-state performance to within **4.4%** of a fixed-membership DeepEP baseline and converts a single-rank failure into bounded pauses of **11 s** for recovery and **8 s** for reintegration [2605.10670].

The cumulative implication is not that DeepEP is slow, but that its performance model is tied to a narrow hardware and runtime contract: GPU-initiated networking, strong ordering, static EP membership, and specialized driver support.

## 6. Successors, alternatives, and terminological scope

Subsequent systems can largely be understood as attempts either to preserve DeepEP’s communication model while relaxing its hardware assumptions, or to replace parts of its interface and runtime with more general abstractions. **UCCL-EP** preserves DeepEP’s GPU-side algorithmic structure, including LL/HT kernels, token deduplication, and hierarchical reduction, but moves communication execution to multithreaded CPU proxies via a high-throughput GPU–CPU FIFO. It reports **comparable performance to the original DeepEP** on an NVIDIA-only platform and the first DeepEP-style token-level MoE communication on EFA, where it outperforms the best existing EP solution by up to **\(2.1\times\)** for dispatch and combine throughput [2512.19849].

**NCCL GIN** and **NCCL EP** pursue a different unification strategy. The GIN work shows that DeepEP’s one-sided GPU-initiated communication can be remapped from pointer-based NVSHMEM operations to NCCL window-and-signal semantics, while keeping the same MoE communication structure [2511.15076]. NCCL EP then presents `ncclEpDispatch` and `ncclEpCombine` as NCCL-native primitives, explicitly positioning itself as “DeepEP + Hybrid-EP, but natively inside NCCL,” with LL adapted from DeepEP and HT adapted from Hybrid-EP [2603.13606].

**FUSCO** frames its critique at a different level. It accepts that DeepEP is a state-of-the-art MoE communication library but argues that DeepEP still inherits a **device-major** communication contract that leaves costly pre- and post-communication rearrangements outside the library. FUSCO fuses transformation and communication through segment descriptors and a pipelined communication engine, reporting up to **\(2.01\times\)** speedup over DeepEP in communication benchmarks and **1.10–1.19×** lower training latency, as well as **1.06–1.16×** lower first-token generation latency in inference, in the evaluated end-to-end settings [2512.22036].

A final terminological point is necessary because the name is overloaded. In large-scale AI systems papers, **DeepEP** denotes the expert-parallel communication library described above. A separate 2025 paper uses “DeepEP” to refer to a framework for **deep Equilibrium Propagation** in convolutional CRNNs, based on intermediate error signals for scalable EP training [2508.15989]. The two uses are unrelated except for the acronym.

Source: https://www.emergentmind.com/topics/deepep