DeepEP: GPU-Initiated Comm for MoE

• DeepEP is a specialized communication system that enables GPU-initiated, token-level RDMA for expert-parallel Mixture-of-Experts models.
• It employs fine-grained, sparse all-to-all token dispatch with deduplication and hierarchical reduction to optimize intra- and inter-node data flow.
• DeepEP has become the performance baseline in MoE research, showcasing significant scalability and low-latency benefits on integrated GPU-NIC platforms.

DeepEP is a specialized communication system for Mixture-of-Experts (MoE) models that implements GPU-initiated, token-level RDMA for expert parallelism. In the systems literature, it functions as the expert-parallel communication substrate for sparse all-to-all token dispatch and combine, and is repeatedly used as the performance baseline for MoE communication on tightly integrated GPU–NIC stacks. It is also described as an open-source expert-parallel communication library from DeepSeek, originally built around NVSHMEM + IBGDA, with distinct Low-Latency (LL) and High-Throughput (HT) kernels for decode versus prefill/training workloads (Mao et al., 22 Dec 2025, Ma et al., 4 Jun 2026).

1. Communication problem and MoE setting

DeepEP is defined by the communication pattern induced by expert parallelism. In an MoE layer, a router selects a small subset of experts for each token; experts are sharded across GPUs; and each layer therefore performs three phases: dispatch from the token’s original GPU to the GPUs hosting its selected experts, expert compute on those destination GPUs, and combine to return expert outputs to the original token owner for merging. The resulting pattern is a sparse all-to-all whose connectivity and volume are determined at runtime by routing decisions on the GPU (Mao et al., 22 Dec 2025).

Several properties make this pattern difficult for conventional collective libraries. Tokens are small—described in one study as approximately 7 KB per activation, with the example of FP8 and hidden size 7168—yet each token may be sent to multiple experts, with an example of top-$K = 8$. The communication is therefore fine-grained, irregular, and frequent, because every MoE layer incurs both dispatch and combine. Standard host-driven collectives such as NCCL AllToAll are characterized as poorly matched to this regime because they assume dense, bulk-synchronous traffic and impose host-side launch and control overheads that are difficult to overlap with GPU-resident routing and expert execution (Mao et al., 22 Dec 2025, Ma et al., 4 Jun 2026).

DeepEP emerged as a response to this exact workload structure. Its purpose is not generic collectives, but MoE-specific token movement under expert parallelism. Later work on DeepSeek-V3 makes this context concrete: tokens are routed to 1 shared expert and 8 routed experts from a pool of 256 routed experts, so expert-parallel all-to-all becomes a first-order systems concern at cluster scale (Zhao et al., 14 May 2025).

2. Architectural model and communication substrate

The central architectural idea of DeepEP is to let the GPU, rather than the CPU, decide and initiate network communication. DeepEP is characterized as a GPU-initiated, token-level RDMA communication system in which GPU threads directly post RDMA work queue entries to the NIC via IBGDA (InfiniBand GPUDirect Async), with no CPU in the critical path. On NVIDIA+Mellanox-style platforms, the GPU can write directly to NIC MMIO doorbells, fill work queue entries from GPU memory, and use GPUDirect RDMA to access remote GPU memory. This enables token transfers to begin as soon as a token or token chunk is ready inside the kernel, rather than after a host round-trip (Mao et al., 22 Dec 2025).

DeepEP’s original implementation, referred to as DeepEP V1 in the NVSHMEM analysis, uses NVSHMEM’s device-side symmetric-memory programming model as a low-level substrate rather than as a high-level collective layer. Symmetric GPU memory is allocated identically across processing elements; DeepEP stores per-peer RDMA ring buffers, head/tail counters, credits, and metadata in that symmetric heap; and the kernels rely on device-side one-sided operations and atomics such as nvshmemi_ibgda_put_nbi_warp and nvshmemi_ibgda_amo_nonfetch_add to implement their own transport protocol (Ma et al., 4 Jun 2026).

A defining optimization is that DeepEP does not merely ship bytes between ranks. It embeds MoE-specific data movement logic directly into GPU kernels. On dispatch, token deduplication ensures that if several experts for the same token reside on GPUs inside the same node, the token is sent only once over the network and then duplicated or forwarded within the node via NVLink. On combine, DeepEP performs hierarchical reduce, first reducing expert outputs intra-node and only then sending the aggregated per-token result across nodes. This structure minimizes redundant inter-node traffic and exploits the large bandwidth asymmetry between NVLink and network RDMA (Mao et al., 22 Dec 2025).

The system has consequently been integrated into frameworks such as Megatron-LM, vLLM, and SGLang, and later papers consistently treat it as the high-performance reference design for expert-parallel MoE communication (Mao et al., 22 Dec 2025, Sun et al., 11 May 2026).

3. Low-latency and high-throughput execution modes

DeepEP exposes two communication modes because decode and training/prefill place different demands on the dispatch/combine path. The HT path targets large batches, where bandwidth is the priority. The LL path targets small batches, where end-to-end layer latency dominates (Mao et al., 22 Dec 2025, Ma et al., 4 Jun 2026).

In the HT path, DeepEP uses a two-stage RDMA + NVLink pipeline. Cross-node communication occurs only between GPUs with the same local index across nodes, implemented in the NVSHMEM-based design through eight parallel NVSHMEM “world” teams, one per GPU slot, assuming 8 P2P-accessible GPUs per node. Once tokens reach the correct node via RDMA, they are redistributed inside the node through NVLink using direct GPU loads and stores into symmetric-memory buffers. The HT kernel is heavily warp-specialized: sender warps pack token messages, sender-coordinator warps aggregate them into larger RDMA bursts, RDMA-to-NVLink forwarder warps move data from inter-node buffers into intra-node ring buffers, and receiver warps place forwarded tokens into their final expert input tensors. A preliminary notify_dispatch phase exchanges token counts and prefix information so that the hot path can stream directly without dynamic routing decisions left unresolved (Ma et al., 4 Jun 2026).

In the LL path, DeepEP adopts a simpler, RDMA-centric structure to minimize latency. It removes the intra-node forwarding stage and instead uses a full all-to-all RDMA mesh in which each GPU can send directly to any expert’s GPU. A single global NVSHMEM world team is used, together with a strided-team overlay for slot-matched GPUs across nodes. Within an SM, warps are grouped by local experts; each warp iterates over tokens, handles one top-$k$ destination, optionally converts activations to FP8, reserves remote receive-buffer slots, and either performs an IBGDA put or a direct NVLink copy when the target expert is intra-node and P2P-accessible. A remote atomic then publishes the final received-token count for that expert, and receiver warp groups poll the count until expert-local work is ready (Ma et al., 4 Jun 2026).

These two modes express the same underlying design principle: communication is driven from inside the GPU kernel at token or token-chunk granularity, and synchronization is performed with lightweight one-sided primitives rather than host-mediated collectives. Later work on NCCL GIN interprets DeepEP precisely in these terms, as a specialized MoE library whose dispatch and combine are fundamentally device-initiated, one-sided, fine-grained, and fused with layout conversion, quantization, and reduction logic (Hamidouche et al., 19 Nov 2025).

4. Performance characteristics and deployment

DeepEP’s performance is discussed across multiple deployment settings, both as a native library and as a reference baseline. In the DeepSeek-V3 hardware analysis, DeepEP is the all-to-all expert-parallel implementation used in a system with eight 400 Gbps InfiniBand NICs per node. On the Multi-Plane Fat-Tree (MPFT) deployment, the reported EP dispatch and combine kernels communicate across 16 to 128 GPUs, with 4096 tokens per GPU, and achieve bandwidth exceeding 40 GB/s per GPU, described as nearly saturating the 400 Gbps NIC bandwidth (Zhao et al., 14 May 2025).

The same DeepSeek-V3 analysis ties DeepEP directly to the system’s overlap strategy. DeepSeek-V3 is described as architected to leverage dual micro-batch overlap, intentionally overlapping communication latency with computation, and DeepEP is cited in that context. The paper also attributes substantial gains to IBGDA, emphasizing that GPU control of the NIC eliminates the latency overhead of GPU–CPU coordination and allows many GPU threads to distribute control-plane work across work queues (Zhao et al., 14 May 2025).

A separate line of work integrates DeepEP with NCCL 2.28 through GPU-Initiated Networking (GIN) and reports that the DeepEP communication pattern can be expressed inside NCCL’s Device API with essentially the same performance envelope. For the NVSHMEM-to-GIN comparison, HT dispatch and combine bandwidths are reported to be typically within 1–2% between DeepEP’s NVSHMEM backend and a GIN backend, while some LL kernels show slightly lower latency with GIN when NVLink is enabled (Hamidouche et al., 19 Nov 2025). In a later NCCL EP study, DeepEP remains the reference point: NCCL EP’s LL dispatch is reported as matching or exceeding DeepEP in some multi-node cases, while end-to-end vLLM inference remains approximately 7–10% behind DeepEP in the tested configuration (Goldman et al., 13 Mar 2026).

DeepEP also serves as the stable communication layer on which other optimization work is built. FEPLB, for example, keeps DeepEP as the standard expert-parallel backend and adds a second, intra-node load-balancing phase using the NVLink Copy Engine and CPU scheduling. On GLM-5’s MoE layers with up to 16 H100 GPUs, FEPLB reports 51–70% reduction in token straggler, 50–68% reduction in GEMM straggler, and no measurable EP communication overhead, with 8–15% MoE layer speedup, all while leaving DeepEP’s inter-node dispatch/combine unchanged (Qi et al., 21 Apr 2026).

5. Portability limits, ordering assumptions, and failure semantics

The strongest criticism of DeepEP in later literature concerns portability. UCCL-EP argues that DeepEP’s architecture is non-portable because it assumes a hard coupling between GPU and NIC. DeepEP requires a NIC that supports GPUDirect RDMA and IBGDA-like GPU-initiated operations, a GPU driver stack that allows kernels to write into NIC MMIO doorbells or driver-defined registers, and strong NIC ordering guarantees. In the formulation used there, the engineering cost of reproducing an IBGDA-style stack across $m$ GPU vendors and $n$ NIC vendors scales as $O(m \times n)$ (Mao et al., 22 Dec 2025).

The ordering assumptions are equally specific. DeepEP kernels are written under the expectation of write-then-atomic ordering, so that token payload writes become visible before a completion atomic or flag, and of per-queue in-order delivery akin to reliable connected InfiniBand semantics. Those assumptions are natural on Mellanox ConnectX RC transport, but they do not hold on fabrics such as AWS EFA + SRD, which is described as reliable but unordered, lacks hardware RDMA atomics, and provides no IBGDA interface for GPU-issued writes and atomics (Mao et al., 22 Dec 2025).

These architectural assumptions affect not only portability but also service semantics. In the fault-tolerance study EEP, DeepEP appears as a fixed-membership expert-parallel backend. There, DeepEP is characterized as executing dispatch and combine over a preconfigured EP group whose communicator, expert placement, routing metadata, and CUDA graph capture are all fixed at initialization. As a result, a single rank failure invalidates peer reachability, expert coverage, and graph-visible routing simultaneously, forcing a full restart in the evaluated serving setup. That paper reports a 348 s unavailability period for the fixed-membership full-restart baseline, whereas EEP preserves steady-state performance to within 4.4% of a fixed-membership DeepEP baseline and converts a single-rank failure into bounded pauses of 11 s for recovery and 8 s for reintegration (Sun et al., 11 May 2026).

The cumulative implication is not that DeepEP is slow, but that its performance model is tied to a narrow hardware and runtime contract: GPU-initiated networking, strong ordering, static EP membership, and specialized driver support.

6. Successors, alternatives, and terminological scope

Subsequent systems can largely be understood as attempts either to preserve DeepEP’s communication model while relaxing its hardware assumptions, or to replace parts of its interface and runtime with more general abstractions. UCCL-EP preserves DeepEP’s GPU-side algorithmic structure, including LL/HT kernels, token deduplication, and hierarchical reduction, but moves communication execution to multithreaded CPU proxies via a high-throughput GPU–CPU FIFO. It reports comparable performance to the original DeepEP on an NVIDIA-only platform and the first DeepEP-style token-level MoE communication on EFA, where it outperforms the best existing EP solution by up to $2.1\times$ for dispatch and combine throughput (Mao et al., 22 Dec 2025).

NCCL GIN and NCCL EP pursue a different unification strategy. The GIN work shows that DeepEP’s one-sided GPU-initiated communication can be remapped from pointer-based NVSHMEM operations to NCCL window-and-signal semantics, while keeping the same MoE communication structure (Hamidouche et al., 19 Nov 2025). NCCL EP then presents ncclEpDispatch and ncclEpCombine as NCCL-native primitives, explicitly positioning itself as “DeepEP + Hybrid-EP, but natively inside NCCL,” with LL adapted from DeepEP and HT adapted from Hybrid-EP (Goldman et al., 13 Mar 2026).

FUSCO frames its critique at a different level. It accepts that DeepEP is a state-of-the-art MoE communication library but argues that DeepEP still inherits a device-major communication contract that leaves costly pre- and post-communication rearrangements outside the library. FUSCO fuses transformation and communication through segment descriptors and a pipelined communication engine, reporting up to $2.01\times$ speedup over DeepEP in communication benchmarks and 1.10–1.19× lower training latency, as well as 1.06–1.16× lower first-token generation latency in inference, in the evaluated end-to-end settings (Zhu et al., 26 Dec 2025).

A final terminological point is necessary because the name is overloaded. In large-scale AI systems papers, DeepEP denotes the expert-parallel communication library described above. A separate 2025 paper uses “DeepEP” to refer to a framework for deep Equilibrium Propagation in convolutional CRNNs, based on intermediate error signals for scalable EP training (Lin et al., 21 Aug 2025). The two uses are unrelated except for the acronym.