UCCL-EP: Portable MoE Communication Transport

• UCCL-EP is a communication substrate enabling dynamic, high-throughput expert-parallel data transfers in Mixture-of-Experts deep learning systems.
• The system employs lock-free GPU–CPU channels and CPU proxy workers that translate 16-byte transfer commands into RDMA operations, ensuring fine-grained ordering and low latency.
• UCCL-EP delivers up to 2.1× performance improvements over traditional GPU collective libraries, supporting diverse GPU and NIC architectures with enhanced portability.

UCCL-EP, in current literature, refers to distinct systems in multiple domains. Most notably, the abbreviation denotes (1) a portable expert-parallel communication system for Mixture-of-Experts (MoE) deep learning workloads, and (2) is occasionally used in high-energy physics to indicate FCC-based electron–proton collider proposals. The predominant technical usage is as a transport layer facilitating high-throughput, fine-grained expert parallelism across heterogeneous GPU and NIC architectures. This entry concentrates on UCCL-EP as the expert-parallel communication substrate, with brief annotation on the collider naming context.

1. Motivation: Expert Parallelism and Transport Demands

MoE architectures partition model capacity across many “experts,” routed dynamically per-token, with experts sharded one-to-one across GPUs. UCCL-EP is designed to address the stringent networking and scheduling demands of MoE, characterized by:

• Sparse, fine-grained all-to-all communication: Each token activation ($\sim$7 KB at hidden size 7168, FP8) is routed to a subset (typically 8) of experts amongst the global pool, producing highly dynamic and irregular traffic (Mao et al., 22 Dec 2025).
• Massive throughput and low-latency requirements: Up to 7 million operations/second per GPU at 400 Gbps link rates; latencies and bottlenecks quickly dominate performance at low message sizes (Table 1).
• Limitations of prior solutions: General-purpose GPU collectives (NCCL, RCCL) are structured for large, symmetric messages and are CPU-initiated, precluding optimal token pipelining. DeepEP, an earlier solution, achieves high performance with GPU-initiated RDMA, but requires tight vertical integration with Mellanox ConnectX NICs, inhibiting portability (Mao et al., 22 Dec 2025).

System	Message Initiation	Hardware Coupling	Portability
NCCL/RCCL	CPU	Loosely coupled	High
DeepEP	GPU	GPU–NIC tightly bound	Very limited
UCCL-EP	GPU→CPU→NIC	Decoupled via proxies	High

Key challenge: Overcoming the $O(m \times n)$ GPU/NIC integration complexity without sacrificing the token-level pipelining and throughput of specialized GPU-initiated protocols.

2. System Architecture and GPU–CPU Decoupling

UCCL-EP introduces a software control plane via high-throughput, lock-free GPU–CPU communication channels (Mao et al., 22 Dec 2025, Zhou et al., 24 Apr 2025).

• GPU-side: Each thread encodes a 16-byte TransferCmd (fields: dest_rank, command_type, q_channel_id, seq_no, offset_src, offset_dst, length) into GPU-resident FIFOs. No direct NIC interaction is performed by GPU threads.
• CPU proxies: Multithreaded CPU proxy workers (default 4 threads per GPU) poll the FIFOs, translate commands into libibverbs RDMA verbs (WRITE, ATOMIC, FENCE, BARRIER), and initiate network operations via GPUDirect.
• Data path: Token payloads remain on the GPU in HBM; RDMA writes move data across the network without intermediate staging (Zhou et al., 24 Apr 2025).
• Control path extensibility: The architecture follows that of the UCCL software transport, supporting plugin-based policy injection for path selection, congestion control, and reliability management.

This decoupling achieves hardware-agnostic control, enabling rapid integration of diverse GPU and NIC pairings (e.g., NVIDIA or AMD GPUs with EFA, IB, or Broadcom Thor-2 NICs), eliminating the need for device-specific kernel–NIC bridges.

3. Emulation of Ordering and Atomic Semantics

Fine-grained MoE/EP protocols require strong ordering guarantees: all writes for an expert must be reliably received before the completion/doorbell atomic is applied (Mao et al., 22 Dec 2025).

• Ordering on different RDMA fabrics: InfiniBand RC channels guarantee in-order delivery; many datacenter fabrics (e.g., AWS EFA) permit out-of-order arrival.
• Immediate-data sequencing: UCCL-EP tags all RDMA verbs with a (channel_id, seq_no) using the 32-bit immediate field. Receiver-side CPU proxies maintain an in-memory buffer per channel and apply operations in strict seq_no order.
• Modes of operation:
  • LL (Low Latency): Per-token WRITE+immediate, receiver applies atomics only after all prior WRITEs have applied.
  • HT (High Throughput): Batching per-expert into ring buffers, WRITEs entire chunk, ATOMICs update counters; partial ordering is enforced per-channel.
• Correctness guarantees: This method emulates strict in-order semantics on unordered NICs through software-managed sequencing buffers and completion polling, as in the following pseudocode segment:

struct ChannelState {
  uint32_t expected_seq_no = 0;
  map<uint32_t, ControlMsg> buffer;
};

void receiver_thread_poll() {
  while (true) {
    WrCompletion c = cq.poll();
    uint32_t imm  = c.imm_data;
    int chan  = imm >> 24;
    int seq   = imm & 0x00FFFFFF;
    ControlMsg m = parse_control_msg(c);
    auto &st = channel_states[chan];
    st.buffer[seq] = m;
    while (st.buffer.count(st.expected_seq_no)) {
      apply_control(st.buffer[st.expected_seq_no]);
      st.buffer.erase(st.expected_seq_no);
      st.expected_seq_no++;
    }
  }
}

This abstraction supports correct operation on both in-order and out-of-order RDMA hardware without requiring hardware modifications.

4. Implementation Specifics and Resource Model

• Source language: C++ (20.8 KLOC, including 2.4 K CUDA/ROCm), plus supporting Python modules.
• GPU support: Both CUDA and ROCm backends; kernel atomics ported to host-driven semantics, with adaptation for warp/wavefront differences.
• Memory model: Per-GPU memory bounded by $k_{max\_inflight} \times 16$ B for FIFOs (typical: 4 K × 16 B = 64 KB), plus ring buffers for HT mode.
• NUMA affinity: CPU proxy threads are NUMA-bound to the GPU's PCIe domain to maximize PCIe throughput.
• Transport substrate: Libibverbs is used for all RDMA verbs; no reliance on vendor-specific user-space drivers, nor dependency on specific MMIO/doorbell maps.
• NIC support: Validated on Mellanox/CX, AWS EFA, and Broadcom Thor-2 hardware.

5. Performance Evaluation Across Heterogeneous Platforms

Empirical results (Mao et al., 22 Dec 2025) establish that UCCL-EP provides high throughput and low latency, consistently matching or outperforming specialized, non-portable solutions.

• NVIDIA+EFA: UCCL-EP outperforms prior PPLX and DeepEP-compatible systems by up to 2.1× on dispatch/combine benchmarks at high token counts (≥512 tokens, EP size ≥16).
• AMD+Broadcom Thor-2: UCCL-EP matches Infiniband performance within ±3% (LL/HT modes); RCCL is consistently 1.8–2× slower at high EP.
• End-to-end ML benchmarks:
  • SGLang inference: Up to 40% higher token throughput on NVIDIA+EFA at EP=32.
  • DeepSeek-V3 training (Megatron-LM based, AMD MI300X, 16 nodes): 27% improvement in TFLOPS/GPU, 44% increase in token throughput per GPU compared to RCCL.

Token dispatch latency is dominated by network delay ($\sim$15 µs over EFA); internal GPU–CPU communication via FIFO incurs sub-2 µs overhead. Four CPU proxy threads per GPU sustain 8 Mops/s; single-threaded proxies saturate at around 1 Mops/s.

6. Extensibility, Congestion Control, and Future Directions

• Congestion and flow control: CPU proxies can globally monitor per-queue-pair outstanding tokens and implement flexible pacing/in-cast mitigation—capabilities that are unavailable to individual GPU threads.
• Elasticity: Proxies can dynamically re-map experts, support elastic scaling, node joins/failures, and handle memory negotiation at runtime.
• Interoperability with new accelerators: The decoupling pattern extends to other AI accelerators (TPUs, AWS Trainium, Habana) by porting the GPU FIFO/push code, as all RDMA orchestration resides in standard host software.
• Potential improvements: Integrating best-effort on-GPU token packing and finer control granularity may yield additional latency benefits at small batch sizes. Planned EFA firmware updates are expected to address inefficiencies that manifest in the low-latency limit of UCCL-EP_LL.

7. UCCL-EP in Other Contexts: FCC-Based ep Colliders

The abbreviation UCCL-EP also appears in parts of the accelerator/HEP literature as a shorthand for "FCC-based electron–proton collider" proposals (Acar et al., 2015). In this context:

• UCCL-EP/e-FCC refers to designs colliding 60–175 GeV electrons (ERL) with 50 TeV protons (FCC-pp), achieving center-of-mass energies up to 5.9 TeV and baseline luminosities of $1 \times 10^{34}$ cm$^{-2}$ s$^{-1}$.
• The focus is on maximizing luminosity using advanced beam optics, energy recovery linacs, and dynamic focusing; the abbreviation is not directly related to expert-parallel or network systems.

• UCCL-EP: Portable Expert-Parallel Communication (Mao et al., 22 Dec 2025)
• An Extensible Software Transport Layer for GPU Networking (Zhou et al., 24 Apr 2025)
• FCC based ep and $\mu$p colliders (Acar et al., 2015)