- The paper introduces a novel CUDA Graph-enabled UCX framework that dynamically orchestrates multi-path GPU communication to reduce CPU overhead and boost bandwidth by up to 2.95×.
- It demonstrates significant application-level speedups of up to 1.28× in MPI-based solvers while utilizing LRU caching to minimize graph instantiation costs.
- Empirical evaluations on modern GPU systems validate the framework's ability to efficiently bridge hardware interconnect heterogeneity for large message transfers.
Accelerating Intra-Node GPU-to-GPU Communication via Multi-Path CUDA Graphs
Introduction and Motivation
Modern HPC systems leverage multi-GPU nodes interconnected by hierarchical topologies (NVLink, NVSwitch, PCIe), creating significant challenges and opportunities for high-throughput intra-node GPU communication. Point-to-point (P2P) GPU communication traditionally saturates a single available path (e.g., one NVLink or PCIe lane), limiting effective concurrency and aggregate bandwidth. While UCX and other MPI transports support basic multi-path striping, concurrent data staging and aggressive overlapping (such as simultaneously using host memory and various device interconnects) remain underexplored and are often bottlenecked by CPU-side launch and synchronization overhead.
The paper introduces a novel framework that extends UCX with seamless CUDA Graph integration for multi-path intra-node GPU communication. By dynamically constructing, executing, and caching CUDA Graphs within the UCX CUDA transport, the framework achieves reduced CPU overhead, increased concurrency, and substantial improvements in bandwidth and application-level efficiency compared to existing single-path or statically pipelined designs.
Technical Contributions
The proposed framework introduces two architectural advances:
- Multi-path communication within UCX: The framework supports dynamically detecting available GPU pairs, interconnect topology, and selecting optimal communication routes between source and destination GPUs. Transfers can be concurrently split across direct NVLink, PCIe, and host-staged paths, supporting fine-grained chunking and pipelined overlap managed by a 2-D pipelining engine.
- CUDA Graph-based execution: For each unique multi-path transfer pattern (determined by involved GPUs, path selection, and chunking), the system dynamically generates a CUDA Graph (DAG) that encapsulates all memory copy and synchronization operations for a single message. This graph is cached using an LRU policy within UCX, minimizing per-transfer graph creation/reinstantiation overhead.
Data integrity is ensured by device synchronization semantics, ordered chunk scheduling, and staged dependency management. The CUDA Graph abstraction significantly reduces launch/synchronization overhead, particularly for large and repetitive bulk transfers. The presented algorithms rely on explicit CUDA Driver API calls for efficient graph construction.
Empirical Evaluation
The proposed design is evaluated across a comprehensive set of micro-benchmarks (UCX Put, OMB Bandwidth/Bidirectional Bandwidth/Latency) and a representative application (MPI-based Jacobi iterative solver) on two multi-GPU systems (Beluga: V100 with dual NVLinks, Narval: A100 with quad NVLinks). Key quantitative findings include:
- Up to 2.95× bandwidth improvement over single-path UCX (UCT::CUDA-IPC) in OMB GPU-to-GPU bandwidth tests for messages up to 512MB.
- Enhanced performance stems from saturating aggregate bandwidth via multiple NVLink and PCIe (via host) paths, with chunk size and path count tuned through runtime and offline configuration.
- The framework achieves up to 1.28× application-level speedup in the distributed Jacobi solver compared to the default single-path approach.
- CUDA Graph overhead (construction, instantiation, launch) is amortized effectively for large messages and high operation counts; for small messages or infrequent patterns, the benefit is marginal or negative due to graph management costs.
Bidirectional tests highlight the risk of PCIe-based host staging introducing contention, particularly under concurrent traffic, indicating NVLink/similar device interconnects are preferable for maximizing benefit.
Bold Findings and Contradictory Claims
- The authors claim first seamless integration of CUDA Graphs in the UCX transport layer, enabling transparent graph-based multi-path intra-node transfers for the broader MPI/UCX ecosystem.
- Host-path inclusion is found to yield only marginal throughput gains (up to 15%), contesting any claims that PCIe host staging alone suffices for maximizing bandwidth in modern GPU nodes.
- For collective communication, the graph-based fine-grained multi-path approach offers only limited gains in regular, low-operation-count scenarios; its benefit scales with pattern complexity and repetition.
Theoretical and Practical Implications
The key implication is the demonstration that deep runtime integration of asynchronous graph-based execution (at the transport layer) can bridge the gap between hardware interconnect heterogeneity and user/application-level bandwidth expectations without resorting to heavyweight computation-overlap or ad hoc manual pipelining. This explicit management of multiple intra-node communication paths becomes increasingly critical as GPU density, hierarchical interconnects, and large-message workloads proliferate in modern exascale and LLM/AI training deployments.
The framework provides a template for extending graph-based runtime scheduling and caching to arbitrary intra-node communication patterns—beyond regular bulk transfer—potentially enabling similar gains for collective patterns (Allreduce, Alltoall, etc.) and for more irregular DAG-based HPC/AI workloads. Proper hardware/software co-design for CUDA Graph instantiation, device/host path balancing, and application-awareness remains a wide-open avenue for further performance scaling.
Limitations and Future Directions
While the results are compelling for large, repetitive P2P transfers, the benefits of the CUDA Graph abstraction are heavily dependent on the number of involved operations and repetition. For small messages, path overlap is insufficient to amortize graph overhead. Further improvements may necessitate:
- Dynamic scheduling: Auto-tuning path/graph selection at runtime based on traffic/load and message characteristics.
- Optimized graph caching: Smarter eviction/caching to reduce creation/instantiation cost in irregular or adaptive workloads.
- Collective integration: Extending the methodology to MPI collectives and supporting computation+communication overlap at the graph level.
The authors also note the necessity of validating scalability on larger GPU configurations (8+ GPUs, NVSwitch/DGX-class systems) and multi-node extensions.
Conclusion
This work demonstrates a significant advancement in intra-node GPU communication by combining multi-path transfer orchestration and CUDA Graph-based runtime acceleration within UCX. The proposed approach achieves order-of-magnitude improvements in GPU-to-GPU bandwidth for large messages and notable application-level speedups, while highlighting the challenges and diminishing returns for small/irregular transfers. Its methodology paves the way for further transport-level innovations in graph-based runtime optimization, with direct relevance to emerging AI/ML, exascale HPC, and dense multi-GPU system architectures.
Citation:
"Accelerating Intra-Node GPU-to-GPU Communication Through Multi-Path Transfers with CUDA Graphs" (2604.22228)
