RAMC: Remote Access Memory Channels over HPE Slingshot

Abstract: In this paper, we present Remote Access Memory Channels (RAMC), an explicit one-sided communication library designed to leverage the capabilities of HPE Cray Slingshot network hardware. Existing one-sided communication frameworks, such as MPI RMA and OpenSHMEM, rely on monolithic shared memory models that introduce scalability and usability challenges. These frameworks often assume symmetric memory regions or require blocking collective operations for window creation, which can mismatch user communication needs and hinder performance. Implicit models, such as PGAS and UPC, aim to simplify programming by treating local and remote memory as a unified region but ultimately rely on explicit mechanisms to implement data movement. MPI's recently-introduced partitioned communication API offers a persistent point-to-point interface but sacrifices the dynamic flexibility of RDMA. RAMC is designed to address these limitations. Based on the core concept of a persistent uni-directional communication channel, RAMC leverages Slingshot's unique memory region counters to enable efficient completion notification. Experiments with a RAMC-based heat diffusion code demonstrate RAMC has no difficulty scaling to 19.6 thousand processes across 250 nodes, and microbenchmark studies across multiple libfabric versions show RAMC can outperform Cray's proprietary MPI implementation (e.g., increases in bandwidth ranging from approx. 100%-130% for 1B-4KiB messages under libfabric 1.15.2, and from approx. 30%-45% under libfabric 2.3.1) while identifying additional areas for improvement, such as small message latencies.

• The paper introduces RAMC, a one-sided communication library that employs persistent uni-directional channels and MR counters for efficient, hardware-assisted notification.
• Empirical evaluations show that RAMC outperforms Cray MPI in bandwidth with gains up to 130% for small-to-medium messages, though latency remains a noted challenge.
• The flexible API and bulletin board mechanism in RAMC enable non-blocking channel establishment and fine-grained synchronization, paving the way for future hardware offload and GPU-centric communication.

Overview of RAMC: Remote Access Memory Channels over HPE Slingshot

The paper "RAMC: Remote Access Memory Channels over HPE Slingshot" (2606.05094) introduces RAMC, an explicit one-sided communication library engineered to exploit the specialized features of the HPE Cray Slingshot network hardware. RAMC addresses inherent limitations in prevailing one-sided communication frameworks such as MPI RMA and OpenSHMEM, notably their monolithic shared memory models and lack of flexibility. The core architectural advance is the persistent uni-directional communication channel, which is augmented with Slingshot’s hardware memory region (MR) counters for efficient completion notification. Empirical benchmarks, scaling tests, and microbenchmarks substantiate RAMC’s superior bandwidth performance and scalability compared to Cray MPI, although small message latencies represent an incumbent challenge that the paper recognizes for further optimization.

Motivation and Context

Traditional point-to-point communication models (e.g., classic MPI) are increasingly inefficient in the face of massive intra-node parallelism and heterogeneous compute (CPU/GPU) architectures. These paradigms induce substantial branching and synchronization complexity, leading to performance degradation in highly multi-threaded deployments. Recent advances in network hardware, exemplified by Slingshot, enable efficient one-sided RDMA operations—bypassing OS involvement, minimizing data movement overhead, and facilitating flexible buffer reuse. However, existing one-sided APIs (MPI RMA, OpenSHMEM, PGAS) enforce symmetric memory mappings and collective window creation, which hinder tailored communication patterns and unnecessarily constrain synchronization. RAMC is motivated by the need for a point-to-point, persistent, and hardware-accelerated interface, directly exposing the capabilities of Slingshot MR counters for lightweight notification and fine-grained completion signaling.

RAMC Design: Channels, Counters, and Synchronization

RAMC’s fundamental construct is the persistent channel, uni-directional and explicitly established between initiator and target without collective window creation. Target-side channels are characterized by data and status buffers, each associated with a hardware MR counter for completion tracking.

Figure 1: Target-side channel data structure, illustrating association of data and status buffers with MR counters for efficient completion notification.

Initiator-side channels hold addressing information (Slingshot offsets rather than VA), local status tracking, and addressing data for remote target buffers.

Figure 2: Initiator-side channel structure, containing addressing metadata and local status for protocol synchronization.

Synchronization in RAMC extends beyond classic fence-based or epoch-based models; it adopts user-configurable status values enabling pair-wise, relaxed synchronization. The protocol supports flexible state semantics (e.g., alternating OK_TO_READ/OK_TO_WRITE), and the status/operation counters facilitate optimized notification and buffer reuse without ancillary signaling messages.

A simple write sequence illustrates initiator/target interplay: the initiator increments its local status, reads the remote status, proceeds with the write if synchronization criteria are met, and leverages hardware counters for completion. Protocol errors (e.g., the target advancing in phase ahead of initiator) can be detected deterministically.

Figure 3: Workflow illustration of a RAMC write, detailing initiator-target synchronization and transmission phases.

The Bulletin Board Mechanism

RAMC employs a bulletin board (BB) system for non-blocking channel establishment and buffer addressing exchange. The BB allows targets to post addressing metadata, which initiators poll and retrieve when the status and tag match, decoupling window creation from program order and supporting dynamic channel instantiation.

Figure 4: The Bulletin Board structure, utilized for decoupled, tag-matched initiation of communication channels.

This mechanism avoids the deadlock risks and ordering constraints of collective operations, facilitating scalability and persistent channel semantics.

API Structure and Protocol Flexibility

RAMC defines orthogonal APIs for targets and initiators, abstracting buffer registration, status management, notification, and explicit channel teardown. The core protocol supports both blocking and non-blocking variants of put/get operations, capitalizing on hardware completion and CXI provider inject semantics. The status-driven synchronization paradigm enables the construction of complex communication (e.g., halo exchanges, collectives) from fundamental point-to-point channels without group-level synchronization.

Empirical Evaluation: Performance and Scalability

RAMC was evaluated on the HPE Cray EX4000 system (Eldorado) under both Cray libfabric 1.15.2 and open-source libfabric 2.3.1. The scalability demonstration leverages a 5-point stencil heat diffusion code, achieving flawless execution across 19.6k processes (250 nodes), confirming RAMC’s resilience and scalability.

Figure 5: Heat diffusion simulation at iteration 1000 using RAMC, showing robust scalability to 19600 processes across 250 nodes for stencil communication.

Microbenchmarks reveal that RAMC outperforms Cray MPI in bandwidth for small-medium messages; bandwidth gains range from approximately 100–130% (1–4KiB) under libfabric 1.15.2 and 30–45% under 2.3.1. For larger messages, bandwidth parity is reached, suggesting RAMC’s advantage is most pronounced in latency-sensitive, fine-grained communication typical of HPC applications.

Latency benchmarks indicate RAMC lags Cray MPI for small messages (<192B), primarily due to polling overhead and CXI provider configuration. The explicit notification variant—using additional atomic increments—shows marked latency penalty beyond inject thresholds, reinforcing MR counters as an efficient notification strategy. Attempts to optimize RAMC through provider configuration and leveraging optimized MRs have not remedied the latency gap, indicating fundamental constraints in underlying hardware implementation.

Related Work and Comparative Analysis

RAMC is positioned against MPI RMA (collective, monolithic, lacking standard notification), OpenSHMEM (symmetric heap, signaled RMA but still monolithic), and MPI persistent/partitioned communication (not genuinely persistent, not compatible with RMA notification in current spec). RAMC’s notification scheme aligns with ongoing proposals for MPI RMA-with-notification, and its design anticipates further hardware offloading (triggered operations and GPU-initiated comm). RAMC’s approach suggests a direction for more granular coordination in explicit one-sided frameworks, especially as hardware capabilities increase.

Challenges and Future Directions

RAMC’s portability is limited by reliance on Slingshot MR counters and architecture-specific features. However, the persistent channel abstraction can be emulated using explicit notification with other RDMA-capable hardware. The passive target design (initiator polling) is suboptimal for reducing network traffic; transitioning to active target signaling would reduce synchronization overhead. Another limitation lies in initiator-side completion granularity: endpoint counters do not distinguish operation targets, compelling further design work to bring RAMC’s fine-grained completion alignment to initiator operations. RAMC also foreshadows triggered communication leveraging NIC offload, which could be pivotal for CPU-free collectives or GPU-centric distributed applications.

Conclusion

RAMC redefines explicit one-sided communication semantics to harness advanced RDMA features in HPE Cray Slingshot hardware. By shifting away from monolithic shared-memory abstractions, RAMC offers high-throughput, persistent channels, lightweight completion notification, and flexible synchronization. Evaluation demonstrates RAMC’s scalability and bandwidth superiority over Cray MPI for relevant messaging regimes. The persistent channel concept, counter-based notification, and relaxed synchronization are promising for future HPC communication substrates, subject to further developments in hardware offload, protocol granularity, and portability. RAMC’s architectural insights and empirical results inform both practical deployment and the evolution of explicit one-sided communication APIs in modern HPC networks.