Cray Slingshot Interconnect for HPC & Datacenters

• Cray Slingshot is an advanced interconnect leveraging a high-radix dragonfly topology and optimized Ethernet protocol for converged HPC, AI/ML, and datacenter workloads.
• It integrates hardware adaptive routing, per-endpoint congestion control, and configurable QoS to maintain low latency and high throughput under diverse conditions.
• Empirical evaluations demonstrate that Slingshot reduces congestion impacts and improves performance compared to earlier systems like Cray Aries across varied network scenarios.

Searching arXiv for the provided Slingshot papers to ground the article in current sources. Cray Slingshot is HPE Cray’s high-radix, low-diameter interconnect for converged HPC, AI/ML, and hyperscale datacenter workloads. It replaces Cray Aries with a fabric that preserves the low diameter and path diversity of dragonfly-like topologies while shifting to an optimized Ethernet protocol that interoperates with standard Ethernet devices and adds hardware-implemented adaptive routing, end-to-end congestion control, and highly tunable QoS traffic classes (Sensi et al., 2020). Subsequent work on one-sided communication has highlighted additional Slingshot-specific capabilities at the NIC and libfabric CXI layers, particularly RDMA with independent progress and target-side memory region counters, which enable communication abstractions not readily captured by monolithic shared-memory APIs (Schonbein et al., 3 Jun 2026). Taken together, these studies position Slingshot as an interconnect architecture aimed at exascale supercomputers and hyperscale datacenters with latency-sensitive and bandwidth-intensive workloads.

1. Design objectives and system position

Slingshot was developed by Cray, now an HPE company, and was identified as the interconnect for the announced US exascale systems Aurora, Frontier, and El Capitan (Sensi et al., 2020). Its stated target is not only exascale supercomputers but also hyperscale datacenters, with an emphasis on converged HPC, AI/ML, and datacenter traffic patterns. In architectural terms, the design intent is to retain the performance characteristics associated with dragonfly-style networks while extending interoperability through an optimized Ethernet protocol.

This positioning distinguishes Slingshot from earlier Cray interconnects and from common alternatives. Relative to Aries, the key shift is not simply higher bandwidth; it is the combination of high-radix switching, Ethernet-native interoperability, hardware adaptive routing, hardware congestion control, and traffic-class-driven QoS. Relative to InfiniBand/RoCE and Omni-Path, the data emphasize that Slingshot is Ethernet-native, mixes enhanced internal traffic with standard Ethernet on all ports, and does not require a gateway for interoperability. A common misconception is that “Ethernet-native” implies an unmodified commodity Ethernet data path. The evidence instead indicates an optimized Ethernet protocol with reduced framing overhead, link-level reliability features, and transport support via RoCEv2 and libfabric, intended to preserve HPC performance while remaining interoperable (Sensi et al., 2020).

The same architecture also exposes NIC features that later software can exploit directly. RAMC, for example, is explicitly designed around Slingshot’s RDMA engines, CXI provider semantics, and target-side counters, suggesting that Slingshot is not only a fabric but also a hardware-software substrate for communication models beyond conventional two-sided MPI (Schonbein et al., 3 Jun 2026). This suggests a broader interpretation of Slingshot’s role: not merely a replacement network, but an attempt to unify datacenter compatibility, HPC transport efficiency, and NIC-resident control mechanisms.

2. Switch microarchitecture and dragonfly deployment

The switch ASIC, Rosetta, provides 64 ports, each at 200 Gb/s per direction (Sensi et al., 2020). Each port uses $4 \times 56$ Gb/s SerDes lanes with PAM-4; because of FEC, the effective throughput is 50 Gb/s per lane, yielding $4 \times 50$ Gb/s $= 200$ Gb/s per port. The ASIC is implemented in TSMC 16 nm and is reported at up to 250 W.

Internally, Rosetta is organized as 32 tile blocks arranged in 4 rows and 8 columns, with each tile handling two ports. Connectivity is provided by per-row buses and per-tile 16-to-8 crossbars, so packets traverse at most two internal hops: a row bus followed by a column crossbar. The switch uses distinct physical crossbars for distinct message types: Request-to-Transmit and Grant-to-Transmit; Data; Request Queue Credits; and End-to-End Acks. The Request-to-Transmit/Grant-to-Transmit path is described as a virtual output-queued architecture that avoids head-of-line blocking, while the Data path is 48-byte wide and parses headers early for speed. This partitioning of control and data movement is central to Slingshot’s claims of isolation, scheduling precision, and rapid congestion response.

The default deployment is a Dragonfly topology. Each Rosetta connects 16 endpoints, typically via up to 2.6 m copper, while the remaining 48 ports are used for inter-switch connectivity. Within a group, switches are fully connected via copper; across groups, full connectivity is provided via optics up to 100 m. Because of full intra-group and inter-group connectivity, the network diameter is at most 3 inter-switch hops for any pair of endpoints (Sensi et al., 2020).

At maximum scale, the largest 1D dragonfly configuration with 64-port switches uses 32 switches per group, giving 512 endpoints per group. Full intra-group mesh consumes 31 ports per switch, leaving 17 ports per switch for inter-group links; this yields $32 \times 17 = 544$ global links per group. The full-global-bandwidth configuration contains 545 groups and $545 \times 512 = 279{,}040$ endpoints, though addressing limits groups to 511 and therefore practical scale to 261,632 endpoints. These figures define the intended scalability regime. A plausible implication is that Slingshot’s high-radix design was chosen not merely to reduce hop count, but to preserve path diversity and global bandwidth in a regime where lower-radix fabrics would require deeper topologies.

Resilience is integrated at both link and node attachment levels. The fabric supports low-latency FEC, Link-Level Retry, lane degrade, and NIC end-to-end retry. Nodes may attach multiple NICs to improve both injection bandwidth and resilience; in the evaluated Shandy system, each node used dual NICs, each connecting to a different switch in the same network (Sensi et al., 2020).

3. Protocol stack, Ethernet optimization, and transport semantics

Slingshot uses an optimized Ethernet protocol that reduces the minimum Ethernet frame to 32 bytes, allows IP packets without an Ethernet header, and removes the inter-packet gap (Sensi et al., 2020). Mixed traffic is supported on all ports: the enhanced Slingshot protocol is used for internal traffic, while standard Ethernet traffic is supported for interoperability, negotiated on a per-port basis. Reliability mechanisms include low-latency FEC, Link-Level Retry, lane degrade, and NIC end-to-end retry, with the stated goal of localizing error handling and minimizing end-to-end retransmissions in 100+ Gb/s, very large systems.

For HPC transports, libfabric verbs and RxM providers support Slingshot, and RoCEv2 is used for HPC payloads with up to 4 KiB payloads per packet plus 62 bytes of headers and trailers: Ethernet (26 bytes including preamble), IPv4 (20), UDP (8), InfiniBand (14), and RoCEv2 CRC (4) (Sensi et al., 2020). The software stack characterization reported that MPI adds only marginal overhead over libfabric for small messages. This is significant because it indicates that the transport stack was engineered to keep software overhead small even while exposing an Ethernet-compatible substrate.

The later RAMC work gives a more detailed view of Slingshot’s one-sided communication semantics through the libfabric CXI provider (Schonbein et al., 3 Jun 2026). RAMC uses CXI endpoints to issue FI_WRITE and FI_READ, registers memory with fi_mr_reg/bind/enable, and relies on Slingshot’s offset-based addressing, in which initiators carry CXI memory keys rather than virtual addresses. Endpoint counters (fi_cntr) count local operation completions for FI_WRITE and FI_READ, while memory region counters (fi_mr) on the target count remote operations or bytes. RAMC uses operation counters associated with target memory regions as a lightweight completion notification mechanism, avoiding explicit follow-up messages.

The completion model is especially important. With the default FI_TRANSMIT_COMPLETE semantics for FI_WRITE, a local completion means that the target NIC has accepted the transfer and the source buffer is reusable. For small messages up to the Slingshot inject threshold of 192 bytes, RAMC uses fi_inject_write; larger messages use fi_write. Optional features such as optimized memory regions with keys 0–99 and an alias endpoint with FI_TC_LOW_LATENCY were explored but not adopted for RAMC’s baseline. In the reported experiments, disabling FI_RMA_EVENT and relying on explicit notification with optimized memory regions worsened latency, so standard memory regions with counters were preferred (Schonbein et al., 3 Jun 2026).

A recurring point of confusion in one-sided programming models is whether “RDMA completion” implies target-side software involvement. In Slingshot’s CXI-based design, the crucial distinction is that once data is received by the NIC, it is delivered into application memory without host involvement; the target CPU and OS need not participate in data movement itself (Schonbein et al., 3 Jun 2026). This underlies both passive-target operation and the claim of independent progress.

4. Routing, congestion control, and traffic isolation

Slingshot’s routing model exploits the path diversity of the dragonfly topology, including multiple minimal and non-minimal paths within and across groups (Sensi et al., 2020). Before sending each packet, the source switch estimates the load on up to four minimal and non-minimal paths and selects the best path based on both congestion and path length. The congestion metric is the total depth of the request queues for each output port. Congestion information is disseminated on-chip by a ring to the forwarding blocks of each input port, and across switches by embedding information in acknowledgement packets, with an average reverse-direction overhead of 4 bytes for every forward packet. To control the cost of extra hops, the adaptive algorithm biases toward minimal routes more frequently, especially as non-minimal routing increases average hop count.

The routing policy is paired with a hardware congestion control mechanism designed to separate endpoint congestion from intermediate congestion. The distinction is operationally important: adaptive routing can detour around intermediate hotspots, but it cannot bypass endpoint congestion when all paths share the last-hop bottleneck. Slingshot therefore tracks all in-flight packets per endpoint pair in hardware, distinguishes “victim” jobs from those contributing to endpoint congestion, and applies fast, stiff backpressure selectively to offending flows rather than to whole applications (Sensi et al., 2020). The stated purpose is to free buffers, avoid head-of-line blocking across the fabric, and reduce tail latency during global synchronization and bursty phases.

The paper formalizes congestion impact as

$C = \frac{T_c}{T_i},$

where $T_c$ is the victim job’s mean execution time under congestion and $T_i$ is its mean execution time in isolation (Sensi et al., 2020). Under endpoint congestion generated by incast, the reported worst-case slowdown on Slingshot is 1.3×, compared with up to 93× on Aries. Across allocations, node counts, and processes per node, congestion impacts on Slingshot were mostly below 2 even with 24 PPN aggressors, whereas Aries reached impacts up to 150; on the smaller Malbec system, the maximum congestion impact dropped from 2.3 to 1.5 for Slingshot, compared with 154 to 40 for Aries. On the 1,024-node Shandy system, an extreme case with 75% of nodes running an incast congestor produced a maximum 3.55× slowdown for LAMMPS under random allocation. For bursty incast, medium-size messages could produce transient congestion before throttling engaged, with a maximum $C \approx 1.21$, but this remained minor relative to previous-generation behavior (Sensi et al., 2020).

Traffic isolation is implemented through hardware traffic classes. Classification uses DSCP tags, and each class has dedicated virtual queues and buffer allocations together with configurable priority, packet ordering, minimum and maximum bandwidth guarantees, lossiness, and routing bias (Sensi et al., 2020). Unassigned buffers are dynamically allocated to best-effort traffic, provided that the administrator ensures the sum of class minimum bandwidths does not exceed available bandwidth. The software stack can map these classes at the granularity of packets, messages, or operations through MPI or libfabric; the paper gives the example of assigning MPI_Barrier and MPI_Allreduce to high-priority, low-bandwidth classes while mapping bulk point-to-point traffic to lower-priority, high-bandwidth classes.

The QoS experiments illustrate the mechanism concretely. On Malbec with 25% tapering, an 8-byte MPI_Allreduce running in the same class as a congesting MPI_Alltoall experienced $C = 2.85$; when the operations were separated into different classes, MPI_Allreduce dropped to $4 \times 50$0 (Sensi et al., 2020). In a bandwidth-reservation experiment with two bisection jobs, configuring TC1 with a minimum of 80% and TC2 with 10% caused TC1 to remain at 80% while TC2 received 20%, including an extra 10% of unreserved bandwidth dynamically allocated to the lower-share class, and then ramped to full bandwidth when TC1 finished. The practical significance is that Slingshot’s QoS is not limited to strict-priority service; it combines isolation, bandwidth reservation, and route selection.

5. Empirical performance and comparative behavior

The microbenchmark characterization reported a per-switch latency of roughly 350 ns mean and median for RoCE traffic when measured with standard RoCE NICs not using Slingshot’s optimized packet format; most values fell in the 300–400 ns range (Sensi et al., 2020). When hop distance was varied across same-switch, same-group, and different-group placements, the worst-case latency impact due to distance was about 40% for 8-byte messages, while for messages of at least 16 KiB the latency difference across distances was below 10%. Bandwidth differences across 1-, 2-, and 3-hop cases were below 15% for all message sizes and were occasionally slightly higher for endpoints in different groups because more disjoint paths were available.

At the system level, the Shandy configuration used in the study had 128 bisection links across the cut, giving

$4 \times 50$1

For all-to-all communication, with 448 global links and a cross-group traffic fraction of $4 \times 50$2,

$4 \times 50$3

Measured MPI_Alltoall achieved more than 90% of the theoretical peak without packet loss; the reported performance drop at 256 B was attributed to an MPI algorithm change to reduce memory usage (Sensi et al., 2020). These results support the claim that the low-diameter dragonfly configuration and adaptive routing preserve high utilization at scale.

Application evaluation covered HPC applications such as MILC, HPCG, LAMMPS, and FFT, an AI proxy based on ResNet, and datacenter tail-latency applications including Silo, Sphinx, Xapian, and Img-dnn (Sensi et al., 2020). Under endpoint congestion, Slingshot maintained stable performance across both HPC and datacenter workloads, while Aries exhibited severe slowdowns and tail-latency inflation. Silo, Xapian, and Img-dnn were reported to degrade severely on Aries under incast, whereas Slingshot showed no relevant effect; Sphinx degraded less on Aries because its communication intensity was lower. For LAMMPS, a 50/50 split incast aggressor made Aries 17× slower, whereas Slingshot showed minimal impact. Under all-to-all aggressors, intermediate congestion did not significantly affect Slingshot because adaptive routing detoured around hotspots.

The comparison with Aries therefore involves more than raw link rate. Both are dragonfly-based and both employ adaptive routing, but Slingshot adds on-chip congestion-credit distribution, per-packet path-load assessment with minimal bias, Ethernet interoperability, and per-endpoint-pair congestion control (Sensi et al., 2020). Relative to Clos/RoCE environments using DCQCN or PFC-based loops, the paper argues that Slingshot’s hardware control avoids the tuning fragility, long control loops, and head-of-line blocking associated with ECN/QCN-style approaches on bursty HPC traffic. This suggests that the main empirical distinction is robustness under congestion rather than only peak throughput.

6. One-sided communication on Slingshot: RAMC and CXI-specific exploitation

The RAMC library was introduced as an explicit one-sided communication library designed to leverage HPE Cray Slingshot hardware (Schonbein et al., 3 Jun 2026). Its premise is that existing frameworks such as MPI RMA and OpenSHMEM rely on monolithic shared-memory models that introduce scalability and usability challenges, including collective window creation, communicator-wide exposure, symmetric heaps, and synchronization structures that can hinder overlap. RAMC instead centers on a persistent uni-directional communication channel between one initiator and one target window.

RAMC’s setup and lifecycle map directly onto Slingshot CXI primitives. During initialization, each process calls ramc_init() to select the CXI provider, create endpoints and counters, allocate NIC resources, and create a process-local bulletin board; PMI is then used so that every process can read every other process’s bulletin board (Schonbein et al., 3 Jun 2026). A target creates a window with ramc_tgt_create_window, which registers a user data buffer and a small status buffer, attaches a memory region counter to count FI_REMOTE_WRITE and FI_REMOTE_READ, initializes the target status to a user-specified value, and caches a tag. The target publishes the addressing information—status and data memory keys, size, and tag—into its bulletin board, and initiators poll the board, match on the tag, and retrieve the keys when the posting becomes active. The target can also use a bulletin-board memory region counter to determine when the expected initiators have read the posting.

Communication then proceeds through RDMA reads and writes to the target’s exposed data buffer and status value. Initiators wait on local endpoint counters for FI_WRITE or FI_READ completion; targets poll or test the data-buffer memory region counter to detect remote completion (Schonbein et al., 3 Jun 2026). Synchronization is status-based. The paper gives a simple convention in which even status means OK_TO_READ and odd status means OK_TO_WRITE, with status increments marking progress. The initiator proceeds only when target status equals initiator status; if the target is behind, the initiator waits, and if the target is ahead, this is treated as an error. The target allows progress by updating its status, then waits for the expected number of memory region counter increments before moving to the next phase.

This design leverages Slingshot’s hardware characteristics in a way that differs sharply from collective window models. The target CPU does not participate in data movement, because the NIC provides independent progress, and the target obtains completion information without explicit notification messages because memory region counters provide “RMA-with-notification” semantics (Schonbein et al., 3 Jun 2026). The paper discusses Portals-style triggered operations as a future avenue, where counters could trigger NIC-executed deferred actions without host involvement. A plausible implication is that Slingshot’s counter model can support increasingly offloaded one-sided runtimes if software is organized around persistent channels rather than monolithic windows.

The performance evaluation on Sandia’s Eldorado system, an HPE Cray EX4000 with 384 AMD MI-300A nodes and 4 Slingshot 200 Gb/s NICs per node, demonstrates both scalability and trade-offs (Schonbein et al., 3 Jun 2026). A RAMC-based heat diffusion application scaled without issue to 19,600 processes across 250 nodes, up to 79 processes per node, using a 5-point stencil and an “earlybird” pattern in which processes permit neighbors to write as they become ready and then await four writes via the memory region counter before updating temperatures. In unidirectional bandwidth microbenchmarks, RAMC on libfabric 1.15.2 consistently outperformed Cray MPI by 100%–130% for 1 B–4 KiB messages, with gains receding toward parity by 32 KiB. On libfabric 2.3.1, the RAMC advantage dropped to 30%–45% for 1–8 KiB, and MPI led slightly by 2%–4% at 64 KiB and 128 KiB. For latency, Cray MPI remained lower for small messages, by roughly 430–540 ns on libfabric 1.15.2 and about 480–500 ns on libfabric 2.3.1, but RAMC achieved up to 32% lower latency for 16–512 KiB, around and above the eager-rendezvous threshold of about 16 KiB (Schonbein et al., 3 Jun 2026).

The small-message latency gap remained unresolved despite several tuning attempts, including limiting requested provider capabilities, matching Cray MPI’s CXI defaults, disabling receiver-side message ordering constraints visible via fi_getinfo, and using an aliased endpoint with FI_TC_LOW_LATENCY; none improved RAMC latency, and some worsened it (Schonbein et al., 3 Jun 2026). Likewise, replacing memory region counters with explicit notification and optimized memory regions increased latency. The resulting picture is therefore mixed but informative: Slingshot exposes primitives that make fine-grained one-sided communication efficient and scalable, yet highly optimized proprietary MPI stacks still retain an advantage for the shortest messages.

7. Scalability, operational behavior, and observed limitations

The aggregate evidence presents Slingshot as a low-diameter dragonfly fabric built for large path diversity, high per-port bandwidth, and bounded switch-to-switch distance, supporting hundreds of thousands of endpoints at full or near-full global bandwidth (Sensi et al., 2020). Operationally, hardware congestion control reduces sensitivity to placement strategy compared with Aries. Random, interleaved, and linear allocations affect Aries strongly under congestion, while Slingshot remains mostly below a congestion impact of 2 in the reported experiments, even under aggressive mixed-workload conditions. This is one reason the paper characterizes Slingshot as suitable for converged HPC and datacenter traffic rather than only for carefully curated batch placements.

Resilience features are integral rather than auxiliary. Low-latency FEC, Link-Level Retry, lane degrade, NIC end-to-end retry, and multi-NIC attachment directly support fault tolerance and sustained injection bandwidth (Sensi et al., 2020). Shandy’s dual-NIC-per-node design is described as both load-balancing and protective against NIC failures. In software terms, libfabric support and RoCEv2 enable integration with existing ecosystems, while CXI-specific capabilities support more specialized runtimes such as RAMC (Schonbein et al., 3 Jun 2026).

Several trade-offs and limitations are explicitly documented. First, the main Slingshot evaluation used standard RoCE NICs rather than Slingshot’s full optimized packet format and on-NIC feature set, so the measurements were described as a lower bound on Slingshot-native performance (Sensi et al., 2020). Second, medium-size incast bursts can still cause transient congestion before control engages, though the measured effect remained small. Third, under extreme aggressor fractions, measurable slowdowns remain possible; on the 1,024-node Shandy system, LAMMPS slowed by as much as 3.55× in the reported worst case. Fourth, in the RAMC work, passive-target polling introduces traffic, endpoint completion counters are coarse because they aggregate all writes or reads issued on the endpoint, and small-message latency trails highly tuned Cray MPI (Schonbein et al., 3 Jun 2026).

These limitations do not negate the overall architectural pattern. Rather, they delimit it. Slingshot’s strength lies in combining an Ethernet-compatible protocol substrate, high-radix dragonfly scaling, per-packet adaptive routing, per-endpoint-pair hardware congestion control, and switch-resident traffic-class enforcement (Sensi et al., 2020). The RAMC results further indicate that these properties extend into one-sided communication, where memory region counters and independent progress enable persistent, point-to-point RDMA channels that are difficult to realize efficiently under collective window abstractions (Schonbein et al., 3 Jun 2026). This suggests that Slingshot is best understood as a fabric whose defining feature is not any single mechanism, but the coordinated interaction of topology, switch microarchitecture, transport semantics, congestion response, and NIC-visible control primitives.