Pollara Networking: High-Performance Interconnect

• Pollara Networking is a high-performance cluster interconnect fabric using AMD Pensando 400 Gbps NICs to connect MI300X GPUs for distributed training.
• It employs a distinctive two-level rails-only switch topology that ensures low latency and multi-Tbps aggregate bandwidth for efficient collective operations.
• Optimized fusion buffers, hierarchical collective algorithms, and topology-aware layouts yield 80-90% of theoretical bandwidth in large-scale pretraining workloads.

Pollara Networking refers to the high-performance cluster interconnect fabric built around AMD Pensando Pollara 400 Gbps NICs, used to connect MI300X GPUs in large-scale distributed training systems. Designed as the centerpiece of AMD’s networking stack for foundational model pretraining, Pollara delivers multi-Tbps total node bandwidth via a distinctive two-level rails-only switch topology. As reported in the context of large-scale mixture-of-experts (MoE) model training, Pollara forms the critical infrastructure for achieving competitive scaling, low-latency collectives, and sustained bandwidth at scale on AMD hardware (Anthony et al., 21 Nov 2025).

1. Hardware Architecture and Topology

Each compute node in the Pollara-connected AMD MI300X cluster integrates eight MI300X GPUs, each directly attached to a dedicated AMD Pensando Pollara NIC providing a 400 Gbps full-duplex link (≈ 50 GiB/s per NIC). These NICs feed into a two-level rails-only Ethernet fabric comprising:

• Four Arista 7060X6-64PE leaf switches, each serving a slice of 30 servers.
• Each leaf switch connects to four spine switches in a non-Clos, rails-only configuration; leaves interconnect exclusively via spines, lacking cross-leaf direct paths.

This arrangement enables a per-node Pollara aggregate bandwidth of 3.2 Tbps (8 × 50 GiB/s). Training-collective data—including gradients, optimizer state, and context-parallel communications—remains on the Pollara network, while a segregated 200 Gbps Pensando DSC fabric handles I/O functions to mitigate potential congestion. Within each node, intra-GPU collectives utilize InfinityFabric (xGMI) links at up to ≈ 450 GiB/s aggregate, reserving Pollara for inter-node traffic (Anthony et al., 21 Nov 2025).

2. Collective Communication Microbenchmarks

The performance of core collectives—including AllReduce, ReduceScatter, AllGather, and Broadcast—was empirically characterized across message sizes (bytes to hundreds of MiB) and world sizes (2, 4, 8, 16 nodes/GPUs). Measurements recorded included bus bandwidth and one-way latency per collective.

• AllReduce: Small-message (128 B) latencies measured ≈ 10 µs (2 nodes) up to ≈ 25 µs (16 nodes), with aggregate bandwidth saturating at ≈ 200 GiB/s (4 nodes), ≈ 350 GiB/s (8 nodes), and ≈ 600 GiB/s (16 nodes). Bandwidth saturation occurred at message sizes around 1–4 MiB per rank, beyond which the communication becomes bandwidth-bound.
• ReduceScatter: Small-message latencies ranged from ≈ 8 µs (2 nodes) to ≈ 20 µs (16 nodes), with peak bandwidths of ≈ 250 GiB/s (4 nodes), ≈ 400 GiB/s (8 nodes), and ≈ 650 GiB/s (16 nodes).
• AllGather and Broadcast: Both operations exhibited latency and bandwidth curves similar to AllReduce, with broadcast detailed to follow the same ~1 MiB crossover and comparable peak bandwidth.

A crucial conclusion is that performance enters the bandwidth-dominated regime for collective operations once call sizes exceed ≈ 1–4 MiB; below this threshold, the fixed startup overhead (α) dominates, as quantified by the standard communication time model.

3. Performance Modeling and Quantitative Metrics

The latency–bandwidth relationship on Pollara is described by established linear models:

$T(m) = \alpha + \beta m$

For tree-based AllReduce across $p$ ranks,

$$T_{\text{all-reduce}}(m, p) = 2 \log_2(p) \alpha + 2 \frac{p-1}{p} m \beta$$

Measured parameters are:

• Pollara startup latency α ≈ 10–15 µs (tiny messages).
• Amortized bandwidth cost β ≈ 1–2 ns/B at saturation.
• Per-hop (leaf–spine–leaf) injection plus switch latency ≈ 2–4 µs, reflected within α.

Maximum per-GPU NIC link bandwidth is 400 Gbps (≈ 50 GiB/s). The effective aggregate bus bandwidth per node reaches ~320 GiB/s (≈ 80–90% of the theoretical aggregate) on large-world collectives. Small-message injection rates are limited to ≈ 50k messages/s per NIC before CPU or driver bottlenecks appear, with efficient coalescing or fused launches enabling 100–200k msg/s (Anthony et al., 21 Nov 2025).

Scaling efficiency, defined as normalized bandwidth per node, remains ≈ 90% at 2–4 nodes, ≈ 80% at 8 nodes, and ≈ 70% at 16 nodes for AllReduce and ReduceScatter.

4. Bottlenecks, Performance Limits, and Mitigation Strategies

Three major limitations were identified in the Pollara rails-only topology and execution context:

Bottleneck	Impact	Recommended Mitigation
Cross-rail traffic	Extra hops, reduced path diversity; increased latency	Group data-parallel or tensor-parallel subgroups under single leaf switch; avoid cross-leaf collective spans where possible
Small-message overhead	RCCL launch and driver latency dominate, poor bandwidth utilization	Configure fusion buffers at 1–4 MiB; operate collective calls above this threshold for bandwidth-bound performance
Hotspot link saturation	Uneven load on individual spines; bottleneck under “stormy” patterns (e.g., expert shuffles)	Pipeline or chunk expert communication; balance routes with token-based scheduling

This suggests that effective partitioning and buffer sizing are crucial for maintaining optimal utilization and aggregate bandwidth.

5. Comparative Context: Pollara vs. Peer Interconnects

Pollara yields a unique position between high-end intra-node fabrics and conventional Ethernet:

• NVIDIA NVLink: Delivers ≈ 600 GB/s per GPU for intra-node, with sub-µs latency; Pollara’s 400 Gbps per GPU for inter-node traffic is an order of magnitude below in bandwidth, yet over an order of magnitude beyond standard 100 GbE Ethernet.
• InfiniBand (HDR/EDR): Pollara’s 400 Gbps bandwidth per NIC matches leading InfiniBand (200–400 Gb/s NICs), with measured Pollara latency/bandwidth curves being competitive for messages >1 MiB.
• AMD xGMI (InfinityFabric): Intra-node links provide up to ≈ 64 GiB/s per GPU, but require all local GPUs for maximal throughput. Pollara’s one-NIC-per-GPU mapping delivers traffic isolation, offset by marginally increased per-hop cost.

A plausible implication is that, for inter-node deep learning collectives, Pollara approaches the efficiency and scalability of favored HPC fabrics at large message sizes (Anthony et al., 21 Nov 2025).

6. Workload Placement and System Design Guidelines

To maximize throughput and minimize collective latency on Pollara-based clusters, the following system and workload design strategies are empirically validated:

• Fusion buffer sizing: Allocate 2–4 MiB buffers per Pollara call for all distributed collectives, ensuring operation within the bandwidth-bound regime while allowing for compute overlap.
• Hierarchical collective algorithms: Employ a two-level scheme—first, run InfinityFabric AllReduce/AllGather within nodes; second, execute Pollara ReduceScatter/AllGather across nodes on fused buffers.
• Topology-aware job layout: Assign each data-parallel group (or tensor-parallel subgroup) to GPUs under the same leaf to maintain full wire speed and path efficiency, avoiding cross-leaf communication unless strictly necessary.
• Collective protocol selection: For large runs, transition from ring collectives to tree or recursive doubling algorithms to limit hop count and avoid multiplicative α increases.

A key outcome is the ability to capture ≈ 80–90% of theoretical Pollara bandwidth in steady-state distributed training workloads when adhering to these best practices (Anthony et al., 21 Nov 2025).

7. Impact and Operational Significance

Through systematic benchmarking and characterization, Pollara networking demonstrates readiness for competitive large-scale deep learning pretraining on pure AMD hardware. Key takeaways include per-GPU Pollara delivery of ≈ 400 Gbps, low startup latency of 10–25 µs, and amortized bandwidth costs of 1–2 ns/B on bandwidth-saturated collectives. Performance scales well to at least 16 nodes, provided fusion buffers are properly sized and topology-aware rank placement is employed. This establishes Pollara as a viable, mature alternative to established inter-node communication fabrics in clustered deep learning workloads (Anthony et al., 21 Nov 2025).