Sustaining Exascale Performance: Lessons from HPL and HPL-MxP on Aurora

Abstract: Sustaining exascale performance in production requires engineering choices and operational practices that emerge only under real deployment constraints and demand coordination across system layers. This paper reports experience from three successive campaigns running HPL and HPL-MxP on Aurora, an Intel-based exascale system featuring the first large-scale deployment of Intel discrete GPUs, CPU-attached network interfaces, and the largest production Slingshot-11 interconnect. Aurora progressed from 0.585EF/s on 5,439 nodes to 1.01EF/s on 9,234 nodes in FP64 HPL, while HPL-MxP reached 11.64EF/s, an 11.5x speedup over FP64 enabled by mixed-precision arithmetic and Intel AMX acceleration. We identify and classify by role at production scale the system-level choices that sustained these results, including deterministic locality-aware resource mapping, explicit CPU-GPU pipelining, mixed-precision orchestration, and a hybrid P2P/collective resilience strategy introduced after synchronization stalls at scale. While some observations are Aurora-specific, the broader lessons are likely to apply to tightly coupled heterogeneous systems at extreme scale.

• The paper demonstrates that system-level optimizations, including NUMA-aware mapping and CPU-mediated GPU pipelining, are crucial for sustaining exascale throughput.
• It shows how mixed-precision techniques, leveraging BF16/FP32 with iterative refinement, yield an 11.5× speedup over traditional FP64 benchmarks.
• The study highlights the importance of hybrid communication strategies combining MPI collectives with direct P2P transfers to mitigate network variability.

Sustaining Exascale Performance on Aurora: System Insights from HPL and HPL-MxP

Introduction

The deployment of exascale systems introduces both architectural opportunities and operational complexity, particularly in the presence of heterogeneous platforms coupling CPUs, GPUs, and novel interconnects. The paper "Sustaining Exascale Performance: Lessons from HPL and HPL-MxP on Aurora" (2604.09517) presents an authoritative account of the engineering, system-level, and operational practices necessary to achieve and maintain sustained exascale throughput on Aurora, Intel’s flagship exascale system at Argonne. The study focuses on FP64 HPL (defining the TOP500) and mixed-precision HPL-MxP, emphasizing cross-layer decisions that enable both peak and robust long-duration performance.

Aurora Platform Architecture

Aurora consists of 10,624 compute nodes, each integrating two Intel Xeon Max CPUs with on-package HBM2e and six Intel Data Center GPU Max Series (Ponte Vecchio) accelerators, interconnected by PCIe Gen5 and Xe-Link fabrics. Eight CPU-attached HPE Slingshot-11 NICs per node connect to a global one-dimensional Dragonfly interconnect, optimized for low-diameter, high-bandwidth communication.

Figure 1: Aurora Exascale Compute Blade (ECB) integrates CPUs, GPUs, HBM, and network interfaces to form the core compute node.

Figure 2: Aurora’s 1-D Dragonfly interconnect topology, designed for scalable, low-diameter system-wide communication.

Distinct features include the attachment of network interfaces to CPUs (not GPUs), the presence of multiple NICs per node, large memory capacity, and the adoption of AMX accelerators for low-precision matrix operations. This architecture imposes unique resource-affinity constraints and demands careful process/grid mapping, NUMA-awareness, and explicit management of communication and data movement.

HPL and HPL-MxP Benchmarks: Algorithmic and Execution Models

HPL benchmarks system FP64 throughput by computing the LU factorization of dense double-precision matrices using a 2D block-cyclic layout, with major phases: panel factorization (PFACT), broadcast (BCAST), row swapping (SWAP), triangular solves (DTRSM), and trailing matrix updates (DGEMM). For large problems, DGEMM dominates; as the matrix reduces, communication and latency-bound phases become critical.

Figure 3: HPL cost model: initial compute-boundedness gives way to communication and synchronization costs as the problem proceeds.

In Aurora’s execution model, all compute-intensive phases (mainly DGEMM) are offloaded to GPUs, while panel factorizations, row swaps, and MPI collectives are CPU-mediated. GPU and network attachment disjointness necessitates explicit CPU-GPU pipelining and careful mapping of ranks to GPUs and NICs, with process, memory, and device affinities enforced to minimize UPI traffic.

HPL-MxP extends this structure by using BF16/FP32 for factorization and matrix updates on GPUs; double-precision correction is enforced through iterative refinement on CPUs, removing pivoting to reduce network pressure. This enables both increased problem sizes and higher arithmetic intensity per GPU kernel.

System-Level and Operational Engineering for Sustained Performance

Communication Stack and MPI Layer

MPI collectives are implemented atop MPICH and libfabric/cxi, with enhancements for high-radix collectives, multi-threaded progress, and host-buffered transfers. For HPL/HPL-MxP, disabling GPU-aware MPI (e.g., Level Zero IPC, GPUDirect) proved beneficial, given host-centric mediation. Key enhancements include enlarged completion queues, deterministic NIC affinity (striping and hashing disabled), and minimization of monitoring/polling overhead to lower communication jitter.

Hybrid collective/P2P resilience mechanisms were deployed to mitigate the significant risk of single-rank stalls in synchronization-heavy phases at extreme scale. Direct P2P exchanges were used for the most synchronization-sensitive panel column, while the rest utilized optimized collectives, providing both forward progress under transient network events and limited global delay propagation.

NUMA-Aware Mapping and Overlap

All resource mapping (rank, memory, GPU, NIC) is performed with strict NUMA affinity, binding CPU cores, GPU, memory allocation, and NIC endpoints to the same package to reduce inter-socket UPI traffic and memory latency. This is particularly important given the CPU-based orchestration of GPU kernel launches and network communication.

Pipeline concurrency is sustained by explicitly overlapping host-device transfers (via OpenCL queues) with computation, mapping data transfers and kernel execution to multiple DMA engines to keep both the PCIe fabric and GPU ALUs well-utilized.

Mixed Precision and AMX Acceleration

HPL-MxP leverages Intel AMX for accelerating BF16 GEMMs on CPUs during iterative refinement and residual correction. This was directly attributable for the 10% performance uplift in HPL-MxP achieved between the ISC24 and SC24 milestones.

NIC Assignment and Communication Phase Specialization

On nodes with multiple NICs, communication traffic for latency-bound (SWAP) and bandwidth-bound (BCAST) phases is partitioned across dedicated interfaces using explicit communicator hints and affinity settings, maximizing available bandwidth and minimizing contention within and across NUMA domains.

Performance Results and Scaling

Aurora achieved 1.01 EF/s FP64 HPL on 9,234 nodes with parallel scaling efficiency of 78.8% (normalized to single-node), and 11.64 EF/s HPL-MxP (BF16/FP32/FP64) on 9,500 nodes—an 11.5× speedup over FP64 baseline—enabled by both mixed-precision and AMX acceleration.

Figure 4: HPL performance on Aurora: strong scaling from 5,439 to 9,234 nodes, maintaining exascale throughput.

Analysis of phase breakdowns demonstrates that early factorizations are compute-bound with DGEMM domination, while late stages are determined by communication, particularly the SWAP phase. Process grid parameters are tuned to prolong compute-dominated regions, further supporting high arithmetic efficiency.

The enabling factors for mixed-precision acceleration include increased block sizes (due to reduced storage requirements), improved arithmetic intensity, and efficient algorithmic orchestration of CPU-GPU workloads during refinement.

Reliability, Network Variability, and Production Orchestration

Multi-hour runs at scale are affected by system reliability (node failures, firmware/driver faults, and correlated system events) and network variability (link flaps, localized congestion, and collective-induced stalls). Aurora employs robust monitoring, repair, and resource isolation frameworks, but application-layer resilience (validation, hybrid communication strategies) and operational controls (pre-screening, targeted scheduling) are required for production success.

These factors are not specific to Aurora but are broad challenges for any tightly coupled system operating at O(10^4) nodes and O(10^5) GPUs/CPUs.

Implications and Future Directions

This study demonstrates that sustained exascale performance is not determined solely by peak hardware but by cross-layer coordination between algorithmic, system, and operational dimensions. Important generalizable lessons include:

• The necessity of deterministic, NUMA/topology-aware mapping on CPU-attached network systems.
• The requirement of explicit pipelined concurrency and overlap, especially with CPU-mediated orchestration.
• The increasing system-wide sensitivity to network variability and the need for hybrid communication resilience (mixing P2P and collectives).
• The nontrivial engineering surrounding mixed-precision arithmetic, including workload balancing and hardware accelerator exploitation (e.g., AMX).

Given the trend towards greater architectural heterogeneity, larger scales, and the application of mixed-precision methods in scientific workloads, these lessons directly inform both system design and future HPC/AI software development. For AI practitioners, the orchestration of multi-precision methods, pipeline management, and collective resilience are increasingly relevant as large foundation models migrate onto similar exascale platforms.

Conclusion

"Sustaining Exascale Performance: Lessons from HPL and HPL-MxP on Aurora" delivers a comprehensive, technically rigorous account of the practical, system-level strategies required to achieve and sustain exascale-class throughput and scaling efficiency on heterogeneous platforms (2604.09517). The Aurora deployment exemplifies the necessity of cross-layer, co-engineered solutions—ranging from resource mapping and communication stack orchestration to pipeline overlap and operational validation—to reach and maintain performance at scale. The operational lessons, particularly concerning communication resilience, NUMA-aware resource coupling, and explicit mixed-precision management, are broadly applicable to the evolving landscape of high-end AI and HPC systems.