Node-Level Performance and Energy Characterization of Flagship Science Applications on SuperMUC-NG Phase 2

Abstract: We present a systematic performance and energy-efficiency characterization of five flagship scientific workloads on SuperMUC-NG phase 2, the 28 PetaFLOPs system at the Leibniz Supercomputing Center (LRZ) equipped with Intel Xeon Platinum 8480+ and Intel Data Center GPU Max 1550 (Ponte Vecchio, PVC) accelerators. The selected codes span molecular dynamics (gromacs, lammps), astrophysics and cosmology (OpenGadget3, AthenaK), and finite-element PDE solvers from the dealii-X Center of Excellence. For each code we measure throughput and energy efficiency expressed as compute-elements per wall-clock second (or per Joule of consumed energy) on a single compute node, comparing CPU-only (SPR) against combined CPU+GPU (SPR+PVC) configurations where available. Energy measurements rely on lightweight code instrumentation with p3em, or the Energy Aware Runtime (EAR) present on the system. Our results show that GPU offload yields $4-12\times$ higher throughput and up to $15\times$ better energy efficiency compared to CPU-only execution, with lammps and AthenaK benefiting most. However, both throughput and energy gains are sensitive to problem granularity: insufficient work per GPU tile erodes the accelerator advantage, as clearly observed in AthenaK at small mesh-block sizes. The power-budget utilization is systematically lower for CPUs than it is for GPUs, indicating that even at peak useful-work rate, most applications running on CPUs leave a significant fraction of the node's thermal envelope unused.

• The paper introduces unified metrics for throughput and energy efficiency to assess node-level performance across diverse scientific workloads.
• It demonstrates that GPU offloading can improve performance up to 12× and energy efficiency up to 15× compared to CPU-only runs.
• The analysis underscores the impact of workload granularity and measurement tools (EAR vs. p3em) in exposing optimization opportunities.

Node-Level Performance and Energy Characterization of Flagship Applications on SuperMUC-NG Phase 2

System Overview and Methodological Approach

This paper provides a systematic characterization of node-level performance and energy efficiency for five production scientific workloads on SuperMUC-NG Phase 2, a heterogeneous system combining Intel Sapphire Rapids Xeon Platinum CPUs and Intel Data Center GPU Max accelerators (Ponte Vecchio, PVC). The benchmark suite includes applications from molecular dynamics (gromacs, lammps), astrophysics and cosmology (OpenGadget3, AthenaK), and high-order finite-element PDE solvers (dealii-X Center of Excellence).

The analysis uses unified throughput ($T$; compute-elements / s) and energy efficiency ($E$; compute-elements / J) metrics, enabling domain-agnostic, application-level comparisons. Measurements are conducted in single-node isolation, excluding network-induced variability. Energy consumption is captured using either EAR (Energy Aware Runtime) or the p3em framework, leveraging hardware counters (RAPL/DRAM/xpu-smi). All benchmarks target both CPU-only (SPR) and combined CPU+GPU (SPR+PVC) configurations where possible.

Application-Specific Results

Molecular Dynamics: gromacs and lammps

gromacs, with a comprehensive SYCL offload of computational kernels, exhibits sustained GPU throughput of approximately $4\times 10^8$ atom-steps/s, yielding a $\sim\!4\times$ performance advantage over the SPR CPU orbit across system sizes. Energy efficiency on PVC plateaus at $2\times 10^5$ atom-steps/J for large systems, whereas the CPU's efficiency declines with increased problem size due to cache effects. For low atom counts, energy efficiency of CPUs approaches or surpasses that of the GPU due to baseline power amortization, revealing a nontrivial trade-off at small granularities.

Figure 1: Throughput and energy efficiency for gromacs on SPR CPU vs. SPR+PVC GPU, confirming a clear GPU advantage for large systems.

For lammps, using the Kokkos+SYCL backend, GPU throughput climbs with system size and saturates at $\sim\!5\times 10^8$ atom-steps/s and $4\times 10^5$ atom-steps/J. The CPU lags by an order of magnitude in both metrics for large sizes. Notably, at minimal system sizes, CPU and GPU energy efficiencies are closely matched; the GPU becomes superior as parallelism increases, reflecting lammps' favorable kernel design for accelerator architectures.

Astrophysics/Cosmology: OpenGadget3 and AthenaK

OpenGadget3 employs hybrid OpenMP offload with selected kernels (gravity tree, hydro, conduction) executed on the GPU. Both throughput and energy efficiency increase as resolution scales, yet the absolute energy efficiency tops out near 660 particle-steps/J for full simulations, though microbenchmarks of individual kernels (e.g., HSML) reach $6\times 10^3$ particle-steps/J, illustrating the drag introduced by less optimized code sections.

AthenaK, leveraging Kokkos+SYCL offload, exhibits pronounced sensitivity to problem granularity. For the GW150914 binary black-hole merger test case, the GPU yields up to $12\times$ higher throughput and $15\times$ better energy efficiency than CPU-only at large mesh-block sizes, peaking near $E$0 zone-cycles/s and $E$1 zone-cycles/J. As mesh-block edge length decreases, GPU gains dissipate due to under-utilization of PVC execution units.

dealii-X Kernels: High-Order Finite-Element Matrix-Free Operators

Matrix-free kernels from dealii-X, evaluated using both Kokkos (various thread-mapping strategies), OpenMP, and the domain-specific Tiny Tensor Compiler, demonstrate that OpenMP and TinyTC attain superior performance and energy usage. Throughput and energy efficiency track each other closely, indicating that instantaneous power is approximately constant and energy is dictated primarily by runtime. The analysis confirms the general suitability of PVC for high-intensity operator workloads amenable to vectorization and fusion.

Power-Budget Utilization and Bottleneck Analysis

The study introduces the power-budget parameter, $E$2, as a normalized measure to evaluate how much of the node TDP is effectively converted to useful computation across system and workload scales.

Figure 2: Power-budget parameter $E$3 as a function of normalized degrees of freedom, highlighting efficiency saturation at large problem sizes.

GPU runs tend to stabilize near unity for large enough workloads, indicating optimal utilization where throughput and energy are coupled and the TDP is a valid power model. However, for small granularities or poorly optimized applications, this ratio drops, suggesting inefficiencies. CPU-only runs, despite a reduced TDP value, do not achieve unity, pointing to power consumed outside of main computational kernels or by non-user processes.

Importantly, the study demonstrates that energy efficiency is a more sensitive diagnostic for kernel-level and occupancy-related bottlenecks than throughput alone. This is particularly visible in applications like OpenGadget3 and AthenaK, where kernel composition and device utilization have non-obvious impacts on energy-per-compute-element scaling.

Measurement Framework Comparison: EAR vs. p3em

A direct comparison between EAR and p3em shows p3em systematically reporting higher energy efficiency, especially for short or fine-grained kernels, attributable to tighter measurement windows and exclusion of initialization overhead. Consistency of trends across both tools validates the data, and the study recommends p3em for fine-grained, intra-application analysis.

Implications and Forward Outlook

This work provides comprehensive baselines for node-level HPC performance and energy analysis on next-generation Intel heterogenous architectures. Practically, the findings underline the importance of exposing sufficient computational granularity—especially for offloaded GPU kernels—to maximize both throughput and energy efficiency. They also suggest that while roofline models remain informative, application-level, phase-accurate energy metrics yield a deeper understanding of bottlenecks and optimization potential.

From a theoretical perspective, the decoupling of throughput and energy efficiency under certain kernel mixes or node configurations highlights the necessity for nuanced resource allocation and scheduling policies in HPC environments as power and thermal constraints become increasingly dominant.

Future research involving multi-node scaling, network contention, and an expanded application set is proposed. Comparative benchmarking across competing hardware generations will also refine interpretations of both empirical and projected energy-performance characteristics for flagship computational science workloads.

Conclusion

The paper systematically demonstrates that, for a heterogeneous system equipped with Intel SSC + PVC, GPU offloading enables order-of-magnitude improvements in both throughput and energy efficiency for a broad set of scientific applications. However, these gains are strongly workload- and granularity-dependent. The unification of throughput and energy metrics across multiple scientific domains enables robust, actionable guidance for code and system optimization. Future work will expand these results into the multi-node regime and deepen analysis of interconnect and memory system effects.