SuperMUC-NG Phase 2 Insights

• SuperMUC-NG Phase 2 is a 28 PFLOPS heterogeneous HPC system that integrates GPU and CPU nodes for scalable language model training and diverse scientific workflows.
• It features 240 GPU-capable nodes with advanced intra-node (Intel Xe Link) and inter-node (HDR InfiniBand) connectivity under strict production power caps.
• Its performance is driven by intra-node tensor parallelism, micro-batching techniques, and reproducible data parallelism, balancing computation and energy efficiency.

Searching arXiv for the cited SuperMUC-NG Phase 2 papers and closely related records. SuperMUC-NG Phase 2, often abbreviated SMNG-P2, is a 28 PFLOPS accelerated system at the Leibniz Supercomputing Centre (LRZ) in Garching, Germany. In the published record, it appears both as a platform for efficient large-scale GPT-style training and as a node-level testbed for performance and energy-efficiency studies across molecular dynamics, astrophysics, cosmology, magnetohydrodynamics, and matrix-free finite-element kernels. The system combines heterogeneous CPU+GPU nodes, Intel XeLink intra-node connectivity, HDR InfiniBand interconnect, and a DAOS storage tier, and it has been studied under production power caps rather than unconstrained peak settings (Rajgopal et al., 8 May 2026, Cielo et al., 22 Jun 2026).

1. System architecture and resource organization

SMNG-P2 comprises 240 GPU-capable nodes, of which 234 are compute nodes, 2 are spare nodes, and 4 are login nodes, for a total of 960 Intel Data Center GPU Max 1550 accelerators and approximately 123 TB aggregate GPU HBM. At the node level, each system contains two 4th Gen Intel Xeon Scalable CPUs with 112 total cores, which the node-level characterization associates with Intel Xeon Platinum 8480+ “Sapphire Rapids” sockets at 56 cores per socket, together with 512 GB DDR5 memory per node (Rajgopal et al., 8 May 2026, Cielo et al., 22 Jun 2026).

Each Intel Data Center GPU Max 1550 accelerator has a multi-tile design with 2 tiles per card, so a node with 4 accelerators exposes 8 tiles. The HBM2e capacity is 128 GB per accelerator, or 64 GB per tile. Intra-node GPU connectivity is provided by Intel Xe Link. Inter-node communication uses a fat-tree NVIDIA/Mellanox HDR InfiniBand network with two HDR interfaces per node and 400 Gbit/s aggregate injection bandwidth. Persistent data services are provided by a DAOS tier of about 1 PB usable capacity and more than 750 GB/s aggregate write bandwidth, with transparent access to the Phase 1 parallel file systems HPPFS and DSS (Rajgopal et al., 8 May 2026).

The platform is operated under a production power envelope. For the accelerators, nominal power is about 600 W per PVC card, but production runs are power-capped to 450 W per accelerator. At the node level, the total single-node power budget is set to $W_{TDP} = 2500\ \mathrm{W}$, while CPU-only analyses use a reduced budget of approximately 700 W for the two SPR sockets. This is significant because both the language-model training results and the application characterization explicitly reflect sustained production operation rather than unconstrained peak measurements (Rajgopal et al., 8 May 2026, Cielo et al., 22 Jun 2026).

2. Software environment and execution model

The operating environment is the standard LRZ production stack based on SLES and SLURM. For large-scale language-model training, the reported runs use public, out-of-the-box software with no custom kernels or hardware-specific code. The framework stack is Megatron-DeepSpeed v2.4, with Megatron providing tensor and pipeline parallelism and DeepSpeed 0.16.9 providing ZeRO sharded data parallelism. The execution backend is PyTorch 2.8.0 with XPU support through Intel Extension for PyTorch (IPEX) 2.8.0, and all training runs use bf16 mixed precision with fused operations and pipeline scheduling as provided by the standard frameworks (Rajgopal et al., 8 May 2026).

The reproducibility model for training is correspondingly direct. The paper specifies use of LRZ modules or containers for PyTorch 2.8.0 and IPEX 2.8.0, installation of DeepSpeed 0.16.9 and Megatron-DeepSpeed v2.4, configuration of tensor-model-parallel-size and pipeline-model-parallel-size, ZeRO stage 1 for the reported large-scale runs, bf16 precision, and tuned micro-batch size and gradient accumulation steps. Jobs are submitted through SLURM with one process per tile, and tensor-parallel groups are restricted to a single node. No special environment variables beyond defaults were required in the reported experiments (Rajgopal et al., 8 May 2026).

The node-level application study uses a different but equally standard software stack. It reports Intel oneAPI compilers v2025.3.0 and Intel MPI v2021.17.0, with PVC targeted via SYCL and OpenMP offload depending on the application. Because each PVC card is subdivided into two logical tiles, the default execution pattern is eight MPI ranks per node, which gives one rank per tile unless otherwise noted. Energy instrumentation is performed either with EAR or with p3em, depending on the application (Cielo et al., 22 Jun 2026).

3. Parallel training methodology for LLMs

The training study examines GPT-style models of 3.6B, 20B, and 175B parameters. The parameter scaling relation is given as

$P_{total} \approx 12 L d^2 + V d,$

where $L$ is the number of layers, $d$ the hidden size, and $V$ the vocabulary size, with the $12Ld^2$ term dominating at scale. Under bf16 mixed-precision memory accounting, each parameter effectively contributes 16 bytes total across master weights, compute weights, gradients, and Adam states. The corresponding total memory footprints reported are 57.6 GB for 3.6B, 320 GB for 20B, and 2.8 TB for 175B. These numbers make clear why model parallelism and sharded optimizer-state techniques are required for the larger models (Rajgopal et al., 8 May 2026).

The training recipe combines tensor parallelism (TP), pipeline parallelism (PP), and sharded data parallelism (SDP) through ZeRO. TP shards the largest projections in attention and MLP layers into per-device slices and requires layer-wise collectives to reassemble outputs. The empirical sweep on the 3.6B model shows that throughput drops sharply when TP exceeds 8, which is the number of tiles per node, because frequent all-reduces begin to cross nodes. The operational rule is therefore to keep tensor parallelism within a node, that is, $TP \leq 8$ (Rajgopal et al., 8 May 2026).

PP stages transformer layers and uses a 1F1B-style schedule. The paper states that the pipeline bubble overhead scales approximately with $PP/M$, where $M$ is the number of micro-batches per optimizer step, and gives the GPipe-style efficiency approximation

$\eta_{pipe} = \frac{m}{m + p - 1},$

with $P_{total} \approx 12 L d^2 + V d,$0 micro-batches and $P_{total} \approx 12 L d^2 + V d,$1 pipeline stages. Empirically, increasing $P_{total} \approx 12 L d^2 + V d,$2 amortizes the bubble until throughput plateaus, increasing PP at fixed $P_{total} \approx 12 L d^2 + V d,$3 degrades throughput, and keeping $P_{total} \approx 12 L d^2 + V d,$4 constant maintains stable throughput. The large-scale runs use ZeRO-1, that is, optimizer-state sharding, because it reduces memory while minimizing the extra communication overhead associated with ZeRO-2 or ZeRO-3 (Rajgopal et al., 8 May 2026).

The tuned 175B configuration was obtained with automated Bayesian optimization using DeepHyper 0.8.1. The selected configuration is $P_{total} \approx 12 L d^2 + V d,$5, $P_{total} \approx 12 L d^2 + V d,$6, micro-batch size $P_{total} \approx 12 L d^2 + V d,$7, gradient accumulation steps $P_{total} \approx 12 L d^2 + V d,$8, and ZeRO-1, with data parallel degree $P_{total} \approx 12 L d^2 + V d,$9 varied for scaling studies. The global batch size follows

$L$0

The study emphasizes that the main enabler is not specialized kernel engineering but the disciplined combination of within-node TP, moderate PP, microbatching, and scale-out through data parallelism (Rajgopal et al., 8 May 2026).

4. Scaling behavior, throughput, and bottlenecks

For decoder-only LLMs, the training compute estimate is summarized as

$L$1

and the paper cites an estimated cost of approximately 120 million exaflops, in aggregate, for a one-trillion-parameter GPT-style model. On SMNG-P2, the best observed result for the 175B model is 57 TFLOPs/s per tile in bf16, reported as approximately 10% of the per-tile theoretical bf16 peak, implying a per-tile theoretical bf16 peak of about 570 TFLOPs/s. The primary reported metric is effective model TFLOPs/s for forward and backward passes as reported by Megatron-DeepSpeed (Rajgopal et al., 8 May 2026).

The scaling results extend to 128 nodes, corresponding to 1024 tiles. With ZeRO-1 and the tuned $L$2 configuration, the reported weak scaling efficiency at 128 nodes is approximately 93%, while the strong scaling efficiency is approximately 82%. The formulas used are

$L$3

for fixed total problem size, and

$L$4

for constant work per node (Rajgopal et al., 8 May 2026).

The dominant bottlenecks are communication and pipeline underfill rather than storage bandwidth. TP communication is dominated by frequent all-reduce and all-gather collectives at every transformer layer, and when TP spans nodes these collectives cross the HDR InfiniBand fabric and reduce compute-communication overlap. The pipeline bubble is the other principal utilization limiter, so increasing micro-batches until throughput plateaus and avoiding unnecessarily deep pipelines are central parts of the recipe. The study explicitly states that cross-node synchronization and collective latency are the main strong-scaling bottlenecks, while the two HDR links per node and DAOS storage do not dominate training iteration time in the reported setup (Rajgopal et al., 8 May 2026).

A recurrent interpretive error is to equate theoretical peak arithmetic throughput with delivered training performance. On SMNG-P2, the 175B result remains at approximately 10% of theoretical per-tile bf16 peak even with a tuned configuration, and the paper treats this as consistent with large-scale studies in which communication and synchronization limit throughput. In this sense, SMNG-P2 is not presented as an exception to distributed-training bottlenecks; rather, it is presented as a system on which those bottlenecks can be managed with a reproducible configuration (Rajgopal et al., 8 May 2026).

5. Node-level application throughput and energy efficiency

The node-level characterization paper adopts compute-element-normalized throughput and energy efficiency as its central metrics. With $L$5 the number of compute elements, $L$6 the number of time steps or iterations, $L$7 the elapsed wall-clock time, and $L$8 the measured node energy, the definitions are

$L$9

and

$d$0

The paper also defines GPU-versus-CPU throughput speedup $d$1, energy-efficiency gain $d$2, average power $d$3, and power-budget utilization

$d$4

These measurements are strictly single-node and therefore isolate intra-node behavior rather than network effects (Cielo et al., 22 Jun 2026).

The workloads span several domains and offload models. In gromacs, the compute element is the atom-step, non-bonded and bonded interactions and coordinate updates are offloaded to PVC, and PME is calculated on a separate GPU tile, with 7 tiles used for short-range PP interactions and 1 tile for PME. In lammps, the compute element is also the atom-step and the Kokkos SYCL back-end targets PVC. OpenGadget3 uses particle-step as the compute element and offloads the gravitational tree, HSML, conduction, hydrodynamics, and density kernels, while keeping other kernels on CPU. AthenaK uses zone-cycle, with a mesh-block edge length as its main granularity knob. The dealii-X kernels use degrees of freedom as the compute element and study several implementation variants, including Kokkos, OpenMP target teams loop, and Tiny Tensor Compiler (Cielo et al., 22 Jun 2026).

The reported single-node results show that accelerator gains are substantial but workload-dependent. For gromacs, $d$5 atom-steps/s across all tested sizes, about $d$6 higher than CPU-only, and $d$7 plateaus at approximately $d$8 atom-steps/J for large systems; however, CPU-only energy efficiency can match or exceed GPU energy efficiency at small sizes. For lammps, GPU throughput plateaus around $d$9 atom-steps/s, GPU energy efficiency around $V$0 atom-steps/J, CPU throughput around $V$1 atom-steps/s, and CPU energy efficiency around $V$2 atom-steps/J, corresponding to about $V$3 throughput speedup and about $V$4 energy-efficiency gain at large sizes (Cielo et al., 22 Jun 2026).

OpenGadget3 reaches a peak whole-simulation throughput of about $V$5 particle-steps/s and a whole-simulation energy efficiency of about 660 particle-steps/J, while the HSML kernel reaches about $V$6 particle-steps/J, roughly $V$7 higher than the aggregate figure. AthenaK reaches GPU peak throughput of about $V$8 zone-cycles/s at mesh-block edge lengths of at least 24, approximately $V$9 higher than the SPR-only peak, with GPU energy efficiency around $12Ld^2$0 zone-cycles/J versus about $12Ld^2$1 zone-cycles/J for CPU-only, for an efficiency gain of about $12Ld^2$2. The dealii-X CEED Bake-off Kernel 1 study reports a best throughput of about $12Ld^2$3 DoF/s and a best energy efficiency of about $12Ld^2$4 DoF/J, with OpenMP and TinyTC outperforming the Kokkos mappings (Cielo et al., 22 Jun 2026).

A central result of the node-level study is the sensitivity of PVC performance to granularity. In AthenaK, the GPU advantage narrows markedly at mesh-block edge lengths of 10 or below because launch overhead and poor occupancy erode the accelerator benefit. In lammps, atom counts below about $12Ld^2$5 per node do not saturate the GPU, and both throughput and energy efficiency rise rapidly only until that range. This makes SMNG-P2 not only a high-throughput platform but also a system on which occupancy and work-per-tile effects are directly visible in both $12Ld^2$6 and $12Ld^2$7 (Cielo et al., 22 Jun 2026).

6. Interpretation, limitations, and portability

The two SMNG-P2 studies jointly highlight a consistent operational pattern: the system is most effective when locality is preserved within the node and when each GPU tile is given enough work to maintain occupancy. For LLM training, this appears as the rule that tensor parallelism should remain intra-node, with scale-out proceeding primarily through data parallelism and ZeRO-1. For node-level scientific applications, it appears as the need for sufficient atoms, zones, particles, or degrees of freedom per tile before the accelerator advantage saturates (Rajgopal et al., 8 May 2026, Cielo et al., 22 Jun 2026).

The power-budget analysis adds an important qualification. In the node-level study, GPU runs on full nodes show $12Ld^2$8 values that are stable or converge near unity at sufficiently large problem sizes, especially for gromacs and AthenaK, whereas CPU-only runs show $12Ld^2$9 systematically below unity even when a reduced CPU-only power budget is used. The paper interprets this as indicating that CPUs often leave part of the node’s thermal envelope unused, while GPU runs at high occupancy make throughput and energy efficiency carry similar information. This does not imply that GPU execution is always superior in every regime: for small gromacs systems, CPU-only execution can match or exceed GPU execution in energy efficiency, and AthenaK explicitly shows loss of accelerator advantage at small mesh-block sizes (Cielo et al., 22 Jun 2026).

Both studies also delimit their claims carefully. The language-model recipe explicitly avoids specialized kernels: activation checkpointing, FlashAttention, and custom fused kernels are not claimed, and the reported performance arises from standard Megatron-DeepSpeed implementations and IPEX’s XPU backend. The application paper is restricted to a single-node scope and therefore does not address inter-node network behavior. On the training side, communication overheads increasingly dominate at higher strong-scaling factors, and the reported 82% strong-scaling efficiency at 128 nodes is presented as evidence of growing collective costs. On the application side, the authors state that future work will extend to multi-node scaling and network interactions (Rajgopal et al., 8 May 2026, Cielo et al., 22 Jun 2026).

The published generalization claims are correspondingly moderate. The training paper states that the recipe of within-node TP, moderate PP with microbatching, and DP plus ZeRO-1 is portable to larger models, but that diminishing returns are expected unless communication is reduced, for example through higher-bandwidth fabrics or algorithmic communication reduction. It also states that moving to ZeRO-2 or ZeRO-3 would increase memory headroom but introduce more communication and may reduce throughput. The same paper further argues that, because the approach uses standard public software, it should port to other XPU/GPU-based HPC systems with high-bandwidth intra-node GPU links and IB-class interconnects without specialized engineering. Quantitative energy-efficiency analysis for the training regime is left to future work (Rajgopal et al., 8 May 2026).

In aggregate, SMNG-P2 is characterized in the literature as a heterogeneous HPC system whose defining technical features are its eight-tile node structure, XeLink-connected PVC accelerators, dual-HDR interconnect, DAOS-backed storage, and production power caps. Its significance lies less in nominal peak numbers than in the empirical recipes and measurement models developed on top of that hardware: a reproducible bf16 large-model training configuration for up to 175B parameters, and a compute-element-normalized framework for comparing throughput and energy efficiency across disparate scientific workloads (Rajgopal et al., 8 May 2026, Cielo et al., 22 Jun 2026).