DPEcho SYCL Benchmark for Astrophysics

• DPEcho benchmark is a realistic MPI+SYCL-driven proxy for GR-MHD simulations, reflecting real astrophysical phenomena like turbulence and black hole flows.
• It employs finite-volume methods with stencil-heavy computations and explicit Runge–Kutta time stepping, using Cartesian MPI for distributed on-node parallelism.
• The benchmark uniquely combines performance (MCUP/s) with energy efficiency (MCUP/J), highlighting the impact of compiler choice and hardware portability across CPUs and GPUs.

DPEcho is an application-driven SYCL benchmark for numerical astrophysics: an MPI+SYCL rewrite of the finite-volume general-relativistic magnetohydrodynamics code ECHO, used in astrophysics for simulations of instabilities, turbulence, wave propagation, stellar winds, magnetospheres, and flows near black holes. In the form benchmarked, it is best viewed as a realistic proxy application rather than a toy kernel. Its stated goals are twofold: to show that SYCL is a viable vendor-agnostic porting route for this class of codes, and to use DPEcho to compare heterogeneous CPUs and GPUs with metrics more meaningful than raw speed alone, especially energy efficiency (Cielo et al., 18 Aug 2025).

1. Purpose and positioning

DPEcho was developed in response to two constraints identified for computational astrophysics. First, energy awareness and efficiency policies are gaining more attention, over pure performance (time-to-solution) Key Performance Indicators when comparing accelerated systems. Second, the field is struggling with code refactorings for GPUs, so viable porting paths have to be shown before first. The benchmark therefore addresses both software portability and hardware evaluation in one instrument (Cielo et al., 18 Aug 2025).

The portability argument is explicit. The benchmark is motivated by the need for portable GPU paths that do not lock codes to CUDA, HIP, or vendor-specific ecosystems. SYCL is treated as the portability mechanism that lets one single-source code run across Intel, AMD, and NVIDIA GPUs as well as CPUs. The paper explicitly pushes back against the idea that SYCL is “for Intel only.” In the reported study, Intel targets were compiled with Intel oneAPI compilers (icpx), while AMD and NVIDIA GPUs were reached through oneAPI plus Codeplay plugins providing HIP and CUDA backends; AdaptiveCpp (acpp) was also tested because compiler choice turned out to matter substantially for GPU performance (Cielo et al., 18 Aug 2025).

The broader claim is not that SYCL eliminates all porting difficulty. The paper identifies lack of native Fortran support, immature multidimensional array vocabulary support at the time, and uneven availability of SYCL compilers on non-Intel systems as practical drawbacks. It also strongly suggests that HPC centers should provide SYCL toolchains rather than forcing users to build them from source. In that sense, DPEcho functions simultaneously as a benchmark and as a case study in the institutional conditions required for portable heterogeneous computing.

2. Workload, physics, and implementation

The benchmarked workload comes from a subset of ECHO physics implemented in SYCL. DPEcho solves the GR-MHD equations on a fixed 3D grid with explicit third-order Runge–Kutta time stepping. Along each spatial dimension it uses a 6-point stencil for interpolation and a 5-point stencil for reconstruction, making the workload stencil-heavy, memory intensive, and structurally representative of structured-grid finite-volume astrophysics (Cielo et al., 18 Aug 2025).

The benchmark case is a standard Alfvén-wave problem: unit wavenumbers in all three directions of a cubic periodic box, wavevector parallel to the box diagonal, full GR-MHD solver enabled, no constrained transport for magnetic-divergence cleaning, and the highest-order reconstruction scheme. Domain decomposition is Cartesian MPI, and on-node parallelism is offloaded through SYCL. This places DPEcho between toy kernels and full production codes: it is small enough to benchmark systematically, but it retains the stencil-rich, MPI-distributed, structured-grid behavior of a real GR-MHD solver.

Implementation choices are central to what the benchmark measures. The code is essentially device-only USM: data stay resident in accelerator memory for the whole computation and I/O is handled by one-way memcpy operations. That choice reduces host-device traffic and makes the measured energy closer to the compute device itself. A practical portability issue arises from the fact that SYCL USM exposes only 1D pointers; since the original Fortran code relied heavily on multidimensional arrays and slices, DPEcho needed its own grid/indexing abstraction. The authors identify this as a real development bottleneck, worsened by the then-incomplete support for std::mdspan. They also note that a corrected flux scheme introduced uneven flux-grid extents such as $512\times512\times513$ instead of $512^3$, which some compilers and runtimes apparently optimize poorly, especially on non-Intel GPUs (Cielo et al., 18 Aug 2025).

3. Metrics and energy-measurement pipeline

The core benchmark KPIs are performance in millions of cell updates per second and energy efficiency in millions of cell updates per joule: $$\mathrm{Performance} = \mathrm{MCUP/s}, \qquad \mathrm{Energy\ efficiency} = \mathrm{MCUP/J}.$$ The implied definitions are

$$\mathrm{MCUP/s} = \frac{N_{\mathrm{cell\ updates}}}{10^6\, T}, \qquad \mathrm{MCUP/J} = \frac{N_{\mathrm{cell\ updates}}}{10^6\, E}.$$

Energy is obtained by integrating measured device power over time,

$$E = \int_0^T P(t)\,dt \approx \sum_i P_i\,\Delta t_i.$$

This numerical integration is performed by the deltaEnergy.sh script (Cielo et al., 18 Aug 2025).

The methodological claim attached to these metrics is that energy efficiency is often more informative than raw performance because it is an intensive metric: performance alone is biased by device size, whereas energy consumed acts as a data-driven proxy for how much hardware is actually being exercised. MCUP/J is therefore treated as the primary comparison metric, analogous in spirit to Green500’s FLOP/W. Time-to-solution remains relevant, but the paper emphasizes throughput and energy-to-solution rather than wall time in isolation.

A major contribution of DPEcho is its energy-measurement pipeline. Because node-level or batch-level “at-the-plug” counters are often inaccessible, site-dependent, and contaminated by non-used hardware, the authors built a portable wrapper that can poll any low-granularity power-reporting command and integrate it over time. Inside DPEcho, an extra Boost thread triggers this polling during the timed compute region. For GPUs they use vendor SMI tools; for CPUs they use perf, likwid, or EAR/econtrol counters, selecting “cores” or “package” energy and explicitly excluding DRAM and unrelated components. The intent is to measure “as close to the cores as possible.” The main caveat is sampling latency: the power meter should sample much faster than a simulation timestep to be statistically accurate. This limitation appears most clearly in the large-scale PVC experiments, where xpu-smi had a minimum granularity of 1 second and inflated the uncertainty on energy efficiency (Cielo et al., 18 Aug 2025).

4. Experimental configuration

The experimental setup spans a wide hardware set on single nodes. The tested devices were Intel CPUs SKX (Xeon Platinum 8174), SPR (Xeon Platinum 8480L), GNR (Xeon 6972P), SRF (Xeon 6780E), and EMR (Xeon Platinum 8592+); AMD CPUs Rome (EPYC 7742) and Milan (EPYC 7773X); Intel GPU PVC (Data Center GPU Max 1550); AMD GPUs MI100 and MI210; and NVIDIA GPUs A40, A100, and H100. Table 1 in the paper reports nominal TDPs, notes that PVC was power-capped to 450 W, and states that results were normalized to a single tile because each card has two tiles (Cielo et al., 18 Aug 2025).

Nodes generally had two CPUs. For the benchmark, one MPI rank was used per device or NUMA domain, which in practice meant one rank per GPU tile or per CPU NUMA region. Problem size was not fixed globally. Instead, for each device the workload was increased to use most of node memory, a weak-scaling-style configuration was run up to full node occupancy, and the result was then divided by the number of devices per node to report per-device figures. This is why the study adopts a throughput metric based on cell updates rather than a fixed-walltime benchmark.

The reported values are medians over 21 measurements, with quartiles and 5th/95th percentiles shown in the main figure. This choice of reporting emphasizes distributional stability rather than a single best-case number. It also reinforces the article’s broader point that DPEcho is intended for application-aware hardware comparison, not for peak-oriented marketing numbers.

5. Performance, energy efficiency, and compiler effects

The main single-node results show a consistent qualitative conclusion: GPUs strongly outperform CPUs in both MCUP/s and MCUP/J. The ranking by raw speed and by energy efficiency is similar, but the separation between GPUs and CPUs is much larger when judged by MCUP/J. Intel PVC shows the highest DPEcho performance and energy efficiency in the reported oneAPI-based measurements, even though results are normalized to a single tile. Among CPUs, Intel generations improve in performance over time, but energy efficiency varies less than performance, which the authors interpret as evidence that higher raw speed is often purchased by higher power draw. AMD CPUs show attractive energy efficiency, likely helped by node power capping (Cielo et al., 18 Aug 2025).

The paper also notes an important comparability issue: DPEcho performance in this work is lower than in the authors’ previous single-device study, because the benchmark here uses a more realistic MPI-enabled node configuration and a corrected, more portable flux scheme. The benchmark is therefore explicitly presented as a more faithful application proxy, even where that reduces the headline throughput.

Compiler choice materially changes GPU conclusions. When icpx and AdaptiveCpp are compared on SKX, SPR, PVC, and A100, CPU differences are small: acpp is slightly slower but similar in MCUP/J. On GPUs, however, A100 benefits dramatically from AdaptiveCpp, and the paper concludes that acpp is the recommended compiler for DPEcho on A100. One of the clearest findings is therefore that SYCL portability is real, but “portable” does not mean performance is compiler-independent. A plausible implication is that any procurement or platform study using SYCL workloads must treat the compiler/runtime stack as part of the platform, not as a neutral implementation detail.

The comparison with High-Performance Linpack sharpens the benchmark’s intended scope. Green500/HPL results had suggested Intel GPUs were less efficient than competitors, apparently conflicting with DPEcho. The paper resolves this by showing that application-specific rankings matter: on DPEcho, PVC gains a larger speed-up over CPU than A100, while on HPL A100 gains more. The interpretation is that HPL is biased toward dense matrix-multiplication-like, highly compute-intensive Type-I kernels, whereas DPEcho is a mixed finite-volume stencil code closer to “Type II” applications. Thus DPEcho reveals something HPL does not: for a realistic structured-grid GR-MHD workload, a device’s ranking can differ substantially from its HPL ranking, and energy efficiency narrows some apparent vendor gaps (Cielo et al., 18 Aug 2025).

6. Scaling, methodological caveats, and significance

The paper includes a multi-node weak-scaling result for DPEcho on SuperMUC-NG phase 2 with PVC, up to 140 nodes and 8 PVC tiles per node. Cells per node are kept constant while the global domain and Cartesian MPI grid grow. Both MCUP/s and MCUP/J scale similarly; performance efficiency stays above 90%, with no appreciable decline until about 32–64 nodes. The larger energy-efficiency error bars are attributed mainly to the coarse xpu-smi sampling interval, not to erratic application behavior (Cielo et al., 18 Aug 2025).

A separate methodological issue concerns the scope of energy accounting. The paper compares device-centric energy with node-level, at-the-plug energy using the EAR framework and finds that the two correlate, but node-level energy can be badly distorted by node composition and runtime behavior. The stated example is an SPR node that also houses four PVC GPUs: even a CPU-only run may still trigger some GPU activity or runtime interrogation, so plug energy is no longer a clean measure of CPU compute efficiency. On that basis, the benchmark chooses device-level energy for cross-platform hardware comparisons, while acknowledging that plug-level energy can be appropriate when comparing complete node designs rather than devices.

The benchmark’s broader significance lies in its intermediate position between toy kernels and full applications. DPEcho captures the stencil-rich, MPI-distributed, structured-grid, device-resident behavior of a real GR-MHD solver and can be run across most modern HPC hardware from one code base. For astrophysicists choosing a GPU-porting path, the paper argues that SYCL is credible and practically valuable, especially for C++ codes needing vendor-agnostic portability. For HPC centers, it argues that hardware selection for SYCL-ported astrophysics workloads should not rely on HPL alone; DPEcho-like application benchmarks plus energy-aware KPIs such as MCUP/J give a better basis for procurement and user guidance.

In the wider SYCL literature, this places DPEcho alongside workload-specific studies showing that kernel orchestration, compiler/runtime choice, and data movement policy materially change conclusions. A closely related example is the ExaHyPE finite-volume kernel study, which found that loop-centric data-parallel formulations outperform straightforward fine-grained task-graph realizations for SYCL GPU execution (Loi et al., 2023). That comparison suggests a broader interpretation of DPEcho’s results: benchmark outcomes are shaped not only by the target hardware, but also by how a scientific computation is lowered into SYCL.