On the Limits of Performance Portability in Directive-Based GPU Programming

Abstract: The transition of scientific applications to GPU-accelerated exascale systems is constrained by trade-offs between performance, portability, and productivity. This work evaluates the performance portability of directive-based GPU programming by porting gPLUTO, a production-grade magnetohydrodynamics code for astrophysical simulations, from OpenACC to OpenMP, and analyzing its performance on NVIDIA A100 (Leonardo Booster) and AMD MI250X (LUMI-G) devices. On NVIDIA platforms, OpenACC and OpenMP achieve comparable performance due to a shared compiler backend, providing a consistent baseline for assessing algorithmic efficiency. In contrast, the same OpenMP implementation is approximately three times slower at the application level on AMD MI250X with respect to the NVIDIA A100 OpenACC baseline, with kernel-level slowdowns reaching up to an order of magnitude, driven by sensitivity to strided memory-access patterns and compiler limitations. Kernel-level profiling shows that the dominant contributors to run-time are memory-latency-bound rather than limited by peak band-width. In low-parallelism kernels, C++ abstraction layers increase register pressure and spilling, leading to extreme slowdowns of up to 47x in specific cases. These results indicate that portable performance across GPU architectures requires not only application-level changes but also continued advances in compiler backends and architecture-aware optimization strategies

• The paper evaluates an OpenACC-to-OpenMP port of the gPLUTO MHD code on NVIDIA A100 and AMD MI250X, combining benchmark workloads with kernel-level hardware-counter profiling.
• The paper finds near-parity on A100 but reports OpenMP slowdowns of up to 47× for individual 2D kernels, a 3× application-level regression on MI250X, and up to 22× sensitivity to directive formulation.
• The paper concludes that compiler backend maturity and architecture-specific effects—especially register spilling, cache capacity, strided memory access, and unified-memory behavior—limit performance portability beyond simple directive translation.

Motivation and scope

This paper examines how far directive-based GPU offloading can deliver performance portability in a production scientific code. The authors port gPLUTO — a finite-volume, Godunov-type magnetohydrodynamics (MHD) code for astrophysical simulations, developed within the EuroHPC SPACE Centre of Excellence — from OpenACC to OpenMP, and evaluate both implementations on NVIDIA A100 (Leonardo Booster) and AMD MI250X (LUMI-G) hardware. The central hypothesis is that observed portability limitations arise primarily from compiler backend maturity rather than programming-model semantics, with architecture characteristics such as cache hierarchy and sensitivity to strided memory access playing a secondary but decisive role. The study is motivated by the fact that over half of TOP500 systems now integrate accelerators, and by the well-documented trade-off between performance, portability, and productivity that legacy C/C++ codes face when adopting GPU offloading.

Methodology and experimental design

Experiments use single CPU-GPU nodes on Leonardo (A100, compiled with NVIDIA HPC SDK nvc++ 24.5) and LUMI-G (MI250X single GCD, ROCm 6.2.4 with the Cray/Clang LLVM OpenMP offload toolchain). Both platforms run gPLUTO under unified memory (NVIDIA Managed Memory; AMD CRAY_ACC_USE_UNIFIED_MEM=1 with HSA_XNACK=1), a requirement imposed by gPLUTO's pointer-intensive C++ data structures. The authors are explicit that this choice introduces vendor-specific runtime behaviors — page migration and fault handling — that cannot be fully disentangled from compiler and architectural effects; this is a stated limitation of the analysis.

The evaluation uses two workloads: the 3D Orszag–Tang vortex problem on a $352^3$ grid (≈43.6 million cells, roughly 170,000 thread blocks, providing 1,500–1,600 blocks per SM/CU) and a 2D Riemann problem ($3200^2$). Profiling combines Nsight Compute/Systems on A100 with rocProfiler/OmniPerf on MI250X, reporting achieved occupancy, FP64 throughput, HBM bandwidth, arithmetic intensity (AI), register usage, spilling, and cache hit rates. Kernels are classified as compute-bound, bandwidth-bound, latency-bound, or control-flow-bound using empirical AI and utilization thresholds. The authors caution that hardware-counter-derived metrics are functionally analogous but not numerically identical across vendors, so cross-architecture comparisons are qualitative rather than absolute.

Porting experience: OpenACC to OpenMP

A substantial methodological contribution is the practical account of the porting process itself. The key findings are:

• Parallelism exposure: OpenMP GPU offloading reliably exposes two parallelism levels (teams and threads), with simd mapping to GPU lanes remaining compiler-dependent. Full occupancy often requires collapse(3) on spatial loop nests, forcing intermediate declarations inside the innermost loop — which improves exposed parallelism but increases register pressure. OpenACC's gang/worker/vector hierarchy achieves equivalent parallelism without such restructuring.
• Data management: C++ wrapper abstractions (the Ary1D/Ary2D array classes) require explicit map clauses or declare [mapper](https://www.emergentmind.com/topics/mapper) directives in OpenMP; without them, compilers generate redundant private copies. OpenACC's implicit deep-copy semantics handle these abstractions more robustly. The authors also note that OpenMP's present modifier silently creates new mappings when data is absent, whereas OpenACC errors out — a correctness hazard. Mapper support varies across compilers, and incorrect definitions can cause silent corruption or device-side faults with limited diagnostics.
• Compiler-specific defects: NVHPC 24.5 internal errors with separated teams distribute + parallel for, incomplete simd support in ROCm 6.2.4, and divergent reduction and device-function semantics.

Results on NVIDIA A100: near-parity and fragility at low parallelism

On the 3D Orszag–Tang problem, OpenACC and OpenMP achieve near-identical performance on A100, which the authors attribute to the shared CUDA backend in NVHPC. Total runtime for ten steps is 13.9 s (OpenACC) versus 17.2 s (OpenMP). Kernel-level profiling shows that no kernel reaches a compute- or bandwidth-bound regime; the dominant HLLD() Riemann solver is instruction-latency-bound, with over 170 registers per thread collapsing occupancy to 12% and triggering roughly $8.8\times10^7$ spill loads and $4.9\times10^7$ spill stores per launch. OpenMP shows a systematic 5–10% slowdown from slightly higher register pressure, with CT_EMF() as the largest single gap (7.8 ms vs. 23.9 ms).

The 2D Riemann test exposes a stark contrast: OpenMP is 47× slower for Reconstruct() and 6× slower for the HLL solver. Importantly, the authors show this cannot be attributed to insufficient parallelism, contrary to common explanations for small-kernel OpenMP underperformance in the literature. Two root causes are identified: massive register spilling induced by conservative privatization of C++ wrapper objects ($8.8\times10^7$ spill loads where OpenACC produces zero), and compiler-driven suboptimal grid mapping (NVHPC selects a $1257\times128$ grid versus $3200\times128$ for OpenACC, exposing 2.5× less parallelism). Directive-variant experiments quantify the fragility: minor syntax changes to the OpenMP formulation vary HLL() runtime by up to 22×, and only one variant enables a shared-memory optimization. Recovering performance requires num_teams/thread_limit tuning or intrusive refactoring of the C++ abstractions into plain pointers — both of which undermine portability or break correspondence with the OpenACC version.

Results on AMD MI250X: a 3× application-level regression

The identical OpenMP implementation on MI250X is approximately 3× slower at the application level (48.1 s vs. 17.2 s on A100 OpenMP for ten Orszag–Tang steps), with kernel-level slowdowns reaching an order of magnitude. The bottleneck profile differs fundamentally from A100. Most kernels sustain substantial HBM traffic (up to 717 GB/s) yet remain latency-bound, with VALU utilization below 5% and IPC below 0.1 for memory-intensive kernels. The most striking result is the 3.7× directional anisotropy in HLLD(): the unit-stride $i$-sweep takes 29.7 ms while the strided $j,k$ sweeps take 110.6 ms, correlating with L1 hit rates of 75% versus 52%. The MI250X's much smaller caches (16 KB vector L1 per CU, 8 MB L2, versus 192 KB and 40 MB on A100) amplify non-unit-stride penalties; on A100 the same anisotropy is only about 20%.

Register pressure is explicitly ruled out as the MI250X bottleneck: VGPR usage is moderate (56–120 per wavefront), spilling is negligible, and the larger CU-level register file (~512 KB) of CDNA2 mitigates the abstraction-induced spilling seen on NVIDIA. Instead, poor memory coalescing under strided access, low cache hit rates, and — possibly — unified-memory page-migration overhead dominate. The authors also demonstrate that aggregate throughput metrics are misleading: Reconstruct() achieves 3,584 GFLOP/s (a 16.5× advantage over A100) on MI250X, yet this does not compensate for latency-bound kernels on the critical path. Occupancy alone does not predict performance either — US() reaches 71% occupancy while remaining latency-bound.

Discussion

Synthesizing the results, the paper argues that performance portability is governed more by compiler backend maturity than by programming-model semantics: on NVIDIA, the shared, mature CUDA backend yields near-parity between OpenACC and OpenMP; on AMD, the younger ROCm OpenMP backend — despite sharing LLVM infrastructure — lacks the allocation/alias-analysis maturity to handle the same code. Architectural characteristics modulate where the bottleneck manifests: C++ abstractions surface as register pressure and spilling on A100, but as coalescing and latency-hiding failures on MI250X. The paper's summary table makes the regime dependence explicit: the same C++ wrapper layer costs 47× in low-parallelism 2D OpenMP kernels, 6% in 3D OpenMP on A100, and 3× application-wide on MI250X. The practical conclusion is that recovering performance across vendors requires loop reordering, data-layout transformations, software transposition, and direction-specific kernel specialization — interventions well beyond pragma translation.

Limitations and open questions

The paper is candid about several constraints. All results are obtained under unified memory, so the contribution of page migration and XNACK fault handling to the MI250X slowdown — particularly for strided accesses — is not isolated; the authors identify explicit data-movement strategies as needed future work. Cross-architecture metric comparisons are qualitative because hardware counters differ between vendors, and IPC is used only intra-architecture. Kernel timings are single-invocation, steady-state measurements under profiling, and absolute times are not directly comparable to aggregated step runtimes. The study covers a single application, two vendors, and one OpenMP compiler per platform; the authors note that testing upstream Clang/LLVM on NVIDIA hardware would be needed to separate NVHPC-specific optimizations from intrinsic OpenMP characteristics, and that extending to Intel GPUs raises non-trivial USM and mapper-maturity challenges. Finally, porting effort is not quantified against performance gain, which limits the actionability of the productivity claims.

Conclusion

This paper provides a kernel-level, counter-based assessment of performance portability in a production MHD code across the two dominant pre-exascale GPU architectures. Its central empirical findings — near-parity between OpenACC and OpenMP on A100 via a shared backend, a 3× application-level and up to 47× kernel-level regression for the same OpenMP code on MI250X, a 22× sensitivity to OpenMP directive formulation in low-parallelism kernels, and a 3.7× strided-access anisotropy unique to MI250X — support the claim that correctness portability is readily achievable while performance portability is not. The results indicate that portable performance depends on continued compiler-backend maturation and architecture-aware optimization, and that directive-based offloading of legacy pointer-heavy C++ codes remains a profiling-driven engineering effort rather than a mechanical translation.