RAJAPerf: Performance-Portability Benchmark Suite

• RAJAPerf is a 64-kernel benchmarking suite spanning algorithms, applications, basic loops, Lcals, Polybench, and streaming workloads to assess performance across diverse computational and memory regimes.
• Researchers use RAJAPerf to compare execution models, compilers, processors, vectorization, NUMA thread placement, GPU offloading, IOMMU systems, and data-movement policies using metrics such as runtime, speedup, efficiency, and transfer cost.
• Its varied kernels reveal workload-specific behavior—including strong effects from NUMA placement, cache design, memory bandwidth, synchronization, and host-device transfers—so robust studies should report per-kernel and per-class results rather than a single aggregate score.

RAJAPerf is a loop-oriented benchmarking suite for evaluating performance-portability frameworks, compilers, processors, vectorization strategies, thread-placement policies, heterogeneous offloading mechanisms, and performance-sensitive configurations. Originally developed to evaluate RAJA, it supports execution models including OpenMP and has been used across CPU, GPU, RISC-V, embedded heterogeneous SoC, and Rust GPU-offload studies. Its kernel diversity spans basic algorithms, application-like workloads, stencil and loop-nest computations, polyhedral kernels, and streaming operations, making it a proxy for multiple computational and memory-performance regimes rather than a single application benchmark (Brown et al., 2023).

1. Scope and organization

RAJAPerf contains 64 kernels grouped into six classes:

Class	Kernels	Representative computational content
Algorithm	6	Memory copies, memory set, sorting, reductions, and other basic algorithms
Apps	13	FIR filtering, halo-exchange packing/unpacking, 3-D diffusion and convection by partial assembly, and a 1-D Laplace solver
Basic	16	DAXPY, matrix multiplication, integer reduction, calculation of $\pi$ by reduction, and elementary mathematical operations
Lcals	11	Livermore Compiler Analysis Loop Suite kernels, including tridiagonal elimination, difference calculations, and minimum/maximum operations
Polybench	13	2MM, 3MM, GEMM, matrix transpose, vector multiplication, Jacobi stencils, Warshall, Heat3D, and an alternating-direction implicit solver
Stream	5	Vectorizable kernels emphasizing memory bandwidth and associated arithmetic

The suite separates compute-heavy arithmetic and reductions, regular dense linear algebra, stencil and loop-nest computations, irregular or algorithmic operations, streaming accesses, and application-inspired packing and partial-differential-equation kernels. This composition allows performance analyses to distinguish architectural behavior that would be obscured by a single aggregate workload.

RAJAPerf does not prescribe one universal physical throughput unit. Studies have primarily reported relative execution performance, class averages, ranges across kernels, wall-clock time, speedup, parallel efficiency, kernel time, transfer time, and total runtime. In one RISC-V evaluation, the principal metric was relative execution performance normalized to a VisionFive V2 single core for RISC-V comparisons and to the Sophon SG2042 for x86 comparisons (Brown et al., 2023).

2. Kernel and workload characteristics

The benchmark classes expose substantially different computational structures. Algorithm kernels include memory movement, sorting, and reductions; Basic and Lcals include arithmetic and loop-analysis kernels; Polybench includes matrix operations, transposes, stencils, and Warshall; Stream emphasizes vectorizable memory-bandwidth behavior; and Apps contains application-like operations such as FIR filtering, halo-exchange packing and unpacking, and PDE-inspired kernels.

The suite has also been used to evaluate heterogeneous accelerator behavior through AXPY, HEAT3D, GESUMMV, GEMM, and merge sort. These workloads were implemented with single-precision floating-point data, input tiling, double buffering between DRAM and accelerator scratchpad or TCDM, and a dedicated DMA engine. AXPY exposes copy, mapping, bandwidth, and synchronization costs; GEMM provides a high-arithmetic-intensity workload; HEAT3D stresses memory-intensive stencil execution; GESUMMV represents matrix-vector traffic; and merge sort exposes irregular memory behavior and synchronization (Koenig et al., 24 Feb 2025).

A separate GPU-offload study selected seven RAJAPerf applications:

• DEL_DOT_VEC_2D
• ENERGY
• FIR
• LTIMES
• MATVEC_3D_STENCIL
• PRESSURE
• VOL3D

The corresponding profiled kernels include deldotvec2d, energycalc1 through energycalc6, fir, ltimes, matvec3d, pressurecalc1, pressurecalc2, and vol3d. The benchmark loop counts follow RAJAPerf defaults and range from 50 to 700 kernel launches. ENERGY contains six kernels sharing many arguments, making it a difficult case for automatic transfer-elimination heuristics because preloading all arguments could reduce transfers while substantially increasing GPU memory consumption (Drehwald et al., 13 Aug 2026).

The workload diversity is important for performance-portability analysis. A framework may perform well on regular vector loops but poorly on sorting, reductions, memory-bound stencils, or repeated offload sequences. This suggests that suite-wide averages should be supplemented with per-class and per-kernel results.

3. Evaluation methodology and metrics

RAJAPerf has been used to compare execution models, compilers, vectorization strategies, memory systems, cache organizations, thread placement, and data-movement policies. Common experimental procedures include running each benchmark repeatedly, averaging execution times, measuring single-thread and multithreaded behavior, and reporting the range across kernels within each class.

For thread scaling, speedup is defined as

$S(n)=\frac{T(1)}{T(n)},$

where $T(1)$ is one-thread execution time and $T(n)$ is execution time with $n$ threads. Parallel efficiency is

$PE(n)=\frac{S(n)}{n} =\frac{T(1)}{nT(n)}.$

An ideal $n$-thread result has $S(n)=n$ and $PE(n)=1$. Because class averages are used in some studies, individual aggregates can produce values above one.

In heterogeneous offloading experiments, application runtime is decomposed into host-to-device copying or mapping, launch and synchronization, device computation, and device-to-host copying. Accelerator execution is further divided into DMA waiting and computation:

$T_{\mathrm{accelerator}}=T_{\mathrm{DMA\mathchar`-wait}}+T_{\mathrm{compute}}.$

This decomposition distinguishes generated kernel quality from offload-interface overhead and data movement. In the Rust GPU-offload evaluation, reported runtime generally includes repeated kernel launches, synchronization, and kernel execution while excluding initial and final bulk transfers. The study consequently separates kernel-level timing from whole-application runtime (Drehwald et al., 13 Aug 2026).

For configurable systems, RAJAPerf can also provide the measurement platform for testing whether a configuration changes execution time, throughput, bandwidth, memory consumption, or scaling. PerfSense, although not evaluated on RAJAPerf, is designed to identify candidate performance-sensitive configurations through call-graph analysis, prompt chaining, and retrieval-augmented generation. A plausible integration is to use its predictions to prioritize RAJAPerf configuration sweeps, then validate them through controlled measurements (Wang et al., 2024).

4. Performance-portability and processor studies

RISC-V processor evaluation

RAJAPerf was used to evaluate the 64-core Sophon SG2042 against other RISC-V and x86 processors. The SG2042 contains four-core clusters of XuanTie C920 cores, four NUMA regions, a 64 MB shared system or L3 cache, a 1 MB L2 cache shared by each four-core cluster, and 128-bit RVV 0.7.1 vector registers.

Relative to a VisionFive V2 SiFive U74 single core, one C920 achieved approximately 4.3–6.5 times the performance in FP64 and 5.6–11.8 times the performance in FP32 across the six RAJAPerf classes. The memory-set benchmark reached approximately 40 times the FP32 performance and 18 times the FP64 performance. The C920 was not reported as slower than the U74 on any kernel in that comparison.

The SG2042’s scaling was strongly dependent on thread placement. With contiguous block placement, 64-thread speedups were only 0.97 for Algorithm and 1.77 for Stream. NUMA-cyclic placement improved 64-thread speedups to 2.09 for Algorithm, 17.29 for Basic, 26.53 for Lcals, and 57.93 for Polybench. Cluster-aware cyclic placement produced 31.76 times speedup for Polybench at 32 threads and 58.26 times speedup at 64 threads. Stream reached 15.18 times speedup at 32 threads but fell to 1.51 times at 64 threads.

The study attributes these effects to memory-controller imbalance, shared-cache contention, and the SG2042 topology. Numerically adjacent core identifiers do not necessarily belong to the same NUMA region, so simple contiguous placement can concentrate traffic in one memory region. This demonstrates that RAJAPerf can expose system-level effects that are not apparent from single-core kernel measurements (Brown et al., 2023).

Compared with x86 systems, the SG2042 generally delivered lower average performance in multithreaded workloads. AMD Rome, Intel Broadwell, and Intel Ice Lake were approximately 5–8 times faster on average, depending on precision and processor. The SG2042 nevertheless exceeded the aggregate performance of the four-core Sandy Bridge system, and some individual regular vectorizable kernels ran faster on the C920.

Heterogeneous RISC-V SoC evaluation

RAJAPerf workloads were also used to evaluate IOMMU-based shared virtual addressing and zero-copy offloading in an embedded heterogeneous RISC-V SoC. The system combines a 64-bit Linux-capable host, a 32-bit eight-core scratchpad-based accelerator, a RISC-V IOMMU, heterogeneous OpenMP target offloading, and FPGA emulation with configurable DRAM latency.

The study compared an IOMMU-disabled physical-address baseline, an IOMMU-enabled configuration without a shared last-level cache, and an IOMMU-enabled configuration with a shared LLC used by the host and IOMMU page-table walker while accelerator DMA bypassed the LLC.

For AXPY with 32,768 elements, zero-copy offloading was 47% faster than copy-based offloading. Without an LLC, IOMMU translation overhead for GEMM ranged from approximately 4.2% at low memory latency to 17.6% at high memory latency. With the shared LLC, the corresponding overhead fell to 0.4–0.7%. Across GEMM, GESUMMV, HEAT3D, and merge sort, the shared LLC reduced the incremental IOMMU cost to below 2% for all selected kernels.

The results show that RAJAPerf can characterize the interaction between arithmetic intensity, DMA behavior, IOTLB capacity, page-table-walk latency, and cache placement. GEMM’s tiled reuse hides much of the translation cost, whereas HEAT3D, GESUMMV, and merge sort are more sensitive to memory-system overhead (Koenig et al., 24 Feb 2025).

5. RAJA and execution-model comparisons

The original performance-portability evaluation compared seven implementations of a particle-in-cell hotspot: sequential C++, OpenMP, OpenACC, CUDA, Kokkos, hybrid OpenMP/Kokkos, and RAJA. Although the study was not a RAJAPerf evaluation, its results provide context for the kinds of behavior RAJAPerf is intended to measure.

The RAJA implementation used RAJA::forall, RAJA::RangeSegment, OpenMP execution policies, CUDA execution policies, views over explicitly allocated storage, CUDA atomics for GPU vector reduction, and per-thread private arrays followed by a final CPU reduction. The reported CUDA policy used RAJA::cuda_exec\<256>, corresponding to 256 threads per CUDA block.

RAJA provided algorithmic portability across CPU and GPU implementations, but the studied version required explicit memory management, architecture-dependent policy selection, and a hand-built vector reduction. On the CPU, false sharing in private state and reduction storage impaired performance. An early RAJA implementation produced approximately 18 GB of last-level-cache hits in a ten-core false-sharing analysis, compared with approximately 200 MB for OpenMP and 10 MB for Kokkos. A per-thread contiguous memory pool improved the placement of private state, but RAJA remained the weakest CPU-scaling implementation in that experiment (Artigues et al., 2019).

The study reported competitive GPU behavior for RAJA. Relative to OpenACC on a K40m, RAJA was 2.02 times slower; on a P100, it was 2.18 times slower than CUDA; and on a V100, it was 2.51 times slower than CUDA. RAJA was comparable to or slightly better than Kokkos across the GPU generations in that kernel. These results illustrate that portable performance is backend- and workload-dependent: RAJA’s GPU behavior was credible, while its CPU behavior was more sensitive to application-managed memory layout and reduction design.

The qualitative comparison rated RAJA highly for portability, medium for code clarity and productivity, and medium for performance in the studied PIC kernel. The authors favored Kokkos for a full particle-in-cell implementation because of its execution-space and memory-space integration, scratch memory, ScatterView, host mirrors, and lower configuration burden. This conclusion does not establish that Kokkos is generally superior to RAJA across RAJAPerf; it is specific to the evaluated particle-loop, scratch-memory, and vector-reduction workload.

6. GPU offload, Rust, and transfer behavior

A Rust GPU-offload framework was evaluated by porting a subset of RAJAPerf to pure Rust and comparing it with native RAJA CUDA and HIP implementations. The framework was implemented in rustc and LLVM, used LLVM Offload infrastructure, and targeted nvptx64-nvidia-cuda and amdgcn-amd-amdhsa.

The evaluation separated generated GPU kernel quality from host-device data movement and offload-runtime overhead. On the AMD MI250X, optimized Rust was between 32% faster and 43% slower than RAJA-HIP when total runtime was considered. Rust was faster than RAJA-HIP for FIR, LTIMES, and VOL3D in the supplied runtime table.

On the NVIDIA H100, kernel-level results were generally close to RAJA CUDA. Rust was approximately 19.6% faster for pressurecalc2, 4.1% faster for matvec3d, and marginally faster for several ENERGY kernels. It was slower for FIR, LTIMES, deldotvec2d, VOL3D, and several other small kernels. The paper attributes the clearest differences to unrolling and vectorization decisions in instruction-light kernels.

The whole-application results were more sensitive to transfers and synchronization. In one H100 experiment, Rust transferred 423 MB host-to-device and 69 MB device-to-host, compared with 468 MB and 99 MB for RAJA. Despite transferring 75 MB less, Rust transfer time was 46 ms versus RAJA’s 16 ms. The result indicates that transfer volume alone does not determine transfer time; memory kinds and asynchronous-transfer behavior also matter (Drehwald et al., 13 Aug 2026).

The framework offered three data-management interfaces. Compiler-managed offload automatically allocates device storage, transfers arguments, launches kernels, and synchronizes. Explicit preloading keeps data resident across multiple kernels. A deliberately naive interface that transferred around every kernel launch could be more than 400 times slower than explicit data management in a repeated-kernel workload. ENERGY, with six kernels and shared arguments, demonstrated the difficulty of automatically determining which data should remain resident.

These findings reinforce the value of RAJAPerf for evaluating the complete execution path rather than only device kernel duration. A programming model can generate competitive GPU instructions while still incurring substantial end-to-end overhead through unnecessary transfers, synchronization, or runtime decisions.

7. Applications, limitations, and research use

RAJAPerf is used in at least four related research roles. First, it serves as a benchmark suite for performance-portability frameworks such as RAJA and for comparisons with OpenMP, CUDA, Kokkos, HIP, and Rust GPU offload. Second, it provides a processor-evaluation workload spanning RISC-V and x86 architectures. Third, it exposes heterogeneous memory-system behavior, including IOMMU translation, shared virtual addressing, zero-copy offloading, DMA, and cache effects. Fourth, it supplies repeated configurable workloads suitable for studying performance-sensitive configurations and regression-testing strategies.

Its principal methodological strength is kernel diversity. Memory-bound, compute-bound, stencil, reduction, sorting, streaming, and application-like kernels respond differently to vector width, compiler maturity, cache capacity, NUMA placement, thread count, data movement, and synchronization. Consequently, RAJAPerf can reveal performance regressions or portability failures that a single kernel would miss.

Several limitations constrain interpretation. Studies commonly evaluate only subsets of the 64 kernels. Problem sizes, compiler flags, runtime settings, device drivers, memory configurations, and exact command lines are not always fully reported. Some results are embedded in figures rather than tabulated. Hardware comparisons may use different compilers or toolchains, and FPGA studies use modeled memory latency rather than production silicon. In the Rust evaluation, the source data contain duplicated profiling tables and an unexplained discrepancy between some raw runtime values and percentage summaries. Such issues require separating directly reported measurements from aggregate interpretations.

RAJAPerf should therefore be treated as a multidimensional experimental instrument rather than as a single score. Appropriate reporting includes per-kernel and per-class execution time, speedup, parallel efficiency, scaling curves, transfer and compute decomposition, memory footprint, cache or coherence effects, compiler and backend configuration, and workload-specific sensitivity. A plausible implication is that future evaluations would benefit from systematic parameter sweeps over thread placement, vectorization, tile size, memory layout, accelerator launch configuration, IOMMU settings, and transfer policies, together with controlled compiler studies and uncertainty estimates.

For configuration-aware performance engineering, a static or LLM-assisted method such as PerfSense could identify candidate parameters, while a data-flow-based method such as ConfFLARE could select configurations implicated by changes to performance-relevant code. Neither paper reports RAJAPerf-specific experiments: PerfSense evaluates seven open-source Java systems, while ConfFLARE evaluates synthetic systems, GNU Coreutils, bzip2, grep, and picoSAT. Their relevance to RAJAPerf is therefore methodological rather than empirical (Wang et al., 2024, Böhm et al., 21 Nov 2025). RAJAPerf can provide the controlled measurements needed to determine whether predicted sensitivity is realized for a specified kernel, workload, hardware platform, compiler, and runtime configuration.