Kokkos Programming Library

• Kokkos is a performance-portable C++ programming model for writing parallel algorithms, data structures, memory placement, and execution policies across heterogeneous computing systems, ranging from CPUs to GPUs. It supports various backend such as OpenMP, CUDA, HIP, SYCL, greatly facilitating cross-architecture compatibility.
• It provides abstractions such as `Kokkos::View`, parallel execution policies like `parallel_for` and `parallel_reduce`, and tools for handling memory spaces and scratch memory. Features like `TeamPolicy` enable hierarchical parallelism, and integration with asynchronous runtimes like HPX allows for complex task scheduling.
• Applications for KOKKOS are diverse, including sparse and dense linear algebra, particle methods, molecular dynamics, and scientific simulations. Optimization involves because architecture-specific policies and understanding data layouts, making Kokkos a versatile tool for domain-specific libraries and production scientific applications.

Kokkos is a C++ performance-portability programming model and library for expressing parallel algorithms, data structures, memory placement, and execution policies across heterogeneous computing systems. It provides abstractions including Kokkos::View, execution and memory spaces, parallel_for, parallel_reduce, parallel_scan, RangePolicy, MDRangePolicy, TeamPolicy, scratch memory, atomics, and SIMD types. The same source can target CPUs and accelerators through backends such as Serial, OpenMP, HPX, CUDA, HIP, and SYCL. Kokkos is used both as an application-level programming model and as the foundation for domain-specific libraries, notably Kokkos Kernels and the Kokkos-based particle library Cabana.

1. Architecture and programming model

Kokkos separates execution spaces, which specify where computation runs, from memory spaces, which specify where data reside. CUDA and HIP execution spaces target NVIDIA and AMD GPUs, respectively; OpenMP, Serial, C++ threads, and HPX execution spaces target CPU-oriented execution; and SYCL provides an additional accelerator backend. Architecture-specific configuration is selected at build time through Kokkos options such as Kokkos_ARCH_SKX, Kokkos_ARCH_Volta70, Kokkos_ARCH_Zen3, Kokkos_ARCH_AMD_GFX942, or Kokkos_ARCH_AMPERE80.

A Kokkos kernel combines a parallel pattern, a computational body, and an execution policy. parallel_for expresses independent work, parallel_reduce combines values generated by parallel iterations, and parallel_scan implements prefix-scan operations. RangePolicy represents a one-dimensional iteration range; MDRangePolicy represents multidimensional iteration spaces; and TeamPolicy expresses hierarchical parallelism over teams, team threads, and vector lanes.

Kokkos::View is a multidimensional array abstraction that carries information about scalar type, rank, layout, memory space, constness, and ownership. Managed Views allocate and automatically manage associated storage; unmanaged Views can wrap externally allocated memory. LayoutLeft and LayoutRight determine the mapping from multidimensional indices to physical memory. DualView maintains host and device representations, while deep_copy and related operations explicitly synchronize data. ScatterView supports concurrent accumulation into vectors or other destinations by using atomic or duplicated partial storage.

Kokkos also provides scratch memory at team and thread scope. This enables temporary data to be staged in hardware- or software-managed local memory, such as GPU shared memory, AMD local data share, or CPU cache-like workspaces. The appropriate use of layouts, policies, scratch memory, and memory spaces remains application- and architecture-dependent; portability of source code does not imply uniform performance without tuning.

The abstraction is intended to replace direct dependence on a single programming model. A conventional loop such as

$6.6\times$0

can be expressed as:

$6.6\times$1

The selected backend determines how the operation is compiled and executed. In production applications, however, achieving good performance may require architecture-specific policies, data layouts, scratch-memory configurations, SIMD types, or algorithmic variants.

2. Hierarchical parallelism, SIMD, and asynchronous execution

TeamPolicy is used when work requires cooperation among threads, team-private storage, or hierarchical decomposition. A team can process a block of particles, a finite-element tile, a sparse matrix row, or a group of small dense matrices. Nested constructs such as TeamThreadRange, TeamVectorRange, and ThreadVectorRange expose additional parallel levels.

Octo-Tiger combines Kokkos with the HPX asynchronous many-task runtime. Kokkos supplies portable numerical kernels and memory abstractions, while HPX supplies futures, continuations, asynchronous scheduling, distributed components, and communication. HPX-Kokkos converts completion of CUDA, HIP, or SYCL operations into HPX futures, avoiding blocking fences that would otherwise occupy worker threads during accelerator execution. This allows GPU kernels, CPU tasks, memory transfers, communication, and dependent computation to coexist in one task graph (Daiß et al., 2022, Daiß et al., 2023, Daiß et al., 2024).

Kokkos supports explicit SIMD through Kokkos::Experimental::simd. CPU instantiations can use AVX, AVX512, NEON, SVE, or related vector types, while GPU instantiations generally use scalar types because GPU execution already supplies thread-level parallelism. In Octo-Tiger, Kokkos SIMD and std::experimental::simd were used to address incomplete SVE support on Fujitsu A64FX systems. A new explicit SVE backend supplied operations through ARM’s SVE ACLE interface, and SIMD masks were added to hydrodynamics kernels with branching behavior (Daiß et al., 2022).

This layered execution model is particularly relevant to irregular applications. Octo-Tiger uses HPX to schedule many small adaptive-mesh tasks, Kokkos to implement portable kernels, SIMD types for CPU vectorization, and dynamic aggregation to combine small GPU operations into larger launches. The resulting architecture separates task scheduling from local kernel execution rather than requiring Kokkos alone to manage distributed asynchronous work.

3. Domain-specific libraries and software ecosystem

Kokkos Kernels extends the programming model with performance-portable sparse linear algebra, dense and batched linear algebra, graph algorithms, sorting, and team-level utilities. It accepts Kokkos Views and dispatches according to their execution and memory spaces. A two-layer design separates user-facing View normalization from implementation dispatch among inline kernels, explicit template instantiation kernels, and third-party library kernels.

Kokkos Kernels integrates vendor libraries including MKL, IBM ESSL, OpenBLAS, cuBLAS, cuSPARSE, MAGMA, and rocSPARSE where advantageous, while retaining native implementations for unsupported scalar types, layouts, architectures, problem sizes, nested execution, and specialized fused operations. Its sparse structures include compressed row storage through CrsMatrix; its dense interfaces support generic scalar types, including automatic-differentiation and ensemble types; and its batched interfaces expose Serial, Team, and TeamVector execution levels (Rajamanickam et al., 2021).

The library implements SpMV, SpGEMM, SpAdd, and SpTRSV, together with structured stencil kernels, fused Jacobi-SpGEMM, supernodal triangular solves, distance-2 coloring, maximal independent set algorithms, and bitonic sorting. Symbolic and numeric sparse phases can be separated so that output structures are reused across repeated computations. Fused operations eliminate intermediate matrices, kernel launches, and possible communication. In the reported fused Jacobi-SpGEMM experiments, the fused implementation was 9–41% faster than the non-fused baseline.

Cabana provides Kokkos-based particle data structures, slices, linked-cell lists, permutation and sorting operations, particle migration, MPI communication, and grid support through Cajita. It has been used for Multi-Particle Collision Dynamics, where particles are grouped into shifted collision cells and cell-level quantities are reduced before collision updates. In that study, GPU execution scaled across up to 16 JUWELS booster nodes, while the Cabana CPU implementation remained substantially slower than the mature Fortran baseline. Concurrent CPU–GPU execution did not outperform pure GPU execution in the tested implementation (Halver et al., 2022).

Other Kokkos-based libraries and applications include the following:

Area	Examples	Principal contribution
Sparse and dense kernels	Kokkos Kernels	Portable linear algebra, graph, sorting, and utility algorithms
Particle methods	Cabana, MPCD, PIC-related implementations	Particle arrays, cell lists, reductions, migration, and communication
Molecular dynamics	LAMMPS KOKKOS	Portable pair potentials, many-body force fields, machine-learned potentials, and GPU-resident execution
Scientific and astrophysical simulation	AthenaK, Octo-Tiger, SACRA-K	AMR, relativistic hydrodynamics, gravity, radiation, and numerical relativity
Legacy-code modernization	NICAM and Fortran-to-Kokkos workflows	Translation of CPU- and OpenACC-oriented scientific applications

LAMMPS uses Kokkos Views, DualViews, ScatterViews, execution policies, scratch memory, and architecture-dependent layouts to support Lennard-Jones, ReaxFF, SNAP, and Moment Tensor Potential calculations across NVIDIA, AMD, and Intel systems. Its design aims to keep neighbor-list construction, force evaluation, communication packing, and accumulation on the device, reducing host–device transfers (Johansson et al., 19 Aug 2025).

4. Applications and performance evidence

Kokkos has been evaluated in computationally diverse workloads whose behavior ranges from regular and bandwidth-bound to irregular, reduction-heavy, and communication-intensive.

A particle-in-cell study implemented a representative Hamiltonian-splitting operator from the GEMPIC discretization of the Vlasov–Maxwell system. The kernel located particles, evaluated spline basis functions, updated particle positions and velocities, deposited current, and reduced particle contributions into a global current vector. Kokkos used TeamPolicy, scratch-backed temporary variables, Views, and ScatterView for vector accumulation. Although Kokkos was not always the fastest kernel implementation, it offered the best overall basis for the full PIC model because it combined CPU/GPU performance, automatic memory management, a single-source model, PIC-specific reduction support, and comparatively low code complexity (Artigues et al., 2019).

In empirical dynamic modeling, kEDM replaced separate ArrayFire GPU and OpenMP CPU implementations with a single Kokkos codebase. It fused delayed embedding with pairwise-distance construction, implemented a custom small-$k$ selection algorithm, used team-local scratch memory, batched lookups across target series, and avoided materializing predictions when only Pearson correlation was required. On the reported workloads, kEDM achieved up to $6.6\times$ faster pairwise distance calculation, up to $6.2\times$ faster partial selection for small embedding dimensions, and up to $5.5\times$ faster convergent cross mapping than mpEDM (Takahashi et al., 2021).

The nanoBragg X-ray tracing application was ported from CUDA to Kokkos and executed through CUDA on NVIDIA A100 GPUs and HIP on AMD MI250X GPUs. The same Kokkos source ran on both systems. On Perlmutter, the Kokkos implementation was approximately 13% faster than the original CUDA implementation; on Crusher, it achieved more than 60% higher performance per node than Perlmutter. Kernel-level profiling associated the improvement on A100 with reduced register use, increased occupancy, and higher compute throughput (Wittwer et al., 2022).

AthenaK is a ground-up Kokkos implementation of the Athena++ block-structured AMR framework. It aggregates blocks into MeshBlockPacks to reduce kernel-launch overhead, keeps computational fields device-resident, and performs ghost-zone operations, AMR interpolation, restriction, and finite-volume updates on the device. Reported single-device hydrodynamics performance reached 1.077 billion cell updates per second on a Grace Hopper processor. The implementation achieved approximately 70% efficiency on 4096 A100 GPUs and approximately 80% efficiency on 65,536 AMD GPU compute dies in the reported weak-scaling experiments (Stone et al., 2024).

SACRA-K translated a Fortran numerical-relativity code into C++/Kokkos while retaining BSSN and Z4c spacetime evolution, relativistic hydrodynamics, Berger–Oliger AMR, and MPI decomposition. Device-resident fields and GPU-aware MPI avoided repeated host staging. On the smallest tested configuration, four A100 GPUs achieved a $13.8\times$ speedup over a 72-core CPU node, while four MI300A APUs achieved a $15.4\times$ speedup. Strong-scaling efficiency declined to approximately 10% at 256 APUs because ghost-zone and communication overhead dominated, whereas weak-scaling efficiency reached 77–83% at 256 devices for representative local problem sizes (Han et al., 9 Jul 2026).

A high-order discontinuous Galerkin fluid solver combined Kokkos with MPI across NVIDIA and AMD GPUs and many-core CPUs. The solver exhibited good GPU performance and MPI weak scaling, with GPUs becoming advantageous for workloads exceeding approximately $10^7$ degrees of freedom. CPU execution was faster for small grids. The study also showed that Kokkos OpenMP policy and layout choices could be suboptimal on CPUs even when the same choices performed well on GPUs (Sporykhin et al., 6 Oct 2025).

5. Performance portability and tuning

Performance portability in Kokkos has several distinct meanings. Source portability means that one implementation can be compiled for multiple execution spaces. Functional portability means that the same numerical method runs correctly across these backends. Performance portability requires useful performance across architectures and therefore depends on data layout, execution policy, memory hierarchy, synchronization, compiler behavior, and workload size.

A comparison of naïve dense matrix multiplication found that Kokkos/OpenMP reached efficiencies of 0.994 in FP64 and 1.014 in FP32 relative to C/OpenMP on AMD EPYC, but only 0.854 and 0.836 on an Ampere Altra CPU. Kokkos/HIP reached efficiencies of 0.842 in FP64 and 0.677 in FP32 relative to native HIP on an AMD MI250X, while Kokkos/CUDA reached only 0.260 in FP64 and 0.208 in FP32 relative to native CUDA on an NVIDIA A100. The study used hand-written naïve kernels rather than optimized vendor BLAS routines, but it illustrates that backend quality and policy selection strongly affect performance (Godoy et al., 2023).

Sparse tensor decomposition provides another example. In SparTen’s CP-APR MU algorithm, the $\Phi^{(n)}$ computation accounted for approximately 81% of the execution time among the four most expensive kernels and was strongly memory-bound. Kokkos policy grid search over league, team, and vector sizes produced average speedups of $2.25\times$ on CPUs and $1.70\times$ on GPUs relative to default SparTen policies. Pressure Point Analysis indicated that cache reuse and regular memory access were more important than atomic overhead on the CPU, while removing atomics reduced GPU performance (Anderson et al., 2023).

Low-order matrix-free finite-element kernels show why identical optimizations can have opposite effects on different devices. Team scratch memory reduced global-memory traffic by staging shared degrees of freedom, but team size, tile shape, occupancy, register pressure, and atomic accumulation required separate tuning. A per-team accumulator reduced atomics from 18,432 to 2,475 for a representative tile. It reduced throughput on MI250X, where atomics were comparatively efficient, but improved throughput on MI300A, where coherent atomic traffic was more expensive (Böhm et al., 18 Sep 2026).

Kokkos therefore provides tuning levers rather than a universal optimization policy. Important parameters include:

• Memory layout: LayoutLeft, LayoutRight, structure-of-arrays, array-of-structures, and hybrid layouts determine coalescing, cache reuse, and SIMD access.
• Execution policy: range, multidimensional, and team policies expose different amounts of hierarchical parallelism.
• Team and vector size: larger teams may improve reuse but increase scratch and register pressure; smaller teams may fail to amortize launch and staging overhead.
• Scratch memory: staging can reduce global loads but lower occupancy or cause resource exhaustion.
• Atomic strategy: atomics, per-thread duplication, per-team accumulation, and subsequent reductions have architecture-dependent costs.
• Work aggregation: combining small operations can improve GPU utilization, particularly in AMR and particle workloads.
• Data residency: keeping state and communication buffers on the device can eliminate substantial transfer and synchronization overhead.
• Backend configuration: architecture flags, launch bounds, compiler options, SIMD extensions, and device-aware MPI can materially change performance.

6. Limitations, modernization, and future directions

Kokkos does not guarantee peak hardware performance, deterministic reduction order, or identical numerical trajectories across architectures. Reduction order can vary, floating-point contraction can differ, and backend compilers can generate materially different code. These effects are particularly important in chaotic simulations such as atmospheric dynamics and binary-neutron-star mergers.

The NICAM migration illustrates both the potential and the limits of automated modernization. An AI coding agent translated approximately 140,000 lines of Fortran into approximately 35,000 lines of C++/Kokkos covering the dynamical core and seven physics packages. A 100-member ensemble comparison found that 95.3% of 5,900 member–quantity scores lay within two standard deviations of the reference ensemble. The Kokkos/CUDA build was 1.71 times faster than the Fortran/OpenACC baseline on GH200, and the Kokkos/HIP build was 1.54 times faster on MI300A relative to that baseline. Nevertheless, the workflow required human architectural decisions, structural review, component tests, short integrations, ensemble validation, and backend-specific tuning (Ren et al., 30 Oct 2026).

A related autonomous Fortran-to-Kokkos workflow used specialized agents to translate, compile, test, debug, and optimize benchmark kernels. GPT-5 and o4-mini-high completed the full workflow for the evaluated kernels and platforms, while Llama 4 Maverick frequently failed to produce functional implementations. The work demonstrates an engineering loop around Kokkos rather than a guarantee of autonomous scientific correctness: compiler feedback, runtime failures, functionality tests, profiler output, and human-defined constraints remain central (Gupta et al., 15 Sep 2025).

Energy analysis adds another dimension to performance portability. Kokkos Performance Tools and the PAPI Connector can associate kernel regions with floating-point operations, memory traffic, execution time, GPU energy, temperature, and execution order. In LOBPCG, FP32 generally produced the most favorable performance–energy behavior, mixed FP32/FP64 retained similar arithmetic intensity with modest penalties, and FP64 generally reduced throughput and increased energy. In ArborX, precision effects varied by kernel and workload, demonstrating that performance and energy decisions should not be made solely from datatype names or temperature measurements (Patrou et al., 22 Oct 2026).

Current limitations identified across Kokkos studies include incomplete backend feature parity, architecture-specific tuning requirements, host–device data conversions, communication overhead at strong scaling, limited support for some numerical algorithms, and insufficiently standardized performance models. Applications may need specialized kernels, vendor-library dispatch, explicit SIMD backends, asynchronous runtimes, memory pools, dynamic aggregation, and selective backend-specific code while retaining Kokkos as the common programming model.

Kokkos is consequently best understood as a layered portability foundation. It supplies portable data structures, execution policies, memory abstractions, parallel patterns, scratch storage, reductions, scans, and backend selection. Domain libraries such as Kokkos Kernels and Cabana provide reusable operations and data structures. Application frameworks such as AthenaK, LAMMPS KOKKOS, Octo-Tiger, SACRA-K, NICAM, and kwqft add domain-specific data ownership, communication, scheduling, numerical methods, and validation. The combined evidence shows that Kokkos can support production scientific applications across NVIDIA, AMD, Intel, ARM, x86, and RISC-V environments, but that effective performance portability depends on deliberate co-design of algorithms, data movement, memory hierarchy, communication, and hardware-aware execution policies.