KHARMA: Flexible, Portable Performance for GRMHD

Abstract: KHARMA (an acronym for "Kokkos-based High-Accuracy Relativistic Magnetohydrodynamics with Adaptive mesh refinement") is a new open-source code for conducting general-relativistic magnetohydrodynamic simulations in stationary spacetimes, primarily of accretion systems. It implements among other options the High-Accuracy Relativistic Magnetohydrodynamics (HARM) scheme, but is written from scratch in C++ with the Kokkos programming model in order to run efficiently on both CPUs and GPUs. In addition to being fast, KHARMA is written to be readable, modular, and extensible, separating functionality into "packages," representing, e.g., algorithmic components or physics extensions. Components of the core ideal GRMHD scheme can be swapped at runtime, and additional packages are included to simulate electron temperature evolution, viscous hydrodynamics, and for designing chained multi-scale "bridged" simulations. This chapter presents the computational environment and requirements for KHARMA, features and design which meet these requirements, and finally, validation and performance data.