---
title: 'AthenaK: Portable GRMHD & NR Framework'
url: https://www.emergentmind.com/topics/athenak
type: topic
---

# AthenaK: Portable GRMHD & NR Framework

to=arxiv_search.search  ചികിത_json
{"query":"AthenaK performance portable Athena++ AthenaK numerical relativity GRMHD arXiv", "max_results": 10} สูตรบาคาร่า
to=arxiv_search.search 平台开号 to=arxiv_search.search code  天天中彩票中奖_json
{"query":"ti:\"AthenaK\" OR abs:\"AthenaK\"", "max_results": 20}
to=arxiv_search.search  天天中彩票不能买_json
{"query":"AthenaK", "max_results": 20}
AthenaK is a performance-portable, open-source reimplementation of the Athena++ block-structured adaptive mesh refinement framework in C++ using the Kokkos programming model. It preserves the Athena-family emphasis on Godunov finite-volume methods, staggered constrained transport, and block-based AMR, while extending the framework to Newtonian, special relativistic, and general relativistic hydrodynamics and magnetohydrodynamics, GR radiation transport, particle modules, and—through companion developments—a Z4c numerical relativity solver and GRMHD in dynamical spacetimes. The “K” denotes Kokkos-based performance portability across CPUs, NVIDIA and AMD GPUs, and ARM systems, and the code is explicitly oriented toward exascale workloads [2409.16053].

## 1. Lineage, scope, and design goals

AthenaK descends directly from Athena++, but it is not a minor port. The framework keeps the core Athena++ ideas—block-based AMR, Godunov finite-volume solvers, flux correction at refinement boundaries, and staggered constrained transport—while rewriting the implementation around a device-first execution model. In this reformulation, blocks are grouped into `MeshBlockPack` aggregates to reduce GPU kernel-launch overhead, dependent variables are stored in device-resident Kokkos views, and ghost-zone communication, prolongation, restriction, and boundary-condition application are performed inside device kernels before MPI exchange. The host is retained primarily for mesh-tree creation and refinement, task scheduling, and I/O [2409.16053].

The implemented physics coverage is correspondingly broad. The base framework includes Newtonian hydrodynamics and magnetohydrodynamics, special relativistic HD and MHD, fixed-metric GRHD and GRMHD, GR radiation hydrodynamics and magnetohydrodynamics, and particle modules for Lagrangian tracers and charged test particles. Companion developments add a Z4c solver for the Einstein equations and a Valencia-formulation GRMHD solver in dynamical spacetimes, enabling binary black hole and binary neutron star applications within the same AMR/Kokkos infrastructure [2409.16053].

At the same time, AthenaK is deliberately narrower than some general-purpose mesh frameworks. In the 2024 framework paper, the code is restricted to Cartesian coordinates; characteristic reconstruction is not yet implemented; and the corner-transport upwind integrator is not included. Those constraints are architectural rather than accidental: the code prioritizes a compact, high-throughput, performance-portable implementation of rectangular-block astrophysical fluid dynamics and numerical relativity [2409.16053].

## 2. Framework architecture and numerical methods

AthenaK organizes the mesh as a binary tree, or an octree in three dimensions, with two-to-one refinement and a uniform timestep across refinement levels. Conservation across AMR interfaces is enforced by flux correction. Physics modules are independent objects with their own device-resident state and task lists, which avoids a combinatorial explosion of monolithic task graphs when multiple modules are combined in one run [2409.16053].

The numerical core is a reconstruct–solve–advance finite-volume scheme. Reconstruction options include PLM, classic PPM4, extremum-preserving PPMX, and WENO-Z; Riemann solvers include LLF, HLLE, HLLC, HLLD, and Roe; and time integration uses method-of-lines SSP Runge–Kutta schemes from first to fourth order. In MHD, AthenaK uses the Athena++ upwind constrained-transport algorithm with face-centered magnetic fields and edge-centered electric fields, preserving $\nabla\cdot B = 0$ to roundoff. For stiff multi-fluid couplings, the framework includes IMEX second- and third-order schemes, including the positivity-preserving IMEX2 method of Krapp et al. as summarized in the framework description [2409.16053].

A representative conservative form used throughout the code family is
$$
\partial_t \mathbf{U} + \partial_i \mathbf{F}^i = \mathbf{S},
$$
with module-specific definitions of $\mathbf{U}$, $\mathbf{F}^i$, and $\mathbf{S}$. In fixed-metric GRMHD, AthenaK follows the conservative HARM-like formulation summarized in the framework paper; in dynamical-spacetime GRMHD it uses the Valencia conservative Eulerian form, evolving densitized variables $\sqrt{\gamma}\,\mathbf{U}$ with metric source terms handled consistently in the 3+1 split [2409.16053].

Robustness in extreme flows is strongly tied to first-order flux correction. If an RK substep would generate negative density or pressure, or fail primitive recovery, AthenaK recomputes local interface fluxes with first-order donor-cell reconstruction and a more diffusive LLF or HLLE flux. In the dynamical-spacetime GRMHD work this FOFC mechanism, optionally combined with a relaxed discrete maximum principle, is central to reducing reliance on artificial atmospheres and stabilizing neutron-star merger calculations [2409.10384].

Radiation transport adds another layer of specialization. The finite-solid-angle GR radiation module, originally developed in Athena++ and ported to AthenaK, discretizes the specific intensity in angle, advances it explicitly in space and angle, and couples matter and radiation through a locally implicit source update. The method is designed for stationary spacetimes and highly anisotropic radiation fields, where moment closures such as M1 are inadequate [2302.04283].

## 3. Relativistic extensions and numerical relativity

AthenaK’s numerical relativity module evolves the vacuum Einstein equations in the Z4c formulation. In the implementation described by Zhu et al., Z4c is combined with the moving-puncture gauge, oct-tree AMR, and cell-centered finite-difference operators designed for GPUs. The module demonstrates sixth-order spatial convergence on linear waves when four ghost cells are used, and fourth-order self-convergence for the gravitational waveform in an equal-mass, non-spinning binary black hole calibration problem. New refinement criteria based on $\chi_{\min}$ and $|\nabla\chi|$ focus resolution near punctures and around smaller holes in high-mass-ratio binaries, while wave-zone derefinement controls protect extraction accuracy at large radius [2409.10383].

The GRMHD extension to dynamical spacetimes uses the Valencia conservative Eulerian formulation,
$$
\partial_t(\sqrt{\gamma}\,\mathbf{U}) + \partial_i(\sqrt{\gamma}\,\mathbf{F}^i) = \sqrt{\gamma}\,\mathbf{S},
$$
with conservative variables $\mathbf{U} = [D, S_i, \tau, B^i]$. In this formulation, spacetime variables are cell-centered rather than vertex-centered, which avoids interpolations between metric and fluid layouts and improves GRMHD performance on block-based AMR meshes. Validation spans magnetized shock tubes, cylindrical blast waves, magnetic-loop advection, oscillating and free-evolving TOV stars, and SANE accretion around a Kerr black hole [2409.10384].

These developments culminated quickly in production applications. The GW150914 target simulation performed with AthenaK uses the moving-puncture approach, Z4c evolution, sixth-order finite differencing in space, and a low-storage fourth-order Runge–Kutta scheme in time. Initial data are constructed with TwoPunctures and matched to the LazEV-targeted parameters of Lovelace et al. (2016); horizon quantities are computed with Einstein Toolkit tools; and waveforms are extracted both from finite-radius $\Psi_4$ and via Cauchy Characteristic Extraction to future null infinity $\mathcal{I}^+$. For the final remnant, the code reports $M_f = 0.951948$, $\chi_f = 0.691914$, and $v_{\rm kick} = 138.68\ {\rm km/s}$, with agreement against the highest-resolution SpEC result at the level of $9\times10^{-5}$ in mass, $2\times10^{-4}$ in spin, and about $3\%$ in recoil [2506.06838].

AthenaK’s relativistic reach is not confined to vacuum or compact-binary waveforms. The code has also been used for fixed-spacetime GRMHD simulations of thin accretion disks in Cartesian Kerr–Schild coordinates, where it validated a gravity-dominated geodesic plunge model and found a non-zero $\delta_{\mathcal J}\approx 5.3\%$ angular-momentum drop across the plunging region, together with an order-of-magnitude rise in the local $\alpha$ parameter inside the ISCO [2505.13701].

## 4. Performance portability and scaling

AthenaK’s central technical claim is that high-order astrophysical fluid dynamics, GRMHD, and numerical relativity can be implemented once and run efficiently across heterogeneous architectures. The framework paper reports over one billion cell updates per second for three-dimensional hydrodynamics on a single NVIDIA Grace Hopper processor and a typical parallel efficiency of $80\%$ on 65,536 AMD GPUs on Frontier [2409.16053].

The reported performance metrics span both framework-level and application-level measurements.

| Context | Reported result | Source |
|---|---:|---|
| Single Grace Hopper processor | over one billion cell updates per second for 3D hydrodynamics | [2409.16053] |
| Framework weak scaling on Frontier | typical parallel efficiency of $80\%$ on 65,536 AMD GPUs | [2409.16053] |
| NR module weak scaling on Frontier | $80\%$ efficiency on up to 65,536 AMD MI250X GPUs, relative to 4 GPUs | [2409.10383] |
| NR module strong scaling | $84\%$ on AMD MI250X and $77\%$ on NVIDIA A100 GPUs | [2409.10383] |
| Dynamical-spacetime GRMHD weak scaling | $\geq 80\%$ to 32,768 GPUs and $74\%$ to 65,536 GPUs | [2409.10384] |
| SuperMUC-NG single-node AthenaK study | $\approx 12\times$ throughput gain and $\approx 15\times$ energy-efficiency gain for GPU offload at mesh-block edge length $\geq 24$ | [2606.23265] |

Several details qualify these headline numbers. First, mesh-block granularity matters. In the SuperMUC-NG node-level study, AthenaK’s peak GPU throughput for the GW150914 BBH workload is $\Theta_{\rm GPU,peak}\approx 6\times10^6$ zone-cycles/s and the peak energy efficiency is $\eta_{\rm GPU,peak}\approx 3\times10^4$ zone-cycles/J, but those gains saturate only once the mesh-block edge length reaches about 24. At edge lengths $\leq 10$, GPU occupancy and launch overhead erode both throughput and per-Joule efficiency [2606.23265].

Second, end-to-end production timings are now concrete rather than aspirational. The GW150914 production run on Aurora used 32 nodes, or 192 GPUs, and required about 130 wall-clock hours for a 10-orbit BBH evolution with CCE worldtube output and post-processing. In the paper’s interpretation, modern GPU-accelerated systems are roughly $100\times$ faster than leadership-scale machines available in 2016 for this class of workflow [2506.06838].

## 5. Scientific applications

AthenaK has been used in a notably diverse set of workflows. In binary black hole merger modeling, the GW150914 calculation established a full “Cauchy-to-infinity” pipeline: TwoPunctures initial data, Z4c/moving-puncture evolution with AMR, isolated-horizon remnant analysis, finite-radius $\Psi_4$ extraction, and CCE waveforms at $\mathcal I^+$. The dominant $(\ell,m)=(2,2)$ mode agrees with SXS:BBH:0305 and RIT:BBH:0062 at the level of maximum dephasing $\Delta\phi \simeq 0.35$ and $\simeq 0.2$ at merger, respectively, with nearly flat amplitude differences of $\Delta A/A \simeq 0.2\%-0.4\%$ through the inspiral. The resulting AthenaK waveform was then used in a bilby re-analysis of GW150914, yielding $M_c = 30.7^{+0.6}_{-0.5}\ M_\odot$, $D_L = 460^{+140}_{-140}\ {\rm Mpc}$, and $\iota = 2.7^{+0.3}_{-0.4}\ {\rm rad}$, broadly consistent with semi-analytic LVK models [2506.06838].

In compact-object GRMHD, the code now spans both black-hole accretion and neutron-star mergers. The plunging-region study around a Schwarzschild black hole used AthenaK for a 20,000 $t_g$ thin-disk evolution and found that a gravity-dominated geodesic plunge remains accurate provided non-adiabatic heating is included, plausibly from grid-scale magnetic dissipation in a mid-plane current sheet. The conclusion that constant $\alpha$-disk models are physically inappropriate inside the ISCO follows directly from the measured rise in entropy and magnetic stress across the plunge [2505.13701]. In magnetized binary neutron star mergers, AthenaK has been used with the SFHo tabulated EOS, trapped-lepton advection, WENOZ reconstruction, HLLE, and upwind constrained transport at resolutions down to $\Delta x_{\min}\simeq 92\ {\rm m}$. Between the two highest resolutions the orbital dephasing over more than seven orbits is only $0.06$ radians, magnetic fields are amplified and a polar funnel forms, but baryonic pollution prevents a magnetically dominated outflow by about 30 ms after merger [2511.06058].

A related neutron-star remnant application used AthenaK to study magnetic flux emergence from a twisted toroidal flux tube in a hot massive post-merger star. In that controlled fixed-metric GRMHD experiment, magnetic buoyant emergence occurs only for extremely large fields significantly exceeding $10^{17}\ {\rm G}$, while more typical fields around $10^{16}\ {\rm G}$ are dominated by hydrodynamic effects. The reported outcome places strong limitations on remnant-driven magnetically powered outflows in binary neutron star remnants [2507.18695].

The code is also used well outside numerical relativity. AthenaK has run fully three-dimensional two-fluid MHD turbulence calculations in partially ionized media at $480^3$, showing how velocity, density, and magnetic-field statistics change across neutral–ion coupling regimes and how decoupling occurs over a range of scales rather than at a single ambipolar-diffusion scale [2306.10010]. It has supported a 4096$^3$ multiphase interstellar turbulence simulation on Frontier, with in situ synthetic dust-polarization map generation, reproducing Planck 353 GHz inertial-range slopes $\alpha_{EE}=-2.45$, $\alpha_{BB}=-2.44$, and $\alpha_{TE}=-2.54$, together with $EE/BB=1.88$ and $r^{TE}=0.33$ [2605.06896]. It has also been used for controlled isothermal turbulence experiments comparing Fourier-space driving and point-source driving, where the canonical turbulence parameter $b$ was shown to be degenerate with both the spatial locality and temporal correlation of the forcing [2606.14946].

## 6. Reproducibility, limitations, and current open problems

AthenaK is public infrastructure. The framework is open source under BSD-3 at `https://github.com/IAS-Astrophysics/athenak`, uses a CMake-based build, selects physics and algorithms at runtime through input files, writes MPI-IO output in a compact binary format, and provides Python translators to HDF5. For the GW150914 application, the authors additionally released a step-by-step tutorial with all necessary input files, analysis scripts, and SpECTRE CCE converters at `https://github.com/dradice/athenak-tutorial-gw150914` [2409.16053].

The code’s current limitations are explicit in the literature. At the framework level, AthenaK remains restricted to Cartesian coordinates, does not yet implement characteristic reconstruction or CTU, and has not yet ported some advanced diffusion machinery such as operator-split implicit solvers and super-time-stepping. In the 2024 framework paper, tabulated microphysical EOS support is described as under test rather than mature base functionality [2409.16053]. In the GW150914 production study, only a single AthenaK resolution is used, so the error budget is established by cross-validation against SXS and LazEV rather than a formal convergence sequence; gauge differences and minor AMR backscattering noise around $t\simeq 700\,M$ in some modes are noted explicitly, and more detailed gauge-invariant waveform comparisons are deferred [2506.06838].

More fundamentally, some AthenaK applications have exposed limitations that are physical and numerical rather than purely infrastructural. In turbulent radiative mixing layers, the apparent resolution independence of total cooling is shown to arise from a cancellation between numerical dissipation and numerical viscosity; the paper argues that this is “remarkable, and perhaps fortuitous,” and recommends resolving the turbulent Field length $\lambda_{\rm F,turb}$, defined by $t_{\rm eddy}(\lambda_{\rm F,turb}) = t_{\rm cool}$, rather than treating converged $\dot E_{\rm cool}$ as sufficient evidence of a well-resolved layer [2606.04087]. In magnetized BNS mergers, the failure to achieve $\sigma \gtrsim 1$ in the polar funnel is traced partly to baryon pollution linked to artificial heating of stellar surfaces, and the authors identify improved Riemann solvers, neutrino cooling and heating, and longer-term evolution as plausible next steps; AthenaK developments under test include HLLD and neutrino transport modules labeled M1 and full Boltzmann via FP_N [2511.06058].

Taken together, these papers define AthenaK less as a single application code than as a unified numerical platform. Its distinguishing feature is not merely GPU acceleration, but the combination of block-structured AMR, constrained transport, relativistic fluid solvers, Z4c numerical relativity, and Kokkos-based performance portability in one code base. The published record already shows that this combination is sufficient for many-orbit BBH waveform production, GPU BNS merger calculations, GR radiation transport, exascale interstellar turbulence, and controlled studies of numerical resolution effects across astrophysical fluid dynamics [2409.16053].

Source: https://www.emergentmind.com/topics/athenak