The Athena++ Adaptive Mesh Refinement Framework: Design and Magnetohydrodynamic Solvers (2005.06651v1)

Abstract: The design and implementation of a new framework for adaptive mesh refinement (AMR) calculations is described. It is intended primarily for applications in astrophysical fluid dynamics, but its flexible and modular design enables its use for a wide variety of physics. The framework works with both uniform and nonuniform grids in Cartesian and curvilinear coordinate systems. It adopts a dynamic execution model based on a simple design called a "task list" that improves parallel performance by overlapping communication and computation, simplifies the inclusion of a diverse range of physics, and even enables multiphysics models involving different physics in different regions of the calculation. We describe physics modules implemented in this framework for both non-relativistic and relativistic magnetohydrodynamics (MHD). These modules adopt mature and robust algorithms originally developed for the Athena MHD code and incorporate new extensions: support for curvilinear coordinates, higher-order time integrators, more realistic physics such as a general equation of state, and diffusion terms that can be integrated with super-time-stepping algorithms. The modules show excellent performance and scaling, with well over 80% parallel efficiency on over half a million threads. The source code has been made publicly available.

• The paper introduces Athena++, an advanced adaptive mesh refinement framework that integrates magnetohydrodynamic solvers for efficient simulations.
• It employs high-order numerical methods and various Riemann solvers to accurately capture fluid dynamics and magnetic fields.
• The framework achieves over 80% parallel efficiency, enabling scalable, high-resolution studies of complex astrophysical phenomena.

Athena++: An Adaptive Mesh Refinement Framework for Astrophysical Magnetohydrodynamics

Athena++ presents a comprehensive framework for adaptive mesh refinement (AMR) in computational astrophysics, with a focus on magnetohydrodynamics (MHD). As an extension of the Athena code, Athena++ incorporates a variety of advanced computational strategies and solvers, enabling high-performance simulations across a range of astrophysical phenomena. Here, we provide an overview of the framework's design, numerical methods, and computational performance, with attention to its implications for future astrophysical research.

Framework Design

Athena++ is designed around a block-based AMR strategy, wherein the computational domain is divided into an array of MeshBlocks. These MeshBlocks can be refined into finer blocks, facilitating efficient resolution of complex flow features without uniformly increasing computational cost. This strategy supports both curvilinear coordinates and nonuniform grids, enhancing its applicability across diverse astrophysical scenarios.

The communication between MeshBlocks, particularly at different refinement levels, is carefully managed to maintain consistency and ensure conservation laws are upheld. This is achieved with a mix of restriction, prolongation, and flux/EMF correction procedures, which ensure that quantities such as momentum and magnetic fields are consistently handled across refinement boundaries.

Numerical Methods

Athena++ incorporates a suite of numerical solvers for both non-relativistic and relativistic MHD. Building on the Godunov-type methods traditionally used in Athena, the framework offers second-order unsplit methods for hydrodynamics and MHD, leveraging the constrained transport (CT) scheme to maintain the divergence-free condition of magnetic fields.

The framework supports various Riemann solvers, including HLLE and HLLC for hydrodynamics, and HLLD for MHD, augmented to accommodate a general equation of state. Higher-order spatial reconstructions like piecewise linear method (PLM) and piecewise parabolic method (PPM) are implemented, alongside time-stepping schemes such as the van Leer (VL2) predictor-corrector and higher-order Runge-Kutta methods, allowing for adaptive choice based on the specific simulation requirements.

Performance and Scaling

The performance of Athena++ is a critical feature, ensuring its suitability for simulations on modern high-performance computing architectures. Detailed benchmarks demonstrate the framework's strong scaling properties—exhibiting parallel efficiency of over 80% on systems with hundreds of thousands of cores—which positions it as one of the leading codes in computational astrophysics.

The dynamic task-scheduling via a task list overlaps communication with computation effectively, maximizing hardware utilization, especially important on exascale systems.

Applications and Future Development

Athena++'s extensive capabilities make it particularly well-suited for simulating complex astrophysical processes such as accretion disk dynamics, stellar evolution, and interstellar medium turbulence. Its flexibility allows for the incorporation of advanced physics, including non-ideal MHD processes, self-gravity, and radiative transfer, all of which are crucial for realistic modeling of astrophysical environments.

The ongoing development aims to integrate additional physics modules and optimize performance portability on emerging architectures like GPUs, ensuring Athena++ meets the evolving demands of computational astrophysics research.

In conclusion, Athena++ represents a robust, versatile framework for MHD simulations, serving the astrophysical community's needs for scalable, high-resolution modeling across diverse astrophysical phenomena. Its development continues to be driven by advancements in numerical methods and computational technology, maintaining its place at the forefront of numerical astrophysics.