GADGET-4: High-Fidelity Astrophysical Simulation

• GADGET-4 is a robust simulation code with modules for gravity, SPH hydrodynamics, time integration, and on-the-fly analysis, essential for cosmic structure studies.
• It employs diverse solvers such as Tree, TreePM, Cartesian FMM, and FMM–PM, ensuring efficient force calculations and high parallel scalability across large domains.
• The integrated KETJU module extends simulation capabilities by accurately resolving SMBH dynamics with post-Newtonian corrections and specialized region-based integration.

GADGET-4 is a sophisticated and scalable codebase designed for high-fidelity simulation of cosmic structure formation, gravitational dynamics, and hydrodynamics in astrophysical systems. Written in C++ with MPI and MPI-3 shared-memory parallelization, GADGET-4 enables simulations of extremely large problem sizes and high dynamic range, with core modules for gravity, smoothed particle hydrodynamics (SPH), time integration, and on-the-fly analysis. Its extensible design, open-source distribution, and integration of advanced solvers have established GADGET-4 as a principal tool for the astrophysical and cosmological simulation community. Recent developments, such as the KETJU module, further extend its capabilities to resolve supermassive black hole (SMBH) dynamics beyond classical softened $N$-body resolution limits (Springel et al., 2020, Mannerkoski et al., 2023).

1. Architectural Structure and Workflow

GADGET-4 is structured around core modules for gravity (Tree, TreePM, FMM, FMM+PM), SPH hydrodynamics, time integration, and analysis, coordinated through a single executable and a flexible parameter file. The workflow encompasses:

• Parsing command-line arguments and the main parameter file.
• Optional initial-conditions (IC) generation via 2LPT or Zel'dovich approximation.
• A three-stage domain decomposition: oct-tree to leaf, Peano–Hilbert domain segmentation, and MPI rank bin-packing for balanced load in gravity and SPH computations.
• Local oct-tree construction per domain, with shared MPI-3 windows facilitating exchange of tree data.
• Main simulation loop: timebin activation, particle drift (with on-demand synchronization), force computation (gravity and SPH), leapfrog integration (kick–drift–kick), on-the-fly analysis routines (FOF groups, SUBFIND, merger trees, power spectra, and lightcones), and data output.

Hybrid hierarchical/global timestepping allows adaption to disparate timescales, efficiently integrating fast-evolving subsystems without unnecessary recomputation for the global particle set (Springel et al., 2020).

2. Gravity and Hydrodynamics Solvers

GADGET-4 incorporates four families of gravity solvers:

• Barnes–Hut Tree: Single-sided multipole expansions up to order $p$ about node center of mass; nodes are opened based on geometric and relative error criteria.
• TreePM: Hybrid Particle-Mesh (PM) for long-range ($k$-space, FFT) and tree for short-range, with a force split at a tunable scale $r_s$.
• Cartesian FMM: Dual-tree walk, symmetric expansions at both sources and sinks, exact momentum conservation, and $O(N)$ complexity for fixed accuracy.
• FMM–PM: Combines PM for long-range and FMM for short-range computation.

SPH hydrodynamics is handled in both “density-formulation” (entropy-conserving) and pressure-based (P-SPH) variants. Kernels include cubic $M_4$ B-spline and higher-order Wendland functions. Artificial viscosity follows Monaghan and time-variable $\alpha_i(t)$ prescriptions. Additional modules enable atomic cooling (H/He), ISM models, and stochastic star formation (Springel et al., 2020).

3. Supermassive Black Hole Dynamics and the KETJU Module

The KETJU module augments GADGET-4 for SMBH dynamics at parsec and subparsec scales. Its principal innovations include:

• Hybrid Region-Based Integration: For each SMBH, spherical "KETJU regions" (radius $r_{\rm ketju}$) are evolved using an algorithmically regularised $N$-body integrator (MSTAR) in place of the standard leapfrog, enabling pure Newtonian (unsoftened) interactions for SMBHs and nearby stars.
• Post-Newtonian Corrections: Up to order 3.5PN (including leading-order spin-orbit and spin–spin terms), with exact N-body non-linear cross-terms at 1PN for multiple SMBH systems.
• Seamless Embedding: Within each GADGET-4 step, region particles are transferred to the MSTAR integrator via libmstar, with negative kicks to subtract region–region interactions (preserving energy-momentum consistency), Gragg–Bulirsch–Stoer extrapolation for time integration, and position/velocity re-synchronization to the global scheme.
• Handling of Region Overlaps: Overlapping regions are merged; MPI resources are distributed based on computational cost.
• Parameterization: Exposed controls in parameterfile.txt allow tuning of $r_{\rm ketju}$, PN order, Gragg–Bulirsch–Stoer tolerances, and region-based timestep limits (Mannerkoski et al., 2023).

KETJU thus allows studies of SMBH binary evolution, hardening, and merger with accuracy down to tens of Schwarzschild radii—removing classical $N$-body resolution limits imposed by gravitational softening.

4. Directory Structure and Code Modularity

Integration of KETJU modifies the GADGET-4 tree as follows:

Directory	File(s)	Function
src/ketju/	ketju.h/.c/.region	Region definition, parameter and interface routines, region search, negative kick application, data pack/unpack, HDF5 output.
src/ketju/	ketju_pn.c/.h	Wrappers for post-Newtonian routines and merge conditions.
libmstar/	mstar.h/.c	MSTAR AR-CHAIN integrator, GBS time-extrapolation, dynamic step order.
libmstar/	pn_terms.c/.h	1PN–3.5PN and spin cross-term accelerations.
src/	time_integration.c	Insertion points for region finding, kicking, integration, and data reconciliation.
src/	timestep.c, allvars.h, param.c	Handles region-based timestep logic and parameter import.

This modular approach permits standalone compilation of libmstar for use in other $N$-body codes (e.g., AREPO, RAMSES) following a minimal interface contract.

5. Gravitational Softening and Force Management

In standard GADGET-4, all gravitational interactions are softened via the cubic spline kernel $W(r, \epsilon)$ up to $2.8\epsilon$. In the KETJU module:

• SMBH–star and SMBH–SMBH interactions within regions are computed with zero softening (pure Newtonian potentials).
• Optionally, star–star interactions inside each region remain softened using GADGET-4's kernel, avoiding potential discontinuities in the potential at region boundaries.
• The KETJU region radius ($r_{\rm ketju}$) must satisfy $r_{\rm ketju} \geq 2.8\,\epsilon$ for force continuity (Mannerkoski et al., 2023).

6. Parallelization, Scalability, and Performance

GADGET-4 employs a hybrid MPI plus shared-memory model for strong and weak scaling to massively parallel systems. Key features include:

• Hierarchical domain decomposition based on multiple workload estimates (number of particles, gravity cost, SPH cost), spatially compact Peano–Hilbert domains, and bin-packing to MPI ranks.
• Shared-memory windows for low-latency data sharing of large static tables (Ewald kernels, FFT grids).
• Efficient tree-packing and communication schemes to avoid global synchronization.
• FFT parallelization using slab or column methods based on mesh/cpu ratio.
• Optional vectorization (SSE/AVX2/AVX512) in intensive SPH kernels (Springel et al., 2020).

With strong and weak scaling demonstrated for cosmological and zoom-in volumes, GADGET-4 sustains high parallel efficiency at scale.

7. Practical Operation, Analysis Tools, and Extensibility

Obtaining, building, and running GADGET-4 (with or without KETJU) involves:

• Download from public repositories.
• CMake or custom build configuration, with Makefile/KETJU additions for libmstar and relevant compilation flags.
• Parameter file specification for physics modules, solver accuracy, SPH settings, and region-based options for KETJU.

On-the-fly modules include analysis packages for dark matter halos (FOF), substructure (SUBFIND), merger trees, lightcone outputs, power spectrum estimation, and IC generation—all accessible through parameter switches (Springel et al., 2020).

KETJU/libmstar can be ported to other $N$-body codes with only minimal data exchange logic, provided region definition, COM transformation, and operator-splitting are properly handled. This further broadens the impact and integration potential of the GADGET-4 ecosystem for high-resolution dynamical studies (Mannerkoski et al., 2023).