---
title: 'SACRA-MPI: Hybrid MPI in Relativistic Hydrodynamics'
url: https://www.emergentmind.com/topics/sacra-mpi
type: topic
---

# SACRA-MPI: Hybrid MPI in Relativistic Hydrodynamics

"SACRA-MPI" (*Editor's term*) denotes the MPI-oriented parallelization and communication design of the SACRA family of fully relativistic hydrodynamics codes. In the available literature, this designation is most directly grounded in two developments: the original SACRA code, which was compared quantitatively with Whisky for binary neutron-star mergers and described as optimized for low-memory, moderate processor environments rather than massive parallel scaling, and SACRA-2D, a new MPI and OpenMP parallelized code for axisymmetric general relativistic hydrodynamics in dynamical spacetime with fixed mesh refinement and the cartoon method [1007.1754; 2502.03223]. In this lineage, MPI is coupled to AMR or FMR layout, buffer-zone synchronization, conservation control across refinement boundaries, and the broader question of how hybrid node-level parallelism should be organized on contemporary multicore systems.

## 1. Code lineage and computational positioning

The original SACRA and SACRA-2D occupy different points in the SACRA code lineage. SACRA, as characterized in the binary neutron-star comparison with Whisky, uses the BSSN formalism, Piecewise Parabolic Method reconstruction, a Kurganov–Tadmor central scheme for hydrodynamic fluxes, and a vertex-centered Berger-Oliger AMR scheme with nested, moving and fixed grids. It typically uses 7 or 8 refinement levels, with about \(7 \times 10^6\) grid points and \(\sim 12\) GB RAM for high resolution, and it was explicitly contrasted with Whisky’s heavier MPI-oriented large-cluster usage [1007.1754].

SACRA-2D is a later, explicitly hybrid design: an MPI and OpenMP parallelized, fully relativistic hydrodynamics code in dynamical spacetime under axial symmetry, using finite-volume shock-capturing hydrodynamics, the Baumgarte-Shapiro-Shibata-Nakamura formalism with Z4c constraint propagation, and fixed mesh refinement. Its implementation makes MPI part of the primary decomposition strategy rather than a secondary execution layer [2502.03223].

| Code | Parallel/computational profile | Numerical scope |
|---|---|---|
| SACRA | Optimized for low-memory, moderate processor environments rather than massive parallel scaling | 3D general-relativistic hydrodynamics with AMR for binary neutron-star mergers |
| SACRA-2D | MPI and OpenMP parallelized with domain-based decomposition and fixed mesh refinement | Axisymmetric GRHD in dynamical spacetime with the cartoon method |

This progression suggests a shift from demonstrating that SACRA could deliver reliable physics with modest hardware to engineering a more explicit hybrid MPI/OpenMP execution model for long-duration, axisymmetric relativistic simulations.

## 2. Numerical foundations of the SACRA MPI lineage

In SACRA, the Einstein equations are evolved in the BSSN formalism, with the conformal factor represented as \(x = e^{-2\phi}\). The hydrodynamic evolution uses conserved variables \(D\), \(S_i\), and \(E\), reconstructing \(p\), \(\tilde{u}_i\), and \(h\). AMR prolongation and restriction are performed on \(D\), \(\tilde{u}_i\), and \(h\), with 5th order Lagrange interpolation downgraded to linear if unphysical values arise. Gauge conditions are the \(1+\log\) slicing for the lapse and a hyperbolic \(\Gamma\)-driver for the shift, with \(\eta \sim 3/M_b\), and the code adopts reflection symmetry across the \(z=0\) plane [1007.1754].

SACRA-2D preserves the emphasis on fully relativistic evolution but modernizes several components. It uses a finite-volume HRSC scheme, 3rd-order PPM reconstruction, HLLC or TVDLF Riemann solvers, sixth-order central finite differences for spacetime evolution, lopsided schemes for advection, eighth-order Kreiss-Oliger dissipation, and fourth-order Runge-Kutta time stepping with Berger-Oliger subcycling for FMR. Axisymmetry is enforced with the cartoon method, using extra \(y\)-layers with rotation and interpolation [2502.03223].

A central methodological distinction in SACRA-2D is the explicit comparison between HLLC and TVDLF. The HLLC solver restores the contact discontinuity and was reported to yield better accuracy than the standard TVDLF solver, resolving neutron-star surfaces and contact discontinuities much more sharply and reducing artificial surface heating and mass drift by orders of magnitude [2502.03223]. This is numerically significant because the communication pattern imposed by MPI decomposition interacts directly with the stencil width, ghost-zone depth, and the robustness of interfacial flux reconstruction.

## 3. MPI decomposition and hybrid execution in SACRA-2D

SACRA-2D uses a domain-based decomposition for MPI parallelization. Each refinement level is divided into \(M_{\mathrm{MPI}} \times M_{\mathrm{MPI}}\) blocks of subdomains when mirror symmetry with respect to the \(z=0\) plane is assumed; without that symmetry, the decomposition is \(M_{\mathrm{MPI}} \times 2M_{\mathrm{MPI}}\). The integer \(M_{\mathrm{MPI}}\) must divide \(N\), the number of grid points per direction per refinement level, such that \(N/M_{\mathrm{MPI}}\) is even. The total number of MPI ranks is therefore \(M_{\mathrm{MPI}}^2\), or \(M_{\mathrm{MPI}} \times 2M_{\mathrm{MPI}}\) if symmetry is not exploited [2502.03223].

OpenMP hybridization is layered on top of this layout. Each MPI process can further use \(N_{\text{thr}}\) OpenMP threads, so the maximum number of cores is \(M_{\text{MPI}}^2 \times N_{\text{thr}}\). The code achieves best efficiency when \(N/M_\mathrm{MPI} \gtrsim 30\), and when the number of OpenMP threads per process satisfies \(N_{\rm thr} \lesssim N/M_\mathrm{MPI}/10\) [2502.03223].

The load-balancing model is static. Each subdomain is assigned to an MPI rank in a regular fashion, and there is no dynamic run-time load balancing. The stated rationale is that balancing is ensured if work per block is approximately the same, which is true for uniform problems [2502.03223]. A common misunderstanding is that such a decomposition automatically guarantees good performance for all relativistic problems. The published characterization is narrower: it is acceptable for uniform problems, while problems with localized activity may produce some imbalance.

## 4. Communication, ghost regions, and refinement interfaces

MPI communication in SACRA-2D is organized around ghost or buffer zones. Each subdomain includes buffer cells—typically 8 per direction—to support high-order differences, reconstruction, and FMR time interpolation over prolongation zones. After each RK4 stage and after updates to grid variables, the buffer-region values at subdomain boundaries are synchronized between adjacent MPI processes via standard MPI communication [2502.03223].

Communication also occurs at refinement boundaries. At FMR interfaces, interpolations are performed for buffer zones between parent and child grids, and those grids may reside on different MPI processes. Metric variables use high-order interpolation, whereas hydrodynamic variables are transferred with conservative summing. The fine-to-coarse conservative restriction is written as
\[
\mathbf{q}^{(l-1)}_{j, k}
=
\frac{1}{\Delta V^{(l-1)}_{j,k}}
\sum_{m = 2j-1}^{2j} \sum_{n = 2k-1}^{2k}
\mathbf{q}^{(l)}_{m, n} \Delta V^{(l)}_{m,n}.
\]
In addition, SACRA-2D records fluxes across fine/coarse boundaries during RK4 and applies a conservative flux fix after time matching to preserve conservation on the coarse grid [2502.03223].

These design choices are integral to the code’s numerical behavior rather than merely infrastructural details. In relativistic hydrodynamics, refinement interfaces are potential sources of secular error in rest mass and angular momentum; the SACRA-2D communication scheme therefore couples MPI synchronization tightly to conservative update rules.

## 5. Accuracy, validation, and resource economics

The original SACRA was validated through a quantitative cross-code comparison with Whisky for binary neutron-star mergers. The paper focused on rest mass, total mass energy, total angular momentum, and the gravitational-wave amplitude and frequency, and reported agreement at a reasonable level, with variations in different quantities but always at better than about \(10\%\). Rest-mass conservation in SACRA at high resolution was \(\sim 10^{-3}\), total-energy conservation was at the \(\sim 0.1\text{–}1\%\) level after merger, and total angular momentum was conserved to \(\sim 1\%\) at the highest resolution. Gravitational-wave amplitudes agreed to \(\lesssim 10\%\), mainly at merger, while inspiral differences were much smaller [1007.1754].

The same study also documented a striking computational contrast. In a high-resolution piecewise-polytropic run, SACRA used 7 million points, 11.6 GB RAM, and \(\sim 2000\) CPU-hours on a 4-core Intel i7, whereas Whisky used 22 million points, 640 GB, and \(\sim 450\) CPU-hours on 320 cores [1007.1754]. The comparison does not imply that SACRA had the stronger large-scale MPI implementation; rather, it indicates that SACRA’s design was resource-frugal while remaining scientifically reliable.

SACRA-2D extends the validation record with benchmark tests and application studies including head-on black-hole collision, neutron-star migration and collapse, and rotating supermassive-star collapse to a massive black hole and a disk. The code was reported to show 6th-order convergence for metric evolution and gravitational-wave extraction, conservation of mass and angular momentum down to \(\lesssim 10^{-10}\) for long evolutions, and strong scaling up to 5120 cores with 70% parallel efficiency for small numbers of OpenMP threads [2502.03223]. A representative efficiency datum is the supermassive-star collapse run with \(N=128\), 10 refinement levels, and \(4 \times 4 \times 5 = 80\) cores, costing only \(\sim 600\) CPU-hours [2502.03223].

## 6. Hybrid MPI context and interpretive significance

The SACRA code family sits within a broader hybrid-MPI design space that has been analyzed explicitly in contemporary MPI research. One line of work studies MPI+Threads endpoint allocation on Mellanox mlx5 InfiniBand and shows that the two conventional extremes—one communication endpoint per process in the MPI-everywhere model and a single shared endpoint for all threads in MPI+threads—are both inefficient in different ways. Dedicated endpoints maximize performance but can incur up to 93.75% hardware resource wastage on ConnectX-4, while a fully shared endpoint can cause up to a 7x reduction in throughput. The proposed scalable endpoints model introduces six categories—MPI-everywhere, 2xDynamic, Dynamic, Shared Dynamic, Static, and MPI+threads—and reports that 2xDynamic can achieve 108% of MPI-everywhere performance with only 31.25% of hardware resources for a 16-thread Global Array benchmark [2002.02509].

A second line of work studies collectives in hybrid MPI+MPI code and proposes context-based collective communication using MPI-3 shared memory windows. Its design splits communicators into a node-local shared-memory communicator, `shmem_comm`, and an inter-node leader communicator, `bridge_comm`, so that leaders execute inter-node collectives while all on-node processes read and write directly in shared memory. The resulting wrappers—such as `Wrapper_Hy_Allgather`, `Wrapper_Hy_Bcast`, and `Wrapper_Hy_Allreduce`—avoid on-node buffer replication and on-node MPI message passing, and the paper reports 27–83% Allreduce gains over pure `MPI_Allreduce` for large systems and/or larger message sizes [2007.11496].

These studies do not describe SACRA or SACRA-2D directly. Their significance is interpretive. They show that hybrid MPI/OpenMP or MPI+MPI codes cannot be analyzed solely in terms of rank count; communication endpoints, on-node shared-memory semantics, and collective organization can dominate performance. This suggests that SACRA-2D’s explicit MPI and OpenMP structure should be read as part of a broader movement away from undifferentiated “MPI everywhere” assumptions toward node-aware, resource-aware communication design.

## 7. Scientific role and limitations

The scientific role of SACRA-MPI is anchored in the fact that numerical relativity and GRHD codes must satisfy simultaneously demanding requirements in physics fidelity, conservation, and computational efficiency. In the original SACRA, this was demonstrated by cross-code agreement with Whisky on conserved quantities and gravitational-wave signals. In SACRA-2D, it is expressed through a hybrid MPI/OpenMP implementation coupled to fixed mesh refinement, conservative refinement-boundary updates, and modern HRSC components such as HLLC [1007.1754; 2502.03223].

The limitations are equally explicit in the published record. SACRA was not presented as a massively parallel MPI-centric code in the style of Whisky; SACRA-2D uses static block assignment and does not mention dynamic load balancing; and hybrid-MPI studies in related literature show that endpoint and collective design remain nontrivial performance bottlenecks even when numerical kernels are otherwise well optimized [1007.1754; 2502.03223; 2002.02509; 2007.11496]. A plausible implication is that future SACRA-family developments will continue to hinge on the co-design of relativistic numerics and communication structure rather than on either component in isolation.

Source: https://www.emergentmind.com/topics/sacra-mpi