XGYRO: Ensemble Fusion Plasma Simulation

• XGYRO is a simulation tool that runs multiple CGYRO instances concurrently by sharing the static collisional tensor (cmat) to reduce memory footprint.
• It decouples the streaming and collisional MPI communicators, lowering latency during streaming without affecting the collisional phase.
• Benchmark results on OLCF Frontier show up to 1.5× speedup overall and 4.4× reduction in streaming communication time through efficient tensor sharing.

Searching arXiv for XGYRO and related CGYRO papers to ground the article in current literature. XGYRO is a tool for first-principles fusion plasma simulation ensembles that executes multiple CGYRO runs as a single HPC job and exploits ensemble-level structure to reduce memory consumption and communication overhead (Sfiligoi et al., 29 Jul 2025). Its central mechanism is the sharing of the collisional constant tensor structure, denoted cmat, across simulations whose collision-operator inputs are identical. Because cmat is typically the dominant memory object in CGYRO, distributing one shared instance across the entire ensemble lowers per-simulation memory and, by changing communicator structure, reduces the cost of communication-intensive phases that would otherwise be incurred independently by each run (Sfiligoi et al., 29 Jul 2025).

1. Computational setting and the role of cmat

XGYRO is defined relative to CGYRO, which is an Eulerian, electromagnetic, multi-species gyrokinetic solver for fusion plasmas that implements the complete Sugama gyrokinetic theory and advances the distribution function with implicit-explicit time integration (Sfiligoi et al., 29 Jul 2025). In the formulation described for XGYRO, each time step is organized into three logical phases: streaming (str), non-linear (nl), and collisional (coll). The streaming phase applies linear operators that require the full configuration-space extent; the non-linear phase performs mode coupling in the toroidal Fourier index; and the collisional phase performs the velocity-space implicit solve using a precomputed constant tensor (Sfiligoi et al., 29 Jul 2025).

Most CGYRO arrays are rank-3 tensors with dimensions labeled by configuration $n_c$, velocity $n_v$, and toroidal index $n_t$. The collisional constant tensor cmat is rank-4 with shape $(n_v \times n_v \times n_c \times n_t)$ and is computed once per simulation, then reused at every collisional step. In the nl03c benchmark, the memory footprint of cmat is about $10\times$ the sum of all other buffers, which is the quantitative fact that motivates XGYRO’s design (Sfiligoi et al., 29 Jul 2025).

The collision operator is described schematically as

$$C_a[h_a] = \sum_b C_{ab}[h_a], \qquad C_{ab}[h_a] = \nu_{ab} L_{ab}[h_a] + \text{field-particle terms},$$

where $h_a$ is the non-adiabatic part of the perturbed distribution, $\nu_{ab}$ are inter-species collision frequencies, and $L_{ab}$ includes pitch-angle scattering and energy diffusion operators in velocity space. In discrete tensor notation,

$$(C h)_\alpha(x,k) = \sum_\beta C_{\alpha\beta}(x,k)\, h_\beta(x,k),$$

with $n_v$0 indexing velocity-space degrees of freedom, $n_v$1 indexing configuration points, and $n_v$2 indexing toroidal modes. The collisional constant tensor is therefore

$n_v$3

It is “constant” because it does not change during time stepping and depends only on static inputs and the discretized velocity-space grid (Sfiligoi et al., 29 Jul 2025).

A common misconception is that XGYRO is a general ensemble scheduler for all CGYRO state. The paper’s description is narrower: all per-simulation data structures other than cmat remain independent; only cmat is shared (Sfiligoi et al., 29 Jul 2025). This suggests that XGYRO is best understood as an ensemble-aware data-layout and communicator transformation rather than a new physics model or a replacement for CGYRO itself.

2. Conditions for tensor sharing across an ensemble

XGYRO relies on the observation that many parameter-sweep ensembles change turbulence-drive parameters without changing the subset of inputs that determine cmat (Sfiligoi et al., 29 Jul 2025). The paper emphasizes that “only a subset of the input parameters influences [cmat]” and that “its values are typically identical between parameter-sweep simulations.” Sharing is therefore valid only when that subset is identical across the ensemble.

The parameters that must be identical are the species set and species properties, including the number of species $n_v$4, masses $n_v$5, and charges $n_v$6; the normalization or base thermodynamic state used to form $n_v$7, including reference densities $n_v$8, temperatures $n_v$9, and the Coulomb logarithm $n_t$0; the collision-operator options; the velocity-space discretization; discretization details relevant to $n_t$1 such as basis, stencils, and operator-splitting choices; and the toroidal/configuration indexing consistency when cmat is stored per $n_t$2 slice (Sfiligoi et al., 29 Jul 2025).

By contrast, profile gradients such as $n_t$3 and $n_t$4, time-step control, total runtime, output cadence, and non-collisional physics toggles may vary without changing cmat, provided they do not alter the base state or collision-operator definition. Resolution changes in $n_t$5 or $n_t$6 that affect non-collisional phases can also vary, so long as the velocity-space grid and operator options remain unchanged (Sfiligoi et al., 29 Jul 2025).

Input category	Sharing requirement
Species, masses, charges	Must be identical
Reference $n_t$7, $n_t$8, and $n_t$9	Must be identical
Collision model and options	Must be identical
Velocity-space grid and quadrature	Must be identical
Profile gradients and runtime controls	May vary
Non-collisional toggles	May vary if they do not alter the collision operator

The practical implication is restrictive but clear. Ensembles that scan base density or temperature generally invalidate sharing, whereas ensembles that scan gradients while holding the reference state fixed are natural XGYRO candidates (Sfiligoi et al., 29 Jul 2025). Another common misunderstanding is that any parameter sweep can benefit from XGYRO; the paper explicitly does not support that conclusion.

3. Ensemble architecture and communicator decomposition

XGYRO wraps an ensemble of CGYRO simulations into one parallel job and changes only the way cmat is managed: cmat is constructed once for the entire ensemble and distributed across all ranks, while all other CGYRO arrays retain the standard per-simulation distribution (Sfiligoi et al., 29 Jul 2025). Achieving this requires decoupling the MPI communicator used in the streaming phase from the communicator used in the collisional phase.

Let $(n_v \times n_v \times n_c \times n_t)$0 denote the total number of MPI ranks and $(n_v \times n_v \times n_c \times n_t)$1 the number of simulations in the ensemble. For the collisional phase, XGYRO uses a single communicator nv_comm of size $(n_v \times n_v \times n_c \times n_t)$2, distributing cmat slabs along $(n_v \times n_v \times n_c \times n_t)$3 across all ranks. For the streaming phase, it creates $(n_v \times n_v \times n_c \times n_t)$4 disjoint communicators str_comm_s, each of size $(n_v \times n_v \times n_c \times n_t)$5, one for each simulation $(n_v \times n_v \times n_c \times n_t)$6. All str-phase collectives, including the AllReduce operations for field and upwind terms and the str↔coll transpose for tensors other than cmat, are executed within these smaller communicators (Sfiligoi et al., 29 Jul 2025).

The construction sequence is correspondingly specialized. cmat is computed once using the shared inputs that define the collision operator and the velocity-space grid; because cmat is needed per toroidal slice, construction can be looped over $(n_v \times n_v \times n_c \times n_t)$7 indices and partitioned in $(n_v \times n_v \times n_c \times n_t)$8 for distribution. The global tensor is then distributed across all ranks in nv_comm, with each rank storing only its local $(n_v \times n_v \times n_c \times n_t)$9 slice for each toroidal index. A barrier on nv_comm ensures that the shared tensor is fully built and distributed before any simulation enters its first collisional step. During time stepping, each simulation binds its own state to its str_comm_s communicator and to views of the shared distributed cmat on nv_comm (Sfiligoi et al., 29 Jul 2025).

This architecture preserves the locality assumptions that CGYRO expects in the collisional solve, while shrinking the participant count of the expensive str-phase collectives. The resulting decomposition is unusual in that the ensemble is treated as the scheduling unit for one phase and as independent simulations for another. A plausible implication is that XGYRO’s benefit depends less on altering numerical kernels than on changing where and at what communicator scale those kernels synchronize.

4. Memory and communication model

The memory model in the paper is framed in terms of $10\times$0, $10\times$1, $10\times$2, the number of species $10\times$3, the element size $10\times$4, the total rank count $10\times$5, and the ensemble size $10\times$6 (Sfiligoi et al., 29 Jul 2025). The total size of cmat is approximated as

$10\times$7

If $10\times$8 is the local configuration extent in the collisional phase, then the per-rank cmat storage for a baseline single CGYRO simulation over $10\times$9 ranks is

$$C_a[h_a] = \sum_b C_{ab}[h_a], \qquad C_{ab}[h_a] = \nu_{ab} L_{ab}[h_a] + \text{field-particle terms},$$0

Without sharing, $$C_a[h_a] = \sum_b C_{ab}[h_a], \qquad C_{ab}[h_a] = \nu_{ab} L_{ab}[h_a] + \text{field-particle terms},$$1 independent runs require total cmat memory that scales as $$C_a[h_a] = \sum_b C_{ab}[h_a], \qquad C_{ab}[h_a] = \nu_{ab} L_{ab}[h_a] + \text{field-particle terms},$$2. With XGYRO, cmat is built once and distributed across all $$C_a[h_a] = \sum_b C_{ab}[h_a], \qquad C_{ab}[h_a] = \nu_{ab} L_{ab}[h_a] + \text{field-particle terms},$$3 ranks. If $$C_a[h_a] = \sum_b C_{ab}[h_a], \qquad C_{ab}[h_a] = \nu_{ab} L_{ab}[h_a] + \text{field-particle terms},$$4 denotes the per-simulation memory of all non-collisional buffers and

$$C_a[h_a] = \sum_b C_{ab}[h_a], \qquad C_{ab}[h_a] = \nu_{ab} L_{ab}[h_a] + \text{field-particle terms},$$5

at $$C_a[h_a] = \sum_b C_{ab}[h_a], \qquad C_{ab}[h_a] = \nu_{ab} L_{ab}[h_a] + \text{field-particle terms},$$6, then the per-simulation footprint under sharing is

$$C_a[h_a] = \sum_b C_{ab}[h_a], \qquad C_{ab}[h_a] = \nu_{ab} L_{ab}[h_a] + \text{field-particle terms},$$7

and the reduction factor relative to the baseline single-simulation footprint is

$$C_a[h_a] = \sum_b C_{ab}[h_a], \qquad C_{ab}[h_a] = \nu_{ab} L_{ab}[h_a] + \text{field-particle terms},$$8

For nl03c, where $$C_a[h_a] = \sum_b C_{ab}[h_a], \qquad C_{ab}[h_a] = \nu_{ab} L_{ab}[h_a] + \text{field-particle terms},$$9, the paper gives, for $h_a$0,

$h_a$1

This is explicitly a per-simulation reduction driven purely by cmat sharing (Sfiligoi et al., 29 Jul 2025).

The communication analysis uses

$h_a$2

where $h_a$3 is the latency term, $h_a$4 the inverse bandwidth, and $h_a$5 an algorithm-dependent scaling factor such as $h_a$6. For baseline CGYRO on $h_a$7 ranks, the streaming phase is modeled as

$h_a$8

Under XGYRO, each simulation performs its streaming-phase AllReduce operations over $h_a$9 ranks, so

$\nu_{ab}$0

The collisional phase, however, still uses nv_comm with $\nu_{ab}$1 and therefore does not receive the same communicator-scaling benefit (Sfiligoi et al., 29 Jul 2025).

The key point is not merely that memory decreases, but that the dominant latency contribution of the str phase decreases with communicator size. This suggests that XGYRO’s performance gain is structurally tied to the particular decomposition of CGYRO phases; it is not a generic claim that all collective communication is accelerated.

5. Frontier benchmark and measured performance

The reported benchmark was run on OLCF Frontier using 32 nodes per job. A single CGYRO nl03c simulation requires at least 32 nodes. The experiment compared eight variants of the nl03c benchmark in two modes: “CGYRO sum,” defined as sequential execution of the eight simulations, each alone on 32 nodes, with per-reporting-step times summed; and “XGYRO,” defined as concurrent execution of an ensemble with $\nu_{ab}$2 as a single job on the same 32 nodes, sharing cmat (Sfiligoi et al., 29 Jul 2025).

Metric at reporting step $\nu_{ab}$3	CGYRO sum	XGYRO
Total time	375 s	250 s
str communication component	145 s	33 s
Derived factor	—	$\nu_{ab}$4 total speedup; $\nu_{ab}$5 str reduction

From these values, the total speedup is

$\nu_{ab}$6

For the dominant streaming communication component, the reduction factor is

$\nu_{ab}$7

The paper states that the coll and nl components showed smaller differences, and that the str-phase AllReduce dominates the observed improvement, matching the communicator-scaling argument (Sfiligoi et al., 29 Jul 2025).

The paper does not report numerical memory counters on Frontier, but it reiterates that for nl03c the constant cmat is approximately $\nu_{ab}$8 larger than all other buffers combined. Using the model above, an ensemble with $\nu_{ab}$9 lowers the per-simulation memory footprint by about $L_{ab}$0, which is presented as enabling higher concurrency on the same node count (Sfiligoi et al., 29 Jul 2025). This is an inference from the model rather than an in situ memory measurement.

6. Practical workflow, limitations, and prospective extensions

XGYRO is described as a thin MPI initialization and partitioning layer around the CGYRO codebase, with minor modifications to separate the str and coll communicators and expose cmat as a globally shared distributed object (Sfiligoi et al., 29 Jul 2025). The practical workflow is correspondingly simple in concept: prepare an ensemble of CGYRO input files that keep all cmat-determining inputs fixed; launch XGYRO as a single MPI job; construct and distribute cmat once on nv_comm; and then enter per-simulation time stepping with independent non-cmat state and shared collisional tensor views (Sfiligoi et al., 29 Jul 2025).

The paper also states several implementation boundaries. It does not specify version numbers, build flags, or a public CLI. It describes an illustrative Slurm script, but explicitly notes that the executable name and flags are not specified in the paper. It also does not describe a concrete built-in enforcement mechanism for checking cmat identity across runs, although it suggests that a practical approach would be to hash or canonicalize the subset of inputs that determine cmat and either proceed when all hashes match or fall back to per-simulation construction if a mismatch is detected (Sfiligoi et al., 29 Jul 2025).

The limitations are substantive. Simulations must share all inputs that affect cmat; otherwise sharing would be incorrect. There is a one-time overhead to construct and distribute cmat; the collisional phase remains at full $L_{ab}$1-rank scale; concurrent execution increases aggregate I/O; and changes in communicator sizes and reduction order may introduce roundoff-level differences relative to independent runs. If memory is provisioned assuming sharing but sharing is disabled at runtime because of an input mismatch, the job may exceed memory limits unless concurrency is reduced (Sfiligoi et al., 29 Jul 2025).

The future-work directions identified in the paper are specific rather than open-ended: extending sharing to other large time-invariant operator blocks; partitioning heterogeneous ensembles into compatibility classes with distinct shared-cmat pools; topology-aware mapping to align cmat slabs and per-simulation compute with sockets or GPUs; portability enhancements using GPU-aware distributions and GPUDirect collectives; and automated pre-launch checking of the cmat invariants (Sfiligoi et al., 29 Jul 2025). Taken together, these indicate that XGYRO is not merely an ensemble launcher but a template for ensemble-level reuse of static operators in large-scale kinetic plasma codes.