libyt: In Situ Analysis for Exascale Simulations

• libyt is an open-source C library that embeds Python in simulations, enabling real-time analysis and visualization without expensive snapshot I/O.
• It provides a bidirectional interface between simulation codes and Python tools like yt and NumPy, efficiently handling AMR data and derived fields.
• The design emphasizes minimal workflow disruption with a simple two-line conversion from post-processing, achieving 20–45% performance improvements in benchmarks.

Searching arXiv for the libyt paper and closely related in situ/yt context to ground the article in current literature. libyt is an open-source C library for in situ analysis in which Python is embedded into a running simulation so that analysis and visualization operate directly on distributed, in-memory data rather than on snapshots written to and reread from storage. It is designed for high-performance and exascale environments, connects simulation codes in C, C++, or Fortran to yt and other Python packages, and provides both automatic analysis hooks and interactive entry points through a Python prompt or a Jupyter Notebook. A central design objective is “reusability”: a post-processing yt script can be converted to an in situ script with just two lines of change, while preserving the researcher’s existing job submission scripts and ordinary Python environment (Tsai et al., 11 Dec 2025).

1. Purpose, scope, and design goals

libyt was introduced to address the increasing difficulty of handling massive simulation datasets in the exascale computing era. Its core premise is that analysis should occur during simulation runtime, thereby bypassing costly intermediate I/O steps and the storage burden associated with traditional post-processing. In this model, analysis is part of the simulation loop rather than a separate downstream stage (Tsai et al., 11 Dec 2025).

The library places particular emphasis on yt-based volumetric analysis and visualization, but its scope is not restricted to yt. It also supports arbitrary Python packages, including NumPy, matplotlib, and machine learning frameworks. This makes libyt less a single analysis tool than a bidirectional interface between production simulation codes and the Python ecosystem. The paper presents this bidirectionality as a means to reuse existing code paths inside the simulation—for example, user-defined C functions for derived data—while still writing orchestration and visualization logic in Python (Tsai et al., 11 Dec 2025).

A defining design constraint is minimal disruption to established workflows. The stated “two-line change” principle is a concrete expression of this constraint: post-processing scripts written for yt should remain largely intact, and ordinary submission scripts should be reusable. This suggests that libyt is intended not merely as a performance optimization, but as an interface layer that reduces migration cost from file-based analysis to runtime analysis.

2. Runtime architecture and execution model

At runtime, libyt consists of four principal elements: the libyt C library itself, a per-rank libyt Python Module created dynamically, the yt_libyt frontend that maps in-memory data into yt’s dataset abstraction, and a libyt kernel for Jupyter connectivity. The simulation launches $N$ MPI ranks; each rank creates its own embedded Python interpreter and its own libyt Python Module instance, and libyt orchestrates these interpreters synchronously. When Python is executing—whether automatically or interactively—the simulation computation pauses on all ranks, and control returns to the simulation only after Python finishes (Tsai et al., 11 Dec 2025).

Component	Role
libyt	Registers metadata and data pointers, binds C callbacks, and triggers Python or interactive entry points
libyt Python Module	Per-rank runtime “shim” storing dictionaries and exposing C-extension methods
yt_libyt	yt frontend that presents the running simulation as a yt dataset
libyt kernel	Jupyter kernel launched by the simulation for Notebook attachment

A key architectural property is that simulation arrays remain owned by the simulation. Intrinsic field and particle arrays are wrapped as NumPy arrays through the NumPy C API without copying, with shapes, dtypes, and strides supplied so that Python can interpret the memory. When data are virtual or derived and do not exist in memory, Python requests invoke user-defined C functions through libyt’s C-extensions; libyt allocates temporary storage, the callback fills it, and ownership is transferred to Python so that deallocation follows Python’s lifetime rules. Only requested data are prepared, and yt’s chunking is reused to bound both on-demand generation and memory usage (Tsai et al., 11 Dec 2025).

The parallel execution model depends on data redistribution across ranks by MPI one-sided communication. This avoids hand-coded sender/receiver pairs and allows Python code to request data independently of where the data reside. The reported procedure per redistribution epoch is: create a window, attach local buffers, gather window addresses and metadata, open the epoch with MPI_Win_fence, fetch remote data, close the epoch, detach buffers, free structures, and close the window. Very large arrays are automatically split into multiple MPI transfers to avoid the 32-bit count limit (Tsai et al., 11 Dec 2025).

3. Data model, AMR representation, and yt integration

libyt targets AMR simulations and stores the grid hierarchy in arrays aligned to yt’s expectations. Each grid has a unique id, parent id, level, left and right edges, and grid dimensions. When a simulation stores only local hierarchy, libyt gathers the global hierarchy so that all ranks can present a coherent dataset to yt. The hierarchy is held in the hierarchy dictionary of the libyt Python Module, while intrinsic per-grid arrays and particle data are kept in grid_data and particle_data dictionaries keyed by grid id and names (Tsai et al., 11 Dec 2025).

Field metadata are explicitly typed and annotated. The reported descriptors include field_name, field_type, field_dtype, contiguous_in_x, field_unit, and ghost-cell metadata. The allowed field_type values include "cell-centered", "face-centered", and "derived_func". Face-centered data are converted to cell-centered data on demand in Python by averaging across faces, which is done to satisfy yt’s cell-centered operations while using minimal extra memory. Derived fields are generated on request through the derived_func C-extension method (Tsai et al., 11 Dec 2025).

Particle support includes multiple particle types with distinct attributes. Intrinsic attributes may be contiguous or sparse; for sparse or derived attributes, libyt calls a user-provided C function through the get_particle C-extension. The required attributes for each particle type include the position components coor_x, coor_y, and coor_z (Tsai et al., 11 Dec 2025).

The mapping to yt frontends is deliberately conservative. yt_libyt reads param_yt and param_user from the libyt Python Module, inherits fields and units from a simulation’s existing yt frontend if one is available, and augments that frontend with fields registered through libyt. This preserves the semantics of established yt-based post-processing. The canonical conversion from post-processing to in situ consists of exactly two edits:

$20$3

in place of an ordinary yt.load(...) call. The data block further notes that fields provided through libyt can be mixed with yt’s own derived fields, as in a SlicePlot over an intrinsic simulation field and another over ("gas", "velocity_x") (Tsai et al., 11 Dec 2025).

4. API, callbacks, and integration into simulation codes

The C-side API is organized as a per-run workflow. Initialization begins with yt_initialize, which creates the Python interpreter and the libyt Python Module and imports the user’s Python script. Global runtime metadata are then set with yt_set_Parameters, while optional simulation-specific parameters are passed with yt_set_UserParameter*. Metadata for fields, particle types, and grids are populated through yt_get_FieldsPtr, yt_get_ParticlesPtr, and yt_get_GridsPtr, after which yt_commit finalizes registration and binds the data structures into the Python module. Python execution is then triggered by one of the runtime entry points, temporary resources are released by yt_free, and final shutdown is performed by yt_finalize before MPI_Finalize (Tsai et al., 11 Dec 2025).

The documented per-step order is precise:

1. yt_initialize
2. yt_set_Parameters
3. yt_get_FieldsPtr
4. yt_get_ParticlesPtr
5. yt_get_GridsPtr
6. yt_commit
7. Trigger Python with yt_run_Function, yt_run_FunctionArguments, yt_run_InteractiveMode, yt_run_ReloadScript, or yt_run_JupyterKernel
8. yt_free
9. yt_finalize

This sequence is significant because several helper functions become usable only after yt_commit. These include accessors such as yt_getGridInfo_Dimensions, yt_getGridInfo_LeftEdge, yt_getGridInfo_RightEdge, yt_getGridInfo_ParentId, yt_getGridInfo_Level, yt_getGridInfo_ProcNum, yt_getGridInfo_ParticleCount, yt_getGridInfo_FieldData, and yt_getGridInfo_ParticleData (Tsai et al., 11 Dec 2025).

Two callback interfaces are central to on-demand data generation. A derived field generator used with field_type="derived_func" has prototype

$20$4

and must fill contiguous-in-x, ghost-free arrays for the requested grid ids. A particle attribute generator for sparse or derived attributes has prototype

$20$5

These interfaces make it possible to expose simulation-native logic directly to Python without materializing all possible derived quantities in advance (Tsai et al., 11 Dec 2025).

5. Interactive operation, prompts, and Jupyter connectivity

libyt supports three interactive modes of use: automatic invocation of named Python functions, an MPI-aware Python prompt, and a Jupyter Notebook interface. In automatic invocation, the simulation calls an inline Python function at specified times. The system supports a fail-fast mode, where any Python error aborts the run, and a fault-tolerant mode, where errors are recorded and the simulation continues (Tsai et al., 11 Dec 2025).

The interactive prompt is MPI-synchronous. The root rank reads user input, broadcasts it, all ranks execute the code, and outputs are gathered and printed. For batch systems, a file-based prompt is provided: flag files can be created to start, reload, or exit, and outputs are captured to files. This is a practical adaptation to environments in which an attached terminal REPL is inconvenient or impossible (Tsai et al., 11 Dec 2025).

The Jupyter pathway is built around a kernel launched by the simulation on the MPI root. The simulation calls yt_run_JupyterKernel("FLAGFILE", use_connection_file), and the kernel writes libyt_kernel_connection.json and libyt_kernel_pid.txt to a directory designated by the environment variable LIBYT_KERNEL_INFO_DIR. Jupyter Notebook or JupyterLab then attaches through the jupyter_libyt provisioner, and the user selects the “Libyt” kernel. Code execution remains MPI-synchronous, outputs from all ranks are gathered and shown, and tab completion is provided through jedi (Tsai et al., 11 Dec 2025).

These interactive modes come with explicit constraints. Interrupts, restarts, and restart-and-run-all are disabled in the Jupyter interface; “Shutdown” cleanly closes the kernel and returns control to the simulation. Outputs are gathered as plain text, so many rich display extensions based on HTML, SVG, or JavaScript do not render. Large text outputs also slow down output gathering. A plausible implication is that libyt’s interactive design prioritizes deterministic MPI coordination and robustness over parity with a conventional local Jupyter kernel (Tsai et al., 11 Dec 2025).

6. Performance characteristics, demonstrated applications, and limitations

The paper reports both benchmarking and application studies. In a strong scaling benchmark using the Plummer test with GAMER 2.2.0, yt 4.3.1, and libyt 0.2.0, the dataset comprised 3,430 grids with a root grid of $640^3$ and four AMR levels, for a total of $9.0 \times 10^8$ cells. Using identical Python workflows for 2D profiles in post-processing and in situ modes, libyt outperformed post-processing by approximately $20$–$45\%$ across core counts. The reported interpretation is that this gain comes from avoiding reads from disk, while deviations from ideal scaling in both modes arise from yt’s internal bottlenecks, including serial fractions, current indexing, and job partitioning (Tsai et al., 11 Dec 2025).

A convenient notation for speedup and overhead is given as

$S = T_{\mathrm{post}} / T_{\mathrm{in\_situ}}$

and

$$O = (T_{\mathrm{in\_situ}} - T_{\mathrm{baseline}}) / T_{\mathrm{baseline}},$$

where $T_{\mathrm{post}}$ is post-processing wall time, $T_{\mathrm{in\_situ}}$ is wall time with libyt, and $T_{\mathrm{baseline}}$ is simulation time without analysis. The article does not present these expressions as a new performance theory, but as a practical characterization of overhead in this setting (Tsai et al., 11 Dec 2025).

Memory profiling indicates that libyt’s grid-metadata footprint is approximately $640^3$0 per grid in a dummy AMR test with $640^3$1 root-level grids, each of size $640^3$2, two fields, and four MPI ranks with even distribution. At the same time, the paper emphasizes that Python memory usage is strongly script-dependent. A concrete example is yt’s print_stats, which can allocate roughly four times the simulation memory on average in the root process because it traverses large grid indices. This guards against the misconception that in situ operation automatically implies low memory overhead (Tsai et al., 11 Dec 2025).

The case studies cover a range of astrophysical workflows. For core-collapse supernovae in GAMER, snapshots every $640^3$3 grow from $640^3$4 to $640^3$5, while instabilities evolve on $640^3$6–$640^3$7. libyt saved 2D planes every $640^3$8 through yt.save_as_dataset, with per-file sizes of $640^3$9–$9.0 \times 10^8$0 and overhead of about $9.0 \times 10^8$1–$9.0 \times 10^8$2 of total runtime. In isolated dwarf galaxy simulations, snapshots every $9.0 \times 10^8$3 were approximately $9.0 \times 10^8$4 each and totaled about $9.0 \times 10^8$5; libyt analyzed every $9.0 \times 10^8$6 with approximately $9.0 \times 10^8$7 runtime overhead. In fuzzy dark matter simulations, average line-of-sight density was monitored every $9.0 \times 10^8$8, about $9.0 \times 10^8$9 the cadence of $20$0 snapshots, with about $20$1 runtime overhead. Additional demonstrations include the Sod shock tube test in ENZO, Kelvin–Helmholtz instability in ENZO, and the AGORA isolated galaxy simulation, for which a conventional workflow would have required about $20$2 of snapshots (Tsai et al., 11 Dec 2025).

The limitations are structural rather than incidental. Because libyt can determine whether redistribution is necessary only after synchronization, all yt data-reading operations are treated as collective. All ranks must therefore call a given yt operation that requires I/O, including some figure-saving operations when annotations trigger data access. The paper explicitly warns that operations such as annotate_streamlines, annotate_quiver/cquiver, annotate_velocity, annotate_magnetic_field, and annotate_particles require save() to be called on all ranks to avoid hangs. For yt volume rendering, some odd-rank configurations can skip I/O in a subset of ranks, so the recommendation is to use an even number of ranks and call save() on all ranks (Tsai et al., 11 Dec 2025).

More broadly, the current architecture is described as flat and synchronous: robust and simple, but enforcing collective participation in I/O. Future directions mentioned in the paper include hierarchical or asynchronous designs to reduce collectivity and improve interactive responsiveness. The article also notes circumstances in which post-processing remains preferable: extensive exploratory analysis long after the run, dependence on rich Jupyter display ecosystems, or the need to isolate Python’s memory footprint from the simulation process. This suggests that libyt is best understood not as a universal replacement for post-processing, but as an infrastructure choice that shifts the trade-off toward lower I/O latency, higher temporal cadence, and tighter coupling between simulation and analysis (Tsai et al., 11 Dec 2025).