---
title: 'libyt: In Situ Analysis for Exascale Simulations'
url: https://www.emergentmind.com/topics/libyt
type: topic
---

# libyt: In Situ Analysis for Exascale Simulations

Searching arXiv for the libyt paper and closely related in situ/yt context to ground the article in current literature.
libyt is an open-source C library for in situ analysis in which Python is embedded into a running simulation so that analysis and visualization operate directly on distributed, in-memory data rather than on snapshots written to and reread from storage. It is designed for high-performance and exascale environments, connects simulation codes in C, C++, or Fortran to yt and other Python packages, and provides both automatic analysis hooks and interactive entry points through a Python prompt or a Jupyter Notebook. A central design objective is “reusability”: a post-processing yt script can be converted to an in situ script with just two lines of change, while preserving the researcher’s existing job submission scripts and ordinary Python environment [2512.11142].

## 1. Purpose, scope, and design goals

libyt was introduced to address the increasing difficulty of handling massive simulation datasets in the exascale computing era. Its core premise is that analysis should occur during simulation runtime, thereby bypassing costly intermediate I/O steps and the storage burden associated with traditional post-processing. In this model, analysis is part of the simulation loop rather than a separate downstream stage [2512.11142].

The library places particular emphasis on yt-based volumetric analysis and visualization, but its scope is not restricted to yt. It also supports arbitrary Python packages, including NumPy, matplotlib, and machine learning frameworks. This makes libyt less a single analysis tool than a bidirectional interface between production simulation codes and the Python ecosystem. The paper presents this bidirectionality as a means to reuse existing code paths inside the simulation—for example, user-defined C functions for derived data—while still writing orchestration and visualization logic in Python [2512.11142].

A defining design constraint is minimal disruption to established workflows. The stated “two-line change” principle is a concrete expression of this constraint: post-processing scripts written for yt should remain largely intact, and ordinary submission scripts should be reusable. This suggests that libyt is intended not merely as a performance optimization, but as an interface layer that reduces migration cost from file-based analysis to runtime analysis.

## 2. Runtime architecture and execution model

At runtime, libyt consists of four principal elements: the libyt C library itself, a per-rank libyt Python Module created dynamically, the yt_libyt frontend that maps in-memory data into yt’s dataset abstraction, and a libyt kernel for Jupyter connectivity. The simulation launches $N$ MPI ranks; each rank creates its own embedded Python interpreter and its own libyt Python Module instance, and libyt orchestrates these interpreters synchronously. When Python is executing—whether automatically or interactively—the simulation computation pauses on all ranks, and control returns to the simulation only after Python finishes [2512.11142].

| Component | Role |
|---|---|
| libyt | Registers metadata and data pointers, binds C callbacks, and triggers Python or interactive entry points |
| libyt Python Module | Per-rank runtime “shim” storing dictionaries and exposing C-extension methods |
| yt_libyt | yt frontend that presents the running simulation as a yt dataset |
| libyt kernel | Jupyter kernel launched by the simulation for Notebook attachment |

A key architectural property is that simulation arrays remain owned by the simulation. Intrinsic field and particle arrays are wrapped as NumPy arrays through the NumPy C API without copying, with shapes, dtypes, and strides supplied so that Python can interpret the memory. When data are virtual or derived and do not exist in memory, Python requests invoke user-defined C functions through libyt’s C-extensions; libyt allocates temporary storage, the callback fills it, and ownership is transferred to Python so that deallocation follows Python’s lifetime rules. Only requested data are prepared, and yt’s chunking is reused to bound both on-demand generation and memory usage [2512.11142].

The parallel execution model depends on data redistribution across ranks by MPI one-sided communication. This avoids hand-coded sender/receiver pairs and allows Python code to request data independently of where the data reside. The reported procedure per redistribution epoch is: create a window, attach local buffers, gather window addresses and metadata, open the epoch with `MPI_Win_fence`, fetch remote data, close the epoch, detach buffers, free structures, and close the window. Very large arrays are automatically split into multiple MPI transfers to avoid the 32-bit count limit [2512.11142].

## 3. Data model, AMR representation, and yt integration

libyt targets AMR simulations and stores the grid hierarchy in arrays aligned to yt’s expectations. Each grid has a unique id, parent id, level, left and right edges, and grid dimensions. When a simulation stores only local hierarchy, libyt gathers the global hierarchy so that all ranks can present a coherent dataset to yt. The hierarchy is held in the `hierarchy` dictionary of the libyt Python Module, while intrinsic per-grid arrays and particle data are kept in `grid_data` and `particle_data` dictionaries keyed by grid id and names [2512.11142].

Field metadata are explicitly typed and annotated. The reported descriptors include `field_name`, `field_type`, `field_dtype`, `contiguous_in_x`, `field_unit`, and ghost-cell metadata. The allowed `field_type` values include `"cell-centered"`, `"face-centered"`, and `"derived_func"`. Face-centered data are converted to cell-centered data on demand in Python by averaging across faces, which is done to satisfy yt’s cell-centered operations while using minimal extra memory. Derived fields are generated on request through the `derived_func` C-extension method [2512.11142].

Particle support includes multiple particle types with distinct attributes. Intrinsic attributes may be contiguous or sparse; for sparse or derived attributes, libyt calls a user-provided C function through the `get_particle` C-extension. The required attributes for each particle type include the position components `coor_x`, `coor_y`, and `coor_z` [2512.11142].

The mapping to yt frontends is deliberately conservative. `yt_libyt` reads `param_yt` and `param_user` from the libyt Python Module, inherits fields and units from a simulation’s existing yt frontend if one is available, and augments that frontend with fields registered through libyt. This preserves the semantics of established yt-based post-processing. The canonical conversion from post-processing to in situ consists of exactly two edits:

```python
import yt_libyt
ds = yt_libyt.libytDataset()
```

in place of an ordinary `yt.load(...)` call. The data block further notes that fields provided through libyt can be mixed with yt’s own derived fields, as in a `SlicePlot` over an intrinsic simulation field and another over `("gas", "velocity_x")` [2512.11142].

## 4. API, callbacks, and integration into simulation codes

The C-side API is organized as a per-run workflow. Initialization begins with `yt_initialize`, which creates the Python interpreter and the libyt Python Module and imports the user’s Python script. Global runtime metadata are then set with `yt_set_Parameters`, while optional simulation-specific parameters are passed with `yt_set_UserParameter*`. Metadata for fields, particle types, and grids are populated through `yt_get_FieldsPtr`, `yt_get_ParticlesPtr`, and `yt_get_GridsPtr`, after which `yt_commit` finalizes registration and binds the data structures into the Python module. Python execution is then triggered by one of the runtime entry points, temporary resources are released by `yt_free`, and final shutdown is performed by `yt_finalize` before `MPI_Finalize` [2512.11142].

The documented per-step order is precise:

1. `yt_initialize`
2. `yt_set_Parameters`
3. `yt_get_FieldsPtr`
4. `yt_get_ParticlesPtr`
5. `yt_get_GridsPtr`
6. `yt_commit`
7. Trigger Python with `yt_run_Function`, `yt_run_FunctionArguments`, `yt_run_InteractiveMode`, `yt_run_ReloadScript`, or `yt_run_JupyterKernel`
8. `yt_free`
9. `yt_finalize`

This sequence is significant because several helper functions become usable only after `yt_commit`. These include accessors such as `yt_getGridInfo_Dimensions`, `yt_getGridInfo_LeftEdge`, `yt_getGridInfo_RightEdge`, `yt_getGridInfo_ParentId`, `yt_getGridInfo_Level`, `yt_getGridInfo_ProcNum`, `yt_getGridInfo_ParticleCount`, `yt_getGridInfo_FieldData`, and `yt_getGridInfo_ParticleData` [2512.11142].

Two callback interfaces are central to on-demand data generation. A derived field generator used with `field_type="derived_func"` has prototype

```c
void my_field(const int len, const long* gids, const char* field, yt_array* data)
```

and must fill contiguous-in-x, ghost-free arrays for the requested grid ids. A particle attribute generator for sparse or derived attributes has prototype

```c
void my_par_attr(const int len, const long* gids, const char* attr, yt_array* data)
```

These interfaces make it possible to expose simulation-native logic directly to Python without materializing all possible derived quantities in advance [2512.11142].

## 5. Interactive operation, prompts, and Jupyter connectivity

libyt supports three interactive modes of use: automatic invocation of named Python functions, an MPI-aware Python prompt, and a Jupyter Notebook interface. In automatic invocation, the simulation calls an inline Python function at specified times. The system supports a fail-fast mode, where any Python error aborts the run, and a fault-tolerant mode, where errors are recorded and the simulation continues [2512.11142].

The interactive prompt is MPI-synchronous. The root rank reads user input, broadcasts it, all ranks execute the code, and outputs are gathered and printed. For batch systems, a file-based prompt is provided: flag files can be created to start, reload, or exit, and outputs are captured to files. This is a practical adaptation to environments in which an attached terminal REPL is inconvenient or impossible [2512.11142].

The Jupyter pathway is built around a kernel launched by the simulation on the MPI root. The simulation calls `yt_run_JupyterKernel("FLAGFILE", use_connection_file)`, and the kernel writes `libyt_kernel_connection.json` and `libyt_kernel_pid.txt` to a directory designated by the environment variable `LIBYT_KERNEL_INFO_DIR`. Jupyter Notebook or JupyterLab then attaches through the `jupyter_libyt` provisioner, and the user selects the “Libyt” kernel. Code execution remains MPI-synchronous, outputs from all ranks are gathered and shown, and tab completion is provided through `jedi` [2512.11142].

These interactive modes come with explicit constraints. Interrupts, restarts, and restart-and-run-all are disabled in the Jupyter interface; “Shutdown” cleanly closes the kernel and returns control to the simulation. Outputs are gathered as plain text, so many rich display extensions based on HTML, SVG, or JavaScript do not render. Large text outputs also slow down output gathering. A plausible implication is that libyt’s interactive design prioritizes deterministic MPI coordination and robustness over parity with a conventional local Jupyter kernel [2512.11142].

## 6. Performance characteristics, demonstrated applications, and limitations

The paper reports both benchmarking and application studies. In a strong scaling benchmark using the Plummer test with GAMER 2.2.0, yt 4.3.1, and libyt 0.2.0, the dataset comprised 3,430 grids with a root grid of $640^3$ and four AMR levels, for a total of $9.0 \times 10^8$ cells. Using identical Python workflows for 2D profiles in post-processing and in situ modes, libyt outperformed post-processing by approximately $20$–$45\%$ across core counts. The reported interpretation is that this gain comes from avoiding reads from disk, while deviations from ideal scaling in both modes arise from yt’s internal bottlenecks, including serial fractions, current indexing, and job partitioning [2512.11142].

A convenient notation for speedup and overhead is given as

$$
S = T_{\mathrm{post}} / T_{\mathrm{in\_situ}}
$$

and

$$
O = (T_{\mathrm{in\_situ}} - T_{\mathrm{baseline}}) / T_{\mathrm{baseline}},
$$

where $T_{\mathrm{post}}$ is post-processing wall time, $T_{\mathrm{in\_situ}}$ is wall time with libyt, and $T_{\mathrm{baseline}}$ is simulation time without analysis. The article does not present these expressions as a new performance theory, but as a practical characterization of overhead in this setting [2512.11142].

Memory profiling indicates that libyt’s grid-metadata footprint is approximately $0.2\ \mathrm{KB}$ per grid in a dummy AMR test with $2 \times 10^6$ root-level grids, each of size $8^3$, two fields, and four MPI ranks with even distribution. At the same time, the paper emphasizes that Python memory usage is strongly script-dependent. A concrete example is `yt`’s `print_stats`, which can allocate roughly four times the simulation memory on average in the root process because it traverses large grid indices. This guards against the misconception that in situ operation automatically implies low memory overhead [2512.11142].

The case studies cover a range of astrophysical workflows. For core-collapse supernovae in GAMER, snapshots every $2\ \mathrm{ms}$ grow from $7.5\ \mathrm{GB}$ to $90\ \mathrm{GB}$, while instabilities evolve on $0.5$–$1.0\ \mathrm{ms}$. libyt saved 2D planes every $0.2\ \mathrm{ms}$ through `yt.save_as_dataset`, with per-file sizes of $50$–$200\ \mathrm{MB}$ and overhead of about $3$–$5\%$ of total runtime. In isolated dwarf galaxy simulations, snapshots every $10\ \mathrm{Myr}$ were approximately $6\ \mathrm{GB}$ each and totaled about $450\ \mathrm{GB}$; libyt analyzed every $1\ \mathrm{Myr}$ with approximately $3\%$ runtime overhead. In fuzzy dark matter simulations, average line-of-sight density was monitored every $32\ \mathrm{Myr}$, about $300\times$ the cadence of $10\ \mathrm{Gyr}$ snapshots, with about $15\%$ runtime overhead. Additional demonstrations include the Sod shock tube test in ENZO, Kelvin–Helmholtz instability in ENZO, and the AGORA isolated galaxy simulation, for which a conventional workflow would have required about $2000\ \mathrm{GB}$ of snapshots [2512.11142].

The limitations are structural rather than incidental. Because libyt can determine whether redistribution is necessary only after synchronization, all yt data-reading operations are treated as collective. All ranks must therefore call a given yt operation that requires I/O, including some figure-saving operations when annotations trigger data access. The paper explicitly warns that operations such as `annotate_streamlines`, `annotate_quiver/cquiver`, `annotate_velocity`, `annotate_magnetic_field`, and `annotate_particles` require `save()` to be called on all ranks to avoid hangs. For yt volume rendering, some odd-rank configurations can skip I/O in a subset of ranks, so the recommendation is to use an even number of ranks and call `save()` on all ranks [2512.11142].

More broadly, the current architecture is described as flat and synchronous: robust and simple, but enforcing collective participation in I/O. Future directions mentioned in the paper include hierarchical or asynchronous designs to reduce collectivity and improve interactive responsiveness. The article also notes circumstances in which post-processing remains preferable: extensive exploratory analysis long after the run, dependence on rich Jupyter display ecosystems, or the need to isolate Python’s memory footprint from the simulation process. This suggests that libyt is best understood not as a universal replacement for post-processing, but as an infrastructure choice that shifts the trade-off toward lower I/O latency, higher temporal cadence, and tighter coupling between simulation and analysis [2512.11142].

Source: https://www.emergentmind.com/topics/libyt