Metatensor: Atomistic ML Data Standard

• Metatensor is a metadata-rich, gradient-aware library designed to standardize data exchange in atomistic machine learning.
• It organizes arrays using Labels, TensorBlock, and TensorMap to effectively manage complex observables with geometric and symmetry metadata.
• Its cross-language architecture and serialization methods enable seamless integration between Python-based ML tools and traditional simulation engines.

Searching arXiv for papers on Metatensor and closely related atomistic ML interoperability. Metatensor is a foundational library for interoperable atomistic machine learning that provides cross-platform and cross-language storage and manipulation of arrays with many potentially sparse indices, designed from the ground up for atomistic ML applications. Its central design choice is to combine numerical values with metadata that describes their nature and facilitates the handling of geometric information and gradients with respect to the atomic positions, thereby providing a common framework for data sharing between ML software—typically written in Python—and established atomistic modeling tools—typically written in Fortran, C or C++ (Bigi et al., 21 Aug 2025).

1. Definition and problem setting

Metatensor addresses a specific interoperability problem in atomistic machine learning. Atomistic ML sits at the intersection of two different software ecosystems: ML frameworks, often Python-based and built around NumPy, PyTorch, and JAX, and atomistic simulation engines, often written in Fortran, C, or C++. In this setting, the exchanged data are not merely dense arrays of floating-point values. They often involve many indices, block sparsity, symmetry-dependent tensor components, per-atom, per-system, or per-neighbor organization, and gradients with respect to atomic positions or strain (Bigi et al., 21 Aug 2025).

The library is therefore organized as a metadata-rich, gradient-aware representation for atomistic observables. Its purpose is not only storage but also preservation of scientific meaning. This includes the physical identity of quantities such as energy, dipole, density coefficients, spherical tensor components, and basis coefficients, together with the geometric organization needed to interpret them correctly in simulation and learning pipelines (Bigi et al., 21 Aug 2025).

A key practical motivation is the integration-complexity problem. Without a common representation, each ML model–simulation engine pair requires a dedicated interface, producing an $\mathcal{O}(M \times N)$ integration problem. Metatensor is introduced as a reusable interoperability layer that reduces this to an $\mathcal{O}(M+N)$-style pattern. This suggests that its role is infrastructural rather than algorithm-specific: it standardizes the representation of atomistic ML data so that models, descriptors, training utilities, and simulation engines can interoperate without bespoke format conversion for every pairwise combination (Bigi et al., 21 Aug 2025).

2. Data model and internal abstractions

Metatensor centers on three core objects: Labels, TensorBlock, and TensorMap. Together they define a structured representation for arrays whose axes carry explicit semantic annotations rather than being interpreted only positionally (Bigi et al., 21 Aug 2025).

Object	Function	Main contents
Labels	Metadata container	Named, multi-dimensional indices
TensorBlock	Basic data unit	One multi-dimensional array, dimension labels, optional gradients
TensorMap	Structured collection	Key/value map from Labels to TensorBlock

Labels store metadata as a set of named, multi-dimensional indices. Conceptually, they behave like a 2D integer array in which each column is a named dimension and each row is one metadata entry. This gives axis labels a formal structure suitable for scientific data exchange rather than treating them as ad hoc annotations (Bigi et al., 21 Aug 2025).

A TensorBlock contains a single multi-dimensional data array, labels on its dimensions, and optional gradients. The dimensions are organized conceptually into samples, components, and properties. Samples identify which object the data pertains to; components encode tensorial directions or rank; properties specify what has been stored about the object. The library also stores gradients as additional TensorBlock instances labeled by a string naming the gradient parameter, such as "positions" or "strain" (Bigi et al., 21 Aug 2025).

A TensorMap is a higher-level container that groups multiple TensorBlock objects into a structured representation. It is effectively a key/value map in which keys are Labels and values are TensorBlocks. This is especially useful for block sparsity, symmetry-decomposed tensors, and situations in which different blocks require different components or properties. The paper gives spherical tensors as a canonical example: each irreducible representation labeled by angular order $\lambda$ can be stored in a separate block (Bigi et al., 21 Aug 2025).

3. Metadata, geometry, and derivatives

Metatensor’s defining principle is that arrays become scientifically meaningful when metadata is carried together with values. In atomistic ML, this is particularly important because observables and descriptors have nontrivial geometric and differential structure (Bigi et al., 21 Aug 2025).

For geometric information, the library uses metadata to encode how atomistic quantities are organized. Samples can identify systems and atoms. Components can represent Cartesian axes or tensor components. Properties encode the physical meaning of the stored output. The examples given include scalar energies with no component axis, per-system dipole moments with a component axis labeled xyz, and vector-like Cartesian quantities for forces or energy gradients with components [0,1,2] for $x$, $y$, and $z$ (Bigi et al., 21 Aug 2025).

The representation of spherical tensors is explicitly symmetry-aware. Spherical observables such as the dipole or electron density can be stored in a block-sparse representation keyed by irreducible representations of $O(3)$, labeled by angular order $\lambda$ and parity $\sigma$. This is significant because symmetry decomposition is a common structural feature of atomistic descriptors and equivariant models, and metatensor makes it a first-class organizational principle rather than an external convention (Bigi et al., 21 Aug 2025).

Gradient handling is a major feature. A TensorBlock can hold gradients in separate TensorBlocks labeled by the gradient parameter. The paper gives two explicit physical identifications: the gradient of energy with respect to "positions" is the negative of forces, and the gradient with respect to cell "strain" is the negative virial. Because gradients are themselves TensorBlocks, higher-order gradients can be stored recursively. A plausible implication is that this reduces ambiguity in force and stress pipelines by binding derivative data to the exact object from which they are derived (Bigi et al., 21 Aug 2025).

4. Software architecture and interoperability mechanisms

Metatensor is designed as a cross-language interoperability layer. The core library is implemented in Rust and exposes a C API. Because most languages can call C, this choice makes the library accessible from many ecosystems while retaining a single systems-level implementation of the metadata and tensor abstractions (Bigi et al., 21 Aug 2025).

Bindings are provided for Python, Rust, and TorchScript. TorchScript is especially important in the intended deployment model because it allows PyTorch models defined in Python to be exported into a form that can run inside compiled simulation engines without requiring a Python interpreter. The practical consequence is that ML developers can work in Python with NumPy or PyTorch while simulation developers can call the same metatensor-backed logic from C, C++, or Fortran-based codes (Bigi et al., 21 Aug 2025).

A further architectural distinction is made between metadata and array storage. Metatensor manages metadata storage itself, while the actual data arrays are handled by backend-specific implementations through a metatensor array interface. This supports NumPy arrays, PyTorch tensors, and potentially GPU-resident arrays. The phrase “treats actual array storage as opaque and delegates it downstream” is central to the design: the library standardizes representation and metadata semantics without imposing a single numerical backend (Bigi et al., 21 Aug 2025).

For persistent exchange, metatensor uses serialization based on NumPy’s npz format, preserving both arrays and metadata. Because npz is language-agnostic, the stored data remain readable even if the library itself disappeared. This serialization strategy reflects the project’s emphasis on portability and long-term interoperability rather than framework-local checkpoints (Bigi et al., 21 Aug 2025).

5. Relation to metatomic and the broader ecosystem

Metatensor is the data layer; metatomic is the model-exchange layer. The distinction is explicit. Metatensor stores labeled arrays, metadata, gradients, and sparse blocks. Metatomic stores an atomistic ML model and model metadata in a portable way, standardizing the interface between ML models and simulation engines (Bigi et al., 21 Aug 2025).

Metatomic is built using TorchScript and allows a model to declare what outputs it can compute, what extra inputs it needs, and what results it returns for a given system. Those outputs are returned as TensorMap objects, which means that model I/O is expressed directly in metatensor’s metadata-rich representation. This division of responsibilities makes the two libraries complementary: one standardizes scientific tensors, the other standardizes executable atomistic models (Bigi et al., 21 Aug 2025).

The ecosystem described around these libraries is broad. metatensor-operations provides metadata-aware operations such as multiply, pow, allclose, zeros_like, sum_over_samples, slice, join, filter_blocks, and sort. metatensor-learn provides higher-level ML building blocks resembling PyTorch’s nn and data-loading APIs, including Linear, ReLU, ModuleMap, symmetry-preserving and $O(3)$-equivariant layers, and Dataset/DataLoader-like utilities (Bigi et al., 21 Aug 2025).

The same ecosystem includes descriptor and feature libraries such as featomic, torch-spex, torch-pme, and sphericart, along with metatrain for training, export, and evaluation using mtt train, mtt export, and mtt eval. Model examples mentioned in the paper include PET-MAD, ShiftML3, and FlashMD. Simulation and analysis integrations include LAMMPS, i-PI, ASE, eOn, TorchSim, PLUMED, and chemiscope. The workflows span energy/force/stress-driven molecular dynamics, path-integral molecular dynamics, geometry optimization, saddle-point searches, collective variable construction, and visualization and exploratory analysis. Taken together, these examples indicate that metatensor is intended as a common substrate across the entire atomistic ML software stack, not as an isolated tensor container (Bigi et al., 21 Aug 2025).

6. Scientific role, scope, and related meanings

The primary meaning of Metatensor as a proper noun is the atomistic ML library described above. The available literature also shows that the word “metatensor” can appear in looser or analogical senses outside this software context, and those usages should be distinguished carefully (Bigi et al., 21 Aug 2025).

In tensor theory, “The Tensor as an Informational Resource” develops a family of hypergraph-based preorders on tensors and describes a “metatensor” viewpoint in which tensors are compared through the larger structures they induce on hypergraphs. In that setting, a metatensor is not a software object but a resource-theoretic perspective on tensor transformation, asymptotic restriction, and induced quantities such as generalized rank and subrank (Christandl, 2023).

In electromagnetics and metamaterials, the literature also uses tensor-valued material parameters in ways described as “metatensor-like.” “Spatiotemporal Isotropic-to-Anisotropic Meta-Atoms” studies a spatiotemporal meta-atom in which the internal state is a time-varying permittivity tensor, while “IE-GSTC Metasurface Field Solver using Surface Susceptibility Tensors with Normal Polarizabilities” treats complete tensorial susceptibility densities in a zero-thickness metasurface model (Pacheco-Peña et al., 2021). These are not uses of Metatensor as the atomistic ML library; they are tensor-centric constructions in different disciplines.

A broader technical background is supplied by work on tensor manipulation languages. “Modeling of languages for tensor manipulation” formally defines a model language for tensor operations, proves that well-typed programs execute correctly without error, and proves equivalence between a reference implementation and a padded-storage implementation (Rink, 2018). This does not define Metatensor itself, but it reflects the same general need for precise semantics and correct implementation of structured tensor operations. This suggests that metatensor belongs to a wider movement toward rigorous, interoperable tensor infrastructure across computational science.

7. Significance for atomistic machine learning

Metatensor’s significance lies in providing a shared, metadata-rich, sparse, gradient-aware data language for atomistic machine learning. By preserving values together with scientific meaning, geometry, symmetry labels, and derivatives, it allows data exchange between ML software and simulation engines without collapsing atomistic observables into generic dense arrays stripped of context (Bigi et al., 21 Aug 2025).

The library is especially relevant in environments where models and simulation packages are developed independently. Because it reduces interoperability effort from $\mathcal{O}(M+N)$0 to $\mathcal{O}(M+N)$1, it changes the scaling of software integration in a field where combinations of descriptors, models, engines, and analysis tools proliferate quickly. In this sense, metatensor functions as infrastructure for reuse and portability (Bigi et al., 21 Aug 2025).

Its scientific importance is also methodological. Atomistic ML frequently depends on symmetry decomposition, sparse block structure, and position- or strain-dependent derivatives. Metatensor makes those structures explicit in the data model itself. A plausible implication is that this reduces the mismatch between model-side abstractions and simulation-side requirements, particularly in workflows involving equivariant descriptors, force training, virial prediction, and deployment into compiled HPC codes (Bigi et al., 21 Aug 2025).

Within the current ecosystem, Metatensor is therefore best understood not as a competitor to a specific ML framework or simulation engine, but as a common representation layer that bridges them. Its role is foundational: it standardizes how atomistic ML quantities are described, stored, serialized, and communicated across languages and software boundaries (Bigi et al., 21 Aug 2025).