---
title: 'Metatensor: Atomistic ML Data Standard'
url: https://www.emergentmind.com/topics/metatensor
type: topic
---

# Metatensor: Atomistic ML Data Standard

Searching arXiv for recent papers on Metatensor and closely related atomistic ML interoperability.
Metatensor is a foundational library for interoperable atomistic machine learning that provides cross-platform and cross-language storage and manipulation of arrays with many potentially sparse indices, designed from the ground up for atomistic ML applications. Its central design choice is to combine numerical values with metadata that describes their nature and facilitates the handling of geometric information and gradients with respect to the atomic positions, thereby providing a common framework for data sharing between ML software—typically written in Python—and established atomistic modeling tools—typically written in Fortran, C or C++ [2508.15704].

## 1. Definition and problem setting

Metatensor addresses a specific interoperability problem in atomistic machine learning. Atomistic ML sits at the intersection of two different software ecosystems: ML frameworks, often Python-based and built around NumPy, PyTorch, and JAX, and atomistic simulation engines, often written in Fortran, C, or C++. In this setting, the exchanged data are not merely dense arrays of floating-point values. They often involve many indices, block sparsity, symmetry-dependent tensor components, per-atom, per-system, or per-neighbor organization, and gradients with respect to atomic positions or strain [2508.15704].

The library is therefore organized as a metadata-rich, gradient-aware representation for atomistic observables. Its purpose is not only storage but also preservation of scientific meaning. This includes the physical identity of quantities such as energy, dipole, density coefficients, spherical tensor components, and basis coefficients, together with the geometric organization needed to interpret them correctly in simulation and learning pipelines [2508.15704].

A key practical motivation is the integration-complexity problem. Without a common representation, each ML model–simulation engine pair requires a dedicated interface, producing an $\mathcal{O}(M \times N)$ integration problem. Metatensor is introduced as a reusable interoperability layer that reduces this to an $\mathcal{O}(M+N)$-style pattern. This suggests that its role is infrastructural rather than algorithm-specific: it standardizes the representation of atomistic ML data so that models, descriptors, training utilities, and simulation engines can interoperate without bespoke format conversion for every pairwise combination [2508.15704].

## 2. Data model and internal abstractions

Metatensor centers on three core objects: `Labels`, `TensorBlock`, and `TensorMap`. Together they define a structured representation for arrays whose axes carry explicit semantic annotations rather than being interpreted only positionally [2508.15704].

| Object | Function | Main contents |
|---|---|---|
| `Labels` | Metadata container | Named, multi-dimensional indices |
| `TensorBlock` | Basic data unit | One multi-dimensional array, dimension labels, optional gradients |
| `TensorMap` | Structured collection | Key/value map from `Labels` to `TensorBlock` |

`Labels` store metadata as a set of named, multi-dimensional indices. Conceptually, they behave like a 2D integer array in which each column is a named dimension and each row is one metadata entry. This gives axis labels a formal structure suitable for scientific data exchange rather than treating them as ad hoc annotations [2508.15704].

A `TensorBlock` contains a single multi-dimensional data array, labels on its dimensions, and optional gradients. The dimensions are organized conceptually into samples, components, and properties. Samples identify which object the data pertains to; components encode tensorial directions or rank; properties specify what has been stored about the object. The library also stores gradients as additional `TensorBlock` instances labeled by a string naming the gradient parameter, such as `"positions"` or `"strain"` [2508.15704].

A `TensorMap` is a higher-level container that groups multiple `TensorBlock` objects into a structured representation. It is effectively a key/value map in which keys are `Labels` and values are `TensorBlock`s. This is especially useful for block sparsity, symmetry-decomposed tensors, and situations in which different blocks require different components or properties. The paper gives spherical tensors as a canonical example: each irreducible representation labeled by angular order $\lambda$ can be stored in a separate block [2508.15704].

## 3. Metadata, geometry, and derivatives

Metatensor’s defining principle is that arrays become scientifically meaningful when metadata is carried together with values. In atomistic ML, this is particularly important because observables and descriptors have nontrivial geometric and differential structure [2508.15704].

For geometric information, the library uses metadata to encode how atomistic quantities are organized. Samples can identify systems and atoms. Components can represent Cartesian axes or tensor components. Properties encode the physical meaning of the stored output. The examples given include scalar energies with no component axis, per-system dipole moments with a component axis labeled `xyz`, and vector-like Cartesian quantities for forces or energy gradients with components `[0,1,2]` for $x$, $y$, and $z$ [2508.15704].

The representation of spherical tensors is explicitly symmetry-aware. Spherical observables such as the dipole or electron density can be stored in a block-sparse representation keyed by irreducible representations of $O(3)$, labeled by angular order $\lambda$ and parity $\sigma$. This is significant because symmetry decomposition is a common structural feature of atomistic descriptors and equivariant models, and metatensor makes it a first-class organizational principle rather than an external convention [2508.15704].

Gradient handling is a major feature. A `TensorBlock` can hold gradients in separate `TensorBlock`s labeled by the gradient parameter. The paper gives two explicit physical identifications: the gradient of energy with respect to `"positions"` is the negative of forces, and the gradient with respect to cell `"strain"` is the negative virial. Because gradients are themselves `TensorBlock`s, higher-order gradients can be stored recursively. A plausible implication is that this reduces ambiguity in force and stress pipelines by binding derivative data to the exact object from which they are derived [2508.15704].

## 4. Software architecture and interoperability mechanisms

Metatensor is designed as a cross-language interoperability layer. The core library is implemented in Rust and exposes a C API. Because most languages can call C, this choice makes the library accessible from many ecosystems while retaining a single systems-level implementation of the metadata and tensor abstractions [2508.15704].

Bindings are provided for Python, Rust, and TorchScript. TorchScript is especially important in the intended deployment model because it allows PyTorch models defined in Python to be exported into a form that can run inside compiled simulation engines without requiring a Python interpreter. The practical consequence is that ML developers can work in Python with NumPy or PyTorch while simulation developers can call the same metatensor-backed logic from C, C++, or Fortran-based codes [2508.15704].

A further architectural distinction is made between metadata and array storage. Metatensor manages metadata storage itself, while the actual data arrays are handled by backend-specific implementations through a metatensor array interface. This supports NumPy arrays, PyTorch tensors, and potentially GPU-resident arrays. The phrase “treats actual array storage as opaque and delegates it downstream” is central to the design: the library standardizes representation and metadata semantics without imposing a single numerical backend [2508.15704].

For persistent exchange, metatensor uses serialization based on NumPy’s `npz` format, preserving both arrays and metadata. Because `npz` is language-agnostic, the stored data remain readable even if the library itself disappeared. This serialization strategy reflects the project’s emphasis on portability and long-term interoperability rather than framework-local checkpoints [2508.15704].

## 5. Relation to metatomic and the broader ecosystem

Metatensor is the data layer; metatomic is the model-exchange layer. The distinction is explicit. Metatensor stores labeled arrays, metadata, gradients, and sparse blocks. Metatomic stores an atomistic ML model and model metadata in a portable way, standardizing the interface between ML models and simulation engines [2508.15704].

Metatomic is built using TorchScript and allows a model to declare what outputs it can compute, what extra inputs it needs, and what results it returns for a given system. Those outputs are returned as `TensorMap` objects, which means that model I/O is expressed directly in metatensor’s metadata-rich representation. This division of responsibilities makes the two libraries complementary: one standardizes scientific tensors, the other standardizes executable atomistic models [2508.15704].

The ecosystem described around these libraries is broad. `metatensor-operations` provides metadata-aware operations such as `multiply`, `pow`, `allclose`, `zeros_like`, `sum_over_samples`, `slice`, `join`, `filter_blocks`, and `sort`. `metatensor-learn` provides higher-level ML building blocks resembling PyTorch’s `nn` and data-loading APIs, including `Linear`, `ReLU`, `ModuleMap`, symmetry-preserving and $O(3)$-equivariant layers, and Dataset/DataLoader-like utilities [2508.15704].

The same ecosystem includes descriptor and feature libraries such as `featomic`, `torch-spex`, `torch-pme`, and `sphericart`, along with `metatrain` for training, export, and evaluation using `mtt train`, `mtt export`, and `mtt eval`. Model examples mentioned in the paper include PET-MAD, ShiftML3, and FlashMD. Simulation and analysis integrations include LAMMPS, i-PI, ASE, eOn, TorchSim, PLUMED, and chemiscope. The workflows span energy/force/stress-driven molecular dynamics, path-integral molecular dynamics, geometry optimization, saddle-point searches, collective variable construction, and visualization and exploratory analysis. Taken together, these examples indicate that metatensor is intended as a common substrate across the entire atomistic ML software stack, not as an isolated tensor container [2508.15704].

## 6. Scientific role, scope, and related meanings

The primary meaning of Metatensor as a proper noun is the atomistic ML library described above. The available literature also shows that the word “metatensor” can appear in looser or analogical senses outside this software context, and those usages should be distinguished carefully [2508.15704].

In tensor theory, “The Tensor as an Informational Resource” develops a family of hypergraph-based preorders on tensors and describes a “metatensor” viewpoint in which tensors are compared through the larger structures they induce on hypergraphs. In that setting, a metatensor is not a software object but a resource-theoretic perspective on tensor transformation, asymptotic restriction, and induced quantities such as generalized rank and subrank [2311.02190].

In electromagnetics and metamaterials, the literature also uses tensor-valued material parameters in ways described as “metatensor-like.” “Spatiotemporal Isotropic-to-Anisotropic Meta-Atoms” studies a spatiotemporal meta-atom in which the internal state is a time-varying permittivity tensor, while “IE-GSTC Metasurface Field Solver using Surface Susceptibility Tensors with Normal Polarizabilities” treats complete tensorial susceptibility densities in a zero-thickness metasurface model [2106.12471]. These are not uses of Metatensor as the atomistic ML library; they are tensor-centric constructions in different disciplines.

A broader technical background is supplied by work on tensor manipulation languages. “Modeling of languages for tensor manipulation” formally defines a model language for tensor operations, proves that well-typed programs execute correctly without error, and proves equivalence between a reference implementation and a padded-storage implementation [1801.08771]. This does not define Metatensor itself, but it reflects the same general need for precise semantics and correct implementation of structured tensor operations. This suggests that metatensor belongs to a wider movement toward rigorous, interoperable tensor infrastructure across computational science.

## 7. Significance for atomistic machine learning

Metatensor’s significance lies in providing a shared, metadata-rich, sparse, gradient-aware data language for atomistic machine learning. By preserving values together with scientific meaning, geometry, symmetry labels, and derivatives, it allows data exchange between ML software and simulation engines without collapsing atomistic observables into generic dense arrays stripped of context [2508.15704].

The library is especially relevant in environments where models and simulation packages are developed independently. Because it reduces interoperability effort from $\mathcal{O}(M \times N)$ to $\mathcal{O}(M+N)$, it changes the scaling of software integration in a field where combinations of descriptors, models, engines, and analysis tools proliferate quickly. In this sense, metatensor functions as infrastructure for reuse and portability [2508.15704].

Its scientific importance is also methodological. Atomistic ML frequently depends on symmetry decomposition, sparse block structure, and position- or strain-dependent derivatives. Metatensor makes those structures explicit in the data model itself. A plausible implication is that this reduces the mismatch between model-side abstractions and simulation-side requirements, particularly in workflows involving equivariant descriptors, force training, virial prediction, and deployment into compiled HPC codes [2508.15704].

Within the current ecosystem, Metatensor is therefore best understood not as a competitor to a specific ML framework or simulation engine, but as a common representation layer that bridges them. Its role is foundational: it standardizes how atomistic ML quantities are described, stored, serialized, and communicated across languages and software boundaries [2508.15704].

Source: https://www.emergentmind.com/topics/metatensor