---
title: 'LeMaterial-Fetcher: Unified Atomistic Pipeline'
url: https://www.emergentmind.com/topics/lematerial-fetcher
type: topic
---

# LeMaterial-Fetcher: Unified Atomistic Pipeline

Searching arXiv for the cited LeMaterial papers and adjacent context.
LeMaterial-Fetcher is a Python library and full data-engineering pipeline for constructing large, unified datasets of atomistic configurations and trajectories from heterogeneous Density Functional Theory sources. It was developed to build LeMat-Traj, and is presented as a general, reproducible framework for fetching, transforming, validating, harmonizing, and distributing very large collections of atomistic data from repositories such as the Materials Project, Alexandria, and OQMD [2508.20875]. Within the LeMaterial ecosystem, its role is infrastructural rather than predictive: it standardizes source-specific calculations into an OPTIMADE-based, trajectory-aware schema, applies quality control and cross-source harmonization, and exports the resulting data in HuggingFace Datasets format for downstream machine-learning interatomic potential workflows [2508.20875].

## 1. Definition and role in the LeMaterial ecosystem

LeMaterial-Fetcher was introduced to address several systemic problems in the use of existing DFT trajectory data for machine learning interatomic potentials. The motivating issues are the fragmentation of data sources, inconsistent formats, heterogeneous DFT settings, missing or inconsistent metadata, and the difficulty of maintaining reproducible aggregation pipelines as upstream repositories evolve [2508.20875]. In response, the library provides a unified, automated pipeline for fetching, transforming, validating, harmonizing, and exporting data, and encapsulates source-specific logic in a modular, extensible framework [2508.20875].

Its immediate function is to underpin the construction of LeMat-Traj, which aggregates more than 120 million atomic configurations from Materials Project, Alexandria, and OQMD [2508.20875]. More generally, it is designed as a reusable community framework: because the pipeline is public and configuration-based, others can reproduce LeMat-Traj or build compatible datasets for their own MLIP pipelines [2508.20875].

This positioning places LeMaterial-Fetcher at the intersection of data engineering, materials informatics, and atomistic machine learning. A plausible implication is that its main contribution is not a new learning objective or architecture, but the conversion of an error-prone, one-off aggregation process into a repeatable data infrastructure layer. In that respect it complements later LeMaterial efforts such as LeMat-Bulk, which extends cross-database standardization and de-duplication for static materials databases using a related fetcher-based integration approach [2511.05178].

## 2. Pipeline architecture and source-specific ingestion

The core conceptual pipeline has five stages: fetching, transformation, validation, harmonization, and push/export [2508.20875]. In the fetching stage, the library interfaces with open APIs or direct downloads from Materials Project, Alexandria, and OQMD, and retrieves tasks or entries together with associated calculations and trajectories [2508.20875]. In transformation, it converts source-specific formats into a unified representation, standardizes atomic structure representations, energy units, and force components, and extracts DFT metadata using Pymatgen and Matminer [2508.20875]. Validation checks physical and numerical plausibility, enforces consistency with an OPTIMADE-like schema, and removes incomplete or inconsistent entries [2508.20875]. Harmonization aligns DFT flags and metadata where available, partitions data into splits by DFT functional, and ensures pseudopotential compatibility following the method of LeMat-Bulk [2508.20875]. Export stores intermediate data in a database backend and writes final outputs in HuggingFace Datasets format for efficient streaming, versioning, and metadata tracking [2508.20875].

The implementation uses PostgreSQL as the default backend for storing both raw downloaded data and transformed, validated structures before export, while MySQL is used specifically to read OQMD because OQMD is published as a MySQL database with multiple tables [2508.20875]. The design is explicitly backend-pluggable. It is also engineered for high parallelism and memory efficiency: the paper reports that it can handle 120M+ rows without exhausting memory or leaking via multiple concurrent database connections, and gives examples ranging from 128 cores and 256 GB RAM to an end-to-end run of approximately 16 hours on an AMD Ryzen 5600G with 12 workers for fetching, processing, and uploading 120M rows [2508.20875].

Source adapters are distinct and tailored. For Materials Project, the pipeline iterates over all tasks, discards deprecated tasks, and collects geometry optimization trajectories that pass filtering criteria [2508.20875]. For Alexandria, it uses all samples except those containing Ytterbium, because of pseudopotential incompatibility with other sources [2508.20875]. For OQMD, it traverses the full database, aggregates relaxation, coarse relaxation, and fine relaxation calculations for each entry, and extracts trajectories from input and output structures when all required targets are present in the correct format [2508.20875].

A recurrent misunderstanding is to treat LeMaterial-Fetcher as merely a downloader for three repositories. That characterization is incomplete. The fetch stage is only the first layer; most of the library’s technical substance lies in transformation, trajectory reconstruction, DFT harmonization, validation, and standardized export [2508.20875].

## 3. Schema, trajectory representation, and standardization

LeMaterial-Fetcher standardizes data according to the OPTIMADE specification, with minimal but important trajectory extensions [2508.20875]. Each row corresponds to a single atomistic configuration; full trajectories are reconstructed by grouping rows with the same trajectory identifier and ordering frames by Relaxation Step [2508.20875]. Two trajectory-specific fields are central. Relaxation Step is an integer index within a given optimization sequence, allowing reconstruction of relaxation paths and energy/force-versus-step curves. Relaxation Number distinguishes different relaxation runs for the same initial structure, reflecting multi-stage workflows such as coarse to fine relaxation or re-relaxation with tighter thresholds or different methods [2508.20875].

The enforced physical fields include structural information, DFT targets, and metadata. Structural content includes atomic species or chemical symbols, atomic positions, and lattice or cell parameters [2508.20875]. DFT targets include total energy per configuration and forces on each atom, while stress is mentioned in modeling experiments and is therefore likely stored when available, though not explicitly spelled out [2508.20875]. Metadata include DFT functional labels—PBE, PBESol, SCAN, or r2SCAN—source provenance, original calculation identifiers, and potentially k-point mesh, cutoff energy, and pseudopotentials when sources expose them [2508.20875].

The trajectory-aware schema is directly motivated by downstream tasks. The paper identifies Relaxation Step and Relaxation Number as central for IS2RE and IS2RS tasks, amortized optimization, and trajectory-aware self-supervised learning, and notes that they also support trajectory-level splitting to avoid leakage [2508.20875]. This suggests that the schema is not only a storage format but an inductive interface between atomistic data curation and trajectory-sensitive ML objectives.

Standardization is deliberately narrow where needed. Functional labels are normalized to a small set, and dataset splits are created per functional [2508.20875]. Energies and forces are converted to uniform units so that shared filtering thresholds are meaningful across sources [2508.20875]. Space groups are computed with `moyo`, described as a faster alternative to Spglib, using a strict tolerance `symprec = 10^{-4}` to enable consistent symmetry labels across the dataset [2508.20875].

## 4. Harmonization, filtering, and quality control

LeMaterial-Fetcher’s quality-control layer defines much of LeMat-Traj’s character. At the most basic level, any configuration lacking energy or atomic forces is removed [2508.20875]. Beyond completeness, filtering is applied at the trajectory level using two explicit convergence criteria [2508.20875].

First, a trajectory is discarded if the energy difference between the penultimate and final steps exceeds
$$
2 \times 10^{-2} \text{ eV}.
$$
Second, a trajectory is discarded if the maximum atomic force norm in the final configuration exceeds
$$
0.2 \, \text{eV/\AA}.
$$
The paper states that this force threshold is intentionally not ultra-strict: it ensures final structures are at least near equilibrium while still allowing inclusion of structures that are not perfectly relaxed, thereby preserving information about the potential-energy surface away from the exact minimum [2508.20875].

All configurations are also validated against OPTIMADE format constraints, and any entry failing schema checks or additional integrity tests is removed [2508.20875]. Harmonization further includes unit conversion, normalization of functional labels, and pseudopotential compatibility logic. An example given is the treatment of Yb-containing structures: they are taken from Materials Project rather than Alexandria because of pseudopotential incompatibilities [2508.20875]. This compatibility logic is closely related to the broader LeMat-Bulk effort, where cross-database filtering and pseudopotential alignment are formalized at larger scale [2511.05178].

These procedures matter because naive aggregation of DFT outputs across repositories would conflate differences in electronic-structure settings, metadata completeness, and relaxation quality. The pipeline does not eliminate all cross-database inconsistencies; the paper explicitly notes that different k-point grids and pseudopotentials can still introduce noise, and that only pseudopotential compatibility is enforced systematically so far [2508.20875]. A plausible implication is that LeMaterial-Fetcher prioritizes a tractable, reproducible harmonization strategy rather than claiming complete physical equivalence across all merged sources.

## 5. Extensibility, reproducibility, and public distribution

LeMaterial-Fetcher is designed to be modular and extensible. Materials Project, Alexandria, and OQMD are handled via distinct source adapters that know how to query or read the source, map source-specific metadata into the standardized schema, and emit configurations in the canonical internal model [2508.20875]. Once an adapter produces structures, energies, forces, and metadata in that canonical format, the same transformation, validation, harmonization, and export logic is reused automatically [2508.20875].

The paper describes a practical extension workflow. To add a new DFT dataset, one would implement a source-specific loader that iterates over entries or files, parses structures and trajectories, extracts energies, forces, stresses, functional labels, and other metadata, and yields configurations in the expected internal format [2508.20875]. That loader is then registered with a provenance label and functional mapping; optional dataset-specific filters such as stricter force cutoffs or custom convergence checks can also be added [2508.20875]. Because the library is Apache-2.0 licensed and hosted on GitHub, existing connectors can serve as templates [2508.20875].

Reproducibility is tied to configuration-driven execution and explicit storage backends. Data source selections, filters, and export options are encoded in code and configuration, allowing the exact pipeline to be rerun to regenerate LeMat-Traj or produce customized variants [2508.20875]. PostgreSQL and MySQL backends facilitate provenance tracking and consistent intermediate storage [2508.20875]. Export to HuggingFace Datasets adds built-in versioning and metadata tracking [2508.20875].

The public distribution model is straightforward:

| Artifact | Location | License |
|---|---|---|
| LeMaterial-Fetcher | `https://github.com/LeMaterial/lematerial-fetcher` | Apache 2.0 |
| LeMat-Traj | `https://huggingface.co/datasets/LeMaterial/LeMat-Traj` | CC-BY 4.0 |
| LeMat-Traj-subset | `https://huggingface.co/datasets/LeMaterial/LeMat-Traj-subset` | Not separately restated in the provided data |

In comparative context, this places LeMaterial-Fetcher closer to reproducible data-integration frameworks than to monolithic materials databases. MatD3, for example, is an open-source database-backed web framework for storing and disseminating materials data generated by individual groups, emphasizing RESTful APIs and curation at the group level [2001.02135]. LeMaterial-Fetcher instead emphasizes automated aggregation, harmonization, and export for very large DFT trajectory collections [2508.20875].

## 6. Downstream MLIP use, limitations, and prospective development

LeMaterial-Fetcher’s outputs are intended primarily for machine-learning interatomic potential training. The paper demonstrates that LeMat-Traj, produced by this pipeline, is complementary to high-force molecular-dynamics or active-learning corpora. In one reported experiment on a held-out PBE-LeMat-Traj test split, a MACE model pretrained on OMat24 alone achieves a force MAE of 42.7 meV/Å, a model trained on LeMat-Traj alone reaches 50.8 meV/Å, and a model pretrained on OMat24 then fine-tuned on LeMat-Traj reaches 27.2 meV/Å [2508.20875]. The stated interpretation is that high-force MD data are effective for initial training, whereas LeMaterial-Fetcher’s curated near-equilibrium trajectories provide critical low-force refinements [2508.20875].

The same paper reports performance on Matbench Discovery stability prediction, with F1 scores of 0.575 for MACE trained on OMat24, 0.694 on MPtrj, 0.768 on LeMat-Traj full, and 0.772 for OMat24 plus LeMat-Traj fine-tuning [2508.20875]. Because LeMaterial-Fetcher annotates and partitions data by functional, the authors also use it to study transfer across PBE, PBESol, and r2SCAN, showing that functional-aware splits support cross-functional and multi-fidelity strategies [2508.20875].

The current pipeline, however, has explicit limits. Its present usage focuses on geometry optimization trajectories; molecular dynamics trajectories are not yet included [2508.20875]. The PBE split is dominated by Alexandria, approximately 92% by volume, which may introduce source-specific bias [2508.20875]. Detailed DFT parameters such as k-points, pseudopotentials, and cutoffs are gathered when available but are not fully exploited for all entries in the current release [2508.20875]. Cross-database inconsistencies beyond pseudopotential compatibility remain a future study area, and tighter integration with active-learning loops and automated DFT workflows is described as a target for future development [2508.20875].

A common misconception is that LeMaterial-Fetcher solves the entire interoperability problem for materials data. The paper does not make that claim. Instead, it provides a reproducible and extensible framework that systematically addresses format heterogeneity, trajectory reconstruction, schema standardization, and selected DFT compatibility constraints, while leaving broader issues such as MD integration, richer metadata conditioning, and quantitative cross-database bias analysis as open directions [2508.20875].

Source: https://www.emergentmind.com/topics/lematerial-fetcher