LeMaterial-Fetcher: Unified Atomistic Pipeline

• LeMaterial-Fetcher is a robust Python framework that standardizes and integrates atomistic trajectories from heterogeneous DFT sources into a unified, trajectory-aware schema.
• It employs a five-stage pipeline—fetching, transformation, validation, harmonization, and export—to ensure data consistency, reproducibility, and quality control.
• The framework powers downstream ML interatomic potential workflows by curating over 120 million configurations and supporting cross-functional data harmonization.

Searching arXiv for the cited LeMaterial papers and adjacent context. LeMaterial-Fetcher is a Python library and full data-engineering pipeline for constructing large, unified datasets of atomistic configurations and trajectories from heterogeneous Density Functional Theory sources. It was developed to build LeMat-Traj, and is presented as a general, reproducible framework for fetching, transforming, validating, harmonizing, and distributing very large collections of atomistic data from repositories such as the Materials Project, Alexandria, and OQMD (Ramlaoui et al., 28 Aug 2025). Within the LeMaterial ecosystem, its role is infrastructural rather than predictive: it standardizes source-specific calculations into an OPTIMADE-based, trajectory-aware schema, applies quality control and cross-source harmonization, and exports the resulting data in HuggingFace Datasets format for downstream machine-learning interatomic potential workflows (Ramlaoui et al., 28 Aug 2025).

1. Definition and role in the LeMaterial ecosystem

LeMaterial-Fetcher was introduced to address several systemic problems in the use of existing DFT trajectory data for machine learning interatomic potentials. The motivating issues are the fragmentation of data sources, inconsistent formats, heterogeneous DFT settings, missing or inconsistent metadata, and the difficulty of maintaining reproducible aggregation pipelines as upstream repositories evolve (Ramlaoui et al., 28 Aug 2025). In response, the library provides a unified, automated pipeline for fetching, transforming, validating, harmonizing, and exporting data, and encapsulates source-specific logic in a modular, extensible framework (Ramlaoui et al., 28 Aug 2025).

Its immediate function is to underpin the construction of LeMat-Traj, which aggregates more than 120 million atomic configurations from Materials Project, Alexandria, and OQMD (Ramlaoui et al., 28 Aug 2025). More generally, it is designed as a reusable community framework: because the pipeline is public and configuration-based, others can reproduce LeMat-Traj or build compatible datasets for their own MLIP pipelines (Ramlaoui et al., 28 Aug 2025).

This positioning places LeMaterial-Fetcher at the intersection of data engineering, materials informatics, and atomistic machine learning. A plausible implication is that its main contribution is not a new learning objective or architecture, but the conversion of an error-prone, one-off aggregation process into a repeatable data infrastructure layer. In that respect it complements later LeMaterial efforts such as LeMat-Bulk, which extends cross-database standardization and de-duplication for static materials databases using a related fetcher-based integration approach (Siron et al., 7 Nov 2025).

2. Pipeline architecture and source-specific ingestion

The core conceptual pipeline has five stages: fetching, transformation, validation, harmonization, and push/export (Ramlaoui et al., 28 Aug 2025). In the fetching stage, the library interfaces with open APIs or direct downloads from Materials Project, Alexandria, and OQMD, and retrieves tasks or entries together with associated calculations and trajectories (Ramlaoui et al., 28 Aug 2025). In transformation, it converts source-specific formats into a unified representation, standardizes atomic structure representations, energy units, and force components, and extracts DFT metadata using Pymatgen and Matminer (Ramlaoui et al., 28 Aug 2025). Validation checks physical and numerical plausibility, enforces consistency with an OPTIMADE-like schema, and removes incomplete or inconsistent entries (Ramlaoui et al., 28 Aug 2025). Harmonization aligns DFT flags and metadata where available, partitions data into splits by DFT functional, and ensures pseudopotential compatibility following the method of LeMat-Bulk (Ramlaoui et al., 28 Aug 2025). Export stores intermediate data in a database backend and writes final outputs in HuggingFace Datasets format for efficient streaming, versioning, and metadata tracking (Ramlaoui et al., 28 Aug 2025).

The implementation uses PostgreSQL as the default backend for storing both raw downloaded data and transformed, validated structures before export, while MySQL is used specifically to read OQMD because OQMD is published as a MySQL database with multiple tables (Ramlaoui et al., 28 Aug 2025). The design is explicitly backend-pluggable. It is also engineered for high parallelism and memory efficiency: the paper reports that it can handle 120M+ rows without exhausting memory or leaking via multiple concurrent database connections, and gives examples ranging from 128 cores and 256 GB RAM to an end-to-end run of approximately 16 hours on an AMD Ryzen 5600G with 12 workers for fetching, processing, and uploading 120M rows (Ramlaoui et al., 28 Aug 2025).

Source adapters are distinct and tailored. For Materials Project, the pipeline iterates over all tasks, discards deprecated tasks, and collects geometry optimization trajectories that pass filtering criteria (Ramlaoui et al., 28 Aug 2025). For Alexandria, it uses all samples except those containing Ytterbium, because of pseudopotential incompatibility with other sources (Ramlaoui et al., 28 Aug 2025). For OQMD, it traverses the full database, aggregates relaxation, coarse relaxation, and fine relaxation calculations for each entry, and extracts trajectories from input and output structures when all required targets are present in the correct format (Ramlaoui et al., 28 Aug 2025).

A recurrent misunderstanding is to treat LeMaterial-Fetcher as merely a downloader for three repositories. That characterization is incomplete. The fetch stage is only the first layer; most of the library’s technical substance lies in transformation, trajectory reconstruction, DFT harmonization, validation, and standardized export (Ramlaoui et al., 28 Aug 2025).

3. Schema, trajectory representation, and standardization

LeMaterial-Fetcher standardizes data according to the OPTIMADE specification, with minimal but important trajectory extensions (Ramlaoui et al., 28 Aug 2025). Each row corresponds to a single atomistic configuration; full trajectories are reconstructed by grouping rows with the same trajectory identifier and ordering frames by Relaxation Step (Ramlaoui et al., 28 Aug 2025). Two trajectory-specific fields are central. Relaxation Step is an integer index within a given optimization sequence, allowing reconstruction of relaxation paths and energy/force-versus-step curves. Relaxation Number distinguishes different relaxation runs for the same initial structure, reflecting multi-stage workflows such as coarse to fine relaxation or re-relaxation with tighter thresholds or different methods (Ramlaoui et al., 28 Aug 2025).

The enforced physical fields include structural information, DFT targets, and metadata. Structural content includes atomic species or chemical symbols, atomic positions, and lattice or cell parameters (Ramlaoui et al., 28 Aug 2025). DFT targets include total energy per configuration and forces on each atom, while stress is mentioned in modeling experiments and is therefore likely stored when available, though not explicitly spelled out (Ramlaoui et al., 28 Aug 2025). Metadata include DFT functional labels—PBE, PBESol, SCAN, or r2SCAN—source provenance, original calculation identifiers, and potentially k-point mesh, cutoff energy, and pseudopotentials when sources expose them (Ramlaoui et al., 28 Aug 2025).

The trajectory-aware schema is directly motivated by downstream tasks. The paper identifies Relaxation Step and Relaxation Number as central for IS2RE and IS2RS tasks, amortized optimization, and trajectory-aware self-supervised learning, and notes that they also support trajectory-level splitting to avoid leakage (Ramlaoui et al., 28 Aug 2025). This suggests that the schema is not only a storage format but an inductive interface between atomistic data curation and trajectory-sensitive ML objectives.

Standardization is deliberately narrow where needed. Functional labels are normalized to a small set, and dataset splits are created per functional (Ramlaoui et al., 28 Aug 2025). Energies and forces are converted to uniform units so that shared filtering thresholds are meaningful across sources (Ramlaoui et al., 28 Aug 2025). Space groups are computed with moyo, described as a faster alternative to Spglib, using a strict tolerance symprec = 10^{-4} to enable consistent symmetry labels across the dataset (Ramlaoui et al., 28 Aug 2025).

4. Harmonization, filtering, and quality control

LeMaterial-Fetcher’s quality-control layer defines much of LeMat-Traj’s character. At the most basic level, any configuration lacking energy or atomic forces is removed (Ramlaoui et al., 28 Aug 2025). Beyond completeness, filtering is applied at the trajectory level using two explicit convergence criteria (Ramlaoui et al., 28 Aug 2025).

First, a trajectory is discarded if the energy difference between the penultimate and final steps exceeds

$2 \times 10^{-2} \text{ eV}.$

Second, a trajectory is discarded if the maximum atomic force norm in the final configuration exceeds

$0.2 \, \text{eV/\AA}.$

The paper states that this force threshold is intentionally not ultra-strict: it ensures final structures are at least near equilibrium while still allowing inclusion of structures that are not perfectly relaxed, thereby preserving information about the potential-energy surface away from the exact minimum (Ramlaoui et al., 28 Aug 2025).

All configurations are also validated against OPTIMADE format constraints, and any entry failing schema checks or additional integrity tests is removed (Ramlaoui et al., 28 Aug 2025). Harmonization further includes unit conversion, normalization of functional labels, and pseudopotential compatibility logic. An example given is the treatment of Yb-containing structures: they are taken from Materials Project rather than Alexandria because of pseudopotential incompatibilities (Ramlaoui et al., 28 Aug 2025). This compatibility logic is closely related to the broader LeMat-Bulk effort, where cross-database filtering and pseudopotential alignment are formalized at larger scale (Siron et al., 7 Nov 2025).

These procedures matter because naive aggregation of DFT outputs across repositories would conflate differences in electronic-structure settings, metadata completeness, and relaxation quality. The pipeline does not eliminate all cross-database inconsistencies; the paper explicitly notes that different k-point grids and pseudopotentials can still introduce noise, and that only pseudopotential compatibility is enforced systematically so far (Ramlaoui et al., 28 Aug 2025). A plausible implication is that LeMaterial-Fetcher prioritizes a tractable, reproducible harmonization strategy rather than claiming complete physical equivalence across all merged sources.

5. Extensibility, reproducibility, and public distribution

LeMaterial-Fetcher is designed to be modular and extensible. Materials Project, Alexandria, and OQMD are handled via distinct source adapters that know how to query or read the source, map source-specific metadata into the standardized schema, and emit configurations in the canonical internal model (Ramlaoui et al., 28 Aug 2025). Once an adapter produces structures, energies, forces, and metadata in that canonical format, the same transformation, validation, harmonization, and export logic is reused automatically (Ramlaoui et al., 28 Aug 2025).

The paper describes a practical extension workflow. To add a new DFT dataset, one would implement a source-specific loader that iterates over entries or files, parses structures and trajectories, extracts energies, forces, stresses, functional labels, and other metadata, and yields configurations in the expected internal format (Ramlaoui et al., 28 Aug 2025). That loader is then registered with a provenance label and functional mapping; optional dataset-specific filters such as stricter force cutoffs or custom convergence checks can also be added (Ramlaoui et al., 28 Aug 2025). Because the library is Apache-2.0 licensed and hosted on GitHub, existing connectors can serve as templates (Ramlaoui et al., 28 Aug 2025).

Reproducibility is tied to configuration-driven execution and explicit storage backends. Data source selections, filters, and export options are encoded in code and configuration, allowing the exact pipeline to be rerun to regenerate LeMat-Traj or produce customized variants (Ramlaoui et al., 28 Aug 2025). PostgreSQL and MySQL backends facilitate provenance tracking and consistent intermediate storage (Ramlaoui et al., 28 Aug 2025). Export to HuggingFace Datasets adds built-in versioning and metadata tracking (Ramlaoui et al., 28 Aug 2025).

The public distribution model is straightforward:

Artifact	Location	License
LeMaterial-Fetcher	https://github.com/LeMaterial/lematerial-fetcher	Apache 2.0
LeMat-Traj	https://huggingface.co/datasets/LeMaterial/LeMat-Traj	CC-BY 4.0
LeMat-Traj-subset	https://huggingface.co/datasets/LeMaterial/LeMat-Traj-subset	Not separately restated in the provided data

In comparative context, this places LeMaterial-Fetcher closer to reproducible data-integration frameworks than to monolithic materials databases. MatD3, for example, is an open-source database-backed web framework for storing and disseminating materials data generated by individual groups, emphasizing RESTful APIs and curation at the group level (Laasner et al., 2019). LeMaterial-Fetcher instead emphasizes automated aggregation, harmonization, and export for very large DFT trajectory collections (Ramlaoui et al., 28 Aug 2025).

6. Downstream MLIP use, limitations, and prospective development

LeMaterial-Fetcher’s outputs are intended primarily for machine-learning interatomic potential training. The paper demonstrates that LeMat-Traj, produced by this pipeline, is complementary to high-force molecular-dynamics or active-learning corpora. In one reported experiment on a held-out PBE-LeMat-Traj test split, a MACE model pretrained on OMat24 alone achieves a force MAE of 42.7 meV/Å, a model trained on LeMat-Traj alone reaches 50.8 meV/Å, and a model pretrained on OMat24 then fine-tuned on LeMat-Traj reaches 27.2 meV/Å (Ramlaoui et al., 28 Aug 2025). The stated interpretation is that high-force MD data are effective for initial training, whereas LeMaterial-Fetcher’s curated near-equilibrium trajectories provide critical low-force refinements (Ramlaoui et al., 28 Aug 2025).

The same paper reports performance on Matbench Discovery stability prediction, with F1 scores of 0.575 for MACE trained on OMat24, 0.694 on MPtrj, 0.768 on LeMat-Traj full, and 0.772 for OMat24 plus LeMat-Traj fine-tuning (Ramlaoui et al., 28 Aug 2025). Because LeMaterial-Fetcher annotates and partitions data by functional, the authors also use it to study transfer across PBE, PBESol, and r2SCAN, showing that functional-aware splits support cross-functional and multi-fidelity strategies (Ramlaoui et al., 28 Aug 2025).

The current pipeline, however, has explicit limits. Its present usage focuses on geometry optimization trajectories; molecular dynamics trajectories are not yet included (Ramlaoui et al., 28 Aug 2025). The PBE split is dominated by Alexandria, approximately 92% by volume, which may introduce source-specific bias (Ramlaoui et al., 28 Aug 2025). Detailed DFT parameters such as k-points, pseudopotentials, and cutoffs are gathered when available but are not fully exploited for all entries in the current release (Ramlaoui et al., 28 Aug 2025). Cross-database inconsistencies beyond pseudopotential compatibility remain a future study area, and tighter integration with active-learning loops and automated DFT workflows is described as a target for future development (Ramlaoui et al., 28 Aug 2025).

A common misconception is that LeMaterial-Fetcher solves the entire interoperability problem for materials data. The paper does not make that claim. Instead, it provides a reproducible and extensible framework that systematically addresses format heterogeneity, trajectory reconstruction, schema standardization, and selected DFT compatibility constraints, while leaving broader issues such as MD integration, richer metadata conditioning, and quantitative cross-database bias analysis as open directions (Ramlaoui et al., 28 Aug 2025).