UniFFBench: Experimental Benchmark for UMLFFs

• UniFFBench is an experimental benchmarking framework for universal machine learning force fields in atomistic simulations, using curated mineral structures to reveal the 'reality gap'.
• It employs the MinX dataset of approximately 1,500 minerals to test model stability, structural accuracy, dynamic behavior, and mechanical fidelity under varied conditions.
• The evaluation protocol exposes key limitations of current UMLFFs, highlighting that good computational performance does not necessarily translate to experimental reliability.

Searching arXiv for the specified paper and closely related benchmark context. arXiv.search(query="(Mannan et al., 7 Aug 2025)", max_results=5) arXiv.search(query="UniFFBench universal machine learning force fields experimental measurements", max_results=10) UniFFBench is an experimental benchmarking framework for universal machine learning force fields (UMLFFs) in atomistic materials simulation. It was introduced to evaluate whether models that perform well on computational benchmarks also remain reliable when confronted with experimentally observed mineral structures, thermodynamic conditions, compositional disorder, and mechanical response. The framework is centered on approximately 1,500 carefully curated mineral structures and is designed to expose a “reality gap” between performance on DFT-centered validation sets and performance against experimental measurements. In the reported evaluation of six state-of-the-art UMLFFs—CHGNet, M3GNet, MACE, MatterSim, SevenNet, and Orb—the benchmark shows that controlled computational accuracy does not reliably transfer to experimental complexity, and that even the strongest models remain above the density-error threshold described as necessary for practical applications (Mannan et al., 7 Aug 2025).

1. Concept and motivation

UniFFBench was proposed as a response to a specific limitation in UMLFF assessment: most existing evaluations are grounded in computational reference data rather than direct experimental observations. In that setting, training and validation can become partially circular, because the same class of calculations underlies both model construction and benchmark targets. UniFFBench instead uses experimental measurements of minerals to test real-world reliability under structural complexity, chemical diversity, extreme environments, and mechanical property constraints (Mannan et al., 7 Aug 2025).

The benchmark is explicitly aimed at atomistic simulation use cases in materials discovery. Its central claim is not that computational benchmarks are uninformative, but that they provide controlled comparisons that may overestimate robustness when models extrapolate to experimentally complex chemical spaces. A recurrent theme is that good force and energy behavior near initialization does not guarantee long-horizon molecular dynamics stability, nor does stable dynamics guarantee accurate mechanical properties. This suggests that “universality” for machine learning force fields cannot be inferred from standard computational leaderboards alone.

2. MinX dataset and experimental coverage

The core dataset in UniFFBench is MinX, which contains approximately 1,500 experimentally determined mineral structures spanning diverse chemical environments, bonding types, structural complexity, and elastic properties (Mannan et al., 7 Aug 2025). MinX is partitioned into four complementary subsets that probe distinct failure modes and deployment regimes.

Subset	Purpose	Size
MinX-EQ	Minerals under standard ambient conditions	Not separately specified
MinX-HTP	Minerals under extreme thermodynamic conditions	75
MinX-POcc	Minerals with partial atomic site occupancies	50
MinX-EM	Minerals with experimentally measured elastic tensors	100

MinX-EQ represents standard ambient conditions. MinX-HTP contains minerals under broader temperature and pressure ranges and is intended to test robustness under extreme thermodynamic conditions. MinX-POcc targets compositional disorder through partial atomic site occupancies; in this subset, supercells are generated using occupancy statistics and random sampling. MinX-EM is curated from the Materials Property Open Database (MPOD) and enables direct comparison between predicted and experimentally measured elastic tensors (Mannan et al., 7 Aug 2025).

Several properties of MinX distinguish it from typical UMLFF training corpora. The dataset is described as having near-complete coverage of the periodic table, with only americium absent in both MinX and the training data. Structures can contain up to 23 unique elements, whereas the computational training set MPtrj is reported to max at 9. Many structures contain hundreds of atoms per unit cell, and the benchmark includes thermodynamic diversity, partial occupancy, and experimentally measured elastic behavior. The benchmark therefore tests extrapolation not merely across compositions, but across scale, disorder, and environmental state.

3. Evaluation protocol and measured quantities

UniFFBench evaluates UMLFFs with a multi-faceted protocol extending beyond static energy and force prediction. Experimental crystallographic information is used directly, including lattice parameters, densities, bond lengths, radial distribution functions (RDFs), and elastic constants. Where possible, molecular dynamics simulations are initialized from experimental CIF files that contain temperature and pressure conditions, and equilibrium is judged relative to experimental averages (Mannan et al., 7 Aug 2025).

The protocol includes seven major components: simulation stability, structural accuracy, atomic-scale organization, dynamic stability and temporal evolution, bond-length accuracy, mechanical properties, and computational efficiency. Stability is measured through the fraction of successful MD runs and through analysis of failure modes such as memory overflow and unphysical forces. Structural accuracy uses mean absolute percentage error (MAPE) for densities and lattice parameters. RDF-based comparison is used to assess atomic-scale organization, with thermal broadening handled through small noise perturbations. Dynamic errors are tracked over 50 ps MD simulations. Bond-length errors are evaluated over all atomic pair combinations. Mechanical validation uses finite-strain calculations of elastic tensor components $C_{ij}$, together with Voigt-averaged moduli. Efficiency is quantified with time per MD step, total resource usage, and an efficiency score defined as success rate divided by time per step (Mannan et al., 7 Aug 2025).

The benchmark explicitly defines several quantitative measures. For density and lattice changes, the reported expressions are

$$\text{Density Error} = \left( \frac{\rho_f - \rho_i}{\rho_i} \right) \times 100$$

and

$$\text{Lattice Error} = \left( \frac{a_f - a_i}{a_i} \right) \times 100.$$

For aggregate simulation reliability, the reported success-rate expression is

$$\textrm{Success Rate (\%)} = \frac{\sum_{i=1}^{3} w_i \cdot s_i}{\sum_{i=1}^{3} w_i},$$

where $w_i$ is the number of minerals in dataset $i$ and $s_i$ is the successful simulation fraction.

For elastic-property reduction, UniFFBench uses Voigt averaging:

$$K_V = \frac{1}{9}\left(C_{11}+C_{22}+C_{33}+2(C_{12}+C_{13}+C_{23})\right),$$

$$G_V = \frac{1}{15}\left(C_{11}+C_{22}+C_{33}-C_{12}-C_{13}-C_{23}+3(C_{44}+C_{55}+C_{66})\right),$$

$E_V = \frac{9K_V G_V}{3K_V + G_V}.$

These definitions are included because the benchmark treats experimental structure, dynamics, and mechanics as a single evaluation surface rather than as isolated tasks.

4. Performance on stability, structure, and efficiency

The six models evaluated in UniFFBench are CHGNet, M3GNet, MACE, MatterSim, SevenNet, and Orb. On simulation stability, Orb and MatterSim achieved 100% completion across all datasets. MACE and SevenNet showed high completion, reported as up to approximately 97% for MinX-HTP and down to approximately 75% for MinX-POcc. CHGNet and M3GNet remained below 15% completion, with most simulations failing (Mannan et al., 7 Aug 2025).

Structural accuracy is reported to be substantially better for Orb, MatterSim, SevenNet, and MACE than for CHGNet and M3GNet. For the stronger models, MAPE for density and lattice parameters is usually below 10%, but still above the 2–3% experimental threshold stated as necessary for practical use. Partial occupancy is a consistent stressor: for all models, MAPE increases by 2–3× relative to ambient-condition cases. RDF behavior also remains imperfect. Even the best models place only about 50% of cases within a 50% RDF-error threshold, while CHGNet and M3GNet exhibit consistently high errors and do not reliably preserve atomic structure under MD (Mannan et al., 7 Aug 2025).

Bond-length analysis reveals a specific form of dataset dependence. The reported benchmark shows a clear correlation between bond error and the frequency of a bond type in the training set; O-containing bonds, which are overrepresented, tend to have lower error. At the same time, frequent pairs can still have high error if the relevant chemical context is absent from training. This result is central to UniFFBench’s interpretation: apparent generality over elements is insufficient if environment-level representation remains poor.

Efficiency is treated as a practical deployment variable rather than a secondary convenience. Orb is reported as the fastest model at 0.736 s per MD step, with 100% reliability and an efficiency score of 135.9. MatterSim is nearly as fast, with 100% reliability and an efficiency score of 128.2. CHGNet and M3GNet are both slow and unreliable, with efficiency below 5. The full UniFFBench campaign required 36,500 CPU-days, underscoring why both success rate and throughput matter for large-scale screening (Mannan et al., 7 Aug 2025).

5. Mechanical validation and the structure–mechanics disconnect

Mechanical-property evaluation is one of the defining features of UniFFBench. The MinX-EM subset contains 100 minerals with experimentally measured elastic tensors, and the benchmark compares predicted $$\text{Density Error} = \left( \frac{\rho_f - \rho_i}{\rho_i} \right) \times 100$$0 components against those measurements. None of the evaluated UMLFFs are explicitly trained on elastic tensors, making this a stringent out-of-distribution test (Mannan et al., 7 Aug 2025).

The reported results reveal a pronounced disconnect between stable structure prediction and mechanical fidelity. Orb, despite strong structural and dynamic stability, performs poorly on elastic prediction, with MAPE of approximately 41.5% for $$\text{Density Error} = \left( \frac{\rho_f - \rho_i}{\rho_i} \right) \times 100$$1, 99.8% for $$\text{Density Error} = \left( \frac{\rho_f - \rho_i}{\rho_i} \right) \times 100$$2, and 100.0% for $$\text{Density Error} = \left( \frac{\rho_f - \rho_i}{\rho_i} \right) \times 100$$3. MatterSim, MACE, and SevenNet are more reasonable on $$\text{Density Error} = \left( \frac{\rho_f - \rho_i}{\rho_i} \right) \times 100$$4, with reported MAPE ranges of about 16–23%, but their errors rise substantially for $$\text{Density Error} = \left( \frac{\rho_f - \rho_i}{\rho_i} \right) \times 100$$5 and $$\text{Density Error} = \left( \frac{\rho_f - \rho_i}{\rho_i} \right) \times 100$$6, reaching approximately 27.1–40.4%. CHGNet and M3GNet produce essentially no reliable predictions because almost all relevant simulations fail (Mannan et al., 7 Aug 2025).

This result is methodologically important because it undermines a common assumption that stable MD trajectories and acceptable density evolution imply accurate derivatives of the energy landscape relevant to elasticity. UniFFBench shows that this implication does not hold in general. A plausible implication is that benchmarks restricted to equilibrium structures, energies, and forces may miss deficiencies in higher-order response behavior that become visible only when finite-strain and experimentally anchored elastic tests are included.

6. Failure modes, curvature analysis, and training-data bias

UniFFBench attributes much of the observed performance variation to training-data representation rather than to model architecture. The benchmark states that success and failure are primarily correlated with the frequency of element pairs and chemical environments in the training data, not with the modeling method itself. Models therefore struggle to generalize to chemical environments that are absent or underrepresented in training, even when the constituent elements are individually common (Mannan et al., 7 Aug 2025).

The framework also reports that simulation failure often has no clear warning signal. Low energy and force errors at initialization do not reliably predict long-term stability. This directly challenges the idea that a short, static pre-check is sufficient for deployment screening. UniFFBench further identifies “mathematical roughness” in learned pair potentials as a diagnostic factor: poorly resolved or rough short-range repulsive behavior is associated with numerical instability in MD and with poor mechanical accuracy. The benchmark formalizes this diagnostic through pairwise curvature measures denoted $$\text{Density Error} = \left( \frac{\rho_f - \rho_i}{\rho_i} \right) \times 100$$7 and $$\text{Density Error} = \left( \frac{\rho_f - \rho_i}{\rho_i} \right) \times 100$$8, based on the second derivative of the potential (Mannan et al., 7 Aug 2025).

These findings organize the reported “reality gap” into three distinct but connected failure modes. First, models can overfit to computational benchmarks and fail under thermal fluctuations, partial occupancy, and extreme environments. Second, structural plausibility can coexist with mechanical inaccuracy. Third, numerical smoothness of the learned potential affects both stability and physical reliability. Together, these observations support the benchmark’s claim that experimental complexity exposes systematic limitations not visible in conventional validation.

7. Position in benchmarking practice and reported outlook

Relative to prior computational benchmarks, UniFFBench is distinguished by three design choices: experimentally grounded reference data, materially broader structural and chemical diversity, and multi-scale evaluation extending from equilibrium structure to temporal evolution and elastic response (Mannan et al., 7 Aug 2025). The benchmark also systematizes quantities that are often omitted from model reports, including failure rates, failure analysis, resource usage, and efficiency. In that sense, it functions not only as a dataset but as a proposed validation standard for UMLFF deployment.

The reported recommendations follow directly from the observed limitations. They include multi-target training protocols that incorporate experimental property prediction, such as elastic tensor components and phonon spectra, into loss functions; diversification of training data toward broader chemical and environmental coverage; architectural innovation that combines force prediction with higher-order property learning while emphasizing invariance and equivariance; community benchmarking standards that include experimental data, failure rates, and practical accuracy thresholds; and transparent reporting of computational cost and failures. These are presented as necessary steps toward genuinely universal force-field capability rather than as optional benchmark extensions (Mannan et al., 7 Aug 2025).

The benchmark’s supporting resources include a Zenodo release of the dataset at 10.5281/zenodo.16733258 and a public code repository at github.com/M3RG-IITD/UniFFBench. Within the literature on machine-learned interatomic potentials, UniFFBench is significant because it reframes evaluation around experimentally realized materials behavior. Its principal conclusion is not that UMLFFs are ineffective, but that present claims of universality remain constrained by training-data bias, incomplete environmental coverage, and a measurable disconnect between simulation stability and experimentally validated mechanics.