MOFSimBench: uMLIP Benchmarking

• MOFSimBench is an open modular framework that benchmarks machine learning interatomic potentials for nanoporous materials, including MOFs, COFs, and zeolites.
• It integrates curated datasets and standardized evaluation tasks such as structure optimization, MD stability, bulk property prediction, and guest–host interaction assessments.
• The framework compares diverse architectures—from equivariant GNNs to kernel models—ensuring reproducible and transparent evaluations for robust materials modeling.

MOFSimBench is an open, modular benchmarking framework for evaluating universal machine learning interatomic potentials (uMLIPs) in nanoporous material modeling, particularly for metal–organic frameworks (MOFs), covalent organic frameworks (COFs), and zeolites. It integrates systematically curated datasets, standardized evaluation tasks, and transparent reporting protocols to enable rigorous and reproducible assessment of uMLIPs across all critical materials modeling workflows, including structure optimization, molecular dynamics (MD) stability, prediction of bulk thermomechanical properties, and guest–host interactions. MOFSimBench sets the reference for comparing architectures—spanning equivariant graph neural networks (GNNs), graph transformers (GTs), graph network simulators (GNS), and kernel models—against both classical force fields and fine-tuned MOF-specific potentials. Its extensible Python platform is publicly available under the MIT license at https://github.com/AI4ChemS/mofsim-bench (Kraß et al., 16 Jul 2025, Edwards et al., 28 Apr 2026).

1. Definition, Scope, and Goals

MOFSimBench was designed to quantify the domain-relevant accuracy and reliability of off-the-shelf uMLIPs within the chemically and structurally diverse space of nanoporous crystals. The central objectives are:

• To enable direct comparison between universal and MOF-specific machine learning potentials, as well as against conventional force fields (e.g., UFF, UFF4MOF).
• To dissect key factors controlling uMLIP performance, including training set diversity, inclusion of out-of-equilibrium conformations, energy–force consistency, and architectural inductive biases.
• To furnish a transparent, reproducible suite of metrics and datasets supporting both static and dynamic materials modeling tasks.

The scope covers MOFs, COFs, and zeolites—materials displaying diverse chemistry, topological complexity, variable coordination environments, and nontrivial guest interactions central to applications in gas storage, catalysis, and beyond (Kraß et al., 16 Jul 2025).

2. Benchmark Datasets and Curation

MOFSimBench integrates five rigorously curated structural and dynamical subsets, each quality-filtered via MOFChecker 2.0 for chemical and topological integrity:

• Four prototypical MOFs: MOF-5, IRMOF-10, UiO-66, HKUST-1.
• A main set of 100 frameworks (83 MOFs, 7 COFs, 10 zeolites), stratified over the largest-pore diameter using data from QMOF, MOASEC-DB, CURATED-COF, and IZA.
• A copper–MOF subset (13 structures, CN = 2–6) for evaluating local coordination changes under MD.
• A heat-capacity set comprising 231 MOFs/COFs/zeolites with DFT C_V references (sourced from Moosavi et al.).
• The GoldDAC host–guest set (26 MOFs × 12 insertions; repulsive, equilibrium, and weak-attractive regimes for CO₂/H₂O) totaling 312 unique structures.

All structures are reduced to primitive cells and checked for missing atoms, overlapping species, or irregular bonding, ensuring robust reference data for each task (Kraß et al., 16 Jul 2025).

3. Task Suite and Evaluation Metrics

MOFSimBench formalizes four primary, workflow-critical tasks, with each associated to physically motivated metrics:

3.1 Structural Optimization

Potentials minimize atomic coordinates and lattice parameters, starting from experimental or hypothetical geometries, using LBFGS convergence (max force ≤10⁻³ eV/Å within 5000 steps). Metrics:

• Success rate: fraction converged.
• Optimization effort: step-count distribution.
• Cell volume deviation: ΔV/V₀ = (V_MLIP – V_DFT)/V_DFT.
• “Accurate” optimizations: |ΔV/V₀| ≤ 10%.

3.2 Molecular Dynamics (MD) Stability

NpT trajectories (50 ps at 300 K, 1 bar) test the robustness of the predicted potential energy surface (PES):

• Volume drift: ΔV/V_init.
• Local coordination change (ΔCN) for copper–MOFs; detected via CrystalNN.
• Outlier trajectory: |ΔV/V| > 10% or |ΔCN| > 0.5.

3.3 Bulk Properties

Bulk Modulus (K)

After optimization, strains (±4%, 11 steps) are applied; energies are fit to the third-order Birch–Murnaghan equation of state:

$$E(V) = E_0 + \frac{9V_0K}{16} \left\{ \left[ \left( \frac{V_0}{V} \right)^{2/3} - 1 \right]^3 K' + \left[ \left( \frac{V_0}{V} \right)^{2/3} - 1 \right]^2 [6 - 4(V_0/V)^{2/3}] \right\}$$

Metrics: MAE and MAPE against DFT-computed K. Unstable fits (volume minimum >1% from V₀) are omitted.

Heat Capacity (C_V)

Phonon density of states from finite displacements (Δ = 0.01 Å) yield the classical-limit heat capacity:

$$C_V = k_B \sum_i \frac{(\hbar \omega_i / k_B T)^2 e^{\hbar \omega_i / k_B T}}{[e^{\hbar \omega_i / k_B T} - 1]^2}$$

Error metrics: MAE (J K⁻¹ g⁻¹) and MAPE to DFT.

3.4 Guest–Host Interactions

Single-point energy and force evaluations on the GoldDAC set:

• Interaction energy: $$E_{\text{int}} = E_{\text{total}}(\text{MOF+guest}) - E_{\text{MOF}} - E_{\text{guest}}$$.
• Force errors: MAE of framework atomic forces. Baselines: UFF + DDEC partial charges, fine-tuned MACE–DAC–1 (Kraß et al., 16 Jul 2025).

4. uMLIP Architectures and Baseline Models

MOFSimBench evaluates >20 uMLIPs, each fitted using reference-quality DFT datasets:

• Equivariant GNNs: MACE–MP–0a/b3, MACE–MPA–0, MACE–OMAT–0, SevenNet variants. Training sets: MPtraj, Alexandria, OMAT24, MATPES.
• Graph Transformers: eqV2–OMsA, eqV2–DeNS, eSEN–OAM/MP (up to 30 M parameters).
• Graph Network Simulators: orb–v2/v3 (including D3-corrected, non-conservative and conservative versions).
• Graph-basis expansions: GRACE–2L variants.
• M3GNet–style: MatterSim–v1 (active-learning database).
• Fine-tuned MOF baseline: MACE–MP–MOF0 (127 MOFs, 4764 DFT points, D3-corrected).

All models implement dispersion corrections consistently, either direct (D3 in output) or at inference (D3(BJ)). Most output energy gradients for conservative force prediction; exceptions (orb–d3–v2, eqV2–OMsA) sacrifice energy–force consistency (Kraß et al., 16 Jul 2025).

5. Quantitative Performance: Tabulated Summary

5.1 Structure Optimization (100 structures; |ΔV/V₀| ≤ 10%)

Model	Success Rate	“Accurate” Rate	Median Steps
eSEN–OAM	95%	89%	350
orb–v3–OMAT	94%	89%	400
MatterSim	92%	86%	420
UFF4MOF	100%	62%	1200
MACE–MP–MOF0	72%	68%	380

5.2 MD Stability (NpT, 50 ps, |ΔV/V| ≤10%)

Model	Stable Trajectories	Outlier Rate	ΔCN_Cu ≤ 0.2
eSEN–OAM	94%	6%	0.05
orb–v3–OMAT	92%	8%	0.08
MatterSim	90%	10%	0.10
UFF4MOF	55%	45%	N/A
MACE–MP–MOF0	68%	32%	0.12

5.3 Bulk Modulus Prediction (100 structures; MAE in GPa)

Model	MAE (GPa)	MAPE (%)	N structures
orb–v3–OMAT	2.97	23.4	98
eSEN–OAM	2.64	22.1	96
SevenNet–ompa	3.35	25.7	99
MACE–MP–MOF0	3.14	24.2	68
MACE–MP–0a	7.15	45.0	100

5.4 Heat Capacity Prediction (231 structures; MAE in J K⁻¹ g⁻¹)

Model	MAE	MAPE (%)	N structures
orb–v3–OMAT	0.018	2.3	231
MACE–MP–MOF0	0.020	2.6	128
eSEN–OAM	0.024	3.0	231

5.5 Host–Guest Interaction (312 snapshots; energy MAE (meV), force MAE (eV/Å))

Model	E_int MAE	F MAE	Outperforms MACE–DAC–1 & UFF+DDEC?
eSEN–OAM	8.5	0.023	Yes
MatterSim	10.2	0.027	Yes
MACE–DAC–1	12.5	0.035	—
UFF+DDEC	35.0	0.120	—

6. Insights, Limitations, and Recommendations

• Training-data diversity, particularly the inclusion of out-of-equilibrium conformations (e.g., OMat24), is the dominant factor driving performance gains across all tasks; model architecture is secondary provided the data is sufficiently broad.
• Force-conservativeness—guaranteed when forces are energy gradients—is essential, as non-conservative models exhibit PES instabilities and poor dynamical behavior.
• Architectural details (e.g., equivariance, message-passing depth) yield only modest improvements on matching datasets, with near-SOTA performance from both GNS, GNN, GT, and graph-basis models.
• Fine-tuning on narrow MOF data corrects PES softening and can improve task accuracy, but typically reduces chemical coverage, while universal models trained on much larger, diverse datastores maintain broader applicability and match or exceed fine-tuned baselines.
• Guest–host energetics and force prediction both benefit from data diversity and force consistency, with top uMLIPs surpassing classical baselines (e.g., UFF+DDEC) in accuracy on the GoldDAC set.
• MOFSimBench exposes gaps in uMLIP robustness at high temperature and during bond-breaking, as benchmarks at 1000–2000 K (e.g., ORB-v3, OMAT on ZIF-8, CALF-20, MOF-5, etc.) show error escalation beyond static validation values (Edwards et al., 28 Apr 2026). This suggests explicit high-temperature/bond-breaking data are required for generative accuracy in extreme regimes.

7. Platform, Extensibility, and Best Practices

MOFSimBench is implemented as a modular Python framework leveraging ASE drivers for structure optimization, MD, EOS fitting, and phonon calculations. Extensibility features include:

• Task modularity: Each evaluation task is a stand-alone module; new metrics/tasks can be added via subclassing an abstract “Task” interface.
• Model integration: Any uMLIP can be wrapped by implementing a basic API (energy, forces, optional stress), with automatic D3(BJ) augmentation supported via torch-dftd.
• Data management: Structural sets stored as CIF+metadata, easily expanded by new YAML configuration files.
• Reporting: Standardized table generation, parity/violin plots, CSV/JSON output for automated analysis and publication-ready figures.

Recommended practices for users:

1. Guarantee energy–force consistency (∂E/∂x) or validate non-conservative force outputs explicitly.
2. Include out-of-equilibrium and high-energy samples in model training to avoid PES softening.
3. Apply D3(BJ) dispersion corrections consistently at both training and test stages.
4. Validate models on all four tasks (geometry, MD, bulk properties, guest–host) before domain deployment.

By providing a standardized, rigorously curated evaluation suite, MOFSimBench accelerates responsible adoption and robust benchmarking of universal ML potentials for nanoporous material simulation workflows, while also highlighting ongoing challenges in the field such as high-temperature decomposition, non-equilibrium PES sampling, and generalization to novel topologies (Kraß et al., 16 Jul 2025, Edwards et al., 28 Apr 2026).