cRMSE: Metric for Crystal Structure Prediction

• The metric cRMSE is defined as the average RMSE over all test structures by incorporating a fixed penalty (stol) for unmatched cases.
• It integrates a match-based criterion (METRe) with RMSE to ensure that evaluation does not ignore failures in structure prediction.
• cRMSE improves benchmark reliability in polymorphic datasets but is sensitive to matching tolerances and dataset duplication.

cRMSE is explicitly defined in the supplied literature as corrected RMSE, an evaluation metric for crystal structure prediction that combines a match-based criterion with a distance/error term so that non-matching structures are not ignored in the aggregate score (Martirossyan et al., 15 Sep 2025). In the same corpus, closely related strings denote different objects: CRMSE is Contrastive Regularized MSE for machine-learning interatomic potentials (Tzivrailis et al., 30 Jun 2026), while several papers discuss CRM as continual reassessment, conditional risk minimization, counterfactual risk minimization, or conformal risk minimization rather than any root-mean-square-error quantity (Koopmeiners et al., 2014, Zimin et al., 2018, Zenati et al., 2023, Noorani et al., 2024). This suggests that, in technical usage, cRMSE should be interpreted only within its immediate methodological context.

1. Definition and naming

In (Martirossyan et al., 15 Sep 2025), cRMSE means “corrected RMSE”. It is introduced in the section “New metric to combine RMSE and match rate” as a metric intended to avoid the standard failure mode in which RMSE is computed only on matched structures and therefore ignores non-matches. The paper’s interpretation is direct: cRMSE is the average RMSE over all test structures, where any non-matching test structure is assigned a penalty equal to stol (Martirossyan et al., 15 Sep 2025).

The same paper presents cRMSE alongside METRe, defined as “match everyone to reference”, and ordinary RMSE. Their intended division of labor is explicit: METRe measures how many reference structures are covered by the generated set, RMSE measures alignment error on matched structures, and cRMSE combines both by incorporating unmatched references through a fixed penalty (Martirossyan et al., 15 Sep 2025).

The broader corpus also shows that the label is not universal. One paper introduces CRMSE, not cRMSE, as a post-training objective for molecular sampling (Tzivrailis et al., 30 Jun 2026). Other papers explicitly state that they do not define cRMSE and instead treat CRPS or various forms of CRM (Friederichs et al., 2012, Noorani et al., 2024). A plausible implication is that exact acronym expansion matters: the literature uses near-identical letter strings for conceptually different metrics and objectives.

2. Formal definition as corrected RMSE

The defining formula in (Martirossyan et al., 15 Sep 2025) is

$$\mathrm{mean\ cRMSE} (\mathtt{stol}) = \dfrac{\sum_{i=1}^{N_{\mathrm{test}}} \mathrm{RMSE}_i + \mathtt{stol}(N_{\mathrm{test}} - N_{\mathrm{ref. match}})}{N_{\mathrm{test}}}$$

and the paper rewrites it as

$$\mathrm{mean\ cRMSE} = \mathrm{METRe} * (\mathrm{mean\ RMSE} - \mathtt{stol}) + \mathtt{stol}$$

with

$$\mathrm{METRe}=\frac{N_\mathrm{ref. match}}{N_\mathrm{test}}$$

and

$$\mathrm{mean\ RMSE}=\frac{\sum_{i=1}^{N_\mathrm{ref. match}} \mathrm{RMSE}_i}{N_\mathrm{ref. match}}.$$

These equations encode the metric’s central correction. For matched reference structures, the observed best RMSE contributes to the average. For unmatched reference structures, the missing RMSE is replaced by the threshold stol. The metric is therefore a full-test-set average rather than a matched-subset average (Martirossyan et al., 15 Sep 2025).

The paper further states the rationale for the penalty choice: stol is used because it “sets the threshold for the computed RMSE of the aligned structures in StructureMatcher (if a mapping can be found).” In this construction, stol functions simultaneously as a StructureMatcher tolerance and as the default penalty for failure to match (Martirossyan et al., 15 Sep 2025).

Special cases are also specified. If every reference matches, cRMSE is close to ordinary mean RMSE. If no reference matches, cRMSE becomes stol. These are not auxiliary interpretations but direct consequences identified in the paper’s discussion of normalization, thresholds, and special cases (Martirossyan et al., 15 Sep 2025).

3. Computation protocol and dependence on matching

The appendix-level procedure in (Martirossyan et al., 15 Sep 2025) makes the computation explicit. The required inputs are a list of generated structures, a list of test or reference structures, and the Pymatgen StructureMatcher tolerances ltol, stol, and angle_tol. For each generated structure, the method compares it against every test structure using StructureMatcher, retaining for each reference the best (lowest) RMSE found so far across all generated structures (Martirossyan et al., 15 Sep 2025).

After all generated structures have been processed, matched test structures have stored RMSE values and unmatched ones remain NaN. The paper then computes METRe as the fraction of test structures that obtained at least one match, computes mean RMSE over matched references only, and finally computes cRMSE by replacing every unmatched structure’s missing RMSE with stol and averaging over all test structures (Martirossyan et al., 15 Sep 2025).

This protocol makes cRMSE fundamentally reference-based. The criterion is not whether each generated structure has a nearby test counterpart, but whether each reference structure is covered by the generated set. In the paper’s terminology, this is what aligns cRMSE with METRe rather than with the older one-to-one match-rate convention (Martirossyan et al., 15 Sep 2025).

The dependence on matching tolerances is also explicit. The reported metric is parameterized as $\mathrm{mean\ cRMSE}(\mathtt{stol})$, and in the experiments the standard benchmark setting is stol = 0.5, ltol = 0.3, and angle_tol = 10.0, so the reported quantity is cRMSE(0.5) (Martirossyan et al., 15 Sep 2025). A plausible implication is that cross-paper comparisons are only meaningful when the tolerance settings are held fixed.

4. Motivation in crystal-structure benchmarking

The motivating critique in (Martirossyan et al., 15 Sep 2025) concerns benchmark design for crystal structure prediction, especially when datasets contain many polymorphs or duplicates. The paper argues that match rate alone can be misleading because it records only whether a structure falls within tolerance and does not quantify closeness within that tolerance. It also argues that RMSE alone ignores failures because non-matching structures are omitted from the average (Martirossyan et al., 15 Sep 2025).

The polymorph setting sharpens this concern. The paper emphasizes that datasets such as carbon-24 and perov-5 contain many polymorphs, so a model can generate a valid polymorph different from the particular reference structure used in the test set and still be scored as a failure under standard one-to-one matching. METRe and cRMSE are proposed as fixes to this evaluation pathology (Martirossyan et al., 15 Sep 2025).

The benchmark redesign in that paper includes revised versions of carbon-24, including a duplicate-removed version, a deduplicated split by number of atoms $N$, and variants containing only identical structures but different unit cells. It also proposes a new split for perov-5 that groups polymorphs within each split subset. The paper’s broader claim is that benchmark design and metric choice are tightly coupled: cRMSE is intended to be meaningful only when the dataset is curated so that duplicates and polymorph structure are treated explicitly (Martirossyan et al., 15 Sep 2025).

The paper recommends that “the primary benchmark for CSP performance should be the mean cRMSE(stol) instead of the match rate and RMSE separately.” That recommendation follows directly from the metric’s design goal: a single scalar should reflect both structural coverage and match quality without discarding failures (Martirossyan et al., 15 Sep 2025).

5. Empirical behavior, interpretation, and limitations

The reported behavior of cRMSE across datasets in (Martirossyan et al., 15 Sep 2025) is diagnostic. On carbon-24-unique, where METRe is very high, cRMSE is close to ordinary RMSE. The paper lists examples including DiffCSP with METRe 98.2%, RMSE 0.231, cRMSE 0.235, and FlowMM with METRe 98.4%, RMSE 0.193, cRMSE 0.198. This is the regime in which nearly all references are matched, so the correction term contributes little (Martirossyan et al., 15 Sep 2025).

On more difficult settings, cRMSE diverges more strongly from RMSE. For MP-20-polymorph-split, the paper reports, for example, OMatG-LinearODE with METRe 70.50%, RMSE 0.056, cRMSE 0.187, and DiffCSP with METRe 53.14%, RMSE 0.084, cRMSE 0.279. The contrast shows the intended effect: lower reference coverage increases cRMSE even when matched-case RMSE appears small (Martirossyan et al., 15 Sep 2025).

The paper also highlights the carbon-NXL and carbon-X cases. carbon-X is described as trivial, with match rate 100% and RMSE and cRMSE both about 0.001. By contrast, carbon-NXL yields much larger cRMSE than RMSE because some cases fail to match, which demonstrates that cRMSE captures failures hidden by matched-only averaging (Martirossyan et al., 15 Sep 2025).

Several limitations are stated directly. First, cRMSE depends on StructureMatcher thresholds; if tolerances are too permissive, reported matches may be physically weak. Second, it is sensitive to dataset duplication because it inherits the same matching machinery as METRe. Third, it is “not a pure physical measure,” but an evaluation heuristic built on geometric matching rather than a thermodynamic or chemical validity metric. Fourth, it “cannot by itself ensure diversity,” since a model may obtain a good cRMSE while still failing to represent full polymorphic variety unless dataset design addresses that separately (Martirossyan et al., 15 Sep 2025).

6. Distinction from CRMSE and other nearby acronyms

A separate and more recent meaning appears in (Tzivrailis et al., 30 Jun 2026), where CRMSE stands for Contrastive Regularized MSE. That work concerns machine-learning interatomic potentials trained to predict energies and forces but deployed as samplers. Its claim is that standard MSE training can achieve chemical accuracy on held-out DFT data while still failing as a sampler because trajectories drift into spurious low-energy minima. CRMSE is introduced as a post-training objective that augments MSE with a contrastive term derived from the Kullback--Leibler divergence between the model’s implicit Boltzmann distribution and the DFT target (Tzivrailis et al., 30 Jun 2026).

The mathematical object in (Tzivrailis et al., 30 Jun 2026) is therefore a training objective, not an evaluation metric of the kind defined in (Martirossyan et al., 15 Sep 2025). Its full objective is

$$\mathcal{L}^{\mathrm{CRMSE}(\theta)} = \mathcal{L}^{\mathrm{MSE}(\theta)} + \lambda\, \mathcal{L}^{\mathrm{CR}(\theta)},$$

with negative samples produced by persistent Langevin chains and Persistent Contrastive Divergence. The paper reports that, on ethanol and aspirin from MD17, CRMSE restores sampled energy and structural distributions while preserving force accuracy and keeping energy errors within chemical accuracy (Tzivrailis et al., 30 Jun 2026). Despite the near-identical spelling, this is not corrected RMSE.

The corpus also includes several papers that explicitly do not define cRMSE. In (Friederichs et al., 2012), the relevant quantity is CRPS, the continuous ranked probability score, used for probabilistic forecast verification of GEV and GPD models; the paper states that RMSE, centered RMSE, and cRMSE are not defined there. In (Noorani et al., 2024), the topic is conformal risk minimization (CRM) and variance-reduced conformal training, and the paper again states that it does not define or use the acronym cRMSE. Likewise, (Zimin et al., 2018) treats conditional risk minimization, (Zenati et al., 2023) treats Sequential Counterfactual Risk Minimization, and (Koopmeiners et al., 2014) concerns the randomized continual reassessment method (Friederichs et al., 2012, Noorani et al., 2024, Zimin et al., 2018, Zenati et al., 2023, Koopmeiners et al., 2014).

Taken together, these usages show that cRMSE is not a stable cross-domain standard. In the provided literature, the exact string is formally anchored by (Martirossyan et al., 15 Sep 2025) as corrected RMSE; other occurrences belong to neighboring but distinct families of abbreviations.