---
title: 'cRMSE: Metric for Crystal Structure Prediction'
url: https://www.emergentmind.com/topics/crmse
type: topic
---

# cRMSE: Metric for Crystal Structure Prediction

cRMSE is explicitly defined in the supplied literature as **corrected RMSE**, an evaluation metric for crystal structure prediction that combines a match-based criterion with a distance/error term so that non-matching structures are not ignored in the aggregate score [2509.12178]. In the same corpus, closely related strings denote different objects: **CRMSE** is **Contrastive Regularized MSE** for machine-learning interatomic potentials [2606.31660], while several papers discuss **CRM** as continual reassessment, conditional risk minimization, counterfactual risk minimization, or conformal risk minimization rather than any root-mean-square-error quantity [1405.1275], [1801.00507], [2302.12120], [2411.01696]. This suggests that, in technical usage, cRMSE should be interpreted only within its immediate methodological context.

## 1. Definition and naming

In [2509.12178], cRMSE means **“corrected RMSE”**. It is introduced in the section **“New metric to combine RMSE and match rate”** as a metric intended to avoid the standard failure mode in which RMSE is computed only on matched structures and therefore ignores non-matches. The paper’s interpretation is direct: cRMSE is the average RMSE over all test structures, where any non-matching test structure is assigned a penalty equal to `stol` [2509.12178].

The same paper presents cRMSE alongside **METRe**, defined as **“match everyone to reference”**, and ordinary **RMSE**. Their intended division of labor is explicit: METRe measures how many reference structures are covered by the generated set, RMSE measures alignment error on matched structures, and cRMSE combines both by incorporating unmatched references through a fixed penalty [2509.12178].

The broader corpus also shows that the label is not universal. One paper introduces **CRMSE**, not cRMSE, as a post-training objective for molecular sampling [2606.31660]. Other papers explicitly state that they do **not** define cRMSE and instead treat **CRPS** or various forms of **CRM** [1204.1022], [2411.01696]. A plausible implication is that exact acronym expansion matters: the literature uses near-identical letter strings for conceptually different metrics and objectives.

## 2. Formal definition as corrected RMSE

The defining formula in [2509.12178] is

\[
\mathrm{mean\ cRMSE} (\mathtt{stol}) = \dfrac{\sum_{i=1}^{N_{\mathrm{test}}} \mathrm{RMSE}_i + \mathtt{stol}(N_{\mathrm{test}} - N_{\mathrm{ref. match}})}{N_{\mathrm{test}}}
\]

and the paper rewrites it as

\[
\mathrm{mean\ cRMSE} = \mathrm{METRe} * (\mathrm{mean\ RMSE} - \mathtt{stol}) + \mathtt{stol}
\]

with

\[
\mathrm{METRe}=\frac{N_\mathrm{ref. match}}{N_\mathrm{test}}
\]

and

\[
\mathrm{mean\ RMSE}=\frac{\sum_{i=1}^{N_\mathrm{ref. match}} \mathrm{RMSE}_i}{N_\mathrm{ref. match}}.
\]

These equations encode the metric’s central correction. For matched reference structures, the observed best RMSE contributes to the average. For unmatched reference structures, the missing RMSE is replaced by the threshold `stol`. The metric is therefore a full-test-set average rather than a matched-subset average [2509.12178].

The paper further states the rationale for the penalty choice: `stol` is used because it “sets the threshold for the computed RMSE of the aligned structures in StructureMatcher (if a mapping can be found).” In this construction, `stol` functions simultaneously as a StructureMatcher tolerance and as the default penalty for failure to match [2509.12178].

Special cases are also specified. If every reference matches, cRMSE is close to ordinary mean RMSE. If no reference matches, cRMSE becomes `stol`. These are not auxiliary interpretations but direct consequences identified in the paper’s discussion of normalization, thresholds, and special cases [2509.12178].

## 3. Computation protocol and dependence on matching

The appendix-level procedure in [2509.12178] makes the computation explicit. The required inputs are a list of generated structures, a list of test or reference structures, and the Pymatgen `StructureMatcher` tolerances `ltol`, `stol`, and `angle_tol`. For each generated structure, the method compares it against every test structure using `StructureMatcher`, retaining for each reference the **best (lowest) RMSE** found so far across all generated structures [2509.12178].

After all generated structures have been processed, matched test structures have stored RMSE values and unmatched ones remain `NaN`. The paper then computes METRe as the fraction of test structures that obtained at least one match, computes mean RMSE over matched references only, and finally computes cRMSE by replacing every unmatched structure’s missing RMSE with `stol` and averaging over all test structures [2509.12178].

This protocol makes cRMSE fundamentally reference-based. The criterion is not whether each generated structure has a nearby test counterpart, but whether each reference structure is covered by the generated set. In the paper’s terminology, this is what aligns cRMSE with METRe rather than with the older one-to-one match-rate convention [2509.12178].

The dependence on matching tolerances is also explicit. The reported metric is parameterized as \(\mathrm{mean\ cRMSE}(\mathtt{stol})\), and in the experiments the standard benchmark setting is `stol = 0.5`, `ltol = 0.3`, and `angle_tol = 10.0`, so the reported quantity is **cRMSE(0.5)** [2509.12178]. A plausible implication is that cross-paper comparisons are only meaningful when the tolerance settings are held fixed.

## 4. Motivation in crystal-structure benchmarking

The motivating critique in [2509.12178] concerns benchmark design for crystal structure prediction, especially when datasets contain many polymorphs or duplicates. The paper argues that **match rate alone can be misleading** because it records only whether a structure falls within tolerance and does not quantify closeness within that tolerance. It also argues that **RMSE alone ignores failures** because non-matching structures are omitted from the average [2509.12178].

The polymorph setting sharpens this concern. The paper emphasizes that datasets such as **carbon-24** and **perov-5** contain many polymorphs, so a model can generate a valid polymorph different from the particular reference structure used in the test set and still be scored as a failure under standard one-to-one matching. METRe and cRMSE are proposed as fixes to this evaluation pathology [2509.12178].

The benchmark redesign in that paper includes revised versions of **carbon-24**, including a duplicate-removed version, a deduplicated split by number of atoms \(N\), and variants containing only identical structures but different unit cells. It also proposes a new split for **perov-5** that groups polymorphs within each split subset. The paper’s broader claim is that benchmark design and metric choice are tightly coupled: cRMSE is intended to be meaningful only when the dataset is curated so that duplicates and polymorph structure are treated explicitly [2509.12178].

The paper recommends that “the primary benchmark for CSP performance should be the mean cRMSE(stol) instead of the match rate and RMSE separately.” That recommendation follows directly from the metric’s design goal: a single scalar should reflect both structural coverage and match quality without discarding failures [2509.12178].

## 5. Empirical behavior, interpretation, and limitations

The reported behavior of cRMSE across datasets in [2509.12178] is diagnostic. On **carbon-24-unique**, where METRe is very high, cRMSE is close to ordinary RMSE. The paper lists examples including DiffCSP with METRe 98.2%, RMSE 0.231, cRMSE 0.235, and FlowMM with METRe 98.4%, RMSE 0.193, cRMSE 0.198. This is the regime in which nearly all references are matched, so the correction term contributes little [2509.12178].

On more difficult settings, cRMSE diverges more strongly from RMSE. For **MP-20-polymorph-split**, the paper reports, for example, OMatG-LinearODE with METRe 70.50%, RMSE 0.056, cRMSE 0.187, and DiffCSP with METRe 53.14%, RMSE 0.084, cRMSE 0.279. The contrast shows the intended effect: lower reference coverage increases cRMSE even when matched-case RMSE appears small [2509.12178].

The paper also highlights the **carbon-NXL** and **carbon-X** cases. `carbon-X` is described as trivial, with match rate 100% and RMSE and cRMSE both about 0.001. By contrast, `carbon-NXL` yields much larger cRMSE than RMSE because some cases fail to match, which demonstrates that cRMSE captures failures hidden by matched-only averaging [2509.12178].

Several limitations are stated directly. First, cRMSE depends on `StructureMatcher` thresholds; if tolerances are too permissive, reported matches may be physically weak. Second, it is sensitive to dataset duplication because it inherits the same matching machinery as METRe. Third, it is “not a pure physical measure,” but an evaluation heuristic built on geometric matching rather than a thermodynamic or chemical validity metric. Fourth, it “cannot by itself ensure diversity,” since a model may obtain a good cRMSE while still failing to represent full polymorphic variety unless dataset design addresses that separately [2509.12178].

## 6. Distinction from CRMSE and other nearby acronyms

A separate and more recent meaning appears in [2606.31660], where **CRMSE** stands for **Contrastive Regularized MSE**. That work concerns machine-learning interatomic potentials trained to predict energies and forces but deployed as samplers. Its claim is that standard MSE training can achieve chemical accuracy on held-out DFT data while still failing as a sampler because trajectories drift into spurious low-energy minima. CRMSE is introduced as a post-training objective that augments MSE with a contrastive term derived from the Kullback--Leibler divergence between the model’s implicit Boltzmann distribution and the DFT target [2606.31660].

The mathematical object in [2606.31660] is therefore a **training objective**, not an evaluation metric of the kind defined in [2509.12178]. Its full objective is

\[
\mathcal{L}^{\mathrm{CRMSE}(\theta)} = \mathcal{L}^{\mathrm{MSE}(\theta)} + \lambda\, \mathcal{L}^{\mathrm{CR}(\theta)},
\]

with negative samples produced by persistent Langevin chains and Persistent Contrastive Divergence. The paper reports that, on ethanol and aspirin from MD17, CRMSE restores sampled energy and structural distributions while preserving force accuracy and keeping energy errors within chemical accuracy [2606.31660]. Despite the near-identical spelling, this is not corrected RMSE.

The corpus also includes several papers that explicitly do **not** define cRMSE. In [1204.1022], the relevant quantity is **CRPS**, the continuous ranked probability score, used for probabilistic forecast verification of GEV and GPD models; the paper states that RMSE, centered RMSE, and cRMSE are not defined there. In [2411.01696], the topic is **conformal risk minimization (CRM)** and variance-reduced conformal training, and the paper again states that it does not define or use the acronym cRMSE. Likewise, [1801.00507] treats **conditional risk minimization**, [2302.12120] treats **Sequential Counterfactual Risk Minimization**, and [1405.1275] concerns the **randomized continual reassessment method** [1204.1022], [2411.01696], [1801.00507], [2302.12120], [1405.1275].

Taken together, these usages show that cRMSE is not a stable cross-domain standard. In the provided literature, the exact string is formally anchored by [2509.12178] as **corrected RMSE**; other occurrences belong to neighboring but distinct families of abbreviations.

Source: https://www.emergentmind.com/topics/crmse