Mean Absolute Deviation Calibration Error

• Mean Absolute Deviation Calibration Error metrics, including ENCE and ZVE, are statistical tools that compare predicted uncertainties with observed errors in regression tasks.
• They employ binning strategies to aggregate local calibration statistics, with ENCE scaling as √B and ZVE using logarithmic deviations to reduce sensitivity to outliers.
• An intercept extraction method is used to obtain bin-invariant error estimates, enabling robust statistical tests for model miscalibration in machine learning uncertainty quantification.

The Mean Absolute Deviation Calibration Error (MAD-based metrics) refers to a class of statistical tools used to assess the calibration quality of predicted uncertainty estimates in regression problems. Most prominent among these is the Expected Normalized Calibration Error (ENCE), which quantifies the mean absolute deviation between predicted and observed uncertainty, and the Z-Variance Error (ZVE), which utilizes the variation of normalized residuals. These metrics rely on binning strategies to aggregate local calibration statistics and are widely applied in machine learning–uncertainty quantification (ML-UQ) contexts. Both metrics exhibit nontrivial dependencies on the binning parameter, which significantly impacts their behavior and interpretation (Pernot, 2023).

1. Mathematical Formulation of MAD-Based Calibration Metrics

The ENCE and ZVE are defined for regression tasks involving observed prediction errors $E_i$ and model-predicted uncertainties $u_i$ for $i=1,\dots,M$ data points. The data is partitioned into $B$ disjoint bins $B_1, ..., B_B$ of approximately equal size $k$. The key statistics within each bin $j$ are:

• Mean squared predicted uncertainty: $\mathrm{MV}_j = \frac{1}{k}\sum_{i\in B_j} u_i^2$
• Mean squared prediction error: $\mathrm{MSE}_j = \frac{1}{k}\sum_{i\in B_j} E_i^2$

The ENCE is defined as:

$$\mathrm{ENCE} = \frac{1}{B} \sum_{j=1}^B \frac{|\sqrt{\mathrm{MV}_j} - \sqrt{\mathrm{MSE}_j}|}{\sqrt{\mathrm{MV}_j}}$$

Equivalently, $\mathrm{ENCE}$ is the mean absolute deviation (MAD) of the ratios $\{\sqrt{\mathrm{MSE}_j}/\sqrt{\mathrm{MV}_j}\}$.

Analogously, ZVE is defined using bin-wise z-score variances:

• For each $i$, $Z_i = E_i/u_i$.
• Within bin $j$, $$v_j = \frac{1}{k-1} \sum_{i\in B_j} (Z_i - \bar{Z}_j)^2$$.
• ZVE is the exponential of the MAD of logarithmic bin-variances:

$$\mathrm{ZVE} = \exp \left( \frac{1}{B}\sum_{j=1}^B |\ln v_j| \right)$$

Under perfect calibration, both statistics should indicate zero deviation from ideal uncertainty calibration.

2. Bin-Dependence and Sampling Noise

Both ENCE and ZVE exhibit an inherent statistical dependence on the number of bins $B$, even for well-calibrated datasets:

• For homoscedastic, unbiased data ($u_i \equiv u$, $E_i \sim N(0, u^2)$), ENCE $\propto \sqrt{B}$, due to the dispersion inherent to sample mean absolute deviation [Eq. (5)].
• For ENCE, the expected absolute deviation of normalized standard deviation estimates $X_j$ scales as $\sim \sqrt{2/(\pi k)}$, leading to:

$$\mathrm{ENCE} \approx \sqrt{\frac{2}{\pi}} \sqrt{\frac{B}{M}}$$

• For ZVE, under calibration, the distribution of sample variances leads to

$$\mathrm{ZVE} \approx \exp\left( c \sqrt{\frac{B}{M}} \right) \sim 1 + c \sqrt{\frac{B}{M}}$$

where $c$ is a distribution-specific constant.

This scaling arises not from model miscalibration but purely from Monte Carlo binning variability, emphasizing the need for correcting for $B$ in practical usage.

3. Correction via Intercept Extraction

To achieve a $B$-independent calibration statistic, Pernot proposes an intercept-extraction method:

• Compute $S(B)$ (either ENCE or $\ln\mathrm{ZVE}$) for a range of $B$.
• Empirically, $S(B)$ displays an approximately linear dependence on $\sqrt{B}$ for well-calibrated datasets:

$S(B) \approx \alpha + \beta \sqrt{B}$

• The intercept $\alpha$ at $\sqrt{B} = 0$ is then interpreted as the true, bin-invariant calibration error.

For practical estimation, $S(B)$ is regressed against $\sqrt{B}$ using ordinary least squares on the linear regime, and $\alpha$ (or $\exp(\alpha)$ for ZVE) is reported as the corrected ENCE or ZVE. The null hypothesis "perfect calibration" corresponds to $\alpha = 0$.

Additionally, the standard error of the intercept provides a statistical test for miscalibration. A $t$-test statistic is computed as

$t = \frac{\hat{\alpha}}{\mathrm{SE}(\hat{\alpha})}$

with significance assessed via the appropriate $t$-distribution degrees of freedom.

4. Sensitivity to Outliers

ENCE exhibits heightened sensitivity to single-bin outliers, especially at small $B$. When an extreme error–uncertainty pair dominates a bin, ENCE can be disproportionately increased; as $B$ increases and the bin splits, the outlier's influence is diluted. In contrast, ZVE, depending on $\ln v_j$ rather than absolute deviations, demonstrates reduced sensitivity to such effects and yields cleaner, more linear scaling even at low $B$.

Empirical findings from the BUS2022 QM9 dataset and others illustrate that ZVE intercept estimates have tighter confidence bands than ENCE, supporting its stability and reliability for datasets with occasional large residual errors.

5. Practical Usage and Guidelines

Optimal binning is critical:

• Bins should not be so small that $k = M/B < 30$, as this leads to excessive estimator noise.
• Bins should not be so large that local calibration effects are averaged out; entropy arguments suggest $B \approx \sqrt{M}$.
• In practice, $B$ is scanned from 2 up to $\lfloor M/30\rfloor$, and only the $\sqrt{B}$-linear regime is used for intercept fitting.

Monte Carlo simulations (e.g., $M=5000$, $N(0,1)$) confirm perfect agreement between theoretical and empirical scaling. Multiple real-world ML-UQ datasets display the anticipated two-phase behavior, with a transient region at small $B$ followed by linear $\sqrt{B}$ scaling, further validating the intercept-extraction approach (Pernot, 2023).

6. Implications and Open Questions

The intrinsic bin-dependence of MAD-based calibration metrics like ENCE and ZVE introduces unavoidable sampling noise, potentially misleading users into interpreting any nonzero statistic as evidence of miscalibration. The intercept-correction addresses this, providing a $B$-invariant calibration error estimate and a valid statistical miscalibration test. ZVE's reduced sensitivity to outliers makes it preferable for applications where rare large residuals are expected.

Open questions remain regarding small-sample corrections for the intercept test, the best practices for bin selection in large datasets, and the extension of bin-invariant strategies to other MAD-based or maximum-error metrics (Pernot, 2023). A plausible implication is that methodological guidelines for metric selection, binning regimes, and statistical testing will benefit from further theoretical and empirical refinement.