Distance to Multicalibration (dMC)

• Distance to Multicalibration (dMC) is a metric that quantifies the minimum L1 modification needed for a predictor to satisfy simultaneous calibration over overlapping subgroup intersections.
• It distinguishes itself from worst-group calibration error by enforcing a single, compatible repair across all subgroups rather than addressing them separately.
• Recent studies explore operational surrogates and algorithms (e.g., HKRR, MC-Pseudolabel) that effectively reduce multicalibration errors while balancing fairness and predictive accuracy.

Searching arXiv for papers on multicalibration distance, auditability, and operational metrics. Distance to Multicalibration (dMC) denotes a family of quantities that measure how far a predictor is from satisfying multicalibration constraints over a collection of overlapping subgroups. In its most direct formalization, for a finite feature space $X$, binary labels $Y=\{0,1\}$, a predictor $f:X\to[0,1]$, and a subgroup family $C=\{S_1,\dots,S_k\}$, dMC is the $L_1(D_x)$-distance from $f$ to the set of perfectly $C$-multicalibrated predictors: $dMC_{C,D}(f)=\inf_{g\in mcal_C(D)}\|f-g\|_1.$ This definition is the multigroup analogue of distance to calibration error $dCE_D(f)=\|f,cal(D)\|_1$, but the literature also uses several operational surrogates for “distance to multicalibration,” including worst-group calibration error, subgroup-bin residuals, and residual-correlation norms over an auditing class (Derhake et al., 21 Sep 2025, Hansen et al., 2024).

1. Formal object and basic setup

Multicalibration strengthens ordinary calibration by requiring simultaneous calibration on each subgroup in a family $C$, possibly with overlap. For a predictor $Y=\{0,1\}$0, perfect calibration requires

$Y=\{0,1\}$1

and perfect $Y=\{0,1\}$2-multicalibration requires the same condition inside every $Y=\{0,1\}$3, namely

$Y=\{0,1\}$4

The corresponding feasible sets are denoted $Y=\{0,1\}$5 and $Y=\{0,1\}$6 (Derhake et al., 21 Sep 2025).

The geometric dMC definition inherits the norm structure of $Y=\{0,1\}$7. If $Y=\{0,1\}$8 is the marginal over $Y=\{0,1\}$9, then

$f:X\to[0,1]$0

The $f:X\to[0,1]$1 case is the primary focus in the dMC literature. Under this formulation, dMC measures the minimum amount one must modify $f:X\to[0,1]$2, in $f:X\to[0,1]$3, to obtain a single predictor that is calibrated on all groups simultaneously, not merely one that is separately close to calibrated on each group (Derhake et al., 21 Sep 2025).

This simultaneous-feasibility requirement is what distinguishes dMC from looser worst-group notions. It also makes the metric sensitive to overlap structure in $f:X\to[0,1]$4: two groupwise repairs that are individually small need not be compatible with each other. In practical studies, the subgroup family is usually finite and explicitly specified, for example by metadata, feature conjunctions, or lexical indicators; the resulting notion is therefore always relative to the chosen family rather than to an infinite concept class (Hansen et al., 2024).

2. Metric variants, geometry, and auditability

A natural alternative to dMC is worst-group distance to calibration,

$f:X\to[0,1]$5

where $f:X\to[0,1]$6. This quantity audits each group separately and then takes a maximum. It is therefore easier to interpret as a worst-case subgroup repair cost, but it is not the same as the distance to a single simultaneously multicalibrated predictor. The hierarchy

$f:X\to[0,1]$7

holds in general (Derhake et al., 21 Sep 2025).

The 2025 analysis of dMC identifies two desiderata for any multicalibration error metric: it should capture how much $f:X\to[0,1]$8 must be modified to become perfectly multicalibrated, and it should be auditable in an information-theoretic sense. Raw dMC satisfies the first desideratum exactly, because it is literally a distance to the feasible set $f:X\to[0,1]$9. It also has a favorable optimization landscape: all local minima of $C=\{S_1,\dots,S_k\}$0 are global minima. Geometrically, $C=\{S_1,\dots,S_k\}$1 is a finite union of truncated affine subspaces, so $C=\{S_1,\dots,S_k\}$2 is generally nonconvex but piecewise linear in the $C=\{S_1,\dots,S_k\}$3 setting (Derhake et al., 21 Sep 2025).

The central negative result is that dMC is not auditable information-theoretically in general. There exist $C=\{S_1,\dots,S_k\}$4, $C=\{S_1,\dots,S_k\}$5, $C=\{S_1,\dots,S_k\}$6, and a family of distributions $C=\{S_1,\dots,S_k\}$7 such that

$C=\{S_1,\dots,S_k\}$8

while

$C=\{S_1,\dots,S_k\}$9

Thus arbitrarily small perturbations of the ground truth can induce constant jumps in dMC, so dMC is not Lipschitz in $L_1(D_x)$0 and fails the second desideratum (Derhake et al., 21 Sep 2025).

To repair this instability, the same paper introduces the continuized distance to multicalibration,

$L_1(D_x)$1

and distance to intersection multicalibration,

$L_1(D_x)$2

where

$L_1(D_x)$3

These two replacements are equivalent: $L_1(D_x)$4 where $L_1(D_x)$5 is the partition generated by $L_1(D_x)$6. They retain the “distance-to-feasibility” interpretation, are $L_1(D_x)$7-Lipschitz in $L_1(D_x)$8, and therefore satisfy both desiderata, at the cost of strengthening the target from multicalibration on $L_1(D_x)$9 to multicalibration on all nonempty intersections of groups in $f$0 (Derhake et al., 21 Sep 2025).

3. Operational surrogates in empirical practice

Before explicit geometric dMC was formalized, empirical work typically operationalized “how far a model is from multicalibration” by worst-group calibration error over a finite subgroup family $f$1. In a broad 2024 empirical study spanning tabular, image, and language tasks, the practical metric is

$f$2

instantiated primarily as

$f$3

with

$f$4

used as an alternative. The paper explicitly states that taking the max avoids fairness concerns associated with the weighted mean of groups of varying size and/or degree of overlap, and it repeatedly uses validation max smECE to select the best post-processed model (Hansen et al., 2024).

In that study, ordinary calibration is measured by binned ECE with 10 equal-width bins of size $f$5, while multicalibration post-processing is formulated on subgroup-bin “categories” $f$6 with bin width

$f$7

For the Hébert-Johnson et al. algorithm (HKRR), the operative residual is the subgroup-bin discrepancy

$f$8

and categories are skipped when

$f$9

This makes the empirical notion of dMC discretization-dependent: closeness to multicalibration is assessed after binning scores at width $C$0 (Hansen et al., 2024).

The main empirical finding is that many already calibrated models are close to multicalibration out of the box under this worst-group surrogate. On MEPS, for example, MLP ERM has

$C$1

HJZ attains

$C$2

and HKRR is worse at

$C$3

RandomForest ERM on the same dataset has

$C$4

and LogisticRegression ERM has

$C$5

By contrast, uncalibrated models can be much farther. On MEPS, SVM ERM has

$C$6

which HKRR reduces to

$C$7

while NaiveBayes ERM has

$C$8

and HKRR reduces it to

$C$9

On larger models, ERM max smECE falls from $dMC_{C,D}(f)=\inf_{g\in mcal_C(D)}\|f-g\|_1.$0 to $dMC_{C,D}(f)=\inf_{g\in mcal_C(D)}\|f-g\|_1.$1 for Amazon Polarity with ResNet-56 under HKRR, and from $dMC_{C,D}(f)=\inf_{g\in mcal_C(D)}\|f-g\|_1.$2 to $dMC_{C,D}(f)=\inf_{g\in mcal_C(D)}\|f-g\|_1.$3 for Camelyon17 with ViT (Hansen et al., 2024).

This operational line of work also emphasizes that low overall calibration error often tracks low worst-group calibration error better than one might expect, though not perfectly. Traditional calibration methods may sometimes provide multicalibration implicitly, and isotonic regression often reduces max-group smECE nearly as much as explicit multicalibration algorithms. At the same time, the exact metric matters: on MEPS, DecisionTree ERM has worst-group calibration error $dMC_{C,D}(f)=\inf_{g\in mcal_C(D)}\|f-g\|_1.$4 by max ECE but only $dMC_{C,D}(f)=\inf_{g\in mcal_C(D)}\|f-g\|_1.$5 by max smECE, so the inferred practical dMC can vary substantially with the estimator (Hansen et al., 2024).

4. Residual-based formulations and theoretical generalizations

A broader theoretical literature treats dMC not as geometric distance to a feasible set, but as a residual aggregated over groups and prediction values. For a scalar property $dMC_{C,D}(f)=\inf_{g\in mcal_C(D)}\|f-g\|_1.$6, one exact calibration condition is

$dMC_{C,D}(f)=\inf_{g\in mcal_C(D)}\|f-g\|_1.$7

and the corresponding approximate multicalibration definition is the weighted $dMC_{C,D}(f)=\inf_{g\in mcal_C(D)}\|f-g\|_1.$8 bound

$dMC_{C,D}(f)=\inf_{g\in mcal_C(D)}\|f-g\|_1.$9

In the identifiable case, the paper gives an equivalent residual in terms of an identification function $dCE_D(f)=\|f,cal(D)\|_1$0: $dCE_D(f)=\|f,cal(D)\|_1$1 These are explicit dMC candidates: weighted squared calibration violations over group-value cells (Noarov et al., 2023).

That framework also gives a sharp structural characterization of when zero dMC is even possible. For continuous scalar properties, the paper proves

$dCE_D(f)=\|f,cal(D)\|_1$2

For non-elicitable properties, there are simple datasets on which even the true predictor is not calibrated. The variance example is canonical: with two points $dCE_D(f)=\|f,cal(D)\|_1$3, deterministic labels $dCE_D(f)=\|f,cal(D)\|_1$4, $dCE_D(f)=\|f,cal(D)\|_1$5, and equal mass, the true pointwise variance predictor is identically $dCE_D(f)=\|f,cal(D)\|_1$6, yet on the prediction cell $dCE_D(f)=\|f,cal(D)\|_1$7 the label variance is $dCE_D(f)=\|f,cal(D)\|_1$8. This yields an irreducibly nonzero calibration residual even for the true predictor. A plausible implication is that any dMC notion based on vanishing calibration equations has a positive floor for variance, CVaR alone, and most distortion risk measures (Noarov et al., 2023).

A second residual-based generalization replaces Boolean groups by real-valued grouping functions. In the joint-grouping formulation, approximate multicalibration is defined by

$dCE_D(f)=\|f,cal(D)\|_1$9

The natural scalar dMC induced by this definition is

$C$0

or its empirical analogue. This formulation includes calibration when $C$1, subgroup calibration for Boolean $C$2, reweighted calibration for real-valued $C$3, and joint covariate-label constraints for $C$4. It also connects multicalibration to robustness under covariate shift and beyond covariate shift via density-ratio grouping classes (Wu et al., 2024).

In regression with squared-error loss, multicalibration can be expressed as residual orthogonality to a weak learner class $C$5. The population error vector is

$C$6

and the empirical analogue is

$C$7

In this setting, $C$8 is the paper’s most direct measurable multicalibration-violation quantity, and therefore a natural empirical dMC proxy (Haimovich et al., 6 Feb 2026).

5. Algorithms, convergence, and computational aspects

Several post-processing and boosting algorithms can be interpreted as dMC-reduction procedures. HKRR is described as an iterative boosting-style post-processing algorithm that works by iteratively searching for and removing subgroup calibration violations until convergence. HJZ instead frames multicalibration as a two-player game with better theoretical sample complexity guarantees and is run with $C$9, 30 iterations, learner decay rates $Y=\{0,1\}$00, and adversary decay rates $Y=\{0,1\}$01. In both cases the effective target is to reduce group-bin residuals and thereby lower worst-group calibration error (Hansen et al., 2024).

For joint grouping functions, MC-Pseudolabel provides a regression-oracle-based post-processing algorithm. It discretizes the current predictor, regresses residuals on each level set using the grouping class, forms pseudolabels, and refits the predictor. Its key stopping certificate is: $Y=\{0,1\}$02 where

$Y=\{0,1\}$03

This converts small one-step improvability in squared error into a bound on worst-case multicalibration violation over bounded groups (Wu et al., 2024).

In multicalibration gradient boosting for squared-loss regression, the iterate update is

$Y=\{0,1\}$04

in the unit-weight case, with Lyapunov objective

$Y=\{0,1\}$05

The central convergence identity is

$Y=\{0,1\}$06

Combined with

$Y=\{0,1\}$07

this implies asymptotic vanishing of empirical multicalibration error and a global $Y=\{0,1\}$08 best-iterate bound on the update gap, with linear convergence under additional smoothness assumptions on $Y=\{0,1\}$09 (Haimovich et al., 6 Feb 2026).

Exact computation of geometric dMC remains largely open, but ordinary calibration distance already exhibits substantial complexity barriers. For the base problem

$Y=\{0,1\}$10

exact computation is possible in $Y=\{0,1\}$11 time when the marginal is uniform and labels are noiseless, yet becomes $Y=\{0,1\}$12-hard when either assumption is removed. The same work gives a PTAS in general and shows that $Y=\{0,1\}$13 samples are sufficient and necessary for one-sided estimation of empirical calibration distance. This does not prove hardness for dMC itself, but the paper explicitly argues that the result strongly suggests analogous barriers for dMC over rich subgroup classes (Qiao, 19 Mar 2026).

6. Fairness refinements, extensions, and scope limitations

A recurrent critique of additive dMC surrogates is that they can mask large relative disparities in low-base-rate groups. Proportional multicalibration replaces absolute calibration error by relative or percent calibration error on group-bin cells. The operative additive proxy for ordinary multicalibration in that paper is

$Y=\{0,1\}$14

while the proportional analogue is

$Y=\{0,1\}$15

The theory shows

$Y=\{0,1\}$16

and

$Y=\{0,1\}$17

This indicates that a small additive dMC need not control multiplicative disparity unless relative calibration is also addressed (Cava et al., 2022).

Higher-moment extensions make the same point in another direction. Moment multicalibration does not calibrate moments on arbitrary subgroups directly; instead it uses mean-conditioned moment multicalibration on bucketed cells

$Y=\{0,1\}$18

The corresponding constraints are

$Y=\{0,1\}$19

and

$Y=\{0,1\}$20

A plausible implication is that dMC for uncertainty estimation is naturally vector-valued: it must track mean violations and moment violations jointly rather than collapse everything into a single mean-calibration number (Jung et al., 2020).

Across the literature, several limitations recur. dMC is always relative to a chosen subgroup family or grouping-function class; a predictor can be close for one family and far for another. Many operational definitions depend on discretization, such as $Y=\{0,1\}$21 score bins or finite bucket counts $Y=\{0,1\}$22. Empirical estimates are sensitive to subgroup sample size, and intersection-based replacements such as $Y=\{0,1\}$23 can incur exponential blow-up in the size of $Y=\{0,1\}$24. Practical post-processing also competes with predictive accuracy because held-out calibration data must be reserved for auditing and repair. These constraints explain why the literature alternates between exact geometric definitions, auditable residual surrogates, and application-specific approximations rather than converging on a single universal scalar metric (Derhake et al., 21 Sep 2025, Hansen et al., 2024).