Multicalibration: Fairness for Predictors

• Multicalibration is a fairness measure that requires predictors to be calibrated across multiple, often overlapping, subpopulations and auditor-defined slices.
• It leverages techniques like boosting, empirical risk minimization, and loss minimization to achieve strong calibration guarantees with proven sample complexity and convergence bounds.
• Extensions such as proportional, extended, and weighted multicalibration address practical issues like missing sensitive attributes and calibration of percent errors in various domains.

to=arxiv_search.search 大发游戏 代理娱乐 to=arxiv_search.search 大发极速_json {"query":"multicalibration fairness calibration arXiv", "max_results": 10, "sort_by": "relevance"}Толуқ to=arxiv_search.search 鸿丰ખ to=arxiv_search.search аҭагылазаашьа code=json {"query":"\"multicalibration\" ", "max_results": 10, "sort_by": "lastUpdatedDate"} Multicalibration is a notion of fairness for predictors that requires them to provide calibrated predictions across a large set of protected groups. In the binary setting, it strengthens ordinary calibration by requiring that predicted probabilities be calibrated not only on the full population but also simultaneously on many, possibly overlapping, subpopulations or more general auditor-defined slices of the data. The topic now spans fairness, calibration theory, property elicitation, boosting and empirical risk minimization, auditing, and domain-specific deployments, with formulations ranging from subgroup/bin constraints to auditor expectations of the form $$\bigl|\mathbb{E}[c(x,f(x))(y-f(x))]\bigr|\le \gamma$$ (Błasiok et al., 2023).

1. Formal definitions and equivalent formulations

Standard calibration asks that for every score $p\in[0,1]$,

$\Pr_{(x,y)\sim D}\bigl(y=1\mid f(x)=p\bigr)=p.$

A common multicalibration formulation fixes a finite collection of possibly overlapping subpopulations $\mathcal G=\{g_1,\dots,g_m\}$ and score bins $I_b$, and requires

$$\Bigl|\mathbb{E}[y\mid x\in g,\;f(x)\in I_b]-\mathbb{E}[f(x)\mid x\in g,\;f(x)\in I_b]\Bigr|\le \alpha$$

for every $g\in\mathcal G$ and every bin $I_b$. In practice, work in this style often reports $\max_{g\in\mathcal G}\mathrm{ECE}_g(f)$ or $\max_{g\in\mathcal G}\mathrm{smECE}_g(f)$ as worst-group calibration errors (Hansen et al., 2024).

A more general formulation fixes an auditor class

$p\in[0,1]$0

and declares $p\in[0,1]$1 to be $p\in[0,1]$2-multicalibrated if for every $p\in[0,1]$3,

$p\in[0,1]$4

Writing $p\in[0,1]$5, the condition is $p\in[0,1]$6 for all $p\in[0,1]$7. In the neural-network setting analyzed by Błasiok, Gollakota, Huiberts, Mao, Nakkiran, Song, and Zhang, the auditors are $p\in[0,1]$8, the family of all ReLU networks of size $p\in[0,1]$9 taking input $\Pr_{(x,y)\sim D}\bigl(y=1\mid f(x)=p\bigr)=p.$0, and the predictors are ReLU networks $\Pr_{(x,y)\sim D}\bigl(y=1\mid f(x)=p\bigr)=p.$1 mapping $\Pr_{(x,y)\sim D}\bigl(y=1\mid f(x)=p\bigr)=p.$2 (Błasiok et al., 2023).

The literature distinguishes multicalibration from multiaccuracy. For a group $\Pr_{(x,y)\sim D}\bigl(y=1\mid f(x)=p\bigr)=p.$3, multiaccuracy controls only the overall mean bias

$\Pr_{(x,y)\sim D}\bigl(y=1\mid f(x)=p\bigr)=p.$4

whereas multicalibration refines this by demanding calibration at each score bucket through quantities such as

$\Pr_{(x,y)\sim D}\bigl(y=1\mid f(x)=p\bigr)=p.$5

Multicalibration implies multiaccuracy by averaging over buckets, but it is strictly stronger because small overall mean error in a group does not guarantee that the predictor’s confidence is calibrated at each score value (Bharti et al., 4 Mar 2025).

2. Statistical, game-theoretic, and structural foundations

A central statistical question is whether empirical multicalibration generalizes. Shabat, Cohen, and Mansour formalize categories $\Pr_{(x,y)\sim D}\bigl(y=1\mid f(x)=p\bigr)=p.$6, where $\Pr_{(x,y)\sim D}\bigl(y=1\mid f(x)=p\bigr)=p.$7 is a subgroup and $\Pr_{(x,y)\sim D}\bigl(y=1\mid f(x)=p\bigr)=p.$8 is a prediction interval, and define $\Pr_{(x,y)\sim D}\bigl(y=1\mid f(x)=p\bigr)=p.$9-multicalibration by requiring $\mathcal G=\{g_1,\dots,g_m\}$0 on every “interesting” category satisfying $\mathcal G=\{g_1,\dots,g_m\}$1 and $\mathcal G=\{g_1,\dots,g_m\}$2. They give uniform-convergence bounds in both realizable and agnostic settings. For finite $\mathcal G=\{g_1,\dots,g_m\}$3, with $\mathcal G=\{g_1,\dots,g_m\}$4 discretized into $\mathcal G=\{g_1,\dots,g_m\}$5 intervals, it suffices to take

$\mathcal G=\{g_1,\dots,g_m\}$6

examples to guarantee simultaneous approximation of empirical and true calibration errors on all interesting categories. For infinite $\mathcal G=\{g_1,\dots,g_m\}$7 with finite graph dimension $\mathcal G=\{g_1,\dots,g_m\}$8, the bound becomes

$\mathcal G=\{g_1,\dots,g_m\}$9

and they also prove a lower bound of

$I_b$0

These results decouple the fairness metric from prediction error: once a predictor is empirically multicalibrated on a sufficiently large sample, it remains nearly multicalibrated with respect to the population distribution regardless of its prediction loss (Shabat et al., 2020).

A second line of work casts multicalibration as multi-objective learning and analyzes it through game dynamics. In the formulation of Gopalan, Huiberts, Ligett, and Roth, multicalibration becomes a two-player zero-sum game between a learner choosing predictors and an adversary choosing mixtures over calibration losses indexed by sign, class, group, and bucket. This yields NRNR, NRBR, and BRNR dynamics, deterministic and randomized batch algorithms, and online algorithms. Their guarantees include non-deterministic batch sample complexity $I_b$1, deterministic batch oracle complexity $I_b$2, and a $I_b$3-scaled multicalibration guarantee obtained by reweighting subgroup constraints (Haghtalab et al., 2023).

The scope of multicalibration is characterized sharply by property elicitation. Noarov and Roth show that, under mild technical conditions, it is possible to produce a multicalibrated predictor for a continuous scalar distributional property $I_b$4 if and only if $I_b$5 is elicitable. Their characterization equates sensibility for calibration, convex level sets, identifiability, and elicitability. The negative side is that for non-elicitable continuous properties there exist simple data distributions on which even the true distributional predictor is not calibrated; variance is the canonical example. The positive side is that conditionally elicitable pairs can still be jointly multicalibrated, which recovers cases such as $I_b$6 even though $I_b$7 is not multicalibratable by itself (Noarov et al., 2023).

Low-Degree Multicalibration refines the landscape further by interpolating between multiaccuracy and full multicalibration. Globus-Harris, Jung, Kearns, Roth, and Wu define the hierarchy

$I_b$8

and relate it to smooth and full multicalibration through results such as $I_b$9. Their main message is that key fairness and accuracy properties of full multicalibration are already low-degree properties, while the sample complexity of low-degree multicalibration improves exponentially in the number of classes over full multicalibration in the multi-class setting (Gopalan et al., 2022).

3. Algorithms, boosting, and empirical risk minimization

The classical algorithmic picture is boosting-style post-processing. Hébert-Johnson-style and Haghtalab–Jordan–Zhao-style methods repeatedly identify a group-bin violation and patch the predictor on the offending slice until no violation remains, with $$\Bigl|\mathbb{E}[y\mid x\in g,\;f(x)\in I_b]-\mathbb{E}[f(x)\mid x\in g,\;f(x)\in I_b]\Bigr|\le \alpha$$0-type convergence guarantees under weak-learning assumptions. In practice, traditional calibration methods—Platt scaling, temperature scaling, and isotonic regression—do not guarantee subgroup calibration, but they are computationally cheap and can sometimes reduce worst-group calibration error implicitly (Hansen et al., 2024).

For squared-error regression, Harrison, Keswani, and Roth show that multicalibration admits a swap-regret characterization. If no $$\Bigl|\mathbb{E}[y\mid x\in g,\;f(x)\in I_b]-\mathbb{E}[f(x)\mid x\in g,\;f(x)\in I_b]\Bigr|\le \alpha$$1 can improve squared error on any level set of the current predictor, then the predictor is exactly multicalibrated with respect to $$\Bigl|\mathbb{E}[y\mid x\in g,\;f(x)\in I_b]-\mathbb{E}[f(x)\mid x\in g,\;f(x)\in I_b]\Bigr|\le \alpha$$2. Their LSBoost procedure discretizes predictions to a grid of size $$\Bigl|\mathbb{E}[y\mid x\in g,\;f(x)\in I_b]-\mathbb{E}[f(x)\mid x\in g,\;f(x)\in I_b]\Bigr|\le \alpha$$3, calls an ordinary least-squares oracle on each level set, and halts after at most $$\Bigl|\mathbb{E}[y\mid x\in g,\;f(x)\in I_b]-\mathbb{E}[f(x)\mid x\in g,\;f(x)\in I_b]\Bigr|\le \alpha$$4 rounds. Under a $$\Bigl|\mathbb{E}[y\mid x\in g,\;f(x)\in I_b]-\mathbb{E}[f(x)\mid x\in g,\;f(x)\in I_b]\Bigr|\le \alpha$$5-weak learning condition, LSBoost converges to within $$\Bigl|\mathbb{E}[y\mid x\in g,\;f(x)\in I_b]-\mathbb{E}[f(x)\mid x\in g,\;f(x)\in I_b]\Bigr|\le \alpha$$6 of Bayes error without any realizability assumption, and the same weak learning condition is necessary and sufficient for multicalibration with respect to $$\Bigl|\mathbb{E}[y\mid x\in g,\;f(x)\in I_b]-\mathbb{E}[f(x)\mid x\in g,\;f(x)\in I_b]\Bigr|\le \alpha$$7 to imply Bayes optimality (Globus-Harris et al., 2023).

A particularly influential result is that loss minimization itself can imply multicalibration. In the setting where auditors are ReLU networks of size $$\Bigl|\mathbb{E}[y\mid x\in g,\;f(x)\in I_b]-\mathbb{E}[f(x)\mid x\in g,\;f(x)\in I_b]\Bigr|\le \alpha$$8 and predictors are ReLU networks of size $$\Bigl|\mathbb{E}[y\mid x\in g,\;f(x)\in I_b]-\mathbb{E}[f(x)\mid x\in g,\;f(x)\in I_b]\Bigr|\le \alpha$$9, Błasiok et al. prove that for all but at most $g\in\mathcal G$0 network sizes $g\in\mathcal G$1, every $g\in\mathcal G$2-loss-optimal $g\in\mathcal G$3 with respect to squared loss is $g\in\mathcal G$4-multicalibrated. The proof uses three ingredients: violation implies a loss drop via the update

$g\in\mathcal G$5

closure of the hypothesis class under this update, and a counting argument over the monotone sequence $g\in\mathcal G$6. The result does not assume realizability of the Bayes predictor, but it does assume infinite data and exact population loss minimization (Błasiok et al., 2023).

A related discretization-free line replaces bucket patching by direct ERM over structured post-processors. “Discretization-free Multicalibration through Loss Minimization over Tree Ensembles” starts from an uncalibrated predictor $g\in\mathcal G$7, optimizes squared loss over an ensemble of depth-two decision trees, and proves that under a loss-saturation condition the resulting predictor satisfies

$g\in\mathcal G$8

The method can be implemented with off-the-shelf tree-ensemble learners such as LightGBM, and its empirical evaluation reports that the loss-saturation condition is always met in practice across the studied datasets (2505.17435).

4. Measuring and auditing multicalibration

As multicalibration moved from a purely existential or algorithmic objective to an audited property, the question of scalar metrics became central. Haghtalab, Jordan, and Zhao derive a multicalibration metric from the classical Kuiper statistic. For each subpopulation $g\in\mathcal G$9, they define the cumulative calibration error process $I_b$0, the corresponding Kuiper statistic

$I_b$1

and a noise scale

$I_b$2

The resulting multicalibration metric is

$I_b$3

This standardizes each subgroup’s raw Kuiper deviation by its signal-to-noise ratio, yields $I_b$4, avoids the user-tuned bandwidths or bin edges of ECE- and KDE-style approaches, and can be computed in $I_b$5 time (Guy et al., 12 Jun 2025).

A complementary line studies “distance to multicalibration.” Derhake, Kim, Lee, and Roth define two natural generalizations of distance to calibration: worst-group distance to calibration,

$I_b$6

and distance to multicalibration,

$I_b$7

They show that each fails one of two desiderata: $I_b$8 fails the “minimal modification” or “local=global” requirement, while $I_b$9 fails information-theoretic auditability because infinitesimal changes in the ground truth can create a jump in $\max_{g\in\mathcal G}\mathrm{ECE}_g(f)$0. Their repair is a continuized metric $\max_{g\in\mathcal G}\mathrm{ECE}_g(f)$1, equivalent to distance to intersection multicalibration

$\max_{g\in\mathcal G}\mathrm{ECE}_g(f)$2

with the closed form

$\max_{g\in\mathcal G}\mathrm{ECE}_g(f)$3

This metric is 1-Lipschitz in the ground-truth predictor $\max_{g\in\mathcal G}\mathrm{ECE}_g(f)$4, and the geometry of the associated loss landscape eliminates non-global local minima (Derhake et al., 21 Sep 2025).

5. Variants and generalizations

One major generalization addresses missing sensitive attributes. In the proxy-group setting, the true groups $\max_{g\in\mathcal G}\mathrm{ECE}_g(f)$5 are unobserved at test time, but proxy classifiers $\max_{g\in\mathcal G}\mathrm{ECE}_g(f)$6 with known error rates are available. La Cava, Lee, and collaborators prove the bounds

$\max_{g\in\mathcal G}\mathrm{ECE}_g(f)$7

where

$\max_{g\in\mathcal G}\mathrm{ECE}_g(f)$8

Consequently,

$\max_{g\in\mathcal G}\mathrm{ECE}_g(f)$9

is an actionable upper bound on worst-case multicalibration violation over the true groups. They also analyze a proxy-based boosting algorithm that discretizes the predictor, scans bucket-proxy pairs, and provably reduces the worst-case bound when proxy-group multicalibration improves and MSE does not increase (Bharti et al., 4 Mar 2025).

Another variant is proportional multicalibration, introduced because ordinary multicalibration constrains absolute error but not percent error. For a group $\max_{g\in\mathcal G}\mathrm{smECE}_g(f)$0, proportional calibration requires

$\max_{g\in\mathcal G}\mathrm{smECE}_g(f)$1

This yields two sharp consequences: $\max_{g\in\mathcal G}\mathrm{smECE}_g(f)$2-PMC implies $\max_{g\in\mathcal G}\mathrm{smECE}_g(f)$3-multicalibration, and it implies $\max_{g\in\mathcal G}\mathrm{smECE}_g(f)$4-differential calibration. The associated PMCBoost procedure converges in

$\max_{g\in\mathcal G}\mathrm{smECE}_g(f)$5

iterations under the paper’s positivity and minimum-prevalence assumptions (Cava et al., 2022).

Extended multicalibration broadens the grouping functions from $\max_{g\in\mathcal G}\mathrm{smECE}_g(f)$6 to $\max_{g\in\mathcal G}\mathrm{smECE}_g(f)$7, thereby connecting multicalibration to out-of-distribution generalization beyond covariate shift. In this framework,

$\max_{g\in\mathcal G}\mathrm{smECE}_g(f)$8

must be small for every $\max_{g\in\mathcal G}\mathrm{smECE}_g(f)$9 in a class $p\in[0,1]$00. The key results show that standard multicalibration yields near-Bayes-optimality under covariate shift when the grouping class is closed under density ratios, while extended multicalibration becomes equivalent to invariance under concept shift when $p\in[0,1]$01 contains joint density-ratio functions. The proposed MC-PseudoLabel algorithm realizes this objective through a sequence of supervised regression steps on pseudolabels (Wu et al., 2024).

Weighted multicalibration extends the framework to vector-valued predictions $p\in[0,1]$02. For matching problems, the condition is

$p\in[0,1]$03

for every selector $p\in[0,1]$04 in a prescribed weight class. With the right choice of selectors, post-processing a base predictor into a weighted-multicalibrated one makes the matching chosen by the Bayes-optimal rule on the new predictor competitive with the best algorithm in a finite comparison class applied to the original predictor (Baldeschi et al., 14 Nov 2025).

6. Empirical picture and application domains

A broad empirical picture is now visible. The study “When is Multicalibration Post-Processing Necessary?” reports three headline findings: models which are calibrated out of the box tend to be relatively multicalibrated without any additional post-processing; multicalibration post-processing can help inherently uncalibrated models and large vision and LLMs; and traditional calibration measures may sometimes provide multicalibration implicitly. In tabular tasks, ERM alone often achieves low worst-group smECE and explicit multicalibration rarely improves it by more than $p\in[0,1]$05, whereas uncalibrated models such as SVMs, Naive Bayes, and decision trees can see max-group smECE drops of $p\in[0,1]$06–$p\in[0,1]$07. On vision and language datasets, late-stage fine-tuned transformers or CNNs often have global ECE $p\in[0,1]$08 but worst-group smECE around $p\in[0,1]$09–$p\in[0,1]$10, and HKRR can reduce worst-group smECE by up to $p\in[0,1]$11, as in the ViT-on-Camelyon17 example $p\in[0,1]$12 (Hansen et al., 2024).

For LLM confidence scoring, multicalibration has been implemented with two group-construction mechanisms: embedding-space clustering and self-annotation through yes/no questions posed to the model itself. The overfitting-resistant IGLB algorithm combines upper/lower cumulative bins, local linear-scaling patches, and early stopping. Across six QA datasets and four LLMs, the paper reports that uncalibrated MSEs range about $p\in[0,1]$13–$p\in[0,1]$14, while IGLB reduces them to about $p\in[0,1]$15–$p\in[0,1]$16; uncalibrated accuracies around $p\in[0,1]$17–$p\in[0,1]$18 rise to about $p\in[0,1]$19–$p\in[0,1]$20; and per-topic $p\in[0,1]$21 on true MMLU topics falls from about $p\in[0,1]$22–$p\in[0,1]$23 to about $p\in[0,1]$24–$p\in[0,1]$25 (Detommaso et al., 2024).

For code generation, multicalibration has been studied on MultiPL-E, McEval, and LiveCodeBench using Qwen3 Coder, GPT-OSS, and DeepSeek-R1-Distill. The compared methods include group-conditional unbiased regression and iterative grouped binning schemes based on language, length, and complexity groups. The reported gains include improvements over uncalibrated token likelihoods of $p\in[0,1]$26 in skill score and over baseline calibrations of $p\in[0,1]$27, with language groups providing the largest marginal benefit in the ablations (Campos et al., 9 Dec 2025).

In insurance pricing, multicalibration is identified with the conjunction of autocalibration and fairness via conditional mean independence. A premium $p\in[0,1]$28 is multicalibrated with respect to a sensitive feature $p\in[0,1]$29 when

$p\in[0,1]$30

Denuit, Michaelides, and Trufin analyze local regression and iterative bias-correction implementations, including credibility shrinkage, and report on French motor insurance data that multicalibration eliminates conditional residual offsets across vehicle-age groups while achieving the best Poisson deviance among the compared methods in the case study (Denuit et al., 17 Mar 2026).

A persistent misconception is that explicit post-processing is always necessary. The current literature does not support that conclusion. Some results show that simply minimizing squared loss already yields multicalibration for large enough neural networks outside a bounded set of “unlucky” sizes, and empirical studies show that calibrated ERM models are often already relatively multicalibrated on realistic finite group families (Błasiok et al., 2023). At the same time, the same literature identifies unresolved problems: sample complexity and computational tractability in realistic settings, efficient algorithms for exact or approximate multicalibrated minima, characterizing which architectures admit efficient “free” multicalibration, and experimental validation on deep networks at broader scales (Błasiok et al., 2023).