Calibrated Multiaccuracy in ML

• Calibrated multiaccuracy is a regularity notion that blends multiaccuracy and global calibration, ensuring subgroup residual control and overall prediction alignment.
• It bridges the gap between standard multiaccuracy and full multicalibration, offering stronger subgroup fairness with reduced algorithmic complexity.
• Practical implementations use methods like low-degree multicalibration, MDCA, and weak supervision techniques to enhance predictive performance and fairness.

Calibrated multiaccuracy is a multigroup predictive regularity notion obtained by combining multiaccuracy with global calibration. In the standard binary formulation, multiaccuracy requires a predictor $p:X\to[0,1]$ to have small subgroup-averaged residuals against every test $c$ in a class $C$, while calibration requires the score $p(x)$ itself to agree with empirical outcome frequencies. This places calibrated multiaccuracy strictly between ordinary multiaccuracy and full multicalibration: $$MA(\alpha)\supseteq calMA(\alpha)\supseteq MC(\alpha)$$. Recent work treats this intermediate notion as a useful compromise: it is substantially stronger than multiaccuracy alone, yet algorithmically much closer to multiaccuracy than to full multicalibration (Gopalan et al., 2022).

1. Formal definition and hierarchy

In the basic binary setting, let $D$ be a distribution over $(x,y^*)\in X\times\{0,1\}$, let $p:X\to[0,1]$ be a predictor, and let $C\subseteq\{c:X\to[-1,1]\}$ be a class of subgroup tests. Multiaccuracy is the requirement

$$\forall c\in C:\quad \Bigl|\mathbb{E}\bigl[c(x)(y^*-p(x))\bigr]\Bigr|\le \alpha.$$

It enforces approximate unbiasedness on every subgroup or test function in $c$0. Approximate calibration is expressed through expected calibration error,

$c$1

and $c$2 is $c$3-calibrated if $c$4. Multicalibration strengthens this by requiring calibration inside each subgroup: $c$5 Calibrated multiaccuracy, abbreviated $c$6, is the conjunction of $c$7-multiaccuracy and $c$8-calibration (Gopalan et al., 2022).

Notion	Defining condition	Position
Multiaccuracy	$c$9	weakest
Calibrated multiaccuracy	multiaccuracy $C$0	intermediate
Multicalibration	calibration within each subgroup and score slice	strongest

This hierarchy is not merely terminological. Multiaccuracy constrains only subgroup means, so opposite errors at different score levels may cancel. Multicalibration removes that cancellation by conditioning on $C$1. Calibrated multiaccuracy retains subgroup mean control while adding a global score-semantics constraint. A closely related formulation in multiclass fairness uses group-wise calibration: $C$2 which conditions on both the predicted probability vector and a protected group $C$3 (Brahmbhatt et al., 2023).

2. Learning-theoretic significance

A central theoretical role of calibrated multiaccuracy appears in the decomposition of Loss Outcome Indistinguishability (Loss OI). For a loss class $C$4 and hypothesis class $C$5, Loss OI decomposes into two parts: hypothesis OI, which is equivalent to multiaccuracy over the derived class $C$6, and decision OI, which is equivalent to weighted calibration with respect to weights of the form $C$7. For generalized linear model losses $C$8, the discrete derivative satisfies $C$9, so $p(x)$0. In that regime, multiaccuracy on $p(x)$1 together with calibration is sufficient for Loss OI, and Loss OI in turn implies omniprediction. The same paper gives a geometric restatement: for GLM losses, Loss OI is equivalent to an approximate Bregman Pythagorean theorem, so a calibrated-multiaccurate predictor behaves as an approximate Bregman projection of the Bayes predictor (Gopalan et al., 2022).

The complement to this positive picture is that multiaccuracy alone is weak as a learning primitive. There are distributions and classes $p(x)$2 for which a predictor can be $p(x)$3-multiaccurate, some $p(x)$4 can have correlation $p(x)$5 with the labels, and yet every post-processing $p(x)$6 has zero correlation with the labels. By contrast, if $p(x)$7 is both $p(x)$8-multiaccurate and $p(x)$9-calibrated, then thresholding at $$MA(\alpha)\supseteq calMA(\alpha)\supseteq MC(\alpha)$$0 recovers strong agnostic learning: $$MA(\alpha)\supseteq calMA(\alpha)\supseteq MC(\alpha)$$1 This result isolates the role of calibration: multiaccuracy aligns the predictor with the projection of $$MA(\alpha)\supseteq calMA(\alpha)\supseteq MC(\alpha)$$2 on $$MA(\alpha)\supseteq calMA(\alpha)\supseteq MC(\alpha)$$3, while calibration turns nontrivial score deviations away from $$MA(\alpha)\supseteq calMA(\alpha)\supseteq MC(\alpha)$$4 into usable predictive correlation (Casacuberta et al., 21 Apr 2025).

3. Fairness, subgroup guarantees, and proxy groups

In fairness applications, calibrated multiaccuracy is typically interpreted as calibration-conditioned subgroup validity. One explicit route is through group-wise calibration with respect to a sensitive attribute $$MA(\alpha)\supseteq calMA(\alpha)\supseteq MC(\alpha)$$5: $$MA(\alpha)\supseteq calMA(\alpha)\supseteq MC(\alpha)$$6 Under the fairness notion called perfect fairness or no amplification,

$$MA(\alpha)\supseteq calMA(\alpha)\supseteq MC(\alpha)$$7

group-wise calibration implies both global calibration and fairness. The same framework defines Proportional Equality

$$MA(\alpha)\supseteq calMA(\alpha)\supseteq MC(\alpha)$$8

so $$MA(\alpha)\supseteq calMA(\alpha)\supseteq MC(\alpha)$$9 means the model does not amplify group disparities already present in the data. In this sense, group-wise calibration can be read as a two-group special case of calibrated multiaccuracy: it is weaker in group richness than general multicalibration, but stronger per protected group because it conditions on the full probability vector (Brahmbhatt et al., 2023).

When sensitive attributes are missing, recent work replaces true groups $D$0 by proxy groups $D$1 and derives explicit upper bounds on true fairness violations. For binary prediction,

$D$2

with

$D$3

Taking maxima over a proxy family $D$4 yields certified upper bounds $D$5 and $D$6 on worst-case true-group multiaccuracy and multicalibration violations. This gives a distributional notion of calibrated multiaccuracy under missing sensitive labels: proxy-group auditing and correction can improve worst-case guarantees on the unknown true groups (Bharti et al., 4 Mar 2025).

4. Efficient relaxations and algorithmic constructions

A central algorithmic question is whether one can obtain multicalibration-style consequences at a cost close to multiaccuracy. One answer is low-degree multicalibration, which defines a hierarchy

$D$7

In the binary case, degree-$D$8 multicalibration adds constraints of the form

$D$9

Degree $(x,y^*)\in X\times\{0,1\}$0 is ordinary multiaccuracy; higher degrees weight residuals by low-degree polynomials of the prediction and therefore encode a calibrated strengthening of multiaccuracy. Degree $(x,y^*)\in X\times\{0,1\}$1 already yields substantive control: it bounds predictor variance relative to the Bayes predictor, enforces positive covariance between predictions and labels within groups, and recovers squared-loss optimality over the class $(x,y^*)\in X\times\{0,1\}$2. In multiclass settings, the paper’s sample-complexity comparison shows that low-degree multicalibration improves exponentially, in the number of classes, over full multicalibration (Gopalan et al., 2022).

The algorithmic counterpart is the $(x,y^*)\in X\times\{0,1\}$3 construction. Starting from a predictor $(x,y^*)\in X\times\{0,1\}$4, it alternates a multiaccuracy routine $(x,y^*)\in X\times\{0,1\}$5, which updates the predictor using a weak agnostic learner for $(x,y^*)\in X\times\{0,1\}$6, with recalibration steps based on discretization and empirical calibration error. The resulting algorithm outputs a $(x,y^*)\in X\times\{0,1\}$7-discrete predictor $(x,y^*)\in X\times\{0,1\}$8, uses $(x,y^*)\in X\times\{0,1\}$9 weak-learner calls, and has outer-loop complexity $p:X\to[0,1]$0. This matches the weak-learner-call complexity of plain multiaccuracy, whereas known multicalibration algorithms require $p:X\to[0,1]$1 weak-learner calls together with stronger assumptions (Gopalan et al., 2022).

A more recent complexity-theoretic refinement argues that even calibrated multiaccuracy can be stronger than necessary in some applications. In pseudoentropy characterizations, multiaccuracy plus weight-restricted calibration—calibration only against a small family of weights $p:X\to[0,1]$2—is sufficient, while full calibrated multiaccuracy or multicalibration incurs exponential dependence on alphabet size. This identifies calibrated multiaccuracy as a meaningful but sometimes overly strong regularity notion when the task only requires calibration against specific loss-derived weights (Hu et al., 8 Jul 2025).

5. Practical procedures and empirical manifestations

Empirically, calibrated multiaccuracy is often approached through post-hoc multicalibration or through train-time surrogates. A broad empirical study across tabular, image, and LLMs reports three findings: models that are calibrated out of the box tend to be relatively multicalibrated without any additional post-processing; multicalibration post-processing can help inherently uncalibrated models and large vision and LLMs; and traditional calibration measures may sometimes provide multicalibration implicitly (Hansen et al., 2024). This study evaluates HKRR and HJZ multicalibration procedures alongside isotonic regression, Platt scaling, and temperature scaling, and it emphasizes worst-group smECE as the operational multicalibration metric.

A train-time route toward calibrated multiaccuracy is MDCA (Multi-class Difference in Confidence and Accuracy), introduced as an auxiliary loss for multiclass classification, domain shift, dense prediction, and NLP: $p:X\to[0,1]$3 This loss aligns, for each class $p:X\to[0,1]$4, the batch-average predicted probability mass with the empirical class frequency, without privileging the argmax class. In the paper’s own interpretation, MDCA is a batch-level loss that pushes the model toward a multi-class, distribution-wide equality between confidence and accuracy, approximating a multiaccuracy-style guarantee at train time. Reported results include an ECE (SCE) of $p:X\to[0,1]$5 $p:X\to[0,1]$6 on CIFAR-100, compared to $p:X\to[0,1]$7 $p:X\to[0,1]$8 by the SOTA, improved calibration under PACS domain shift, and a $p:X\to[0,1]$9 reduction in calibration error on PASCAL-VOC segmentation relative to focal loss (Hebbalaguppe et al., 2022).

Weak supervision introduces a different practical challenge: the calibration moments

$C\subseteq\{c:X\to[-1,1]\}$0

are not directly observable when clean labels are unavailable. A unified weak-supervision framework rewrites these moments via contamination-matrix operators for PU, UU, and positive-confidence learning, thereby defining weakly supervised estimators of multicalibration error and a boosting-style post-hoc algorithm, WLMC. In the PU case, for example,

$C\subseteq\{c:X\to[-1,1]\}$1

and the corresponding empirical supremum over $C\subseteq\{c:X\to[-1,1]\}$2 estimates multicalibration under weak labels. This extends calibrated multiaccuracy from clean-label auditing to weak-label auditing and correction (Futami et al., 11 May 2026).

6. Limitations and frontier directions

The main conceptual limitation is that calibrated multiaccuracy is still weaker than multicalibration. It controls subgroup mean error and global score semantics, but it does not enforce calibration on every subgroup–score slice. This gap matters in applications that require the full conditional structure of multicalibration, such as the strongest omniprediction or swap-agreement consequences. Conversely, multiaccuracy alone is too weak for several of the same goals. Much of the recent literature therefore treats calibrated multiaccuracy as an intermediate regime whose sufficiency is application-dependent (Casacuberta et al., 21 Apr 2025).

A second limitation concerns scaling and representation. In the pseudoentropy literature, exponential dependence on alphabet size is unavoidable for multicalibration and also for the weaker notion of calibrated multiaccuracy. This lower-bound perspective motivates weaker substitutes such as weight-restricted calibration when only a structured family of losses matters (Hu et al., 8 Jul 2025). In fairness settings, a parallel limitation is group specification: many positive results assume either explicit protected groups or proxy groups with known error rates, and extending guarantees to rich, overlapping, or automatically discovered subgroup families remains difficult (Bharti et al., 4 Mar 2025).

Emerging applications show that new subgroup structures can create calibration failures not captured by classical formulations. In open-ended question answering with multiple valid answers, answer cardinality acts as a hidden axis along which many confidence estimators systematically fail: accuracy increases with the number of valid answers, while estimated confidence decreases. The proposed Semantic Confidence Aggregation remedies this by aggregating confidence over multiple sampled semantic responses, improving mixed-answer calibration while preserving strong calibration on single-answer questions. A plausible implication is that calibrated multiaccuracy in generative models will increasingly be studied over latent subgroup axes such as answer multiplicity, domain, and semantic equivalence classes rather than only over explicit demographic groups (Wang et al., 8 Feb 2026).