---
title: Class-Wise Inequality Loss for Calibration
url: https://www.emergentmind.com/topics/class-wise-inequality-loss
type: topic
---

# Class-Wise Inequality Loss for Calibration

Searching arXiv for the primary paper and closely related class-wise loss-balancing work.
Class-wise inequality loss is not a single canonical formula in the literature, but a class of objectives that explicitly penalize disparities among class-level losses. In the most direct recent usage, it denotes the absolute difference between the positive-class and negative-class cross-entropy losses in a minibatch, introduced to mitigate both class imbalance and class-feature bias in medical diagnosis [2508.06943]. Closely related formulations synchronize class-wise losses for post-hoc calibration by minimizing their empirical standard deviation [2306.10989], while earlier deep hashing work used the label “class-wise inequality” for a class-wise likelihood with cubic and quantization constraints in binary code learning [1803.04137]. Across these usages, the common theme is that unequal class-level optimization signals are treated as a training pathology rather than as an incidental by-product of skewed data.

## 1. Terminological scope and problem setting

In the medical-diagnosis setting, class-wise inequality loss is introduced to address **class-feature bias**, defined as models’ potential reliance on features that are strongly correlated with only a subset of classes, leading to biased performance and poor generalization on other classes. The proposed objective promotes equal contributions of classification loss from positive-class and negative-class samples, and is paired with a class-wise group distributionally robust optimization objective that upweights underperforming classes [2508.06943].

A separate calibration framework uses class-wise loss inequality in a different sense. There, the class-wise training losses often diverge because of different levels of intra-class and inter-class appearance variation, and the method measures inequality by the empirical standard deviation of the per-class losses, then learns class-wise scaling factors to synchronize them during post-hoc calibration [2306.10989].

An earlier and technically distinct usage appears in deep supervised hashing, where the “class-wise inequality” or “cubic-constraint” loss models each class as an isotropic Gaussian in the Hamming-code space and regularizes continuous codes so that they remain inside a slightly enlarged Hamming cube before final binarization [1803.04137].

This suggests that the phrase should be interpreted through the surrounding optimization problem rather than as a single standardized loss.

## 2. Binary class-wise inequality loss in medical diagnosis

As formulated in the medical-diagnosis work, the minibatch $\mathcal{B}$ is partitioned into positive-class and negative-class subsets,
\[
\mathcal{B}^{\rm pos} = \{(x_i,y_i)\in\mathcal{B}\mid y_i=1\},\qquad
\mathcal{B}^{\rm neg} = \{(x_i,y_i)\in\mathcal{B}\mid y_i=0\}.
\]

The per-class cross-entropy losses are
\[
\mathcal{L}^{\rm pos}
= -\frac{1}{|\mathcal{B}^{\rm pos}|}\sum_{(x,y)\in\mathcal{B}^{\rm pos}}
\bigl[y\log\hat P(y\!=\!1\mid x)\bigr],
\]
\[
\mathcal{L}^{\rm neg}
= -\frac{1}{|\mathcal{B}^{\rm neg}|}\sum_{(x,y)\in\mathcal{B}^{\rm neg}}
\bigl[(1-y)\log\hat P(y\!=\!0\mid x)\bigr].
\]

The class-wise inequality loss is then defined as
\[
\mathcal{L}_{\rm cls\text{-}ineq}
=
\bigl|\mathcal{L}^{\rm pos}-\mathcal{L}^{\rm neg}\bigr|.
\]

Its immediate effect is to penalize the model whenever one class’s loss is much higher or lower than the other’s. The stated intuition is twofold. First, if a classifier exploits class-specific or spurious features that strongly benefit one class but not the other, one side can incur very low loss while the opposite side suffers high loss. Second, in an imbalanced dataset, the majority class tends to have lower loss while the minority class has high loss. Driving $\mathcal{L}^{\rm pos}\approx\mathcal{L}^{\rm neg}$ is therefore intended to force the model toward shared discriminative signals and to increase attention to the minority group without resorting to naive oversampling [2508.06943].

As presented, this formulation is explicitly binary: it is built from positive-class and negative-class losses rather than from a general $C$-class dispersion functional.

## 3. Coupling with class-weighted group DRO

The medical formulation does not use the inequality term in isolation. It is integrated with a **group distributionally robust optimization** term that automatically up-weights the worse-performing class. The per-class weights are defined by a softmax over the two losses, with stop-gradient:
\[
w^{c}
=
\mathrm{sg}\Biggl(
\frac{\exp(\tau\,\mathcal{L}^{c})}
{\exp(\tau\,\mathcal{L}^{\rm pos})+\exp(\tau\,\mathcal{L}^{\rm neg})}
\Biggr),
\qquad c\in\{\rm pos,neg\}.
\]

The resulting group DRO objective is
\[
\mathcal{L}_{\rm g\text{-}dro}
=
w^{\rm pos}\mathcal{L}^{\rm pos}
+
w^{\rm neg}\mathcal{L}^{\rm neg},
\]
and the overall training objective is
\[
\mathcal{L}_{\rm total}
=
\mathcal{L}_{\rm g\text{-}dro}
+
\alpha\,\mathcal{L}_{\rm cls\text{-}ineq}.
\]

The temperature parameter $\tau$ controls how aggressively the optimization focuses on the higher-loss class: as $\tau\to\infty$, all weight goes to the class with larger $\mathcal{L}^c$; as $\tau\to0$, the objective approaches a uniform per-class average. The coefficient $\alpha$ controls the strength of the inequality penalty. In practice, both are annealed over epochs: $\alpha$ is increased from $0$ toward a final $\alpha_{\rm end}$, while $\tau$ is decreased so that early training emphasizes stable class weighting and later training becomes more even across classes [2508.06943].

The gradient of the inequality term is
\[
\frac{\partial}{\partial \theta}\mathcal{L}_{\rm cls\text{-}ineq}
=
\mathrm{sign}(L_{\rm pos}-L_{\rm neg})
\Bigl(
\frac{\partial L_{\rm pos}}{\partial \theta}
-
\frac{\partial L_{\rm neg}}{\partial \theta}
\Bigr),
\]
which makes explicit that the update direction depends on which class currently has the larger loss.

The theoretical rationale is stated in information-theoretic terms. Equality of class-conditional entropies $H(Y\mid X)$ implies equality of class losses when the model approximates the true posterior, so large loss differences indicate reliance on class-specific features or under-use of minority evidence. Under mild regularity and symmetry assumptions, the related calibration work further shows that a reweighting proportional to $-(L_c-\overline{L})$ is the steepest-descent direction for reducing class-wise loss inequality while preserving total loss magnitude [2306.10989].

## 4. Empirical behavior and reported gains

The medical-diagnosis evaluation spans a synthetic toy setting and several real datasets: DAIC-WOZ, MODMA, ADReSS, BreastMNIST, and RetinaMNIST. Baselines include plain cross-entropy, class-weighted ERM, per-class-average ERM, and G-DRO only. Reported metrics are Macro F1, per-class F1, and accuracy [2508.06943].

On the toy example, the balanced setting reports ERM positive-class accuracy of $0.703$ versus $0.780$ for the class-unbiased method; in the imbalanced setting, ERM positive-class accuracy is $0.000$ versus $0.708$ for the proposed method. On DAIC-WOZ, described as a severe-imbalance setting, ERM collapses to the negative class; ERM with class weighting gives $\mathrm{MF1}\approx0.483$, G-DRO only gives $\mathrm{MF1}\approx0.510$, ERM plus the inequality term gives $\mathrm{MF1}\approx0.588$, and G-DRO plus the inequality term gives $\mathrm{MF1}\approx0.627$. MODMA, ADReSS, BreastMNIST, and RetinaMNIST all show consistent Macro F1 gains of $2$–$6$ points over the best ERM variant. In a subsampled RetinaMNIST case study, Macro F1 improves from $0.621$ to $0.633$ for balanced $100$-sample training and from $0.568$ to $0.604$ in the imbalanced $25\%$-positive setting [2508.06943].

The post-hoc calibration framework reports a different empirical pattern but a closely related conclusion. Across CIFAR-10, CIFAR-100, ImageNet, and long-tailed CIFAR10-LT/CIFAR100-LT, synchronizing class-wise losses reduces Expected Calibration Error while preserving accuracy. For example, CWLS+TS reduces average ECE on CIFAR-10 from $0.97\%$ to $0.72\%$, on CIFAR-100 from $1.52\%$ to $0.80\%$, and on ImageNet from $2.15\%$ to $1.94\%$, with accuracy change $\le 0.03\%$. On CIFAR10-LT, average ECE drops from $6.11$ to $3.06$; on CIFAR100-LT, from $9.51$ to $4.09$, while accuracy is maintained or slightly improved by up to $+0.85\%$ [2306.10989].

Taken together, these results support a recurrent empirical claim: synchronizing class-level losses can improve either downstream classification balance or calibration reliability without the accuracy degradation often associated with naive reweighting.

## 5. Class-wise loss synchronization and other related objectives

The calibration framework defines per-class negative log-likelihood losses on a validation set,
\[
L_c
=
-\sum_{(x,y)\in\mathcal{D}_c}\mathbf{1}\{y=c\}\ln\bigl(\phi_c(x)_c\bigr),
\]
collects them into $\mathbf{L}=(L_1,\dots,L_C)$, and measures inequality by
\[
\mathrm{Ineq}(\mathbf{L})
=
\sqrt{
\frac{1}{C}\sum_{c=1}^C (L_c-\overline{L})^2
},
\qquad
\overline{L}=\frac{1}{C}\sum_{c=1}^C L_c.
\]
The method normalizes each class loss, maps it through a sigmoid-shaped gain with parameters $\alpha$ and $\beta$, and trains with
\[
L_{\rm total}=\sum_{c=1}^C (1+w_c)L_c,
\]
where $\sum_c w_c\approx 0$ preserves the integrated loss scale. Classes with $L_c>\overline{L}$ receive extra gradient, whereas over-fitted classes with $L_c<\overline{L}$ have reduced gradient [2306.10989].

Other imbalance-oriented objectives use different proxies for class-level underperformance. The **Class-Wise Difficulty-Balanced loss** defines class difficulty as $d_{c,t}=1-A_{c,t}$, where $A_{c,t}$ is validation accuracy for class $c$ at epoch $t$, and sets class weights to $w_{c,t}=(d_{c,t})^{\tau_t}$ with a dynamic focusing exponent
\[
\tau_t=\frac{2}{1+e^{-b_t}},
\qquad
b_t=\frac{\max_c A_{c,t}}{\min_c A_{c,t}+\epsilon}-1,
\]
so that harder classes receive larger weight as training evolves [2010.01824].

**OWAdapt** takes class-level cross-entropies $F_c(y,p)$, sorts them, and applies an Ordered Weighted Average,
\[
L_{OWA}=\sum_{c=1}^C w_c F_{(c)}(y,p),
\]
reassigning the largest weights to the currently highest class-losses at each iteration. The intended effect is again to focus optimization pressure on the worst-performing classes, whether due to imbalance or intrinsic difficulty [2305.19443].

From a theoretical perspective, the balanced-loss literature studies surrogate objectives such as **Generalized Logit-Adjusted (GLA)** and **Generalized Class-Aware (GCA)** losses. Both are presented as Bayes-consistent for balanced-loss targets, but their $H$-consistency guarantees differ markedly: GLA requires complete hypothesis sets and yields bounds scaling at least as $1/\mathsf p_{\min}$, whereas GCA is $H$-consistent for any bounded or complete hypothesis set with bounds scaling as $1/\sqrt{\mathsf p_{\min}}$ [2512.23947].

These objectives are not the same loss, but they share a structural commitment: class-wise optimization signals should adapt to class-specific underperformance rather than remain fixed by raw frequency alone.

## 6. Earlier hashing usage and conceptual boundaries

In deep supervised hashing, the term “class-wise inequality” refers to a different construction. The method models each class as an isotropic Gaussian in Hamming-code space and minimizes the negative log-likelihood
\[
L_{\rm cls}
=
-\sum_{n=1}^{N}
\log
\Biggl[
\frac{
\exp\bigl(-\|r_n-\mu_{y_n}\|^2/(2\sigma^2)\bigr)
}{
\sum_{i=1}^{C}
\exp\bigl(-\|r_n-\mu_i\|^2/(2\sigma^2)\bigr)
}
\Biggr],
\]
where $r_n\in\{-1,+1\}^L$ is the $L$-bit code and $\mu_i$ is the binary center of class $i$. Training proceeds in two stages: a “cubic” stage constrains codes and centers to remain in $[-\alpha,+\alpha]^L$ with ReLU penalties, and a “vertex” stage adds a quantization term $\eta_2\sum_n\|b_n-r_n\|_2^2$ with $b_n=\mathrm{sign}(r_n)$ [1803.04137].

This formulation is class-wise because it clusters samples around class centers and repels them from other class centers. It is called “class-wise inequality” in the cited summary, but it does not equalize positive- and negative-class losses or minimize a dispersion statistic over class losses. Its primary domain is semantics-preserving hashing rather than calibration or debiasing under class imbalance.

A common misconception is therefore to treat “class-wise inequality loss” as a single, universally accepted object. The literature instead contains several paper-specific definitions. Another misconception is to equate all such losses with class-frequency weighting. Related work explicitly argues that the number of available training data may not always be a good clue to determine the weighting strategy, because some minority classes may already be sufficiently represented and overweighting them can drop overall performance [2010.01824]. In that sense, the broader class-wise-inequality perspective is less about frequency per se than about unequal optimization pressure, unequal fittedness, or unequal class-specific reliability.

The most defensible general characterization is therefore narrow and operational: class-wise inequality losses are objectives that monitor some class-level discrepancy—between two class losses, among many class losses, or between class centers in an embedding space—and convert that discrepancy into a training signal. Which discrepancy is used, and why, depends entirely on the task: debiasing in medical diagnosis [2508.06943], synchronization for calibration [2306.10989], adaptive class emphasis under imbalance [2010.01824; 2305.19443; 2512.23947], or semantics-preserving hashing [1803.04137].

Source: https://www.emergentmind.com/topics/class-wise-inequality-loss