---
title: Class-Weighted Cross-Entropy Loss
url: https://www.emergentmind.com/topics/class-weighted-cross-entropy-loss
type: topic
---

# Class-Weighted Cross-Entropy Loss

Class-weighted cross-entropy loss is a modification of standard cross-entropy loss designed to address class imbalance in classification and segmentation tasks. By reweighting the loss contributions from each class, either globally or at the pixel/instance level, it ensures improved gradient signal for under-represented or high-cost classes. Numerous variants and extensions—ranging from simple inverse-frequency schemes, adaptive weighting, spatially-structured weighting, to hierarchical and class-distance-based formulations—have been developed for different modalities and downstream objectives.

## 1. Mathematical Definition and Formulation

In multiclass classification, given $C$ classes, softmax predictions $p(c_i)$, and one-hot ground-truth $t_i$, the standard categorical cross-entropy is:
$$
L_\mathrm{CE} = -\sum_{i=1}^C t_i \log p(c_i)
$$
The class-weighted variant introduces per-class weights $w_{c_i}$:
$$
L_\mathrm{WCE} = -\sum_{i=1}^C w_{c_i} t_i \log p(c_i)
$$
where $w_{c_i}$ is typically a function of class frequency or domain-defined costs. For binary classification, this simplifies to the familiar positive/negative class weighting. Class weights can be constant (e.g., $w_{c} \propto 1/\text{freq}(c)$), derived from effective number of samples, or computed adaptively per iteration.

A comprehensive theoretical treatment shows that weighted cross-entropy, for any positive vector $\omega_c$, minimizes the expected weighted error, formalized in score-oriented loss theory [2305.13472]. This includes the classical cost-sensitive and frequency-balanced losses as special cases, with explicit guarantees on expected error minimization in the weighted metric.

## 2. Weight Computation: Static, Adaptive, and Structured Approaches

The choice and computation of class weights are critical for practical effectiveness:

- **Static Inverse-Frequency:** $w_c = N_\mathrm{All}/(N_\mathrm{Labels}N_c)$ for class $c$, ensuring that rare classes contribute more to the loss [2312.02266].
- **Effective Number of Samples:** $w_c = (1-\beta)/(1-\beta^{n_c})$ with $\beta\to 1$ amplifying weights for small $n_c$ [2006.01413].
- **Heuristic Weights:** Manually assign $w_c=\alpha>1$ for minority classes, $w_c=1$ for majority/background [2006.01413].
- **Ordered Weighted Average (OWA):** Weights are reassigned each iteration to classes with the current largest loss contributions, governed by a linguistic quantifier function [2305.19443].
- **Recall-based Adaptive Weights:** Per-class weights are computed dynamically as $w_{c,t}=1-\text{recall}_c$, focusing on classes with low recall and relaxing as recall improves [2106.14917].
- **Spatial and Geometric Weighting:** Pixel-wise or edge/boundary weights for segmentation or structure-aware applications [2412.06045, 1802.07465, 2507.06569].
- **Hierarchical Weights:** When class labels are structured in a tree, class weights and level weights are combined, enabling cross-entropy over each parent-child softmax [2312.02266].

In many implementations, weights are normalized (for example, their mean is set to unity) to avoid loss scaling issues.

## 3. Extensions: Hierarchical, Distance, and Similarity-weighted Losses

### Hierarchical Cross-Entropy

In hierarchical taxonomies, Villar et al. introduce weighted hierarchical cross-entropy (WHXE) where class weights $W(c^{(h)})$ and level weights $\lambda(c^{(h)})$ are applied at all tree levels:
$$
\mathcal{L}_\mathrm{WHXE} = -\sum_{h=0}^{H-1} W(c^{(h)}) \lambda(c^{(h)}) \log p(c^{(h)} | c^{(h+1)})
$$
This structure allows for flexible classification in tree-structured domains, such as astrophysical transient taxonomies, generalizing the flat class-weighted CE as a special case ($H=1$) [2312.02266].

### Class Distance-weighted Cross-Entropy

For ordinal tasks, class distance weighted cross-entropy (CDW-CE) penalizes errors by distance in label space:
$$
L_\mathrm{CDW-CE}(\hat{y}, y) = -\sum_{i=0}^{N-1} |i - c|^\alpha \log(1 - \hat{y}_i)
$$
where $c$ is the true class, $\alpha$ adjusts penalty sharpness. This directly embeds ordinal structure; distant misclassifications are penalized more than near ones. CDW-CE outperforms classical CE, CORN, CO2, and HO2 ordinal losses on both accuracy and interpretability (CAM quality) in medical imaging [2202.05167, 2412.01246].

### Similarity-weighted Cross-Entropy

SimLoss generalizes class-weighted CE with a class similarity matrix $S$: 
$$
L_\mathrm{SimLoss} = -\frac{1}{N}\sum_{i=1}^N \log\left(\sum_{c=1}^C S_{y_i,c}p_i[c]\right)
$$
This allows explicit modeling of semantic/ordinal relations: $S_{i,j}$ encodes the similarity between classes $i$ and $j$, constructed via knowledge or embedding similarity. SimLoss strictly generalizes class-weighted CE (retrieved with $S=I$) [2003.03182].

## 4. Applications and Empirical Performance

Class-weighted cross-entropy and its variants have been systematically validated in domains suffering from severe label imbalance or needing specialized error costs:

- **Object Detection:** Weighted CE, Focal Loss, and class-balanced loss improve minority-class recall (e.g., “Bike” recall from 19.1% to 49.1% or higher), with focal loss providing best overall recall when combined with effective sample weighting [2006.01413].
- **Medical and Biological Segmentation:** Multiclass weighted losses using per-pixel class weights, and spatial/shape-aware schemes (DWM, SAW, DBCE) yield substantial improvements in boundary F1 and instance recall, outperforming focal and standard CE losses [2412.06045, 1802.07465].
- **Semantic Segmentation:** Fixed class-weighted CE can lead to excessive false positives for minority classes; recall-adaptive weighting rectifies over-emphasis and improves both mean accuracy and mean IoU vs. static weighting [2106.14917].
- **Ordinal/Hierarchical Classification:** Class distance-weighted and hierarchical cross-entropy losses consistently provide superior scores such as Quadratic Weighted Kappa, macro-F1, and accuracy in both disease severity and astrophysics [2312.02266, 2202.05167, 2412.01246].

## 5. Implementation, Pseudocode, and Theoretical Guarantees

Implementation is functionally straightforward—introducing a per-sample or per-pixel weight vector/matrix in the CE loss computation. Representative PyTorch/numpy-style pseudocode appears in all cited works [2312.02266, 2305.19443, 2412.06045]. For instance, in the flat case:

```python
def weighted_cross_entropy(logits, targets, class_weights):
    pred_probs = softmax(logits, dim=1)
    # targets: one-hot or integer-encoded
    weights = class_weights[targets]  # shape: (batch_size,)
    loss = -weights * log(pred_probs[range(len(targets)), targets])
    return loss.mean()
```

Theoretical work demonstrates that, for any choice of positive weights, weighted CE loss is the unique continuous convex surrogate optimally minimizing the expected weighted error metric when the score is linear in entries of the confusion matrix [2305.13472].

## 6. Limitations, Best Practices, and Evolving Variants

- **Class-weight Selection:** Overly large inverse frequency weights may destabilize optimization if rare classes have extremely low counts. Normalization and hyperparameter clipping are standard remedies.
- **Adaptive Weighting:** Dynamic (epoch/batch-wise) weighting schemes—e.g., OWAdapt and recall-based losses—address shifting class difficulty, automatically focusing learning on underperforming classes [2305.19443, 2106.14917].
- **Spatial/Structured Weighting:** For segmentation, adding spatial structure via geometric or morphological priors substantially boosts precision, especially for small and complex objects [2412.06045, 1802.07465, 2507.06569].
- **Calibration Caveats:** Hierarchical and per-level weighted losses may produce pseudo-probabilities requiring post-hoc calibration if probability estimates are required for downstream tasks [2312.02266].
- **Hybrid Schemes:** Combining class-weighted CE with region-based or focus losses (Dice, Focal) is common in practice, particularly in medical image segmentation [2412.06045].

| Variant         | Weight Source                  | Application Example                    |
|-----------------|-------------------------------|----------------------------------------|
| Static WCE      | Inverse class frequency       | Generic imbalance, object detection    |
| DBCE, DWM, SAW  | Morphological, pixel geometry | Med. image segmentation, cell analysis |
| Hierarchical CE | Taxonomy/level weighted       | Astrophysical transients               |
| SimLoss         | Class similarity matrix       | Ordinal/semantic classification        |
| CDW-CE          | Class distance in label space | Disease severity, ordinal regression   |
| OWAdapt         | Per-class/batch adaptive      | Imbalanced multiclass                  |

## 7. Connections to Implicit Geometry and Modern Training Regimes

Recent advances explore the effect of class-weighted and logit-adjusted cross-entropy variants on the learned classifier and embedding geometry. Weighted CE can yield only marginal improvements in class separation in overparameterized regimes. Multiplicative logit-scaling (e.g., label-dependent temperature, LDT) can enforce symmetric geometry (Simplex Equiangular Tight Frames), overcoming the limitations of standard weighting, especially for large models and extreme imbalance [2303.07608].

A direct implication is that simple weighting strategies offer most value in small to moderate scale and in the presence of modest imbalance, while advanced geometric or adaptive methods are called for in challenging or high-performance settings.

---

**References**

- "Hierarchical Cross-entropy Loss for Classification of Astrophysical Transients" [2312.02266]
- "SimLoss: Class Similarities in Cross Entropy" [2003.03182]
- "Class Distance Weighted Cross-Entropy Loss for Ulcerative Colitis Severity Estimation" [2202.05167]
- "Multiclass Weighted Loss for Instance Segmentation of Cluttered Cells" [1802.07465]
- "Edge-Boundary-Texture Loss: A Tri-Class Generalization of Weighted Binary Cross-Entropy for Enhanced Edge Detection" [2507.06569]
- "OWAdapt: An adaptive loss function for deep learning using OWA operators" [2305.19443]
- "Resolving Class Imbalance in Object Detection with Weighted Cross Entropy Losses" [2006.01413]
- "A comprehensive theoretical framework for the optimization of neural networks classification performance with respect to weighted metrics" [2305.13472]
- "Dilated Balanced Cross Entropy Loss for Medical Image Segmentation" [2412.06045]
- "Striking the Right Balance: Recall Loss for Semantic Segmentation" [2106.14917]
- "Class Distance Weighted Cross Entropy Loss for Classification of Disease Severity" [2412.01246]
- "On the Implicit Geometry of Cross-Entropy Parameterizations for Label-Imbalanced Data" [2303.07608]

Source: https://www.emergentmind.com/topics/class-weighted-cross-entropy-loss