---
title: 'Label Smoothing++: Adaptive Regularization'
url: https://www.emergentmind.com/topics/label-smoothing-3e7f26a1-955a-4186-af2f-119cf6e3d193
type: topic
---

# Label Smoothing++: Adaptive Regularization

Label Smoothing++ denotes a family of label-regularization schemes that extend standard label smoothing beyond a single global, uniform perturbation of one-hot targets. In standard label smoothing, a one-hot label $\bar{y}$ is replaced by a convex combination with a uniform distribution $u$, typically $\bar{y}^{ls} = (1-\alpha)\bar{y} + \alpha u$, or equivalently by assigning $1-\alpha$ or $1-\alpha+\alpha/K$ to the target class and uniform nonzero mass to non-target classes. The “++” designation is used in the literature both for a specific class-wise learnable method called “Label Smoothing++” and, more broadly, for structured, adaptive, or post-hoc corrected variants that preserve the regularization benefits of label smoothing while addressing its uniformity, calibration, ranking, or robustness limitations [2509.05307].

## 1. Standard label smoothing and the rationale for going beyond it

Standard label smoothing is a regularized cross-entropy procedure in which “hard” one-hot labels are “smoothed” by uniformly distributing probability mass to other classes, reducing overfitting. In a $K$-class problem, one common form is
\[
y^{\text{LS}}_k = (1-\alpha) y_k + \frac{\alpha}{K},
\]
or, equivalently for target class $y$, $q_c = 1-\beta$ and $q_{k\neq c} = \beta/(K-1)$ under a different parametrization. This regularizer is simple to implement and has been reported to improve test accuracy, calibration, and generalization across image classification, translation, and speech settings [2403.14715].

The same literature also identifies recurrent limitations. Conventional label smoothing assumes that each non-target class is equally likely; several later works take this assumption as the main deficiency to be corrected. It has also been shown to encourage penultimate-layer representations of training examples from the same class to group in tight clusters, improving calibration while reducing information in logits about resemblances between instances of different classes, which is detrimental to knowledge distillation [1906.02629]. Other analyses show that standard label smoothing can degrade selective classification, especially in low-risk regimes, by distorting the uncertainty rank ordering of correct and incorrect predictions [2403.14715]. These observations motivate richer forms of label regularization rather than abandoning smoothing altogether.

## 2. Main design patterns of Label Smoothing++

Across the literature, Label Smoothing++ methods replace the single global uniform smoothing rule with one or more of the following: non-uniform class structure, instance-specific smoothing, geometry- or objectness-conditioned smoothing, layer-wise or posterior-driven adaptation, or post-hoc correction of label-smoothing side effects.

| Family | Core mechanism | Representative source |
|---|---|---|
| Class-wise learned smoothing | Learn a class-dependent non-target distribution | [2509.05307] |
| Rank-aware smoothing | Zipf-shaped non-target targets from predicted ranks | [2207.12980] |
| Instance-specific smoothing | Learn $P_{\text{ls}}(\cdot\mid x_i)$ by bi-level optimization | [2305.04971] |
| Objectness-conditioned smoothing | Set $\alpha_i$ from relative object size | [2009.06432] |
| Noise-aware adaptive smoothing | Use EMA confidence and auxiliary classifiers | [2206.07277] |
| Pairwise / midpoint smoothing | Learn smoothing on interpolated samples | [2106.13913] |
| Structural smoothing | Choose cluster-wise smoothing from BER estimates | [2001.01900] |
| Post-hoc correction | Normalize logits after LS for selective classification | [2403.14715] |

This taxonomy suggests that Label Smoothing++ is best understood not as a single algorithmic template but as a research direction: retain the regularization effect of soft targets while replacing uniform, static, label-agnostic smoothing by structured or adaptive mechanisms grounded in class similarity, sample difficulty, uncertainty, or downstream decision criteria.

## 3. Learnable and structured non-target distributions

The 2025 method explicitly titled “Label Smoothing++” parameterizes a learnable class-wise distribution over non-target classes. For each true class $y$, it learns a vector $C_y$ over the other $K-1$ classes, inserts a zero at the target position, and forms
\[
\bar{y}^{ls++} = (1-\alpha)\bar{y} + \alpha C_y^{(K)}.
\]
The target-class probability is fixed to $1-\alpha$, while the non-target mass $\alpha$ is allocated non-uniformly according to a learned $C$-matrix. To avoid collapse of the learned label distribution, the method uses a symmetric cross-entropy objective with decoupled gradient flow: $H(\bar{y}^{ls++},\hat{y})$ updates network parameters only, while $H(\hat{y},\bar{y}^{ls++})$ updates the $C$-matrix only. The reported experiments span image, video, text, and audio tasks, and the learned $C$-matrices visibly encode inter-class relationships such as elevated Dog$\rightarrow$Cat mass [2509.05307].

Other “++” variants implement structure differently. Zipf’s Label Smoothing replaces uniform non-target smoothing by a rank-dependent Zipf law,
\[
\tilde{p}_c \propto r_c^{-\alpha}, \quad c\neq y,
\]
with ranks obtained by Dense Classification Ranking from local predictions on feature maps. The target class remains under standard cross-entropy, and only the non-target tail is regularized through a KL term to match the Zipf-shaped distribution [2207.12980]. LABO formulates label smoothing as a bi-level optimization over an instance-specific smoothing distribution and derives a closed-form optimum,
\[
P_{\text{ls}}(j\mid x_i) = \frac{p_\theta(j\mid x_i)^{\alpha/\beta}}{\sum_{j'} p_\theta(j'\mid x_i)^{\alpha/\beta}},
\]
equivalently a temperature-softmax under $\beta=\alpha T$, thereby turning uniform smoothing into learned, instance-dependent label regularization [2305.04971].

Related structured variants operate at different granularities. Adaptive Regularization of Labels learns a residual correlation matrix whose rows define class-specific residual labels over wrong classes, updated online from the model’s own erroneous probabilities; the resulting ALR-S combination with conventional label smoothing is explicitly described as a stronger label-regularization scheme than static LS [1908.05474]. Structural Label Smoothing instead computes cluster-specific smoothing strengths $\hat{\alpha}_c$ from estimated cluster-level Bayes error rates, with closed-form allocation under a fixed global average smoothing budget [2001.01900].

## 4. Sample-dependent, uncertainty-dependent, and context-conditioned smoothing

A second major line of Label Smoothing++ makes smoothing depend on the input rather than only on the class. Adaptive Label Smoothing for image classification ties the smoothing factor to objectness derived from bounding boxes:
\[
\alpha_i = 1 - \frac{w_i h_i}{W_i H_i}.
\]
When the object occupies most of the crop, $\alpha_i\approx 0$ and the target remains close to one-hot; when the object is tiny or absent, $\alpha_i\approx 1$ and the target approaches uniform. An optional interpolation coefficient $\beta$ mixes adaptive smoothing with hard labels. In ImageNet experiments, this design produced an order of magnitude reduction in confidence on context-only images and improved downstream MS COCO detection by $+2.1$ mAP over hard-label pretraining [2009.06432].

ALASCA extends the idea to noisy labels and internal representations. It interprets label smoothing as inducing a regularizer
\[
\Omega(\mathbf{f}) = L\log\Big[\sum_{i=1}^{L} e^{f_i(\cdot)}\Big] - \sum_{i=1}^{L} f_i(\cdot),
\]
and argues that this implicitly promotes Lipschitz regularization of the feature extractor. Its adaptive smoothing coefficient is sample-wise,
\[
\tilde{\alpha}(\mathbf{x}) = \big[1-\mathcal{S}(\mathbf{f}(\mathbf{x}))\big]_y,
\]
stabilized by an EMA of logits and applied mainly through auxiliary classifiers attached to intermediate layers. The result is a noise-aware, layer-wise adaptive smoothing scheme that improves robustness of feature extractors and systematically strengthens a range of learning-with-noisy-labels baselines [2206.07277].

A related but conceptually distinct route is variational learning. Under a variational objective
\[
\mathcal{L}(q) = \sum_{i=1}^N \mathbb{E}_{q(\theta)}[\ell_i(\theta)] + D_{\mathrm{KL}}\bigl(q(\theta)\,\|\,p(\theta)\bigr),
\]
the expected gradient can be rewritten as standard training with example-specific label perturbations
\[
\tilde{y}_{i,t} = y_i + \varepsilon_{i,t},
\]
and, in generalized linear models,
\[
\varepsilon_{i,t} = A'(f_i(\theta_t)) - \mathbb{E}_{q_t}[A'(f_i(\theta))].
\]
This induces larger smoothing near decision boundaries and under high posterior variance, without explicitly designing an adaptive LS rule [2502.07273]. In domain adversarial training, an analogous idea is applied to environment labels rather than task labels: Environment Label Smoothing replaces one-hot domain labels with smoothed targets $\tilde{y}_d$, reducing discriminator over-confidence and improving stability, local convergence, and robustness to noisy environment labels [2302.00194].

## 5. Geometric and information-theoretic interpretations

Several analyses explain Label Smoothing++ in terms of geometry, uncertainty, or representation compression rather than only target perturbation. Pairwise Label Smoothing, also called midpoint regularization, constructs midpoint samples
\[
x_{ij} = \frac{x_i+x_j}{2}, \qquad q(y\mid x_{ij}) = \frac{y_i+y_j}{2},
\]
and learns a smoothing distribution for each midpoint through an auxiliary head. The resulting targets are deliberately high-uncertainty, and the method empirically produces very low winning softmax scores for both in- and out-of-distribution samples while improving classification performance relative to uniform LS and Mixup baselines [2106.13913].

A different theoretical lens shows that the label-augmentation component of both label smoothing and Mixup pushes models toward minimum-variance features. In binary linear settings with both low-variance and high-variance separating features, label smoothing and Mixup provably converge to solutions that use only the low-variance feature, whereas standard training with weight decay can learn higher-variance features. The same work shows that, more generally, the losses of label smoothing and Mixup are lower bounded by a function of model output variance, and empirically that these methods can therefore be more susceptible to low variance spurious correlations in the training data even while performing strongly on standard image benchmarks [2402.06855].

Other papers frame the phenomenon more positively. One study interprets label smoothing as encouraging class-wise compression in the penultimate layer, with tight clusters around class templates; this improves calibration and beam-search behavior but erases inter-class resemblance information useful for distillation [1906.02629]. Another recent analysis revisits label smoothing through the information bottleneck and argues that, under sufficient model flexibility and no conflicting labels for the same input, the model output obtained through label smoothing explores the optimal solution of the information bottleneck; in that sense, label smoothing becomes a practical approach to the information bottleneck and is empirically insensitive to nuisance factors that do not contain target information [2508.14077].

## 6. Failure modes, corrections, and the current role of Label Smoothing++

The most prominent documented failure mode is selective classification. In large-scale image classification and semantic segmentation, label smoothing often improves top-1 accuracy at full coverage but consistently degrades selective classification: RC curves shift upward, AURC increases, and degradation is strongest in the low-risk regime. The explanation given is logit-level: relative to cross-entropy, label smoothing applies stronger downward pressure to the maximum logit when a prediction is likely correct and weaker pressure when it is likely wrong, thereby harming the uncertainty ranking needed by maximum-softmax-probability rejection. A simple post-hoc correction, logit normalization,
\[
v' = \frac{v}{\|v\|_p}, \qquad U(x) = -v'_{\max},
\]
recovers most of the lost selective-classification performance on LS-trained models without retraining; this train-time LS plus test-time logit normalization is explicitly characterized as a practical “Label Smoothing++” [2403.14715].

A second correction targets the probabilistic interpretation of label smoothing itself. One line of work argues that standard label smoothing conflates a robustified loss with a modified estimator. In logistic regression, it introduces a modified LSLR and proposes MLSLR as an improvement: MLSLR uses the same smoothed-KL loss as LSLR and the same consistent estimator as ordinary logistic regression, while not squeezing the logits. The theoretical conclusion is that larger smoothing levels reduce efficiency under correctly specified models but increase robustness under model misspecification; experimentally, the estimator modification and logit squeezing in standard LSLR have negative effects on both probability estimation and classification performance, motivating MLSLR as a more principled variant [2305.08501].

Taken together, these results position Label Smoothing++ as a mature but heterogeneous research area rather than a single settled method. Its defining move is to replace uniform, class-agnostic, globally fixed smoothing by mechanisms that learn inter-class structure, adapt to instance difficulty or uncertainty, exploit auxiliary geometry, or correct downstream pathologies introduced by LS. The unifying theme is not the abandonment of label smoothing, but its reformulation into a structured, adaptive, and task-aware form of label regularization.

Source: https://www.emergentmind.com/topics/label-smoothing-3e7f26a1-955a-4186-af2f-119cf6e3d193