Label Smoothing++: Adaptive Regularization

• Label Smoothing++ is a family of label-regularization techniques that refines traditional smoothing by replacing uniform perturbations with adaptive, structured distributions.
• The approach includes methods such as class-wise learned smoothing, rank-aware and instance-specific smoothing to improve calibration, uncertainty ranking, and model robustness.
• Advanced variants address limitations like selective classification failure via strategies such as post-hoc logit normalization, enhancing model performance in diverse tasks.

Label Smoothing++ denotes a family of label-regularization schemes that extend standard label smoothing beyond a single global, uniform perturbation of one-hot targets. In standard label smoothing, a one-hot label $\bar{y}$ is replaced by a convex combination with a uniform distribution $u$, typically $\bar{y}^{ls} = (1-\alpha)\bar{y} + \alpha u$, or equivalently by assigning $1-\alpha$ or $1-\alpha+\alpha/K$ to the target class and uniform nonzero mass to non-target classes. The “++” designation is used in the literature both for a specific class-wise learnable method called “Label Smoothing++” and, more broadly, for structured, adaptive, or post-hoc corrected variants that preserve the regularization benefits of label smoothing while addressing its uniformity, calibration, ranking, or robustness limitations (Chhabra et al., 22 Aug 2025).

1. Standard label smoothing and the rationale for going beyond it

Standard label smoothing is a regularized cross-entropy procedure in which “hard” one-hot labels are “smoothed” by uniformly distributing probability mass to other classes, reducing overfitting. In a $K$-class problem, one common form is

$$y^{\text{LS}}_k = (1-\alpha) y_k + \frac{\alpha}{K},$$

or, equivalently for target class $y$, $q_c = 1-\beta$ and $q_{k\neq c} = \beta/(K-1)$ under a different parametrization. This regularizer is simple to implement and has been reported to improve test accuracy, calibration, and generalization across image classification, translation, and speech settings (Xia et al., 2024).

The same literature also identifies recurrent limitations. Conventional label smoothing assumes that each non-target class is equally likely; several later works take this assumption as the main deficiency to be corrected. It has also been shown to encourage penultimate-layer representations of training examples from the same class to group in tight clusters, improving calibration while reducing information in logits about resemblances between instances of different classes, which is detrimental to knowledge distillation (Müller et al., 2019). Other analyses show that standard label smoothing can degrade selective classification, especially in low-risk regimes, by distorting the uncertainty rank ordering of correct and incorrect predictions (Xia et al., 2024). These observations motivate richer forms of label regularization rather than abandoning smoothing altogether.

2. Main design patterns of Label Smoothing++

Across the literature, Label Smoothing++ methods replace the single global uniform smoothing rule with one or more of the following: non-uniform class structure, instance-specific smoothing, geometry- or objectness-conditioned smoothing, layer-wise or posterior-driven adaptation, or post-hoc correction of label-smoothing side effects.

Family	Core mechanism	Representative source
Class-wise learned smoothing	Learn a class-dependent non-target distribution	(Chhabra et al., 22 Aug 2025)
Rank-aware smoothing	Zipf-shaped non-target targets from predicted ranks	(Liang et al., 2022)
Instance-specific smoothing	Learn $u$0 by bi-level optimization	(Lu et al., 2023)
Objectness-conditioned smoothing	Set $u$1 from relative object size	(Krothapalli et al., 2020)
Noise-aware adaptive smoothing	Use EMA confidence and auxiliary classifiers	(Ko et al., 2022)
Pairwise / midpoint smoothing	Learn smoothing on interpolated samples	(Guo, 2021)
Structural smoothing	Choose cluster-wise smoothing from BER estimates	(Li et al., 2020)
Post-hoc correction	Normalize logits after LS for selective classification	(Xia et al., 2024)

This taxonomy suggests that Label Smoothing++ is best understood not as a single algorithmic template but as a research direction: retain the regularization effect of soft targets while replacing uniform, static, label-agnostic smoothing by structured or adaptive mechanisms grounded in class similarity, sample difficulty, uncertainty, or downstream decision criteria.

3. Learnable and structured non-target distributions

The 2025 method explicitly titled “Label Smoothing++” parameterizes a learnable class-wise distribution over non-target classes. For each true class $u$2, it learns a vector $u$3 over the other $u$4 classes, inserts a zero at the target position, and forms

$u$5

The target-class probability is fixed to $u$6, while the non-target mass $u$7 is allocated non-uniformly according to a learned $u$8-matrix. To avoid collapse of the learned label distribution, the method uses a symmetric cross-entropy objective with decoupled gradient flow: $u$9 updates network parameters only, while $\bar{y}^{ls} = (1-\alpha)\bar{y} + \alpha u$0 updates the $\bar{y}^{ls} = (1-\alpha)\bar{y} + \alpha u$1-matrix only. The reported experiments span image, video, text, and audio tasks, and the learned $\bar{y}^{ls} = (1-\alpha)\bar{y} + \alpha u$2-matrices visibly encode inter-class relationships such as elevated Dog$\bar{y}^{ls} = (1-\alpha)\bar{y} + \alpha u$3Cat mass (Chhabra et al., 22 Aug 2025).

Other “++” variants implement structure differently. Zipf’s Label Smoothing replaces uniform non-target smoothing by a rank-dependent Zipf law,

$\bar{y}^{ls} = (1-\alpha)\bar{y} + \alpha u$4

with ranks obtained by Dense Classification Ranking from local predictions on feature maps. The target class remains under standard cross-entropy, and only the non-target tail is regularized through a KL term to match the Zipf-shaped distribution (Liang et al., 2022). LABO formulates label smoothing as a bi-level optimization over an instance-specific smoothing distribution and derives a closed-form optimum,

$\bar{y}^{ls} = (1-\alpha)\bar{y} + \alpha u$5

equivalently a temperature-softmax under $\bar{y}^{ls} = (1-\alpha)\bar{y} + \alpha u$6, thereby turning uniform smoothing into learned, instance-dependent label regularization (Lu et al., 2023).

Related structured variants operate at different granularities. Adaptive Regularization of Labels learns a residual correlation matrix whose rows define class-specific residual labels over wrong classes, updated online from the model’s own erroneous probabilities; the resulting ALR-S combination with conventional label smoothing is explicitly described as a stronger label-regularization scheme than static LS (Ding et al., 2019). Structural Label Smoothing instead computes cluster-specific smoothing strengths $\bar{y}^{ls} = (1-\alpha)\bar{y} + \alpha u$7 from estimated cluster-level Bayes error rates, with closed-form allocation under a fixed global average smoothing budget (Li et al., 2020).

4. Sample-dependent, uncertainty-dependent, and context-conditioned smoothing

A second major line of Label Smoothing++ makes smoothing depend on the input rather than only on the class. Adaptive Label Smoothing for image classification ties the smoothing factor to objectness derived from bounding boxes: $\bar{y}^{ls} = (1-\alpha)\bar{y} + \alpha u$8 When the object occupies most of the crop, $\bar{y}^{ls} = (1-\alpha)\bar{y} + \alpha u$9 and the target remains close to one-hot; when the object is tiny or absent, $1-\alpha$0 and the target approaches uniform. An optional interpolation coefficient $1-\alpha$1 mixes adaptive smoothing with hard labels. In ImageNet experiments, this design produced an order of magnitude reduction in confidence on context-only images and improved downstream MS COCO detection by $1-\alpha$2 mAP over hard-label pretraining (Krothapalli et al., 2020).

ALASCA extends the idea to noisy labels and internal representations. It interprets label smoothing as inducing a regularizer

$1-\alpha$3

and argues that this implicitly promotes Lipschitz regularization of the feature extractor. Its adaptive smoothing coefficient is sample-wise,

$1-\alpha$4

stabilized by an EMA of logits and applied mainly through auxiliary classifiers attached to intermediate layers. The result is a noise-aware, layer-wise adaptive smoothing scheme that improves robustness of feature extractors and systematically strengthens a range of learning-with-noisy-labels baselines (Ko et al., 2022).

A related but conceptually distinct route is variational learning. Under a variational objective

$1-\alpha$5

the expected gradient can be rewritten as standard training with example-specific label perturbations

$1-\alpha$6

and, in generalized linear models,

$1-\alpha$7

This induces larger smoothing near decision boundaries and under high posterior variance, without explicitly designing an adaptive LS rule (Yang et al., 11 Feb 2025). In domain adversarial training, an analogous idea is applied to environment labels rather than task labels: Environment Label Smoothing replaces one-hot domain labels with smoothed targets $1-\alpha$8, reducing discriminator over-confidence and improving stability, local convergence, and robustness to noisy environment labels (Zhang et al., 2023).

5. Geometric and information-theoretic interpretations

Several analyses explain Label Smoothing++ in terms of geometry, uncertainty, or representation compression rather than only target perturbation. Pairwise Label Smoothing, also called midpoint regularization, constructs midpoint samples

$1-\alpha$9

and learns a smoothing distribution for each midpoint through an auxiliary head. The resulting targets are deliberately high-uncertainty, and the method empirically produces very low winning softmax scores for both in- and out-of-distribution samples while improving classification performance relative to uniform LS and Mixup baselines (Guo, 2021).

A different theoretical lens shows that the label-augmentation component of both label smoothing and Mixup pushes models toward minimum-variance features. In binary linear settings with both low-variance and high-variance separating features, label smoothing and Mixup provably converge to solutions that use only the low-variance feature, whereas standard training with weight decay can learn higher-variance features. The same work shows that, more generally, the losses of label smoothing and Mixup are lower bounded by a function of model output variance, and empirically that these methods can therefore be more susceptible to low variance spurious correlations in the training data even while performing strongly on standard image benchmarks (Chidambaram et al., 2024).

Other papers frame the phenomenon more positively. One study interprets label smoothing as encouraging class-wise compression in the penultimate layer, with tight clusters around class templates; this improves calibration and beam-search behavior but erases inter-class resemblance information useful for distillation (Müller et al., 2019). Another recent analysis revisits label smoothing through the information bottleneck and argues that, under sufficient model flexibility and no conflicting labels for the same input, the model output obtained through label smoothing explores the optimal solution of the information bottleneck; in that sense, label smoothing becomes a practical approach to the information bottleneck and is empirically insensitive to nuisance factors that do not contain target information (Kudo, 12 Aug 2025).

6. Failure modes, corrections, and the current role of Label Smoothing++

The most prominent documented failure mode is selective classification. In large-scale image classification and semantic segmentation, label smoothing often improves top-1 accuracy at full coverage but consistently degrades selective classification: RC curves shift upward, AURC increases, and degradation is strongest in the low-risk regime. The explanation given is logit-level: relative to cross-entropy, label smoothing applies stronger downward pressure to the maximum logit when a prediction is likely correct and weaker pressure when it is likely wrong, thereby harming the uncertainty ranking needed by maximum-softmax-probability rejection. A simple post-hoc correction, logit normalization,

$1-\alpha+\alpha/K$0

recovers most of the lost selective-classification performance on LS-trained models without retraining; this train-time LS plus test-time logit normalization is explicitly characterized as a practical “Label Smoothing++” (Xia et al., 2024).

A second correction targets the probabilistic interpretation of label smoothing itself. One line of work argues that standard label smoothing conflates a robustified loss with a modified estimator. In logistic regression, it introduces a modified LSLR and proposes MLSLR as an improvement: MLSLR uses the same smoothed-KL loss as LSLR and the same consistent estimator as ordinary logistic regression, while not squeezing the logits. The theoretical conclusion is that larger smoothing levels reduce efficiency under correctly specified models but increase robustness under model misspecification; experimentally, the estimator modification and logit squeezing in standard LSLR have negative effects on both probability estimation and classification performance, motivating MLSLR as a more principled variant (Yamasaki et al., 2023).

Taken together, these results position Label Smoothing++ as a mature but heterogeneous research area rather than a single settled method. Its defining move is to replace uniform, class-agnostic, globally fixed smoothing by mechanisms that learn inter-class structure, adapt to instance difficulty or uncertainty, exploit auxiliary geometry, or correct downstream pathologies introduced by LS. The unifying theme is not the abandonment of label smoothing, but its reformulation into a structured, adaptive, and task-aware form of label regularization.