---
title: Adaptive Label Smoothing
url: https://www.emergentmind.com/topics/adaptive-label-smoothing
type: topic
---

# Adaptive Label Smoothing

Adaptive label smoothing is a family of training procedures that generalizes standard label smoothing by making the softening of target labels depend on the sample, the label structure, the model state, or the training regime. In standard label smoothing, a one-hot target is replaced by a convex mixture of the hard label and a fixed prior, typically uniform; adaptive variants instead let the smoothing strength, the smoothing distribution, or both depend on model confidence, neighboring labels, graph structure, object extent, posterior uncertainty, or schedule design. The resulting methods are used to regulate overconfidence, improve calibration, stabilize learning under label noise, preserve useful structure among alternative classes, and, in some formulations, avoid the fixed confidence ceiling imposed by ordinary label smoothing [2003.02819, 2210.13459, 2410.06134].

## 1. From fixed label smoothing to adaptive targets

Standard label smoothing replaces a one-hot label with a softened target of the form
\[
q'(y\mid x_i) = (1-\alpha)\,q(y\mid x_i)+\alpha\,u(y\mid x_i),
\]
where \(u(y\mid x_i)=1/K\) in the uniform case and \(\alpha\in[0,1]\) is a global hyperparameter. In the label-noise literature, the same mechanism can be written as a linear transformation of the loss vector,
\[
\ell^{\mathrm{SM}}(\mathbf{f}) = M\,\ell(\mathbf{f}), \qquad
M=(1-\alpha)I+\frac{\alpha}{L}J,
\]
which makes explicit that a uniform component is added to every training label [2012.01559, 2003.02819].

The fixed formulation is effective as a regularizer, but its assumptions are restrictive. It uses the same \(\alpha\) for all samples and all training stages; it typically smooths toward a uniform prior; and it considers one gold label at a time. Several later works identify these as the core limitations motivating adaptive variants: different inputs require different amounts of smoothing, class relationships are often non-uniform, and the model’s own confidence changes substantially over the course of training [2012.01559, 2210.13459].

A further complication is that the effect of smoothing is task-dependent. In noisy-label classification, standard label smoothing is competitive with loss-correction methods even though it is a biased objective rather than an unbiased denoiser [2003.02819]. In out-of-distribution detection, however, the fixed reduction of the true-class target tends to shrink the maximal probability and maximal logit, which can directly damage score-based OOD separation [2410.06134]. Adaptive label smoothing emerged largely to resolve such mismatches between the uniform, global nature of standard smoothing and the heterogeneity of actual training examples.

## 2. Principal axes of adaptation

Adaptive label smoothing methods differ along a small number of recurring design axes. Some methods adapt the scalar smoothing strength, some adapt the target distribution itself, and some alter when smoothing is applied rather than what the softened target is. In many cases, these mechanisms are combined.

| Adaptation axis | Representative mechanism | Representative papers |
|---|---|---|
| Strength adaptation | Per-instance \(\alpha\) from entropy, confidence, or posterior uncertainty | [2210.13459], [2206.07277], [2502.07273] |
| Distribution adaptation | Learned smoothing prior, self-knowledge prior, or context-dependent non-target distribution | [2012.01559], [2105.14556], [2210.13459] |
| Structural adaptation | Local \(k\)-NN labels, graph-propagated labels, object-size-aware labels, non-maximal-probability equalization | [2102.05140], [2108.13555], [2009.06432], [2410.06134] |
| Schedule or condition adaptation | Smooth pacing, alternating hard/soft supervision, margin-conditioned smoothing | [2108.13555], [2508.17846], [2308.11911] |

This taxonomy is not merely organizational. It reflects distinct hypotheses about what fixed label smoothing fails to model. Confidence-based methods assume that the model’s predictive entropy or posterior uncertainty is a good proxy for how much regularization is needed. Structure-aware methods assume that smoothing should respect geometry in input, feature, graph, or class space. Schedule-based methods assume that the timing of soft supervision is as important as its magnitude, especially when early soft labels are unreliable or when full-time smoothing conflicts with pretraining dynamics.

## 3. Instance- and context-dependent target construction

A direct form of adaptive label smoothing learns a data-dependent soft target rather than prescribing a fixed prior. Pairwise Label Smoothing (PLS) constructs a synthetic paired example from two samples,
\[
x_{ij}=\frac{x_i+x_j}{2}, \qquad q(y\mid x_{ij})=\frac{y_i+y_j}{2},
\]
then uses an additional head to learn a pair-dependent smoothing distribution \(u'_\theta(y\mid x_{ij})\), and trains on
\[
q'(y\mid x_{ij})=\frac{q(y\mid x_{ij})+u'_\theta(y\mid x_{ij})}{2}.
\]
The method therefore preserves the special role of two ground-truth labels while redistributing mass to the remaining classes in an input-dependent manner. The authors also train with alternating mini-batches of original inputs and averaged pair inputs, and report that removing the original images significantly hurts performance [2012.01559].

In sequence generation, the same principle appears in more explicitly contextual form. AdaLabel for dialogue generation replaces the one-hot target \(\mathbf{q}\) by
\[
\mathbf{q'} = \varepsilon \cdot \mathbf{q} + (1-\varepsilon)\cdot \mathbf{v},
\]
where \(\mathbf{v}\) is produced by a lightweight bi-directional auxiliary decoder with target-masked attention, and \(\varepsilon\) is computed from the current model confidence through \(p_{\max}\), \(\lambda\), and a refinement factor \(\alpha=\bigl[p(y_t\mid y_{<t},X)/p_{\max}\bigr]^2\). The target distribution is thus time-step specific and context-aware, rather than globally uniform [2105.14556]. A related natural-language-generation formulation makes the smoothing coefficient itself instance-specific,
\[
\alpha^{(n)} = 1-\frac{H(P_\theta(\cdot \mid^{(n)}))}{\log |C|},
\]
and replaces the uniform prior by self-knowledge from the past checkpoint that performs best on validation, yielding a dynamic mixture of hard labels and self-distilled soft labels [2210.13459].

Local label geometry provides another route to adaptation. Locally adaptive label smoothing for prediction churn replaces uniform smoothing by a mixture of the one-hot label, the uniform prior, and a deep \(k\)-nearest-neighbor label estimate,
\[
y^{\text{kNN}}_{a,b} = (1-a)\cdot y + a\cdot\left(b\cdot \frac{1}{L}\mathbf{1}_L + (1-b)\cdot \eta_k(x)\right).
\]
This uses a preliminary network to build neighborhoods in logit space, then retrains on the resulting locally softened targets [2102.05140]. In noisy-label learning, ALASCA makes the smoothing strength confidence-dependent through
\[
\tilde{\alpha}(\mathbf{x}_i)=1-\mathcal{S}\!\left(\frac{\mathbf{t}_i}{\tau}\right)_y,
\]
where \(\mathbf{t}_i\) is an EMA of model outputs; the resulting adaptive smoothing is applied through auxiliary classifiers to regularize intermediate representations [2206.07277]. A more Bayesian variant derives example-specific smoothing directly from variational learning: for logistic regression the induced label noise is
\[
\epsilon_{it}=\sigma(f_i(\theta_t))-\mathbb{E}_{q_t}[\sigma(f_i(\theta))],
\]
which becomes larger near ambiguous decision regions and for examples with larger posterior-induced logit variance [2502.07273].

## 4. Structure-aware and task-specific formulations

Graph learning has produced a particularly explicit adaptive label smoothing pipeline. For large-scale mini-batch GNN training, ALS first propagates observed labels,
\[
\mathbf{Y}^{(k+1)}=(1-\beta)\mathbf{D}^{-1}\mathbf{A}\mathbf{Y}^{(k)}+\beta \mathbf{Y}^{(0)},
\]
then refines the propagated label through a learnable class-relevance matrix,
\[
\mathbf{y}_i^{\mathrm{soft}}=\mathrm{Softmax}(\mathbf{W}\mathbf{y}_i^{(K)}),
\]
and finally forms
\[
\mathbf{y}_i^{ALS}=(1-\alpha)\mathbf{y}_i+\alpha \mathbf{y}_i^{\mathrm{soft}}.
\]
A pacing schedule such as \(\alpha_t=\min(rt,\alpha_{\max})\) or \(\alpha_t=\min(be^{rt},\alpha_{\max})\) delays strong smoothing until the propagated labels become more informative [2108.13555].

In visual classification, one of the earliest explicit adaptive label smoothing schemes ties \(\alpha\) to object extent. Using bounding-box annotations and transformed object masks, the method sets
\[
\alpha = 1-\frac{wh}{WH},
\]
where \(w,h\) are object width and height and \(W,H\) are image width and height. The softened label is then blended with the hard-label loss through a parameter \(\beta\). This grounds confidence in the visible object proportion rather than background context learned from random crops [2009.06432].

Some task-specific variants modify not only the target distribution but the label type being smoothed. Environment Label Smoothing (ELS) applies smoothing to domain labels in domain adversarial training rather than class labels, replacing one-hot environment supervision by a \(\gamma\)-weighted soft target over domains. Its purpose is not ordinary calibration of class probabilities but stabilization of the adversarial discriminator under noisy or ambiguous environment assignments [2302.00194]. For OOD detection, a different ALS formulation keeps the true-class confidence unconstrained and instead penalizes the dispersion of non-maximal probabilities:
\[
\mathcal{L}_{ALS}=H(\mathbf{y},\mathbf{p})+\lambda \cdot \mathcal{L}_{NMPC}, \qquad
\mathcal{L}_{NMPC}=\sqrt{\frac{1}{N-1}\sum_{i\neq k}(p_i-\bar p)^2}.
\]
This enforces equality among non-maximal probabilities while avoiding the fixed ceiling on maximal confidence created by ordinary label smoothing [2410.06134].

Prompt tuning for vision-language models motivates yet another variant. ATLaS alternates between one-hot and softened supervision rather than using soft labels at every step:
\[
\mathbf{y}_{t}^{\mathrm{ATLaS}}=\xi \mathbf{y}_t + (1-\xi)\mathbf{y}_{t}^{\mathrm{LS}},
\]
where every \(K\)-th epoch uses soft labels and the other \(K-1\) epochs use hard labels. The soft labels can be vanilla LS, class-wise soft labels (CSL) derived from CLIP text similarities, or instance-wise soft labels (ISL) derived from CLIP image-conditioned predictions [2508.17846].

## 5. Theoretical interpretations

A recurring theoretical theme is that adaptive label smoothing is not just target corruption. In the label-noise setting, standard label smoothing resembles symmetric label noise and can be expressed in the same label-smearing formalism as backward correction, but the two differ in sign on the average-loss term:
\[
\ell^{\mathrm{LS}}(y,\mathbf{f}) \propto \ell(y,\mathbf{f}) + \frac{\alpha}{(1-\alpha)L}\sum_{y'}\ell(y',\mathbf{f}),
\]
whereas backward correction subtracts an average-loss term. This makes smoothing a biased objective that trades exact denoising for regularization rather than an unbiased loss-correction method [2003.02819].

A second viewpoint interprets smoothing as shrinkage or implicit Lipschitz regularization. For softmax cross-entropy, label smoothing can be written as the original risk plus
\[
\beta\,\Omega(\mathbf{f}), \qquad
\Omega(\mathbf{f})=\mathbb{E}_x\!\left[L\log\!\left(\sum_{y'}e^{f_{y'}(x)}\right)-\sum_{y'}f_{y'}(x)\right],
\]
and for linear classifiers the regularizer is minimized at zero weights [2003.02819]. ALASCA strengthens this interpretation: under a surrogate decomposition \(\mathbf{f}=\mathbf{g}\circ\mathbf{h}\), the paper proves that \(\mathbf{h}=0\) is the unique minimizer of the induced regularizer under an affine-basis assumption on classifier weights, and under a Lipschitz-gradient assumption the Jacobian norm of \(\mathbf{f}\) shrinks as training points densify [2206.07277].

A third viewpoint comes from robust statistics and model misspecification. In logistic regression, the paper “Label Smoothing is Robustification against Model Misspecification” separates the loss-function modification from the probability-estimator modification. Its modified LSLR keeps the smoothed loss but uses the standard softmax estimator, and the analysis shows a clear trade-off: increasing \(\alpha\) reduces asymptotic efficiency under correct specification but increases robustness under contamination or misspecification. The paper does not propose a concrete adaptive algorithm, but it gives a principled rationale for adapting \(\alpha\) to the severity of misspecification or noise [2305.08501].

A fourth line of theory makes adaptivity emerge from Bayesian optimization itself. Variational learning over a parameter distribution \(q(\theta)\) yields an expected gradient that is equivalent to ordinary gradient descent with an example-specific noisy label. In generalized linear models,
\[
\epsilon_{it}=A'(f_i(\theta_t))-\mathbb{E}_{q_t}[A'(f_i(\theta))],
\]
and in multiclass classification \(A'(f)\) is the softmax. This connects posterior uncertainty directly to adaptive label smoothing, with stronger smoothing near the decision boundary and for examples with larger logit variance under the posterior [2502.07273].

Calibration theory supplies a fifth interpretation. ACLS shows that several regularization-based calibration losses can be rewritten as adaptive or conditional label smoothing in gradient form, then proposes a margin-conditioned piecewise-linear smoothing function. Its indicator function activates only when a logit margin exceeds a threshold, so smoothing is both adaptive to the output geometry and conditional on apparent miscalibration [2308.06549].

## 6. Empirical behavior, limitations, and controversies

Across domains, adaptive label smoothing often improves both predictive performance and confidence behavior. PLS reports up to **30.06%** relative classification error reduction on CIFAR-10 and substantial gains across MNIST, Fashion, SVHN, CIFAR-10, and CIFAR-100, while producing much lower winning softmax scores than baseline or uniform label smoothing [2012.01559]. Graph ALS improves test accuracy over both plain training and standard label smoothing in **18 out of 20** backbone–dataset settings and adds only \(\mathcal{O}(|\mathcal{B}|C^2)\) time and \(\mathcal{O}(C^2)\) memory [2108.13555]. Adaptive label smoothing with self-knowledge improves machine translation quality and calibration, including **+3.75 BLEU** on Multi30K DE\(\to\)EN and a reduction of ECE on IWSLT14 DE\(\to\)EN from **12.98** to **1.76** [2210.13459]. Object-size-based adaptive smoothing yields an order-of-magnitude reduction in confidence on context-only ImageNet images and gains **2.1mAP** on MS COCO transfer learning relative to hard labels [2009.06432]. For OOD detection, the probability-variance ALS improves average max-probability OSCR from **89.65** for the baseline to **90.12**, while standard label smoothing drops it to **85.47** [2410.06134].

The same literature also shows that smoothing is not uniformly beneficial, and several of the sharpest findings are explicitly cautionary. Ordinary label smoothing is competitive with loss-correction under label noise and beneficial when applied to noisy teachers in distillation, but it is not a principled unbiased replacement for correction, and excessive smoothing can worsen calibration [2003.02819]. In NLP robustness, both standard LS and adversarial label smoothing reduce attack success rate and adversarial confidence, but automatic selection of the optimal \(\alpha\) remains future work [2212.10258]. In privacy, positive label smoothing can increase leakage to model inversion attacks, while negative smoothing factors impede extraction and outperform specialized defenses in the reported experiments [2310.06549]. In augmentation-heavy vision pipelines, adaptive label smoothing helps homogeneous strong augmentations such as Random Erasing, yet its benefits largely vanish under TrivialAugment and excessive smoothing harms corruption robustness [2507.16427]. Prompt tuning provides a related negative result: vanilla label smoothing can weaken prompt generalization, which motivated alternating hard/soft supervision in ATLaS [2508.17846].

This suggests that adaptive label smoothing is best understood not as a single recipe but as a design space. The decisive choices are which quantity is adapted, what structure defines the soft target, and whether the downstream objective values reduced confidence, preserved confidence scale, stability under noise, or robustness to adversaries and privacy attacks. The literature consistently supports the underlying premise that one-size-fits-all smoothing is too coarse, but it equally consistently shows that the “right” adaptation mechanism is highly task-dependent.

Source: https://www.emergentmind.com/topics/adaptive-label-smoothing