Adaptive Label Smoothing

• Adaptive Label Smoothing is a training method that refines standard label smoothing by adjusting the softening process based on sample features, model state, or label structure.
• It improves model calibration and reduces overconfidence while stabilizing learning under label noise, with reported error reductions up to 30% on datasets like CIFAR-10.
• Various adaptation axes—such as strength, distribution, schedule, and structure—are combined to tailor smoothing to task-specific requirements and data heterogeneity.

Adaptive label smoothing is a family of training procedures that generalizes standard label smoothing by making the softening of target labels depend on the sample, the label structure, the model state, or the training regime. In standard label smoothing, a one-hot target is replaced by a convex mixture of the hard label and a fixed prior, typically uniform; adaptive variants instead let the smoothing strength, the smoothing distribution, or both depend on model confidence, neighboring labels, graph structure, object extent, posterior uncertainty, or schedule design. The resulting methods are used to regulate overconfidence, improve calibration, stabilize learning under label noise, preserve useful structure among alternative classes, and, in some formulations, avoid the fixed confidence ceiling imposed by ordinary label smoothing (Lukasik et al., 2020, Lee et al., 2022, Xu et al., 2024).

1. From fixed label smoothing to adaptive targets

Standard label smoothing replaces a one-hot label with a softened target of the form

$$q'(y\mid x_i) = (1-\alpha)\,q(y\mid x_i)+\alpha\,u(y\mid x_i),$$

where $u(y\mid x_i)=1/K$ in the uniform case and $\alpha\in[0,1]$ is a global hyperparameter. In the label-noise literature, the same mechanism can be written as a linear transformation of the loss vector,

$$\ell^{\mathrm{SM}}(\mathbf{f}) = M\,\ell(\mathbf{f}), \qquad M=(1-\alpha)I+\frac{\alpha}{L}J,$$

which makes explicit that a uniform component is added to every training label (Guo, 2020, Lukasik et al., 2020).

The fixed formulation is effective as a regularizer, but its assumptions are restrictive. It uses the same $\alpha$ for all samples and all training stages; it typically smooths toward a uniform prior; and it considers one gold label at a time. Several later works identify these as the core limitations motivating adaptive variants: different inputs require different amounts of smoothing, class relationships are often non-uniform, and the model’s own confidence changes substantially over the course of training (Guo, 2020, Lee et al., 2022).

A further complication is that the effect of smoothing is task-dependent. In noisy-label classification, standard label smoothing is competitive with loss-correction methods even though it is a biased objective rather than an unbiased denoiser (Lukasik et al., 2020). In out-of-distribution detection, however, the fixed reduction of the true-class target tends to shrink the maximal probability and maximal logit, which can directly damage score-based OOD separation (Xu et al., 2024). Adaptive label smoothing emerged largely to resolve such mismatches between the uniform, global nature of standard smoothing and the heterogeneity of actual training examples.

2. Principal axes of adaptation

Adaptive label smoothing methods differ along a small number of recurring design axes. Some methods adapt the scalar smoothing strength, some adapt the target distribution itself, and some alter when smoothing is applied rather than what the softened target is. In many cases, these mechanisms are combined.

Adaptation axis	Representative mechanism	Representative papers
Strength adaptation	Per-instance $\alpha$ from entropy, confidence, or posterior uncertainty	(Lee et al., 2022, Ko et al., 2022, Yang et al., 11 Feb 2025)
Distribution adaptation	Learned smoothing prior, self-knowledge prior, or context-dependent non-target distribution	(Guo, 2020, Wang et al., 2021, Lee et al., 2022)
Structural adaptation	Local $k$-NN labels, graph-propagated labels, object-size-aware labels, non-maximal-probability equalization	(Bahri et al., 2021, Zhou et al., 2021, Krothapalli et al., 2020, Xu et al., 2024)
Schedule or condition adaptation	Smooth pacing, alternating hard/soft supervision, margin-conditioned smoothing	(Zhou et al., 2021, Chen et al., 25 Aug 2025, Park et al., 2023)

This taxonomy is not merely organizational. It reflects distinct hypotheses about what fixed label smoothing fails to model. Confidence-based methods assume that the model’s predictive entropy or posterior uncertainty is a good proxy for how much regularization is needed. Structure-aware methods assume that smoothing should respect geometry in input, feature, graph, or class space. Schedule-based methods assume that the timing of soft supervision is as important as its magnitude, especially when early soft labels are unreliable or when full-time smoothing conflicts with pretraining dynamics.

3. Instance- and context-dependent target construction

A direct form of adaptive label smoothing learns a data-dependent soft target rather than prescribing a fixed prior. Pairwise Label Smoothing (PLS) constructs a synthetic paired example from two samples,

$$x_{ij}=\frac{x_i+x_j}{2}, \qquad q(y\mid x_{ij})=\frac{y_i+y_j}{2},$$

then uses an additional head to learn a pair-dependent smoothing distribution $u'_\theta(y\mid x_{ij})$, and trains on

$$q'(y\mid x_{ij})=\frac{q(y\mid x_{ij})+u'_\theta(y\mid x_{ij})}{2}.$$

The method therefore preserves the special role of two ground-truth labels while redistributing mass to the remaining classes in an input-dependent manner. The authors also train with alternating mini-batches of original inputs and averaged pair inputs, and report that removing the original images significantly hurts performance (Guo, 2020).

In sequence generation, the same principle appears in more explicitly contextual form. AdaLabel for dialogue generation replaces the one-hot target $u(y\mid x_i)=1/K$0 by

$u(y\mid x_i)=1/K$1

where $u(y\mid x_i)=1/K$2 is produced by a lightweight bi-directional auxiliary decoder with target-masked attention, and $u(y\mid x_i)=1/K$3 is computed from the current model confidence through $u(y\mid x_i)=1/K$4, $u(y\mid x_i)=1/K$5, and a refinement factor $u(y\mid x_i)=1/K$6. The target distribution is thus time-step specific and context-aware, rather than globally uniform (Wang et al., 2021). A related natural-language-generation formulation makes the smoothing coefficient itself instance-specific,

$u(y\mid x_i)=1/K$7

and replaces the uniform prior by self-knowledge from the past checkpoint that performs best on validation, yielding a dynamic mixture of hard labels and self-distilled soft labels (Lee et al., 2022).

Local label geometry provides another route to adaptation. Locally adaptive label smoothing for prediction churn replaces uniform smoothing by a mixture of the one-hot label, the uniform prior, and a deep $u(y\mid x_i)=1/K$8-nearest-neighbor label estimate,

$u(y\mid x_i)=1/K$9

This uses a preliminary network to build neighborhoods in logit space, then retrains on the resulting locally softened targets (Bahri et al., 2021). In noisy-label learning, ALASCA makes the smoothing strength confidence-dependent through

$\alpha\in[0,1]$0

where $\alpha\in[0,1]$1 is an EMA of model outputs; the resulting adaptive smoothing is applied through auxiliary classifiers to regularize intermediate representations (Ko et al., 2022). A more Bayesian variant derives example-specific smoothing directly from variational learning: for logistic regression the induced label noise is

$\alpha\in[0,1]$2

which becomes larger near ambiguous decision regions and for examples with larger posterior-induced logit variance (Yang et al., 11 Feb 2025).

4. Structure-aware and task-specific formulations

Graph learning has produced a particularly explicit adaptive label smoothing pipeline. For large-scale mini-batch GNN training, ALS first propagates observed labels,

$\alpha\in[0,1]$3

then refines the propagated label through a learnable class-relevance matrix,

$\alpha\in[0,1]$4

and finally forms

$\alpha\in[0,1]$5

A pacing schedule such as $\alpha\in[0,1]$6 or $\alpha\in[0,1]$7 delays strong smoothing until the propagated labels become more informative (Zhou et al., 2021).

In visual classification, one of the earliest explicit adaptive label smoothing schemes ties $\alpha\in[0,1]$8 to object extent. Using bounding-box annotations and transformed object masks, the method sets

$\alpha\in[0,1]$9

where $$\ell^{\mathrm{SM}}(\mathbf{f}) = M\,\ell(\mathbf{f}), \qquad M=(1-\alpha)I+\frac{\alpha}{L}J,$$0 are object width and height and $$\ell^{\mathrm{SM}}(\mathbf{f}) = M\,\ell(\mathbf{f}), \qquad M=(1-\alpha)I+\frac{\alpha}{L}J,$$1 are image width and height. The softened label is then blended with the hard-label loss through a parameter $$\ell^{\mathrm{SM}}(\mathbf{f}) = M\,\ell(\mathbf{f}), \qquad M=(1-\alpha)I+\frac{\alpha}{L}J,$$2. This grounds confidence in the visible object proportion rather than background context learned from random crops (Krothapalli et al., 2020).

Some task-specific variants modify not only the target distribution but the label type being smoothed. Environment Label Smoothing (ELS) applies smoothing to domain labels in domain adversarial training rather than class labels, replacing one-hot environment supervision by a $$\ell^{\mathrm{SM}}(\mathbf{f}) = M\,\ell(\mathbf{f}), \qquad M=(1-\alpha)I+\frac{\alpha}{L}J,$$3-weighted soft target over domains. Its purpose is not ordinary calibration of class probabilities but stabilization of the adversarial discriminator under noisy or ambiguous environment assignments (Zhang et al., 2023). For OOD detection, a different ALS formulation keeps the true-class confidence unconstrained and instead penalizes the dispersion of non-maximal probabilities: $$\ell^{\mathrm{SM}}(\mathbf{f}) = M\,\ell(\mathbf{f}), \qquad M=(1-\alpha)I+\frac{\alpha}{L}J,$$4 This enforces equality among non-maximal probabilities while avoiding the fixed ceiling on maximal confidence created by ordinary label smoothing (Xu et al., 2024).

Prompt tuning for vision-LLMs motivates yet another variant. ATLaS alternates between one-hot and softened supervision rather than using soft labels at every step: $$\ell^{\mathrm{SM}}(\mathbf{f}) = M\,\ell(\mathbf{f}), \qquad M=(1-\alpha)I+\frac{\alpha}{L}J,$$5 where every $$\ell^{\mathrm{SM}}(\mathbf{f}) = M\,\ell(\mathbf{f}), \qquad M=(1-\alpha)I+\frac{\alpha}{L}J,$$6-th epoch uses soft labels and the other $$\ell^{\mathrm{SM}}(\mathbf{f}) = M\,\ell(\mathbf{f}), \qquad M=(1-\alpha)I+\frac{\alpha}{L}J,$$7 epochs use hard labels. The soft labels can be vanilla LS, class-wise soft labels (CSL) derived from CLIP text similarities, or instance-wise soft labels (ISL) derived from CLIP image-conditioned predictions (Chen et al., 25 Aug 2025).

5. Theoretical interpretations

A recurring theoretical theme is that adaptive label smoothing is not just target corruption. In the label-noise setting, standard label smoothing resembles symmetric label noise and can be expressed in the same label-smearing formalism as backward correction, but the two differ in sign on the average-loss term: $$\ell^{\mathrm{SM}}(\mathbf{f}) = M\,\ell(\mathbf{f}), \qquad M=(1-\alpha)I+\frac{\alpha}{L}J,$$8 whereas backward correction subtracts an average-loss term. This makes smoothing a biased objective that trades exact denoising for regularization rather than an unbiased loss-correction method (Lukasik et al., 2020).

A second viewpoint interprets smoothing as shrinkage or implicit Lipschitz regularization. For softmax cross-entropy, label smoothing can be written as the original risk plus

$$\ell^{\mathrm{SM}}(\mathbf{f}) = M\,\ell(\mathbf{f}), \qquad M=(1-\alpha)I+\frac{\alpha}{L}J,$$9

and for linear classifiers the regularizer is minimized at zero weights (Lukasik et al., 2020). ALASCA strengthens this interpretation: under a surrogate decomposition $\alpha$0, the paper proves that $\alpha$1 is the unique minimizer of the induced regularizer under an affine-basis assumption on classifier weights, and under a Lipschitz-gradient assumption the Jacobian norm of $\alpha$2 shrinks as training points densify (Ko et al., 2022).

A third viewpoint comes from robust statistics and model misspecification. In logistic regression, the paper “Label Smoothing is Robustification against Model Misspecification” separates the loss-function modification from the probability-estimator modification. Its modified LSLR keeps the smoothed loss but uses the standard softmax estimator, and the analysis shows a clear trade-off: increasing $\alpha$3 reduces asymptotic efficiency under correct specification but increases robustness under contamination or misspecification. The paper does not propose a concrete adaptive algorithm, but it gives a principled rationale for adapting $\alpha$4 to the severity of misspecification or noise (Yamasaki et al., 2023).

A fourth line of theory makes adaptivity emerge from Bayesian optimization itself. Variational learning over a parameter distribution $\alpha$5 yields an expected gradient that is equivalent to ordinary gradient descent with an example-specific noisy label. In generalized linear models,

$\alpha$6

and in multiclass classification $\alpha$7 is the softmax. This connects posterior uncertainty directly to adaptive label smoothing, with stronger smoothing near the decision boundary and for examples with larger logit variance under the posterior (Yang et al., 11 Feb 2025).

Calibration theory supplies a fifth interpretation. ACLS shows that several regularization-based calibration losses can be rewritten as adaptive or conditional label smoothing in gradient form, then proposes a margin-conditioned piecewise-linear smoothing function. Its indicator function activates only when a logit margin exceeds a threshold, so smoothing is both adaptive to the output geometry and conditional on apparent miscalibration (Islam et al., 2023).

6. Empirical behavior, limitations, and controversies

Across domains, adaptive label smoothing often improves both predictive performance and confidence behavior. PLS reports up to 30.06% relative classification error reduction on CIFAR-10 and substantial gains across MNIST, Fashion, SVHN, CIFAR-10, and CIFAR-100, while producing much lower winning softmax scores than baseline or uniform label smoothing (Guo, 2020). Graph ALS improves test accuracy over both plain training and standard label smoothing in 18 out of 20 backbone–dataset settings and adds only $\alpha$8 time and $\alpha$9 memory (Zhou et al., 2021). Adaptive label smoothing with self-knowledge improves machine translation quality and calibration, including +3.75 BLEU on Multi30K DE$\alpha$0EN and a reduction of ECE on IWSLT14 DE$\alpha$1EN from 12.98 to 1.76 (Lee et al., 2022). Object-size-based adaptive smoothing yields an order-of-magnitude reduction in confidence on context-only ImageNet images and gains 2.1mAP on MS COCO transfer learning relative to hard labels (Krothapalli et al., 2020). For OOD detection, the probability-variance ALS improves average max-probability OSCR from 89.65 for the baseline to 90.12, while standard label smoothing drops it to 85.47 (Xu et al., 2024).

The same literature also shows that smoothing is not uniformly beneficial, and several of the sharpest findings are explicitly cautionary. Ordinary label smoothing is competitive with loss-correction under label noise and beneficial when applied to noisy teachers in distillation, but it is not a principled unbiased replacement for correction, and excessive smoothing can worsen calibration (Lukasik et al., 2020). In NLP robustness, both standard LS and adversarial label smoothing reduce attack success rate and adversarial confidence, but automatic selection of the optimal $\alpha$2 remains future work (Yang et al., 2022). In privacy, positive label smoothing can increase leakage to model inversion attacks, while negative smoothing factors impede extraction and outperform specialized defenses in the reported experiments (Struppek et al., 2023). In augmentation-heavy vision pipelines, adaptive label smoothing helps homogeneous strong augmentations such as Random Erasing, yet its benefits largely vanish under TrivialAugment and excessive smoothing harms corruption robustness (Siedel et al., 22 Jul 2025). Prompt tuning provides a related negative result: vanilla label smoothing can weaken prompt generalization, which motivated alternating hard/soft supervision in ATLaS (Chen et al., 25 Aug 2025).

This suggests that adaptive label smoothing is best understood not as a single recipe but as a design space. The decisive choices are which quantity is adapted, what structure defines the soft target, and whether the downstream objective values reduced confidence, preserved confidence scale, stability under noise, or robustness to adversaries and privacy attacks. The literature consistently supports the underlying premise that one-size-fits-all smoothing is too coarse, but it equally consistently shows that the “right” adaptation mechanism is highly task-dependent.