Label Smoothing (ε=0.06)

Updated 2 February 2026

Label smoothing with ε=0.06 is a regularization technique that replaces one-hot labels with a weighted mix of true and uniform distributions to reduce overconfidence.
It applies a convex combination using ε=0.06, thereby boosting the true class loss gradient while uniformly diminishing others to promote better calibration and faster convergence.
Empirical evidence shows that ε=0.06 not only accelerates convergence but also improves model calibration, generalization, and robustness against label noise and adversarial conditions.

Label smoothing with $\epsilon = 0.06$ refers to a regularization technique in which ground-truth one-hot labels for a classification task are replaced by a convex combination of the original one-hot vector and a uniform distribution over classes, with the mixing strength parameterized by $\epsilon = 0.06$ . This modification systematically reduces the confidence (entropy) of model predictions, penalizing overconfidence and leading to well-calibrated probabilities and improved generalization across tasks and domains.

1. Mathematical Formulation and Loss Construction

Uniform label smoothing, as introduced by Szegedy et al., replaces the one-hot target $y \in \{0,1\}^K$ for class $y$ with a "soft" target $\tilde y$ defined by

$\tilde y_j = (1-\epsilon)\cdot 1_{j=y} + \frac{\epsilon}{K},$

where $K$ is the number of classes and $j \in \{1, \ldots, K\}$ . For $\epsilon = 0.06$ :

The target class receives $1-\epsilon + \epsilon/K$ (e.g., $\epsilon = 0.06$ 0 out of $\epsilon = 0.06$ 1 for $\epsilon = 0.06$ 2).
Each non-target class receives $\epsilon = 0.06$ 3 (e.g., $\epsilon = 0.06$ 4 for $\epsilon = 0.06$ 5).

This smoothed label is used in the cross-entropy loss:

$\epsilon = 0.06$ 6

where $\epsilon = 0.06$ 7 denotes the predicted probability for class $\epsilon = 0.06$ 8. For binary or $\epsilon = 0.06$ 9-class logistic regression, smoothed targets take analogous forms, with the loss function modified accordingly (Liu et al., 2020, Chen et al., 2020).

The modification imposes a regularization gradient, inflating the true class loss gradient by $y \in \{0,1\}^K$ 0 and deflating others by $y \in \{0,1\}^K$ 1, effectively applying a uniform regularization push toward higher-entropy (less confident) outputs (Xia et al., 2024).

2. Theoretical Underpinnings: Calibration, Generalization, and Information Bottleneck

Label smoothing at $y \in \{0,1\}^K$ 2 acts as a bias-variance regularizer by shifting probability mass from the true class to other classes, controlling generalization loss in the presence of label noise (Chen et al., 2020), and capping the maximum attainable confidence for any class. In the information-theoretical paradigm, uniform label smoothing is equivalent to a variational information bottleneck with bias parameter $y \in \{0,1\}^K$ 3, efficiently trading off between compression and sufficiency of the learned representation (Kudo, 12 Aug 2025).

Uniform label smoothing can be viewed as optimizing

$y \in \{0,1\}^K$ 4

where $y \in \{0,1\}^K$ 5 is the uniform prior, $y \in \{0,1\}^K$ 6 is cross-entropy, and $y \in \{0,1\}^K$ 7 is Kullback–Leibler divergence (Meister et al., 2020). This yields an output distribution strictly bounded away from zero, prohibiting exact sparsity in predicted class probabilities.

Moreover, in the generalized framework, label smoothing is one endpoint in a family of entropy-promoting regularizers (α–Jensen divergences)—the $y \in \{0,1\}^K$ 8 case—leading to full density in the output and forbidding zeros, which may be undesirable in some structured prediction or interpretable ML tasks (Meister et al., 2020).

3. Practical Effects: Calibration, Robustness, and Empirical Performance

Calibration and Confidence Control

With $y \in \{0,1\}^K$ 9, label smoothing significantly improves confidence calibration and reduces Expected Calibration Error (ECE). On CIFAR-100 (ResNet-18), ECE with uniform label smoothing (ε=0.06) drops from baseline values (e.g., $y$ 0 to $y$ 1 in hist-based prediction ECE), with further gains from similarity-aware smoothing schemes (Liu et al., 2020). These findings reflect that smoothing eliminates over-confident errors and yields more diagonal reliability diagrams, a property empirically confirmed in vision, NLP, and code summarization models (Haque et al., 2023).

Generalization and Robustness

Label smoothing yields improved generalization, especially under label noise or adversarial scenarios. In noisy-label benchmarks, static ε=0.06 label smoothing increases CIFAR-10 test accuracy from $y$ 2 to $y$ 3 at $y$ 4 symmetric noise; more aggressive smoothing induces underfitting (e.g., $y$ 5 reduces accuracy) (Ko et al., 2022). In text and code generation, ε=0.06–0.1 consistently produces higher BLEU, ROUGE-L, or classification accuracy, with $y$ 6 typically providing an optimal trade-off between under- and over-smoothing (Gao et al., 2023, Haque et al., 2023).

In adversarial NLP robustness, higher smoothing (e.g., $y$ 7) further lowers attack success rates and adversarial confidence, but the effect is monotonic and plateaus above $y$ 8—no evidence suggests $y$ 9 is sub-optimal, though moderate values between $\tilde y$ 0– $\tilde y$ 1 are empirically robust (Yang et al., 2022).

Model Dynamics and Convergence

Label smoothing accelerates convergence: in sentiment classification, LS models reach peak validation accuracy $\tilde y$ 2 faster than one-hot baselines (Gao et al., 2023). Final-layer representations under LS are more separable, as t-SNE projections show tighter, less-overlapping clusters.

Risk-Coverage Trade-off and Selective Classification

A documented downside is that label smoothing at $\tilde y$ 3 degrades selective classification, i.e., the ability to use model confidence for error rejection. The mechanism involves stronger suppression of the max logit for correct predictions than incorrect ones, collapsing the margin required for accurate scoring of model uncertainty. Post-hoc logit normalization (e.g., $\tilde y$ 4-norm or mean subtraction at inference) can recover lost selective coverage without hurting top-1 accuracy (Xia et al., 2024).

4. Adaptive and Structured Smoothing Variants

Uniform $\tilde y$ 5 is rarely optimal across example types or feature-space regions. Several approaches introduce adaptivity:

Structural Label Smoothing (SLS): The smoothing parameter $\tilde y$ 6 adapts per-data cluster, increasing for regions of high Bayes error overlap and decreasing for reliably classified areas, mitigating bias introduced by uniform smoothing (Li et al., 2020).
Adaptive LS via Instance Uncertainty: Smoothing strength is set per-example as a function of model entropy, growing as the model becomes overconfident during training and suppressing over-confident predictions when needed (Lee et al., 2022).
Similarity-Based Smoothing: Replaces the uniform distribution with a class-similarity–weighted prior, further enhancing calibration by assigning more probability mass to semantically/feature-similar classes rather than distributing it evenly (Liu et al., 2020).

These variants empirically outperform fixed uniform smoothing in test error, calibration, and noisy-label robustness, especially in settings with complex, heterogeneous class boundaries.

5. Recommendations for Hyperparameter Selection and Implementation

Practical Guidelines

Hyperparameter sweep: Empirically, moderate $\tilde y$ 7 (including $\tilde y$ 8) maximizes accuracy and generalization across image, text, and code classification/generation tasks (Gao et al., 2023, Haque et al., 2023, Ko et al., 2022, Li et al., 2020).
Label noise: For label noise at rate $\tilde y$ 9, theory and experiment suggest setting $\tilde y_j = (1-\epsilon)\cdot 1_{j=y} + \frac{\epsilon}{K},$ 0 or slightly above, guided by estimated clean rate $\tilde y_j = (1-\epsilon)\cdot 1_{j=y} + \frac{\epsilon}{K},$ 1 via $\tilde y_j = (1-\epsilon)\cdot 1_{j=y} + \frac{\epsilon}{K},$ 2 (Chen et al., 2020).
Implementation: Most ML frameworks natively support label smoothing. For PyTorch: $K$ 6 or, for non-integrated cases, manually smooth labels using one-hot plus $\tilde y_j = (1-\epsilon)\cdot 1_{j=y} + \frac{\epsilon}{K},$ 3 formula (Kudo, 12 Aug 2025).

Trade-offs

Too small $\tilde y_j = (1-\epsilon)\cdot 1_{j=y} + \frac{\epsilon}{K},$ 4 ( $\tilde y_j = (1-\epsilon)\cdot 1_{j=y} + \frac{\epsilon}{K},$ 5): Marginal benefit over one-hot; latent overfitting.
Too large $\tilde y_j = (1-\epsilon)\cdot 1_{j=y} + \frac{\epsilon}{K},$ 6 ( $\tilde y_j = (1-\epsilon)\cdot 1_{j=y} + \frac{\epsilon}{K},$ 7): Under-represents the true class, risks underfitting, reduced accuracy and class discrimination.
Instance-adaptive $\tilde y_j = (1-\epsilon)\cdot 1_{j=y} + \frac{\epsilon}{K},$ 8 / structured smoothing: Generally yields superior sample efficiency and calibration, eliminates need for tight ε grid search (Lee et al., 2022, Lu et al., 2023, Li et al., 2020).

6. Extensions, Limitations, and Future Directions

Entropy Regularization Connections: Label smoothing is a special case ( $\tilde y_j = (1-\epsilon)\cdot 1_{j=y} + \frac{\epsilon}{K},$ 9) of a broader α-Jensen divergence entropy regularizer. Alternative entropic and sparsity-promoting regularizers (e.g., confidence penalty, generalized entropy) may be preferable when sparsity of output distribution is essential (Meister et al., 2020).
Calibration versus Uncertainty Ranking: Label smoothing can impair the fidelity of softmax confidence as a selective-rejection criterion. Simple post-hoc logit normalization fully restores the risk-coverage trade-off (Xia et al., 2024).
Model Misspecification and Robustness: The main benefit of label smoothing, especially at moderate $K$ 0 (e.g., $K$ 1), is robustification against label noise and model misspecification. Modifying only the loss function (not probability estimation—e.g., using MLSLR, not LSLR) further improves both calibration and generalization (Yamasaki et al., 2023).
Bi-level and Data-driven Regularization: Advanced approaches like LABO generate the entire smoothing distribution optimally per-instance, strictly outperforming any fixed ε baseline at negligible extra cost (Lu et al., 2023).
Information Bottleneck View: Label smoothing implements the pragmatic discrete information bottleneck, balancing output compression and sufficiency, and is IB-optimal for models with enough capacity and no label conflicts (Kudo, 12 Aug 2025).

7. Empirical Comparison Table for $K$ 2 (Image Classification)

Model/Method	Hist-P ECE (%)	Top-1 Acc (%)	Calibration Strategy
One-hot baseline	22.96	—	None
Uniform LS ( $K$ 3)	2.35	—	Uniform LS
Class-similarity (word2vec)	1.74	—	Informed LS

Uniform LS at $K$ 4 reduces ECE by an order of magnitude relative to the one-hot baseline, with structured/similarity-based smoothing providing an additional $K$ 5 advantage in calibration, especially valuable in safety-critical decision-making scenarios (Liu et al., 2020).