---
title: Label Smoothing (ε=0.06)
url: https://www.emergentmind.com/topics/label-smoothing-epsilon-0-06
type: topic
---

# Label Smoothing (ε=0.06)

Label smoothing with $\epsilon = 0.06$ refers to a regularization technique in which ground-truth one-hot labels for a classification task are replaced by a convex combination of the original one-hot vector and a uniform distribution over classes, with the mixing strength parameterized by $\epsilon = 0.06$. This modification systematically reduces the confidence (entropy) of model predictions, penalizing overconfidence and leading to well-calibrated probabilities and improved generalization across tasks and domains.

## 1. Mathematical Formulation and Loss Construction

Uniform label smoothing, as introduced by Szegedy et al., replaces the one-hot target $y \in \{0,1\}^K$ for class $y$ with a "soft" target $\tilde y$ defined by
$$
\tilde y_j = (1-\epsilon)\cdot 1_{j=y} + \frac{\epsilon}{K},
$$
where $K$ is the number of classes and $j \in \{1, \ldots, K\}$. For $\epsilon = 0.06$:
- The target class receives $1-\epsilon + \epsilon/K$ (e.g., $0.9406$ out of $1$ for $K=100$).
- Each non-target class receives $\epsilon/K$ (e.g., $0.0006$ for $K=100$).

This smoothed label is used in the cross-entropy loss:
$$
\mathcal{L}_{LS} = - \sum_{j=1}^K \tilde y_j\, \log p_j,
$$
where $p_j$ denotes the predicted probability for class $j$. For binary or $K$-class logistic regression, smoothed targets take analogous forms, with the loss function modified accordingly [2006.14028][2010.12648].

The modification imposes a regularization gradient, inflating the true class loss gradient by $+\epsilon$ and deflating others by $-\epsilon/(K-1)$, effectively applying a uniform regularization push toward higher-entropy (less confident) outputs [2403.14715].

## 2. Theoretical Underpinnings: Calibration, Generalization, and Information Bottleneck

Label smoothing at $\epsilon=0.06$ acts as a bias-variance regularizer by shifting probability mass from the true class to other classes, controlling generalization loss in the presence of label noise [2010.12648], and capping the maximum attainable confidence for any class. In the information-theoretical paradigm, uniform label smoothing is equivalent to a variational information bottleneck with bias parameter $\beta = \epsilon/(1-\epsilon)$, efficiently trading off between compression and sufficiency of the learned representation [2508.14077].

Uniform label smoothing can be viewed as optimizing
$$
\mathcal{L}_{LS}(\theta; \alpha) = (1-\alpha) H(p_{\text{data}}, q_\theta) + \alpha\, D_{KL}(u \| q_\theta) + \mathrm{const},
$$
where $u$ is the uniform prior, $H$ is cross-entropy, and $D_{KL}$ is Kullback–Leibler divergence [2005.00820]. This yields an output distribution strictly bounded away from zero, prohibiting exact sparsity in predicted class probabilities.

Moreover, in the generalized framework, label smoothing is one endpoint in a family of entropy-promoting regularizers (α–Jensen divergences)—the $\alpha\rightarrow1$ case—leading to full density in the output and forbidding zeros, which may be undesirable in some structured prediction or interpretable ML tasks [2005.00820].

## 3. Practical Effects: Calibration, Robustness, and Empirical Performance

### Calibration and Confidence Control

With $\epsilon=0.06$, label smoothing significantly improves confidence calibration and reduces Expected Calibration Error (ECE). On CIFAR-100 (ResNet-18), ECE with uniform label smoothing (ε=0.06) drops from baseline values (e.g., $22.96\%$ to $2.35\%$ in hist-based prediction ECE), with further gains from similarity-aware smoothing schemes [2006.14028]. These findings reflect that smoothing eliminates over-confident errors and yields more diagonal reliability diagrams, a property empirically confirmed in vision, NLP, and code summarization models [2303.16178].

### Generalization and Robustness

Label smoothing yields improved generalization, especially under label noise or adversarial scenarios. In noisy-label benchmarks, static ε=0.06 label smoothing increases CIFAR-10 test accuracy from $78.2\%$ to $84.0\%$ at $50\%$ symmetric noise; more aggressive smoothing induces underfitting (e.g., $\epsilon=0.15$ reduces accuracy) [2206.07277]. In text and code generation, ε=0.06–0.1 consistently produces higher BLEU, ROUGE-L, or classification accuracy, with $\epsilon=0.06$ typically providing an optimal trade-off between under- and over-smoothing [2312.06522][2303.16178].

In adversarial NLP robustness, higher smoothing (e.g., $\epsilon=0.10-0.45$) further lowers attack success rates and adversarial confidence, but the effect is monotonic and plateaus above $\epsilon \sim 0.45$—no evidence suggests $\epsilon=0.06$ is sub-optimal, though moderate values between $0.05$–$0.1$ are empirically robust [2212.10258].

### Model Dynamics and Convergence

Label smoothing accelerates convergence: in sentiment classification, LS models reach peak validation accuracy $2-3\times$ faster than one-hot baselines [2312.06522]. Final-layer representations under LS are more separable, as t-SNE projections show tighter, less-overlapping clusters.

### Risk-Coverage Trade-off and Selective Classification

A documented downside is that label smoothing at $\epsilon=0.06$ degrades selective classification, i.e., the ability to use model confidence for error rejection. The mechanism involves stronger suppression of the max logit for correct predictions than incorrect ones, collapsing the margin required for accurate scoring of model uncertainty. Post-hoc logit normalization (e.g., $p$-norm or mean subtraction at inference) can recover lost selective coverage without hurting top-1 accuracy [2403.14715].

## 4. Adaptive and Structured Smoothing Variants

Uniform $\epsilon$ is rarely optimal across example types or feature-space regions. Several approaches introduce adaptivity:
- **Structural Label Smoothing (SLS):** The smoothing parameter $\epsilon_c$ adapts per-data cluster, increasing for regions of high Bayes error overlap and decreasing for reliably classified areas, mitigating bias introduced by uniform smoothing [2001.01900].
- **Adaptive LS via Instance Uncertainty:** Smoothing strength is set per-example as a function of model entropy, growing as the model becomes overconfident during training and suppressing over-confident predictions when needed [2210.13459].
- **Similarity-Based Smoothing:** Replaces the uniform distribution with a class-similarity–weighted prior, further enhancing calibration by assigning more probability mass to semantically/feature-similar classes rather than distributing it evenly [2006.14028].

These variants empirically outperform fixed uniform smoothing in test error, calibration, and noisy-label robustness, especially in settings with complex, heterogeneous class boundaries.

## 5. Recommendations for Hyperparameter Selection and Implementation

### Practical Guidelines
- **Hyperparameter sweep:** Empirically, moderate $\epsilon \in [0.05, 0.1]$ (including $\epsilon=0.06$) maximizes accuracy and generalization across image, text, and code classification/generation tasks [2312.06522][2303.16178][2206.07277][2001.01900].
- **Label noise:** For label noise at rate $r$, theory and experiment suggest setting $\epsilon \approx r$ or slightly above, guided by estimated clean rate $a$ via $\epsilon=1-a$ [2010.12648].
- **Implementation:** Most ML frameworks natively support label smoothing. For PyTorch:
  ```python
  import torch
  criterion = torch.nn.CrossEntropyLoss(label_smoothing=0.06)
  loss = criterion(logits, y_true)
  ```
  or, for non-integrated cases, manually smooth labels using one-hot plus $\epsilon/K$ formula [2508.14077].

### Trade-offs
- **Too small $\epsilon$ ($<0.01$):** Marginal benefit over one-hot; latent overfitting.
- **Too large $\epsilon$ ($>0.1$):** Under-represents the true class, risks underfitting, reduced accuracy and class discrimination.
- **Instance-adaptive $\epsilon$ / structured smoothing:** Generally yields superior sample efficiency and calibration, eliminates need for tight ε grid search [2210.13459][2305.04971][2001.01900].

## 6. Extensions, Limitations, and Future Directions

- **Entropy Regularization Connections:** Label smoothing is a special case ($\alpha \to 1$) of a broader α-Jensen divergence entropy regularizer. Alternative entropic and sparsity-promoting regularizers (e.g., confidence penalty, generalized entropy) may be preferable when sparsity of output distribution is essential [2005.00820].
- **Calibration versus Uncertainty Ranking:** Label smoothing can impair the fidelity of softmax confidence as a selective-rejection criterion. Simple post-hoc logit normalization fully restores the risk-coverage trade-off [2403.14715].
- **Model Misspecification and Robustness:** The main benefit of label smoothing, especially at moderate $\epsilon$ (e.g., $0.06$), is robustification against label noise and model misspecification. Modifying only the loss function (not probability estimation—e.g., using MLSLR, not LSLR) further improves both calibration and generalization [2305.08501].
- **Bi-level and Data-driven Regularization:** Advanced approaches like LABO generate the entire smoothing distribution optimally per-instance, strictly outperforming any fixed ε baseline at negligible extra cost [2305.04971].
- **Information Bottleneck View:** Label smoothing implements the pragmatic discrete information bottleneck, balancing output compression and sufficiency, and is IB-optimal for models with enough capacity and no label conflicts [2508.14077].

## 7. Empirical Comparison Table for $\epsilon=0.06$ (Image Classification)

| Model/Method                   | Hist-P ECE (%) | Top-1 Acc (%) | Calibration Strategy      |
|--------------------------------|---------------|---------------|--------------------------|
| One-hot baseline               | 22.96         | —             | None                     |
| Uniform LS ($\epsilon=0.06$)   | 2.35          | —             | Uniform LS               |
| Class-similarity (word2vec)    | 1.74          | —             | Informed LS              |

Uniform LS at $\epsilon=0.06$ reduces ECE by an order of magnitude relative to the one-hot baseline, with structured/similarity-based smoothing providing an additional $20\!-\!40\%$ advantage in calibration, especially valuable in safety-critical decision-making scenarios [2006.14028].

---

References: [2006.14028], [2312.06522], [2206.07277], [2001.01900], [2212.10258], [2210.13459], [2005.00820], [2305.04971], [2403.14715], [2303.16178], [2508.14077], [2010.12648], [2305.08501]

Source: https://www.emergentmind.com/topics/label-smoothing-epsilon-0-06