Alternating Training-based Label Smoothing (ATLaS)

• The paper introduces ATLaS, which alternates hard-label and soft-label epochs to reconcile CLIP’s training consistency with enhanced regularization.
• ATLaS uses structured soft labels—Class-wise Soft Labels (CSL) and Instance-wise Soft Labels (ISL)—to capture inter-class and instance-class relationships during prompt tuning.
• Empirical results show that ATLaS improves base-to-new generalization and test accuracy dynamics, mitigating overfitting in prompt-tuned vision-language models.

Searching arXiv for the ATLaS paper and closely related label smoothing work to ground the article in published sources. Alternating Training-based Label Smoothing (ATLaS) is a training regime for prompt tuning in pre-trained vision-LLMs in which supervision alternates between standard one-hot labels and soft labels generated by label smoothing. Introduced in the context of prompt tuning for CLIP-like models, it was motivated by the observation that vanilla label smoothing can even weaken the generalization ability of prompt tuning, despite label smoothing’s broader role as a regularizer that prevents models from becoming over-confident. ATLaS therefore retains hard-label supervision for most epochs while periodically injecting soft-label supervision, and it augments vanilla soft labels with two efficient offline constructions—Class-wise Soft Labels (CSL) and Instance-wise Soft Labels (ISL)—that encode inter-class or instance-class relationships for prompt tuning (Chen et al., 25 Aug 2025).

1. Problem setting and motivation

ATLaS is formulated for prompt tuning in pre-trained vision-LLMs such as CLIP, where a visual encoder $E_v$ maps an image $\mathbf{x}$ to a visual embedding $\mathbf{I} = E_v(\mathbf{x})$, and a text encoder $E_t$ maps a prompt to a text embedding $\mathbf{t}$. In zero-shot classification with $C$ classes, CLIP uses text prompts such as “a photo of a [CLS]”, producing class embeddings $\mathbf{t}_c$, and predicts

$$f_c(\mathbf{x}) = \cos(\mathbf{t}_c, \mathbf{I}), \quad p(y=c\mid\mathbf{x}) = \frac{\exp(f_c(\mathbf{x})/\tau)}{\sum_{j=1}^C \exp(f_j(\mathbf{x})/\tau)}.$$

Prompt tuning freezes both encoders and learns textual prompt vectors $\mathbf{v}_1,\dots,\mathbf{v}_M$, replacing the fixed textual prefix with a learnable sequence $[\mathbf{v}_1,\dots,\mathbf{v}_M,[\mathrm{CLS}]]$. This is parameter-efficient, but prompt tuning methods such as CoOp often overfit the tuned distribution and degrade on unseen datasets, out-of-distribution domains, and new classes (Chen et al., 25 Aug 2025).

The motivation for ATLaS is specific to this setting. Label smoothing is widely used to reduce over-confidence, but the reported empirical finding is that directly applying vanilla LS to prompt tuning does not improve generalization and can slightly hurt it. The explanation given is threefold: CLIP was originally trained using hard one-hot labels in its contrastive objective; uniform LS ignores inter-class similarities and instance-level relations; and naively smoothing labels during prompt tuning can over-regularize a model that already has strong zero-shot generalization. ATLaS addresses this by alternating hard and soft supervision rather than replacing hard supervision altogether, thereby preserving consistency with CLIP’s pre-training regime while still injecting regularization (Chen et al., 25 Aug 2025).

2. Alternating supervision mechanism

ATLaS starts from the standard prompt-tuning cross-entropy objective. For one-hot labels $\mathbf{x}$0, the loss is

$\mathbf{x}$1

Vanilla LS replaces $\mathbf{x}$2 by

$\mathbf{x}$3

where $\mathbf{x}$4 is typically uniform,

$\mathbf{x}$5

The key change in ATLaS is not a new per-step loss, but an alternating schedule over epochs. With alternation period $\mathbf{x}$6, the first $\mathbf{x}$7 epochs use pure hard labels and the $\mathbf{x}$8-th epoch uses pure soft labels. Defining

$\mathbf{x}$9

ATLaS forms the supervision target

$\mathbf{I} = E_v(\mathbf{x})$0

Thus, if $\mathbf{I} = E_v(\mathbf{x})$1, the target is exactly one-hot; if $\mathbf{I} = E_v(\mathbf{x})$2, the target is the selected soft label. Prompt parameters are then updated by

$\mathbf{I} = E_v(\mathbf{x})$3

The default setting reported for ATLaS uses $\mathbf{I} = E_v(\mathbf{x})$4 and $\mathbf{I} = E_v(\mathbf{x})$5, so training alternates one hard-label epoch and one soft-label epoch. The method does not mix hard and soft losses in the same iteration through a weighted sum; instead, it switches the label regime itself. This distinction is important in the paper’s empirical analysis, where joint optimization with vanilla LS and one-hot labels yields almost no gain, whereas alternation improves prompt generalization (Chen et al., 25 Aug 2025).

3. Structured soft labels: CSL and ISL

ATLaS supports vanilla LS, but its principal soft-label constructions are CSL and ISL. Both are computed offline and are designed to replace the uniform prior by structured distributions aligned with CLIP’s multimodal geometry (Chen et al., 25 Aug 2025).

Class-wise Soft Labels encode inter-class relations through CLIP text embeddings. Let $\mathbf{I} = E_v(\mathbf{x})$6 denote the CLIP text embedding for class $\mathbf{I} = E_v(\mathbf{x})$7, and let $\mathbf{I} = E_v(\mathbf{x})$8 be cosine similarity. ATLaS computes a class similarity matrix $\mathbf{I} = E_v(\mathbf{x})$9 with

$E_t$0

For each class $E_t$1, the CSL distribution is the $E_t$2-th row,

$E_t$3

The reported default temperature is $E_t$4. CSL is purely class-level and textual: it places mass on semantically similar classes rather than distributing it uniformly (Chen et al., 25 Aug 2025).

Instance-wise Soft Labels are image-dependent and combine visual and textual information. For each image $E_t$5, the fixed zero-shot CLIP classifier produces

$E_t$6

To ensure compatibility with the ground-truth label, ATLaS rectifies this distribution. Let $E_t$7 be the one-hot ground truth and define

$E_t$8

Then the ISL target is

$E_t$9

If CLIP’s top class already matches the ground-truth class, $\mathbf{t}$0 and the ISL target is exactly the zero-shot CLIP distribution. If CLIP’s top class is wrong, ATLaS adds mass $\mathbf{t}$1 to the true class and renormalizes. The reported default rectification coefficient is $\mathbf{t}$2. CSL therefore supplies inter-class structure, whereas ISL supplies instance-class structure (Chen et al., 25 Aug 2025).

4. Optimization analysis and staged label-smoothing perspective

The theoretical analysis of ATLaS is given for the expected hard-label loss $\mathbf{t}$3 under two assumptions: $\mathbf{t}$4 is $\mathbf{t}$5-smooth, and stochastic gradients under one-hot labels have bounded variance $\mathbf{t}$6. A corresponding bound is then established for stochastic gradients induced by soft labels: $\mathbf{t}$7 where $\mathbf{t}$8 depends on soft-label quality. The resulting convergence bound for ATLaS, with learning rate $\mathbf{t}$9, smoothing parameter $C$0, and alternation period $C$1, is

$C$2

For vanilla LS without alternation, the corresponding variance term is

$C$3

When $C$4, the ATLaS variance term is attenuated by a factor $C$5, and the paper concludes that ATLaS enjoys a smaller variance bound, especially when soft labels are suboptimal (Chen et al., 25 Aug 2025).

This optimization perspective places ATLaS in direct continuity with earlier staged label-smoothing analyses. In particular, Two-Stage LAbel smoothing algorithm (TSLA) uses LSR in the early training epochs and drops it off in the later training epochs, and its analysis attributes the benefit of LSR to variance reduction in the first stage and faster convergence in the second stage (Xu et al., 2020). ATLaS differs in mechanism—periodic alternation rather than a single early-stage/late-stage split—but the shared rationale is that label smoothing can be optimization-friendly without needing to dominate the entire training trajectory. A plausible implication is that ATLaS can be viewed as a periodic variant of staged label smoothing specialized to prompt tuning, where hard-label consistency with CLIP remains a primary design constraint.

5. Empirical behavior in prompt tuning

The reported evaluation uses CLIP with a ViT-B/16 visual backbone and integrates ATLaS with five prompt tuning methods: CoOp, CoCoOp, DePT, IVLP, and MaPLe. Experiments cover 11 classification datasets and four settings: cross-dataset generalization, domain generalization, base-to-new generalization, and few-shot classification. The cross-dataset setting tunes prompts on ImageNet with 16-shot per class and tests on 10 target datasets. In that setting, CoOp averages $C$6, CoOp+ATLaS averages $C$7, CoOp+ATLaS-CSL averages $C$8, and CoOp+ATLaS-ISL averages $C$9. On domain generalization to ImageNetV2, ImageNet-Sketch, ImageNet-A, and ImageNet-R, CoOp averages $\mathbf{t}_c$0, while CoOp+ATLaS, CoOp+ATLaS-CSL, and CoOp+ATLaS-ISL reach $\mathbf{t}_c$1, $\mathbf{t}_c$2, and $\mathbf{t}_c$3, respectively (Chen et al., 25 Aug 2025).

The strongest reported gains appear in base-to-new generalization. Averaged over 11 datasets, CoOp yields Base $\mathbf{t}_c$4, New $\mathbf{t}_c$5, and harmonic mean $\mathbf{t}_c$6. CoOp+ATLaS reaches Base $\mathbf{t}_c$7, New $\mathbf{t}_c$8, and $\mathbf{t}_c$9; CoOp+ATLaS-CSL reaches Base $$f_c(\mathbf{x}) = \cos(\mathbf{t}_c, \mathbf{I}), \quad p(y=c\mid\mathbf{x}) = \frac{\exp(f_c(\mathbf{x})/\tau)}{\sum_{j=1}^C \exp(f_j(\mathbf{x})/\tau)}.$$0, New $$f_c(\mathbf{x}) = \cos(\mathbf{t}_c, \mathbf{I}), \quad p(y=c\mid\mathbf{x}) = \frac{\exp(f_c(\mathbf{x})/\tau)}{\sum_{j=1}^C \exp(f_j(\mathbf{x})/\tau)}.$$1, and $$f_c(\mathbf{x}) = \cos(\mathbf{t}_c, \mathbf{I}), \quad p(y=c\mid\mathbf{x}) = \frac{\exp(f_c(\mathbf{x})/\tau)}{\sum_{j=1}^C \exp(f_j(\mathbf{x})/\tau)}.$$2; and CoOp+ATLaS-ISL reaches Base $$f_c(\mathbf{x}) = \cos(\mathbf{t}_c, \mathbf{I}), \quad p(y=c\mid\mathbf{x}) = \frac{\exp(f_c(\mathbf{x})/\tau)}{\sum_{j=1}^C \exp(f_j(\mathbf{x})/\tau)}.$$3, New $$f_c(\mathbf{x}) = \cos(\mathbf{t}_c, \mathbf{I}), \quad p(y=c\mid\mathbf{x}) = \frac{\exp(f_c(\mathbf{x})/\tau)}{\sum_{j=1}^C \exp(f_j(\mathbf{x})/\tau)}.$$4, and $$f_c(\mathbf{x}) = \cos(\mathbf{t}_c, \mathbf{I}), \quad p(y=c\mid\mathbf{x}) = \frac{\exp(f_c(\mathbf{x})/\tau)}{\sum_{j=1}^C \exp(f_j(\mathbf{x})/\tau)}.$$5. The same pattern extends to CoCoOp, DePT, IVLP, and MaPLe, for which all ATLaS variants improve harmonic mean. The paper emphasizes this as high compatibility with prevalent prompt tuning methods (Chen et al., 25 Aug 2025).

Ablations clarify the role of alternation. For CoOp, vanilla LS without alternation reduces the harmonic mean from $$f_c(\mathbf{x}) = \cos(\mathbf{t}_c, \mathbf{I}), \quad p(y=c\mid\mathbf{x}) = \frac{\exp(f_c(\mathbf{x})/\tau)}{\sum_{j=1}^C \exp(f_j(\mathbf{x})/\tau)}.$$6 to $$f_c(\mathbf{x}) = \cos(\mathbf{t}_c, \mathbf{I}), \quad p(y=c\mid\mathbf{x}) = \frac{\exp(f_c(\mathbf{x})/\tau)}{\sum_{j=1}^C \exp(f_j(\mathbf{x})/\tau)}.$$7. A joint optimization baseline, described as “vanilla LS + y”, yields $$f_c(\mathbf{x}) = \cos(\mathbf{t}_c, \mathbf{I}), \quad p(y=c\mid\mathbf{x}) = \frac{\exp(f_c(\mathbf{x})/\tau)}{\sum_{j=1}^C \exp(f_j(\mathbf{x})/\tau)}.$$8, almost unchanged from CoOp. By contrast, CSL alone gives $$f_c(\mathbf{x}) = \cos(\mathbf{t}_c, \mathbf{I}), \quad p(y=c\mid\mathbf{x}) = \frac{\exp(f_c(\mathbf{x})/\tau)}{\sum_{j=1}^C \exp(f_j(\mathbf{x})/\tau)}.$$9, ISL alone gives $\mathbf{v}_1,\dots,\mathbf{v}_M$0, ATLaS-CSL gives $\mathbf{v}_1,\dots,\mathbf{v}_M$1, and ATLaS-ISL gives $\mathbf{v}_1,\dots,\mathbf{v}_M$2. Training curves on SUN397 new classes are reported to show that ATLaS improves test accuracy dynamics and mitigates overfitting. These results support the central ATLaS claim: in prompt tuning, structured soft labels help, but alternation is crucial when LS is used as a supervision mechanism rather than a constant replacement for hard targets (Chen et al., 25 Aug 2025).

6. Relation to broader label-smoothing research

ATLaS belongs to a broader shift from static, globally uniform label smoothing toward schedule-based, data-dependent, and learned smoothing. Earlier schedule-based work showed that LSR can help to speed up convergence by reducing the variance, and proposed a two-stage strategy that uses LSR early and removes it later (Xu et al., 2020). Data-dependent approaches include Adaptive Label Smoothing, which computes a sample-dependent smoothing factor from objectness and relative object size in the crop (Krothapalli et al., 2020), and Pairwise Label Smoothing / Midpoint Regularization, which alternates original and midpoint batches while learning a smoothing distribution for each midpoint sample (Guo, 2021). In sequence modeling, adaptive label smoothing with self-knowledge makes the smoothing parameter instance-wise and entropy-dependent, while using a best-checkpoint self-teacher as the prior distribution (Lee et al., 2022). ATLaS is narrower in scope—it is designed for prompt tuning in vision-LLMs—but it shares with these methods the move away from fixed uniform smoothing.

The prompt-tuning setting also makes several broader caveats about label smoothing especially salient. One is that LS is not uniformly helpful: the explicit empirical point of departure for ATLaS is that vanilla LS can weaken prompt generalization (Chen et al., 25 Aug 2025). Related image-classification work likewise reports that adaptive label smoothing can help under a limited, homogeneous set of image transformation types, but that its benefits vanish under diverse augmentation policies such as TrivialAugment, and excessive smoothing harms robustness to common corruptions (Siedel et al., 22 Jul 2025). A second caveat concerns robustness claims. Label smoothing can produce striking accuracy under standard gradient-based attacks, but this robustness can be incomplete and volatile under stronger attacks, diversified initializations, and transfer attacks (Fu et al., 2020). A third caveat concerns representation structure: label smoothing improves calibration and can help generalization, yet it can also make knowledge distillation into a student much less effective in noise-free settings because it reduces information in the logits about resemblances between instances of different classes (Müller et al., 2019). Under label noise, however, smoothing the teacher can be beneficial and competitive with loss-correction methods (Lukasik et al., 2020).

Taken together, these results define the specific significance of ATLaS. It is not simply another label-smoothing variant; it is a prompt-tuning-specific response to a negative result about constant LS in frozen-encoder vision-language adaptation. Its defining contribution is the claim that alternating hard and soft supervision can reconcile CLIP-consistent training with the regularization benefits of soft labels, especially when those soft labels are structured by class-level or instance-level multimodal information (Chen et al., 25 Aug 2025).