Class-Wise Soft Labels (CSL)

• Class-wise Soft Labels (CSL) are a set of techniques that replace one-hot labels with probability vectors encoding class-structured uncertainty.
• They redistribute label mass according to class-specific ambiguity, semantic similarity, and learned confidence scores to enhance model calibration and robustness.
• CSL methods utilize various estimation strategies—from multi-annotator vote averaging to model-driven updates—improving accuracy, fairness, and interpretability across domains.

Searching arXiv for the cited papers to ground the article and ensure up-to-date citation context. Class-wise Soft Labels (CSL) denote a family of supervision schemes in which training targets retain class-structured uncertainty instead of collapsing each example to a one-hot label. Across the literature, the term covers several related constructions: class-conditional probability targets in the simplex, class-shared soft targets learned or estimated per class, class-wise temperatures in knowledge distillation, and, in some multi-label settings, independent per-class confidence scores rather than a normalized categorical distribution. What unifies these usages is the replacement of uniform or purely hard supervision by targets that encode structure in ambiguity, confusion, semantic proximity, or annotation uncertainty at the class level (Grossmann et al., 2022, Li et al., 2020, Zhao et al., 10 Jun 2025, Chen et al., 25 Aug 2025, Fard et al., 2024).

1. Definition and formal scope

In the standard $C$-class setting, a hard label for an example $x$ is a one-hot vector $y \in \{e_c : c \in \{1,\dots,C\}\}$, whereas a soft label is a probability vector in the simplex $\Delta^C$:

$$y = (y_1,\dots,y_C), \qquad \sum_{k=1}^C y_k = 1,\qquad y_k \ge 0.$$

This basic distinction underlies several CSL formulations (Grossmann et al., 2022).

One influential class-conditional formulation assigns the non-true-class mass according to a class-dependent ambiguity distribution $s_c$ rather than uniformly. For a sample whose majority class is $c$, the target is

$$y'_k = (1-\alpha)\mathbf{1}_{k=c} + \alpha\, s_c(k), \qquad s_c \in \Delta^C,\quad \alpha \in [0,1].$$

This contrasts with uniform label smoothing, which redistributes mass through $u(k)=1/C$ instead of a class-specific $s_c$ (Grossmann et al., 2022).

A stricter class-wise interpretation appears in COLAM, where all samples of class $x$0 share the same soft label vector $x$1, so that each instance $x$2 with $x$3 uses $x$4. In that formulation, CSL is explicitly a per-class target encoding inter-class confusion learned from peer samples of the same class (Li et al., 2020).

Other papers operationalize class-wise structure differently. In fairness-aware adversarial distillation, CSL replaces a single distillation temperature by a class-wise temperature $x$5, yielding

$x$6

with separate temperatures for clean and adversarial samples (Zhao et al., 10 Jun 2025). In prompt tuning for vision-LLMs, CSL is an offline similarity distribution $x$7 derived from text embeddings, with paper-default target $x$8 (Chen et al., 25 Aug 2025). In automatic 3D annotation, CSL is a point-wise target where the nearest object class receives probability $x$9, background receives $y \in \{e_c : c \in \{1,\dots,C\}\}$0, and all other classes receive $y \in \{e_c : c \in \{1,\dots,C\}\}$1 (Humblot-Renaux et al., 2023).

Not all CSL formulations are categorical distributions. AffectNet+ defines an 8-dimensional class-wise confidence vector for facial expressions in which the components are independent one-vs-rest scores and are explicitly not constrained to sum to 1. This makes the representation suitable for compound expressions and multi-label interpretation, but it departs from simplex-based CSL used in standard classification (Fard et al., 2024).

2. Construction and estimation mechanisms

A central source of CSL is multi-annotator supervision. When each example receives $y \in \{e_c : c \in \{1,\dots,C\}\}$2 annotations, empirical soft labels can be formed by vote averaging,

$y \in \{e_c : c \in \{1,\dots,C\}\}$3

or, equivalently,

$y \in \{e_c : c \in \{1,\dots,C\}\}$4

where $y \in \{e_c : c \in \{1,\dots,C\}\}$5 is the annotator’s one-hot vote. This captures ambiguity and, in real data, label noise (Grossmann et al., 2022). A closely related construction augments primary labels with annotator confidence, optional secondary labels, and disagreement-derived reliability. In that setting, annotator-level soft labels are calibrated by a two-stage Bayesian procedure and then averaged across annotators to obtain the final CSL target (Wu et al., 2023).

When multiple annotations are unavailable, several papers estimate CSL from model behavior. COLAM updates class-wise targets from the current network by aggregating predictions over peer samples of the same class. Under its original objective, the class prototype is the average of peer logits followed by temperature-scaled softmax; under KL-based variants, the update becomes a normalized geometric mean of peer probabilities (Li et al., 2020). Meta-learning-based soft-label learning takes a different route: it treats labels as learnable parameters and updates class-wise smoothing coefficients $y \in \{e_c : c \in \{1,\dots,C\}\}$6 using meta-gradients computed through a one-step look-ahead on a held-out set, producing time-varying class-wise targets

$y \in \{e_c : c \in \{1,\dots,C\}\}$7

(Vyas et al., 2020).

Class-wise structure can also be imported from external priors. In automatic 3D segmentation, per-point class-wise soft labels are derived from fitted CAD models. A point is first assigned to its nearest object instance, then an object confidence $y \in \{e_c : c \in \{1,\dots,C\}\}$8 is computed by averaging available geometric and spatial cues—region score, distance score, and Platt-scaled SVM score—and finally embedded into a class target with object-vs-background ambiguity (Humblot-Renaux et al., 2023). In prompt tuning for CLIP-like models, CSL is computed offline from text-side class similarities:

$y \in \{e_c : c \in \{1,\dots,C\}\}$9

This produces a class-dependent target that respects semantic proximity in the frozen text embedding space (Chen et al., 25 Aug 2025).

In semi-supervised fine-grained learning, CSL is constructed dynamically by selecting a subset of plausible classes and renormalizing the current prediction over that subset:

$\Delta^C$0

The subset is obtained from class-transition tracking and confidence-aware clustering, so the support of the soft label shrinks as confidence increases (Duan et al., 2023). AffectNet+ uses yet another pipeline: an ensemble of binary classifiers and an Action Unit-based classifier each produce per-class scores, which are then fused into the final soft-label vector

$\Delta^C$1

(Fard et al., 2024).

3. Optimization objectives and learning dynamics

The canonical loss for simplex-based CSL is soft cross-entropy,

$\Delta^C$2

which reduces to $\Delta^C$3 under one-hot labels. Up to a target-dependent constant, this is the forward KL divergence $\Delta^C$4 (Grossmann et al., 2022). The same form appears in wrapper-based soft-label learning and in many class-wise constructions, where the supervision differs only in how $\Delta^C$5 is produced (Vries et al., 2024).

Several papers use KL directly as the training criterion. The 3D segmentation pipeline trains PointNet++ with

$\Delta^C$6

and masks out unlabeled points in its weak-label regime (Humblot-Renaux et al., 2023). ABSLD uses KL between student and teacher distributions for both clean and adversarial examples, with class-wise teacher temperatures controlling the smoothness of supervisory targets (Zhao et al., 10 Jun 2025).

An important gradient-level interpretation appears in ambiguity-aware classification. With softmax outputs, the per-example gradient with respect to logits satisfies

$\Delta^C$7

Under hard labels, all non-target classes are pushed away from the target uniformly; under soft labels, the gradient direction and magnitude are modulated by the full target distribution, so ambiguous examples generate smaller and more structured updates (Grossmann et al., 2022). This mechanism is used to explain smoother decision boundaries and more regular feature manifolds.

Collision Cross-Entropy (CCE) proposes a different treatment of soft labels. Instead of Shannon cross-entropy, it uses

$\Delta^C$8

CCE agrees with standard CE for one-hot labels but differs on uncertain targets; in particular, if $\Delta^C$9 is uniform, $$y = (y_1,\dots,y_C), \qquad \sum_{k=1}^C y_k = 1,\qquad y_k \ge 0.$$0 is constant in $$y = (y_1,\dots,y_C), \qquad \sum_{k=1}^C y_k = 1,\qquad y_k \ge 0.$$1, so such samples contribute zero logits gradient. This makes CCE a robust alternative when soft labels reflect uncertainty that should not be copied literally into the model (Zhang et al., 2023).

Optimization schedules also vary. COLAM alternates between model updates and class-wise soft-label updates across stages (Li et al., 2020). Meta-learned soft labels update label parameters at every step by differentiating through a one-step look-ahead on held-out data (Vyas et al., 2020). ATLaS alternates epoch-wise between one-hot supervision and offline CSL to avoid the mismatch that vanilla label smoothing introduces in prompt tuning (Chen et al., 25 Aug 2025). ABSLD updates class-wise temperatures once per epoch from per-class optimization error risk, sharpening targets for hard classes and smoothing them for easy classes (Zhao et al., 10 Jun 2025).

4. Empirical behavior across application domains

Empirical evidence for CSL is broad but highly task-dependent. In ambiguity-aware image classification, learning from annotator-derived soft labels improved both accuracy and distributional fit. On the synthetic dataset, accuracy increased from $$y = (y_1,\dots,y_C), \qquad \sum_{k=1}^C y_k = 1,\qquad y_k \ge 0.$$2 to $$y = (y_1,\dots,y_C), \qquad \sum_{k=1}^C y_k = 1,\qquad y_k \ge 0.$$3, and KL divergence dropped from $$y = (y_1,\dots,y_C), \qquad \sum_{k=1}^C y_k = 1,\qquad y_k \ge 0.$$4 to $$y = (y_1,\dots,y_C), \qquad \sum_{k=1}^C y_k = 1,\qquad y_k \ge 0.$$5. On the MiceBone dataset, macro accuracy improved from $$y = (y_1,\dots,y_C), \qquad \sum_{k=1}^C y_k = 1,\qquad y_k \ge 0.$$6 to $$y = (y_1,\dots,y_C), \qquad \sum_{k=1}^C y_k = 1,\qquad y_k \ge 0.$$7, while KL divergence fell from $$y = (y_1,\dots,y_C), \qquad \sum_{k=1}^C y_k = 1,\qquad y_k \ge 0.$$8 to $$y = (y_1,\dots,y_C), \qquad \sum_{k=1}^C y_k = 1,\qquad y_k \ge 0.$$9 (Grossmann et al., 2022).

These gains were accompanied by qualitative changes in representation geometry. In the synthetic setting, soft labels yielded six distinct clusters arranged around a cyclic interpolation region that mirrored the data-generation process, whereas hard labels fragmented the interpolation manifold. On MiceBone, soft labels strengthened separation and clarified transitions between fiber classes and the “not relevant” class. The paper attributes this to a more regular, interpretable feature space aligned with human uncertainty (Grossmann et al., 2022).

In 3D semantic segmentation, automatically generated class-wise soft point labels improved downstream performance relative to hard automatic labels. On BeamsHooks, PointNet++ trained with auto-soft CSL achieved $s_c$0 mIoU versus $s_c$1 for auto-hard, with boundary IoU improving from $s_c$2 to $s_c$3. On Scan2CAD, auto-soft reached $s_c$4 mIoU, exceeding the manual baseline at $s_c$5 and auto-hard at $s_c$6 (Humblot-Renaux et al., 2023).

In co-learning and meta-learning settings, class-wise soft labels yield smaller but consistent improvements on standard benchmarks. COLAM reported top-1 gains over hard-label SGD and label smoothing on CIFAR-100 and ImageNet, including ResNeXt29-8×64d on CIFAR-100 from $s_c$7 to $s_c$8 and ResNet50 on ImageNet from $s_c$9 to $c$0 (Li et al., 2020). Meta-learned labels improved ResNet18 on CIFAR100 from $c$1 to $c$2 for the instance-wise version and to $c$3 for the class-wise version, while also improving robustness under synthetic label noise (Vyas et al., 2020).

Prompt tuning provides a different empirical pattern: vanilla label smoothing weakened generalization, but alternating training with CSL reversed that effect. With CLIP ViT-B/16, CoOp plus ATLaS-CSL improved harmonic mean in base-to-new generalization from $c$4 to $c$5, and cross-dataset generalization from ImageNet to ten targets improved by $c$6 average accuracy over CoOp (Chen et al., 25 Aug 2025).

Robustness- and fairness-oriented CSL likewise changes the metric profile rather than only average accuracy. ABSLD improved worst-class robustness on CIFAR-10 and CIFAR-100, with reductions in Normalized Standard Deviation and gains such as $c$7 worst-class robustness under PGD for CIFAR-10 ResNet-18 relative to the best baseline (Zhao et al., 10 Jun 2025). This suggests that class-wise smoothness can be used not only for ambiguity modeling but also for rebalancing optimization pressure across classes.

5. Relation to adjacent methods and recurrent misconceptions

CSL is often conflated with label smoothing, but the distinction is explicit across multiple papers. Uniform label smoothing redistributes a fixed amount of probability mass uniformly across non-target classes, whereas CSL redistributes mass according to class ambiguity, semantic similarity, geometric evidence, or learned confusion structure. In MiceBone-like settings, for example, confusion is concentrated between fiber subtypes rather than spread evenly to “not relevant”; in CLIP prompt tuning, the redistribution is defined by text-embedding similarities rather than a uniform prior (Grossmann et al., 2022, Chen et al., 25 Aug 2025).

CSL is also distinct from standard knowledge distillation. Distillation usually transfers teacher outputs produced with a single temperature, while class-wise distillation methods adjust the softness by class. ABSLD explicitly argues that the smoothness degree of soft labels for different classes significantly impacts robust fairness and therefore uses varying temperatures for different classes, with separate clean and adversarial temperature sets (Zhao et al., 10 Jun 2025). This makes the class-wise supervisory signal part of the optimization policy rather than a fixed teacher property.

Nor is CSL identical to pseudo-labeling. In the CAD-based 3D pipeline, the soft labels are derived from external geometric priors and can be used from scratch without any pre-trained model; the paper explicitly contrasts this with pseudo-labeling, which feeds model predictions back into training (Humblot-Renaux et al., 2023). By contrast, COLAM and SoC do rely on model outputs, but they aggregate or filter them through class-wise structure rather than using raw hard pseudo-labels (Li et al., 2020, Duan et al., 2023).

A further misconception is that soft labels are always normalized categorical targets. AffectNet+ uses independent class-wise confidences that do not sum to 1, because facial expressions may be compound and one-vs-rest interpretation is desired (Fard et al., 2024). This suggests that the phrase “CSL” names a design principle—retaining class-structured uncertainty—rather than a single algebraic object.

Evaluation can also be misunderstood. In ambiguity-aware classification, soft-label training improved KL to soft ground truth robustly, but Expected Calibration Error did not improve uniformly across all settings; on the synthetic dataset, hard ECE was approximately $c$8 while soft ECE was approximately $c$9. The paper therefore stresses that calibration must be evaluated consistently with the target notion, hard or soft (Grossmann et al., 2022). A plausible implication is that CSL changes the semantics of “well-calibrated” prediction, so metric choice becomes part of the problem definition rather than a post hoc diagnostic.

6. Limitations, design trade-offs, and open directions

A persistent practical limitation is data acquisition. Multiple annotations per example are costly, and several papers treat this as a central bottleneck. Ambiguity-aware classification notes that more datasets, modalities, and training paradigms should be tested, while also suggesting semi/self-supervised strategies to bootstrap soft labels (Grossmann et al., 2022). The meta-learning framework requires a reasonably clean and representative meta set; if the meta set is biased or noisy, learned labels can be misdirected (Vyas et al., 2020). Annotation-reliability modeling also breaks down when annotator overlap is sparse: in the CDS dataset, Bayesian calibration degraded macro-F1 from $$y'_k = (1-\alpha)\mathbf{1}_{k=c} + \alpha\, s_c(k), \qquad s_c \in \Delta^C,\quad \alpha \in [0,1].$$0 to $$y'_k = (1-\alpha)\mathbf{1}_{k=c} + \alpha\, s_c(k), \qquad s_c \in \Delta^C,\quad \alpha \in [0,1].$$1 because Stage 2 reliability estimates were unstable (Wu et al., 2023).

Over-smoothing remains a recurrent risk. Excessive $$y'_k = (1-\alpha)\mathbf{1}_{k=c} + \alpha\, s_c(k), \qquad s_c \in \Delta^C,\quad \alpha \in [0,1].$$2 in class-conditional soft targets can blur decision boundaries, especially for minority classes (Grossmann et al., 2022). In COLAM-style or temperature-based CSL, too large a temperature or entropy regularizer can approach uniformity, while too small a value can collapse back to near-hard labels (Li et al., 2020). ABSLD frames this as a fairness–robustness trade-off: improving worst-class fairness can reduce average robustness slightly, and teacher reliability is an explicit assumption (Zhao et al., 10 Jun 2025).

The quality of the prior structure used to build CSL is equally important. In prompt tuning, CSL depends on the quality of class text embeddings; ambiguous or overly generic class names can misallocate probability mass (Chen et al., 25 Aug 2025). AffectNet+ soft labels are algorithmically generated rather than directly human-rated per-class intensities, and the paper notes that future work should study per-class calibration and uncertainty quantification (Fard et al., 2024). In semi-supervised fine-grained learning, class-transition tracking can be noisy early in training, so warmup and averaging are needed to prevent unstable clusters (Duan et al., 2023).

At scale, storage and computation introduce another trade-off. In large-scale dataset distillation, auxiliary soft labels can exceed image storage by more than an order of magnitude; ImageNet-1K IPC200 required $$y'_k = (1-\alpha)\mathbf{1}_{k=c} + \alpha\, s_c(k), \qquad s_c \in \Delta^C,\quad \alpha \in [0,1].$$3 GB of labels versus $$y'_k = (1-\alpha)\mathbf{1}_{k=c} + \alpha\, s_c(k), \qquad s_c \in \Delta^C,\quad \alpha \in [0,1].$$4 GB of images. Class-wise supervision during synthesis reduced within-class similarity and enabled random pruning of soft labels, compressing storage to $$y'_k = (1-\alpha)\mathbf{1}_{k=c} + \alpha\, s_c(k), \qquad s_c \in \Delta^C,\quad \alpha \in [0,1].$$5 GB with a $$y'_k = (1-\alpha)\mathbf{1}_{k=c} + \alpha\, s_c(k), \qquad s_c \in \Delta^C,\quad \alpha \in [0,1].$$6 performance gain (Xiao et al., 2024). This suggests that CSL is not only a target representation but can also be a mechanism for controlling redundancy in distilled supervision.

Open questions recur across the surveyed work. Recommended experiments include ablations over softness parameters and per-class schedules, comparisons between class-wise and per-example soft labels, robustness studies under synthetic noise, and quantitative analyses of representation regularity beyond qualitative visualization (Grossmann et al., 2022). More broadly, the literature suggests three unresolved axes: how to estimate class structure when annotations are scarce, how to reconcile calibration with robustness and fairness objectives, and how to choose between simplex-based, class-shared, and independent-confidence formulations in domains where class membership is not strictly mutually exclusive.