Instance-wise Soft Labels Overview

• Instance-wise Soft Labels are per-instance target representations that use continuous probability vectors to encode uncertainty, ambiguity, and graded class compatibility.
• They are applied in diverse learning settings such as sound event detection, superset label learning, and prompt tuning to improve model robustness and performance.
• Constructed from human annotation or model-generated predictions, these soft labels enable adaptive supervision while demanding careful calibration and method selection.

Searching arXiv for recent and foundational papers on instance-wise soft labels, soft-label learning, and related formulations. arxiv_search query="instance-wise soft labels OR soft label learning OR instance-based label smoothing OR soft labels from annotators" max_results=10

arxiv_search query="(Martín-Morató et al., 2023)" max_results=5

Instance-wise Soft Labels (ISL) are per-instance target representations that replace a single hard decision with a non-binary label object, typically a probability vector, a per-class confidence vector, or another continuous target attached to an individual training example. Across the recent literature, ISL appears in several distinct but related forms: segment-wise activity indicators in sound event detection, candidate-label distributions in superset label learning, feature-conditioned label distributions in noisy-label learning, CLIP-derived instance-dependent targets for prompt tuning, per-point object scores in 3D segmentation, and per-instance confidence distributions elicited from annotators (Martín-Morató et al., 2023, Gong et al., 2019, Algan et al., 2021, Vyas et al., 2020, Chen et al., 25 Aug 2025, Humblot-Renaux et al., 2023, Collins et al., 2022). What unifies these formulations is that supervision is attached to each instance as a soft object encoding uncertainty, ambiguity, disagreement, or graded class compatibility rather than as a one-hot or binary label.

1. Conceptual scope and variants

The term “Instance-wise Soft Labels” does not denote a single universal formalism. In the literature considered here, it covers several mathematically different target types. In sound event detection, the basic instance is a 1-second segment, and the label is a multi-label vector $\mathbf{a}_t = (a_{t,1},\dots,a_{t,C})$ with $a_{t,c}\in[0,1]$, where each component is an independent activity indicator rather than part of a simplex-valued class distribution (Martín-Morató et al., 2023). In superset label learning, by contrast, ISL refers to a simplex-constrained vector over candidate classes, initially

$$\mathbf{Y}_{ij}= \begin{cases} \frac{1}{|\mathcal S_i|}, & j\in \mathcal S_i \ 0, & j\notin \mathcal S_i \end{cases}$$

and then refined by graph-based disambiguation (Gong et al., 2019). In meta-learned classification, ISL takes the form of a learnable per-example distribution $\mathbf{p}_i^t\in\Delta^{c-1}$ or $\boldsymbol{\alpha}_i^t\in\Delta^{c-1}$ that changes during optimization (Vyas et al., 2020). In binary semi-supervised least squares, each unlabeled instance receives a scalar soft target $u_i\in[0,1]$ rather than a vector (Krijthe et al., 2016).

A further distinction concerns whether softness encodes epistemic disagreement, aleatoric ambiguity, or model-imposed regularization. In crowdsourced sound event detection, soft labels reflect aggregated annotator uncertainty and disagreement (Martín-Morató et al., 2023). In CIFAR-10S, each annotator provides a per-image probability distribution, so softness is explicitly elicited from humans rather than inferred from vote counts (Collins et al., 2022). In prompt tuning for vision-LLMs, ISL is computed from CLIP probabilities and rectified by the ground-truth one-hot label (Chen et al., 25 Aug 2025). In 3D semantic segmentation, soft labels arise from geometric agreement between points and fitted CAD models rather than from annotator uncertainty (Humblot-Renaux et al., 2023).

Setting	Instance	Soft-label object
Sound event detection	1-second audio segment	$\mathbf{a}_t\in[0,1]^C$, independent per-class activity scores
Superset label learning	Training example $\mathbf{x}_i$	$\mathbf{Y}_i,\mathbf{F}_i\in\mathbb{R}^c$, simplex-constrained label distributions
Prompt tuning	Image $\mathbf{x}$	$a_{t,c}\in[0,1]$0 from CLIP probabilities and rectification
Meta-learned classification	Example $a_{t,c}\in[0,1]$1	Learnable $a_{t,c}\in[0,1]$2 or $a_{t,c}\in[0,1]$3
3D segmentation	Point $a_{t,c}\in[0,1]$4 relative to a CAD instance	Soft class vector from object score $a_{t,c}\in[0,1]$5 and background mass $a_{t,c}\in[0,1]$6

This heterogeneity is important. A plausible implication is that ISL is better understood as a supervision principle—assigning each instance a soft target—than as a single canonical label type.

2. Mathematical formulations and optimization objectives

A central mathematical distinction is whether the soft target is simplex-valued or only component-wise bounded. In multi-label sound event detection, the soft target for class $a_{t,c}\in[0,1]$7 at time segment $a_{t,c}\in[0,1]$8 is

$a_{t,c}\in[0,1]$9

with annotator competence $$\mathbf{Y}_{ij}= \begin{cases} \frac{1}{|\mathcal S_i|}, & j\in \mathcal S_i \ 0, & j\notin \mathcal S_i \end{cases}$$0 estimated by MACE and binary votes $$\mathbf{Y}_{ij}= \begin{cases} \frac{1}{|\mathcal S_i|}, & j\in \mathcal S_i \ 0, & j\notin \mathcal S_i \end{cases}$$1 (Martín-Morató et al., 2023). Here, $$\mathbf{Y}_{ij}= \begin{cases} \frac{1}{|\mathcal S_i|}, & j\in \mathcal S_i \ 0, & j\notin \mathcal S_i \end{cases}$$2 in general. Hard labels are obtained by thresholding at $$\mathbf{Y}_{ij}= \begin{cases} \frac{1}{|\mathcal S_i|}, & j\in \mathcal S_i \ 0, & j\notin \mathcal S_i \end{cases}$$3. In prompt tuning, the ISL target is instead a class distribution

$$\mathbf{Y}_{ij}= \begin{cases} \frac{1}{|\mathcal S_i|}, & j\in \mathcal S_i \ 0, & j\notin \mathcal S_i \end{cases}$$4

where $$\mathbf{Y}_{ij}= \begin{cases} \frac{1}{|\mathcal S_i|}, & j\in \mathcal S_i \ 0, & j\notin \mathcal S_i \end{cases}$$5 is CLIP’s probability vector, $$\mathbf{Y}_{ij}= \begin{cases} \frac{1}{|\mathcal S_i|}, & j\in \mathcal S_i \ 0, & j\notin \mathcal S_i \end{cases}$$6 is the one-hot ground-truth label, $$\mathbf{Y}_{ij}= \begin{cases} \frac{1}{|\mathcal S_i|}, & j\in \mathcal S_i \ 0, & j\notin \mathcal S_i \end{cases}$$7 indicates whether CLIP’s top prediction is wrong, and $$\mathbf{Y}_{ij}= \begin{cases} \frac{1}{|\mathcal S_i|}, & j\in \mathcal S_i \ 0, & j\notin \mathcal S_i \end{cases}$$8 is a rectification coefficient (Chen et al., 25 Aug 2025). In superset label learning, the refined distribution $$\mathbf{Y}_{ij}= \begin{cases} \frac{1}{|\mathcal S_i|}, & j\in \mathcal S_i \ 0, & j\notin \mathcal S_i \end{cases}$$9 is constrained by $\mathbf{p}_i^t\in\Delta^{c-1}$0 and $\mathbf{p}_i^t\in\Delta^{c-1}$1, so each $\mathbf{p}_i^t\in\Delta^{c-1}$2 is a probability vector over labels (Gong et al., 2019).

Training objectives vary accordingly. Sound event detection compares hard-label BCE, soft-label BCE, and regression-style MSE on soft targets: $\mathbf{p}_i^t\in\Delta^{c-1}$3

$\mathbf{p}_i^t\in\Delta^{c-1}$4

with the latter directly treating ISL as regression targets (Martín-Morató et al., 2023). In MetaLabelNet, the base classifier is trained against generated soft labels using KL divergence,

$\mathbf{p}_i^t\in\Delta^{c-1}$5

while the label generator is updated by a meta-objective defined on clean meta-data (Algan et al., 2021). In “Learning Soft Labels via Meta Learning,” the labels themselves are learned parameters, and the training loss is written as

$\mathbf{p}_i^t\in\Delta^{c-1}$6

with a one-step look-ahead meta-loss used to update $\mathbf{p}_i^t\in\Delta^{c-1}$7 (Vyas et al., 2020). In semi-supervised least squares classification, unlabeled targets enter directly as latent regression targets in

$\mathbf{p}_i^t\in\Delta^{c-1}$8

with $\mathbf{p}_i^t\in\Delta^{c-1}$9 updated by projection of current predictions (Krijthe et al., 2016).

For dense prediction, the loss itself may need modification to remain valid under soft targets. “Jaccard Metric Losses” introduces JML1 and JML2 because standard soft Jaccard losses do not properly support soft labels in segmentation. JML1 is

$\boldsymbol{\alpha}_i^t\in\Delta^{c-1}$0

and is identical to the soft Jaccard loss in the hard-label setting while remaining a metric on $\boldsymbol{\alpha}_i^t\in\Delta^{c-1}$1 for soft labels (Wang et al., 2023).

3. How ISL are obtained

One major source of ISL is human annotation. In sound event detection, multiple annotators label 10-second audio segments, and the system reconstructs 1-second segment-wise opinions, then aggregates them with competence weights estimated by MACE (Martín-Morató et al., 2023). In CIFAR-10S, each annotator provides a top-1 label and probability, an optional top-2 label and probability, and “definitely not” classes; a redistribution function then maps these partial judgments into a full per-image distribution $\boldsymbol{\alpha}_i^t\in\Delta^{c-1}$2, and the dataset-level soft label is

$\boldsymbol{\alpha}_i^t\in\Delta^{c-1}$3

(Collins et al., 2022). This is an especially direct form of ISL because the individual annotator distribution is already an instance-wise soft label before aggregation.

A second source is model-generated soft labels. In prompt tuning, CLIP probabilities supply an instance-specific class distribution, optionally rectified when CLIP’s top prediction is incorrect (Chen et al., 25 Aug 2025). In satellite object detection, each retained detection instance is effectively a soft label consisting of bounding box, confidence, and class distribution, although the downstream YOLOv5 training pipeline ultimately uses the argmax class label plus the model-generated geometry (Ciolino et al., 2022). In ASR continuous pseudo-labeling, the relevant ISL are per-utterance, per-frame probability distributions produced by a teacher acoustic model (Likhomanenko et al., 2022).

A third source is latent estimation under weak or indirect supervision. In MetaLabelNet, a frozen feature extractor $\boldsymbol{\alpha}_i^t\in\Delta^{c-1}$4 feeds a single-layer perceptron $\boldsymbol{\alpha}_i^t\in\Delta^{c-1}$5, yielding $\boldsymbol{\alpha}_i^t\in\Delta^{c-1}$6, so ISL are generated from instance features and optimized by a meta-objective on clean meta-data (Algan et al., 2021). In meta-learned label optimization, the per-instance label vector is itself a learnable parameter (Vyas et al., 2020). In multiple instance learning, BIMIL learns a probabilistic instance classifier $\boldsymbol{\alpha}_i^t\in\Delta^{c-1}$7 from bag labels alone, so the instance-level soft label is inferred rather than observed (Peng et al., 2019). In 3D segmentation, per-point object scores are derived from geometry, region growing, and an SVM rather than from humans, then converted into soft semantic labels with object-class mass $\boldsymbol{\alpha}_i^t\in\Delta^{c-1}$8 and background mass $\boldsymbol{\alpha}_i^t\in\Delta^{c-1}$9 (Humblot-Renaux et al., 2023).

This diversity of construction mechanisms has methodological consequences. Human-elicited ISL preserve subjective uncertainty; teacher-generated ISL preserve model geometry; and latent or geometric ISL preserve constraints of the inference procedure that created them. A plausible implication is that the inductive bias of ISL depends at least as much on their source as on their numerical format.

4. Representative methodological families

In superset label learning, ISL originally refers to iterative label propagation over a similarity graph, starting from uniform mass over candidate labels. “RegISL” retains the instance-wise soft-label idea but adds graph smoothness, fidelity to the candidate set, and a discrimination term $u_i\in[0,1]$0, explicitly encouraging peaked per-instance distributions (Gong et al., 2019). The optimization is non-convex and solved by augmented Lagrangian methods together with CCCP.

In sound event detection, ISL are implemented at 1-second temporal resolution inside a CRNN with mel-spectrogram features, 3 convolutional layers, and 2 bidirectional GRU layers. Three training setups are compared: hard labels with BCE and sigmoid output, soft labels with BCE and sigmoid output, and soft labels with MSE and linear output (Martín-Morató et al., 2023). The MSE formulation is the clearest regression-style realization of ISL in that setting.

In prompt tuning for CLIP-like models, ISL are integrated through ATLaS, an alternating schedule in which epochs $u_i\in[0,1]$1 to $u_i\in[0,1]$2 use one-hot labels and epoch $u_i\in[0,1]$3 uses soft labels. The ATLaS target is

$u_i\in[0,1]$4

and setting $u_i\in[0,1]$5 yields alternating hard/soft supervision (Chen et al., 25 Aug 2025). This is notable because the paper explicitly reports that vanilla label smoothing weakens prompt generalization, whereas alternating training with ISL improves it.

In noisy-label learning, MetaLabelNet and meta-learned label parameters represent two closely related but distinct strategies. MetaLabelNet learns a feature-to-label mapping with an SLP and optimizes it so that gradient updates induced by the generated soft labels reduce meta loss on clean data (Algan et al., 2021). “Learning Soft Labels via Meta Learning” instead treats every label vector $u_i\in[0,1]$6 as a learnable parameter and updates it through one-step look-ahead meta-gradients (Vyas et al., 2020). Both approaches turn soft labels into dynamic, optimization-dependent objects rather than fixed preprocessing artifacts.

For dense prediction, 3D semantic segmentation and 2D semantic segmentation illustrate two different uses of ISL. In the CAD-based 3D pipeline, each point is hard-assigned to the nearest CAD instance and then scored by region, distance, and SVM cues, producing an object score $u_i\in[0,1]$7; the final soft semantic label places mass $u_i\in[0,1]$8 on the object class and $u_i\in[0,1]$9 on background (Humblot-Renaux et al., 2023). In 2D semantic segmentation, JMLs are not themselves ISL, but they are explicitly designed to make IoU optimization compatible with soft labels arising from label smoothing, knowledge distillation, and semi-supervised learning (Wang et al., 2023).

5. Empirical behavior and observed benefits

The empirical literature attributes several recurring benefits to ISL. In sound event detection, soft-label regression preserves label distributions and helps with rare or ambiguous events. On MAESTRO-Real, the hard-label baseline H_BCE_sig achieved ER $\mathbf{a}_t\in[0,1]^C$0, F1 $\mathbf{a}_t\in[0,1]^C$1, and KLD $\mathbf{a}_t\in[0,1]^C$2, whereas S_MSE_lin with a $\mathbf{a}_t\in[0,1]^C$3 threshold achieved ER $\mathbf{a}_t\in[0,1]^C$4, F1 $\mathbf{a}_t\in[0,1]^C$5, and KLD $\mathbf{a}_t\in[0,1]^C$6; with class-dependent thresholds, S_MSE_lin produced non-zero F1 for all classes and detected sounds missed in the typical binary target training setup (Martín-Morató et al., 2023). The reported interpretation is that soft supervision enables learning of rare classes whose activity indicators often lie below $\mathbf{a}_t\in[0,1]^C$7.

In superset label learning, RegISL consistently improves over ISL-style propagation on several datasets. On Lost, ISL reported $\mathbf{a}_t\in[0,1]^C$8 training accuracy and $\mathbf{a}_t\in[0,1]^C$9 test accuracy, while RegISL reported $\mathbf{x}_i$0 and $\mathbf{x}_i$1. On Bird Song, ISL reported $\mathbf{x}_i$2 training accuracy and $\mathbf{x}_i$3 test accuracy, whereas RegISL reached $\mathbf{x}_i$4 and $\mathbf{x}_i$5 (Gong et al., 2019). The paper attributes the gain to more discriminative disambiguation and better use of mutual exclusiveness among candidate labels.

In prompt tuning, ATLaS-ISL improves several generalization regimes for CoOp. In 16-shot cross-dataset generalization, CoOp averaged $\mathbf{x}_i$6 and CoOp + ATLaS-ISL averaged $\mathbf{x}_i$7. In domain generalization, the average improved from $\mathbf{x}_i$8 to $\mathbf{x}_i$9. In base-to-new evaluation averaged over 11 datasets, CoOp’s new-class accuracy rose from $\mathbf{Y}_i,\mathbf{F}_i\in\mathbb{R}^c$0 to $\mathbf{Y}_i,\mathbf{F}_i\in\mathbb{R}^c$1, and the harmonic mean increased from $\mathbf{Y}_i,\mathbf{F}_i\in\mathbb{R}^c$2 to $\mathbf{Y}_i,\mathbf{F}_i\in\mathbb{R}^c$3, although base accuracy decreased from $\mathbf{Y}_i,\mathbf{F}_i\in\mathbb{R}^c$4 to $\mathbf{Y}_i,\mathbf{F}_i\in\mathbb{R}^c$5 (Chen et al., 25 Aug 2025). The reported pattern is regularization against overfitting to base classes while retaining CLIP’s structure.

In noisy supervision, feature-conditioned and meta-learned ISL improve both accuracy and robustness to corrupted labels. The abstract of “Learning Soft Labels via Meta Learning” reports that dynamically learned labels improve ResNet18 by $\mathbf{Y}_i,\mathbf{F}_i\in\mathbb{R}^c$6 on CIFAR100 (Vyas et al., 2020). Its detailed results show CIFAR100 ResNet18 improving from $\mathbf{Y}_i,\mathbf{F}_i\in\mathbb{R}^c$7 with one-hot labels to $\mathbf{Y}_i,\mathbf{F}_i\in\mathbb{R}^c$8 with instance labels. MetaLabelNet reports $\mathbf{Y}_i,\mathbf{F}_i\in\mathbb{R}^c$9 on Clothing1M and $\mathbf{x}$0 / $\mathbf{x}$1 top-1/top-5 on WebVision, outperforming the baselines discussed there (Algan et al., 2021).

Human-elicited ISL can be especially beneficial in low-annotator regimes. CIFAR-10S achieves comparable performance to CIFAR-10H while using about $\mathbf{x}$2 annotators per image instead of about $\mathbf{x}$3, albeit with significant temporal costs per elicitation (Collins et al., 2022). In de-aggregated training with only one annotator per image per batch, CIFAR-10S substantially outperformed single hard labels from CIFAR-10H on CE, calibration, and FGSM loss on both CIFAR-10H and CIFAR-10S evaluation sets (Collins et al., 2022). This directly supports the claim that an individual annotator’s uncertainty distribution can be more informative than an individual hard label.

Outside standard classification, automatically generated ISL can retain much of the value of manual labels. In satellite object detection, models trained exclusively on instance-wise soft labels remained within about $\mathbf{x}$4 mAP and $\mathbf{x}$5–$\mathbf{x}$6 F1 of models trained on the original data (Ciolino et al., 2022). In CAD-based point-cloud segmentation, auto-soft labels achieved mIoU overall $\mathbf{x}$7 on Scan2CAD, versus $\mathbf{x}$8 for auto-hard labels, and auto-soft or auto-weak generally outperformed auto-hard (Humblot-Renaux et al., 2023).

6. Limits, failure modes, and open directions

ISL are not uniformly beneficial, and several papers identify specific failure modes. The strongest negative result comes from ASR: replacing hard pseudo-labels with soft per-frame label distributions in continuous pseudo-labeling can lead to divergence and collapse to a degenerate token distribution per frame (Likhomanenko et al., 2022). The paper hypothesizes that hard-label CTC training imposes sequence-level consistency that prevents this collapse, whereas frame-wise soft distribution matching does not. Blending soft and hard losses ameliorates the problem but does not outperform hard pseudo-labeling. This is a direct caution against treating all soft supervision as automatically superior.

Another important limitation concerns interpretation and calibration. “Practical estimation of the optimal classification error with soft labels and calibration” shows that calibration guarantee is not enough: even perfectly calibrated soft labels can result in a substantially inaccurate Bayes error estimate (Ushio et al., 27 May 2025). Under monotone corruption $\mathbf{x}$9, isotonic calibration yields a statistically consistent estimator, but raw or merely calibrated probabilities may still be misleading. This result matters beyond Bayes-error estimation. It implies that ISL quality cannot be judged by calibration alone if the downstream task depends on preserving the structure of the posterior distribution across instances.

Collection and aggregation also remain nontrivial. CIFAR-10S reduces annotator count but incurs much longer elicitation time per label (Collins et al., 2022). MAESTRO-Real contains only 75 recordings, 17 classes, and high imbalance, so rare-class behavior is fragile and threshold-dependent (Martín-Morató et al., 2023). CAD-based soft labels depend on fitted CAD models and currently assume one winning instance per point (Humblot-Renaux et al., 2023). Satellite soft labels inherit teacher bias and miss unlabeled objects (Ciolino et al., 2022). In superset label learning, graph construction and the assumption that the true label belongs to the candidate set remain open issues (Gong et al., 2019).

Several explicit future directions recur across the literature: more sophisticated annotator modeling and aggregation, dynamic or online ISL instead of purely offline targets, extensions to other modalities and architectures, and stronger treatment of imbalance and rare events (Martín-Morató et al., 2023, Chen et al., 25 Aug 2025, Gong et al., 2019). A plausible implication is that the next phase of ISL research will be less about proving that soft per-instance targets can help and more about identifying which source of softness, which loss, and which structural constraints are appropriate for a given task.

ISL therefore occupies a broad methodological space rather than a single technique. Its common premise is simple—replace hard per-instance supervision with a richer target—but the empirical record shows that the value of doing so depends on the semantics of the soft target, the mechanism that produces it, the objective used to learn from it, and the structure of the prediction problem itself.