Soft-Label Learning for Calibrated Decisions (SLCD)

• Soft-Label Learning for Calibrated Decisions (SLCD) is a framework for evaluating decisions derived from models trained with soft targets, emphasizing calibration for downstream usage rather than classification accuracy alone.
• Soft targets in SLCD originate from diverse sources like expert confidence, hidden feature marginalization, and uncertain supervision, ensuring models handle ambiguity better than hard labels.
• SLCD evaluates models based on proper scoring rules, distributional divergences, and decision-level actions, making it suitable for hierarchical, relational, and multimodal predictions and ensuring ethical and reliable decision-making.

Soft-Label Learning for Calibrated Decisions (SLCD) is a framework for training and evaluating predictive systems whose outputs are probability distributions intended to support downstream decisions. It replaces or augments one-hot supervision with soft targets, incorporates uncertainty in labels or supervision sources, and evaluates confidence according to calibration and decision utility rather than classification accuracy alone. In its directly named formulation, SLCD trains schema-conditioned models on uncertain decision distributions using soft-label cross-entropy and Brier loss, followed by temperature scaling and confidence-gated delegation to a more expensive reasoning system (Cao et al., 4 Oct 2026). More broadly, SLCD encompasses meta-learned labels, human annotation distributions, label-distribution learning, calibrated pseudo-labeling, decision-aware evaluation, and calibration of structured outcomes such as rankings.

1. Conceptual foundations and scope

Hard labels and soft labels

For an input $x_i$ with $K$ possible classes, a hard label is a one-hot vector,

$\mathbf{y}_i=(0,\ldots,1,\ldots,0),$

where one class is assigned probability one and all alternatives probability zero. A soft label is a probability vector

$$\mathbf{q}_i=(q_{i1},\ldots,q_{iK}), \qquad q_{ik}\geq 0,\qquad \sum_{k=1}^{K}q_{ik}=1.$$

The component $q_{ik}$ represents the available estimate that instance $x_i$ belongs to class $k$. Soft labels may originate from expert confidence scores, annotator disagreement, crowdsourced label frequencies, teacher-model posteriors, hidden-feature marginalization, or uncertain synthetic supervision (Vries et al., 2024).

The distinction is consequential when an instance is ambiguous. A distribution such as $(0.75,0.25)$ preserves evidence for both classes, whereas the corresponding hard label $(1,0)$ records only the plurality class. Similarly, an empirical annotator distribution such as $(0.6,0.4)$ differs materially from its majority-vote reduction $K$0: the former represents disagreement, while the latter implies certainty.

SLCD treats the target distribution as potentially decision-relevant rather than merely as a regularizer. A downstream policy may use the distribution to choose an action, abstain, request additional evidence, or delegate the case to a more expensive model. Consequently, SLCD distinguishes predictive performance from the reliability of the probabilities used in those actions.

The supervision–inference mismatch

Under ordinary cross-entropy with a one-hot label $K$1, the loss is

$K$2

This objective rewards increasing the predicted probability of the observed class toward one. At inference time, however, the maximum predicted probability is often interpreted as a confidence estimate. When examples are ambiguous, classes overlap, annotations are noisy, or inputs are transformed, the one-hot training target can encourage confidence that exceeds the available evidence.

Label-distribution learning addresses this mismatch with the soft-label objective

$K$3

For fixed $K$4, minimizing soft-label cross-entropy is equivalent, up to the constant entropy of the target, to minimizing

$K$5

The target distribution therefore supplies graded supervision for plausible alternatives instead of forcing the model to suppress them completely.

Calibration and decision quality

Calibration concerns the relationship between predicted probabilities and empirical event frequencies. In the binary case, a predictor $K$6 is calibrated when

$K$7

For multiclass soft labels, calibration can instead concern whether the predicted vector matches the conditional distribution of outcomes or annotations. These notions are not interchangeable. A model can have high top-1 accuracy and poor probability calibration, or be calibrated after collapsing information into a constant prediction while losing instance-specific posterior structure (Ushio et al., 27 May 2025).

Decision quality adds another layer. Given actions $K$8 and utility $K$9, a distributional predictor supports the decision rule

$\mathbf{y}_i=(0,\ldots,1,\ldots,0),$0

If probabilities are miscalibrated, thresholds, abstention rules, risk estimates, and cost-sensitive actions can be systematically distorted even when classification accuracy or AUC is unchanged.

2. Target construction and learning methodologies

SLCD includes several methods for constructing or learning soft targets. They differ in whether targets are fixed, generated by teachers, learned through meta-optimization, derived from human judgments, or produced conditionally for transformed inputs.

Fixed and normalized soft targets

Conventional label smoothing replaces a one-hot target with a mixture of the target and a uniform distribution:

$\mathbf{y}_i=(0,\ldots,1,\ldots,0),$1

or, in an alternative convention,

$\mathbf{y}_i=(0,\ldots,1,\ldots,0),$2

Label smoothing discourages extreme probabilities, but it assigns the same mass to every incorrect class and generally applies the same smoothing level to every example. It therefore does not represent instance-specific ambiguity or structured semantic alternatives.

“Normalized Label Distribution: Towards Learning Calibrated, Adaptable and Efficient Activation Maps” proposes normalized label-smoothing cross-entropy, referred to as norm. LSCE, normLS, or NLSCE (Uppal et al., 2020). Its intended role is to scale the smoothed target loss or target-related quantity so that soft supervision improves generalization and confidence behavior without the accuracy losses sometimes associated with ordinary label smoothing. The paper reports accuracy and F1 improvements over cross-entropy and ordinary label smoothing on a skewed stubble dataset and accuracy improvements on CIFAR-10, but does not report standard calibration metrics such as ECE, Brier score, NLL, or reliability diagrams.

Meta-learned labels

“Learning Soft Labels via Meta Learning” treats labels as learnable parameters rather than fixed targets (Vyas et al., 2020). It introduces both class-level labels and instance-level labels.

For class-level labels, a class-specific smoothing parameter $\mathbf{y}_i=(0,\ldots,1,\ldots,0),$3 determines the target:

$\mathbf{y}_i=(0,\ldots,1,\ldots,0),$4

For instance-level labels, each example has a distribution

$\mathbf{y}_i=(0,\ldots,1,\ldots,0),$5

implemented through a softmax parameterization.

The label parameters are updated using a clean meta minibatch distinct from the training minibatch. A one-step model update is first computed using current labels; the look-ahead model is then evaluated on the meta set; and the meta loss is differentiated through the look-ahead update to obtain label gradients. In schematic form,

$\mathbf{y}_i=(0,\ldots,1,\ldots,0),$6

followed by

$\mathbf{y}_i=(0,\ldots,1,\ldots,0),$7

The method uses one-step unrolling rather than differentiating through the entire training trajectory. The learned labels generally apply more smoothing early in training and reduce smoothing as model performance improves. They can also move probability mass away from incorrect annotations and toward plausible alternatives. On CIFAR100 with ResNet18, instance labels improve accuracy from $\mathbf{y}_i=(0,\ldots,1,\ldots,0),$8 to approximately $\mathbf{y}_i=(0,\ldots,1,\ldots,0),$9, while noisy-label experiments report gains over comparison methods at several noise levels.

The method provides evidence for adaptive targets, generalization, semantic structure, and noise correction. It does not directly optimize calibration or decision risk and does not report ECE, reliability diagrams, Brier score, selective prediction, or utility-sensitive outcomes.

Human annotation distributions

Human soft labels are obtained by retaining the empirical distribution of multiple annotators. If annotators provide labels $$\mathbf{q}_i=(q_{i1},\ldots,q_{iK}), \qquad q_{ik}\geq 0,\qquad \sum_{k=1}^{K}q_{ik}=1.$$0, the distribution is

$$\mathbf{q}_i=(q_{i1},\ldots,q_{iK}), \qquad q_{ik}\geq 0,\qquad \sum_{k=1}^{K}q_{ik}=1.$$1

“Soft-Label Training Preserves Epistemic Uncertainty” argues that this empirical distribution should be treated as the target when disagreement reflects legitimate ambiguity rather than annotation error (Singh et al., 18 Nov 2025). Its experiments on ChaosNLI, POPQUORN, and CIFAR-10H-Hard show lower KL divergence from human distributions and stronger correlation between model entropy and annotation entropy than majority-label training, while matching accuracy on two of the three datasets and improving it on ChaosNLI.

“An Assessment of Human vs. Model Uncertainty in Soft-Label Learning and Calibration” separates two effects that are often conflated: correction of an erroneous modal label and preservation of uncertainty after the modal class is fixed (Pavlovic et al., 18 May 2026). Its comparisons among original labels, human majority labels, human soft labels, and synthetic soft labels show that human soft supervision produces its strongest calibration benefits on high-human-label-variation examples. The study reports lower KL divergence and Brier score, lower confidence on difficult examples, and greater alignment between human uncertainty and model training dynamics. It also finds that synthetic soft labels may encode the uncertainty of a generating model without reproducing human perceptual uncertainty.

Online and offline label-distribution learning

“Rethinking Soft Label in Label Distribution Learning Perspective” treats label smoothing, knowledge distillation, ensembles, and label-mixing augmentation as forms of label distribution learning (Hong et al., 2023). Offline LDL generates a fixed teacher distribution for each original input. Online LDL repeatedly transforms an input through mixup, CutMix, or RICAP and generates a new teacher distribution for the transformed input.

For an augmented input $$\mathbf{q}_i=(q_{i1},\ldots,q_{iK}), \qquad q_{ik}\geq 0,\qquad \sum_{k=1}^{K}q_{ik}=1.$$2, the target is generated as

$$\mathbf{q}_i=(q_{i1},\ldots,q_{iK}), \qquad q_{ik}\geq 0,\qquad \sum_{k=1}^{K}q_{ik}=1.$$3

and the student minimizes

$$\mathbf{q}_i=(q_{i1},\ldots,q_{iK}), \qquad q_{ik}\geq 0,\qquad \sum_{k=1}^{K}q_{ik}=1.$$4

The paper reports simultaneous improvements in accuracy and ECE across CIFAR-10, CIFAR-100, STL-10, and ImageNet in many configurations, particularly for large models and long training. It also finds that knowledge-distillation methods can improve accuracy while worsening calibration. The results support input-dependent and augmentation-aware soft targets, but the method remains dependent on teacher quality, augmentation design, model capacity, and training duration.

Soft targets for noisy-label benchmarks

“Generating the Ground Truth: Synthetic Data for Soft Label and Label Noise Research” introduces SYNLABEL, which constructs controlled datasets with clean hard labels, clean soft labels, observed soft labels, and observed hard labels (Vries et al., 2023). A deterministic ground-truth function is applied to complete inputs. Feature Hiding then hides selected features, repeatedly samples their values, evaluates the ground-truth function, and aggregates the resulting labels into a per-instance distribution:

$$\mathbf{q}_i=(q_{i1},\ldots,q_{iK}), \qquad q_{ik}\geq 0,\qquad \sum_{k=1}^{K}q_{ik}=1.$$5

The framework separately represents uncertainty caused by hidden information and noise introduced through NCAR, NAR, NNAR, or annotation-based mechanisms. It quantifies distributional changes using entropy and total variation distance. SYNLABEL is primarily a controlled experimental laboratory rather than a calibration algorithm, but it provides a reference distribution against which soft-label predictors can be evaluated.

Calibrated pseudo-labeling

“Pseudo-label Learning with Calibrated Confidence Using an Energy-based Model” proposes energy-based pseudo-labeling (EBPL) (Toba et al., 2024). A classifier and an energy-based model share a feature extractor. The classifier estimates $$\mathbf{q}_i=(q_{i1},\ldots,q_{iK}), \qquad q_{ik}\geq 0,\qquad \sum_{k=1}^{K}q_{ik}=1.$$6, while the energy-based model represents the marginal input distribution $$\mathbf{q}_i=(q_{i1},\ldots,q_{iK}), \qquad q_{ik}\geq 0,\qquad \sum_{k=1}^{K}q_{ik}=1.$$7. The EBM is trained using SGLD-based likelihood estimation and is coupled to the classifier through Gaussian feature-space parameters.

At each pseudo-labeling step, unlabeled examples are ranked by maximum class confidence. Selected examples receive either hard pseudo-labels or the full posterior as a soft pseudo-label. The procedure recalculates pseudo-labels at every step rather than permanently retaining early assignments. Across seven image datasets, EBPL improves accuracy relative to curriculum-labeling baselines, and its soft variant has the best ECE on all reported datasets. Its limitations include the Gaussian feature-space assumption, computational cost from SGLD, dependence on confidence ranking, and the absence of explicit utility-sensitive or abstention objectives.

3. Hierarchical, structured, and decision-specific soft labels

Soft-label learning is not restricted to flat multiclass classification. SLCD becomes more demanding when outputs are hierarchical, relational, multimodal, or consumed at a particular decision granularity.

Hierarchical conditional prediction

“AI Wizards at EXIST 2026” models multimodal sexism identification in memes as hierarchical conditional soft-label prediction (Fasulo et al., 5 Jul 2026). The system predicts:

$$\mathbf{q}_i=(q_{i1},\ldots,q_{iK}), \qquad q_{ik}\geq 0,\qquad \sum_{k=1}^{K}q_{ik}=1.$$8

then conditional source intention,

$$\mathbf{q}_i=(q_{i1},\ldots,q_{iK}), \qquad q_{ik}\geq 0,\qquad \sum_{k=1}^{K}q_{ik}=1.$$9

and conditional sexism categories,

$q_{ik}$0

The joint downstream probabilities are obtained by multiplying conditional probabilities. For example,

$q_{ik}$1

The model uses frozen Gemini Embedding 2 representations, a lightweight Gated MLP, KL-based soft-label training, conditional masking, and homoscedastic uncertainty weighting. It ranked first on the official Task 2.3 Soft-Soft leaderboard and fourth on Tasks 2.1 and 2.2. However, it does not report ECE, reliability diagrams, Brier score, decision utility, or calibration under distribution shift. Its results demonstrate hierarchical soft-label prediction rather than complete calibrated decision-making.

Hierarchical modeling makes conditional semantics explicit. A downstream probability should not be interpreted as unconditional when it is defined only within a parent class. For multi-label categories, independently predicted category probabilities need not sum to one, and hierarchical coherence must be distinguished from ordinary multiclass calibration.

Probabilistic label ranking

“Calibrated Preference Learning: The Case of Label Ranking” extends calibration to probabilistic distributions over rankings (Thies et al., 28 May 2026). A label-ranking model produces

$q_{ik}$2

where $q_{ik}$3 is the set of all complete rankings of a finite item set. Since $q_{ik}$4, treating every ranking as an unrelated class is computationally and statistically difficult.

The paper distinguishes:

• Full-ranking calibration: calibration of the complete ranking distribution.
• Rankwise calibration: calibration of one ranking probability at a time.
• Sub-ranking calibration: calibration of the relative ordering of a selected subset.
• Top-$q_{ik}$5 calibration: calibration of ordered lists occupying the first $q_{ik}$6 positions.
• Rankwise sub-$q_{ik}$7 and rankwise top-$q_{ik}$8 calibration: weaker event-specific forms.

Full-ranking calibration implies sub-ranking and top-$q_{ik}$9 calibration, but the converse does not generally hold. Sub-ranking and top-$x_i$0 calibration are incomparable because they describe different events. A model calibrated for pairwise comparisons may be poorly calibrated for top-$x_i$1 selection, and calibration of pairwise components does not guarantee calibration after reconstructing a full Plackett–Luce distribution.

This result establishes a general SLCD principle: the calibrated probability object should match the event consumed by the downstream decision. Pairwise preference decisions require pairwise calibration; shortlist selection requires top-$x_i$2 calibration; arbitrary decisions over complete rankings require full-ranking calibration.

Search decisions and selective delegation

“SearchJev: A Fast and Calibrated System-1 Model for Search Agents” explicitly defines SLCD for repeated search decisions (Cao et al., 4 Oct 2026). SearchJev scores legal options directly from a language-model vocabulary-logit vector rather than generating structured text. For a schema field $x_i$3 with legal options $x_i$4,

$x_i$5

The training target is a source-dependent soft distribution $x_i$6, and the per-field loss is

$x_i$7

The implementation uses $x_i$8 and assigns lower weights to LLM-assigned labels, demonstrations, and constructed rewrites. After training, temperatures are fitted separately for Choice, Score, and Noul output types. The calibrated distribution is

$x_i$9

Confidence controls whether the decision is accepted or delegated to System 2:

$k$0

On SearchDecision-Bench, SearchJev reports lower latency and lower ECE than same-size autoregressive baselines in the main in-distribution comparisons. In BrowseComp-Plus, dual-system agents using SearchJev reduce active search time and, for the 4B model, improve reported answer accuracy from $k$1 for System-2-only to $k$2. The OOD calibration results are inconsistent, and the paper does not provide component-isolating ablations for soft targets, the Brier term, source weights, or temperature scaling.

4. Evaluation criteria for calibrated decisions

Accuracy is insufficient

Accuracy evaluates a thresholded decision under one prevalence and one error weighting. AUC evaluates ranking and is invariant under strictly increasing transformations of the score. Neither metric establishes that a probability is numerically meaningful.

Calibration evaluation should therefore include proper scoring rules, distributional divergences, reliability analyses, and decision-level criteria. Common quantities include:

$k$3

$k$4

and

$k$5

ECE compares empirical accuracy and confidence across bins, but it is bin-dependent and usually concentrates on top-class confidence. For soft targets, the empirical reference should reflect the target distribution rather than only the majority class.

“Temperature Scaling Is Not Enough: Calibration Gaps Under Human Label Distributions” shows why the calibration target matters (Dogah, 15 Jul 2026). Temperature scaling fitted on hard labels produces a positive soft-label calibration gap in all nine tested configurations. The gap ranges from $k$6 to $k$7 Brier-score points, with much larger gaps for ChaosNLI than CIFAR-10H. Multiclass isotonic regression exhibits the same qualitative effect. The findings imply that calibration against majority-vote labels can misstate reliability when human disagreement is structural.

Decision-oriented calibration

“Efficient Calibration for Decision Making” defines calibration decision loss relative to a family $k$8 of post-processors (Gopalan et al., 17 Nov 2025):

$k$9

This measures the largest improvement available by post-processing a prediction before applying any proper loss in the specified family. It distinguishes ordinary numerical calibration from actionable miscalibration. A prediction may have a small aggregate calibration error while still admitting a useful structured recalibration, or may have a calibration defect that no allowed decision procedure can exploit.

The paper shows that unrestricted post-processing is computationally difficult, while structured families such as monotone functions or functions with boundedly many intervals can be tractable. Pool Adjacent Violators and uniform-mass binning obtain omniprediction guarantees for certain post-processing classes. For SLCD, the implication is that calibration should be evaluated relative to the post-processing and decision families that deployment actually permits.

Prevalence shift and asymmetric costs

“Aligning Evaluation with Clinical Priorities: Calibration, Label Shift, and Error Costs” develops a bounded, cost-sensitive, prevalence-averaged log score (Flores et al., 17 Jun 2025). Under label shift,

$(0.75,0.25)$0

while class prevalence changes from $(0.75,0.25)$1 to $(0.75,0.25)$2. A calibrated score can be adjusted through prior odds:

$(0.75,0.25)$3

The paper averages decision performance over a clinically specified prevalence range, preferably uniformly in log odds, and incorporates asymmetric costs. Its framework connects proper scoring rules to thresholded decisions through the Schervish representation.

For soft targets, a distributional version of a cost-sensitive loss can replace the hard outcome with its expectation:

$(0.75,0.25)$4

This provides a route from soft-label training to prevalence- and cost-aware evaluation. The paper is primarily an evaluation framework rather than a complete soft-label training algorithm, and its empirical analysis is limited to an illustrative eICU study.

Bayes-error estimation and the limits of calibration

“Practical estimation of the optimal classification error with soft labels and calibration” studies instance-free estimation of the binary Bayes error (Ushio et al., 27 May 2025). With clean posterior values $(0.75,0.25)$5, the estimator is

$(0.75,0.25)$6

The paper shows that a perfectly calibrated score can still be unsuitable for estimating Bayes error. A constant score equal to the class prevalence is perfectly calibrated but can yield an estimated error of $(0.75,0.25)$7 even when the Bayes error is zero. Thus, calibration does not guarantee preservation of instance-specific posterior structure.

When corrupted soft labels preserve posterior ordering, isotonic calibration provides a statistically consistent estimator. This result is relevant to SLCD because it distinguishes reliability conditional on a score from preservation of the information needed for aggregate decision limits.

5. Empirical evidence and applications

Generalization, noise correction, and distillation

Meta-learned labels improve classification accuracy across CIFAR-10 and CIFAR-100 architectures and correct some synthetic annotation errors. Instance-level labels can encode semantic relationships, such as assigning mass to pear or orange for an apple image. The learned labels also improve teacher models for knowledge distillation, with learned-label teachers performing comparably to teacher predictions in reported CIFAR-100 experiments (Vyas et al., 2020).

The evidence is strongest for generalization, synthetic-noise robustness, semantic target structure, and distillation. It is not direct evidence of deployment calibration because the paper does not report ECE, Brier score, reliability diagrams, selective risk, or cost-sensitive utility.

Normalized soft labels

Normalized soft-target training improves accuracy over one-hot cross-entropy and ordinary label smoothing on the reported stubble dataset and CIFAR-10 experiments (Uppal et al., 2020). The strongest reported stubble result is MobileNetV2 with $(0.75,0.25)$8 accuracy and $(0.75,0.25)$9 F1 under normalized soft targets. The paper presents these outcomes as evidence that soft-label regularization can improve accuracy and the qualitative behavior associated with calibration.

The calibration evidence remains indirect. No standard calibration metrics, formal adversarial evaluation, or OOD uncertainty study is reported, and the normalization equation is not sufficiently unambiguous for exact reproduction.

Label-distribution learning

Online LDL frequently improves both accuracy and ECE relative to hard-label training and several knowledge-distillation baselines (Hong et al., 2023). Reported results include lower ECE on CIFAR-10, CIFAR-100, STL-10, and ImageNet, with improvements often most pronounced for large models, long training schedules, and augmentation methods such as CutMix or RICAP.

The study also reports cases in which temperature scaling obtains lower ECE in a particular configuration, while LDL improves accuracy as well as calibration. It emphasizes that no single calibration metric is sufficient: MC-dropout LDL can reduce ECE while producing poor NLL and Brier values.

Human uncertainty alignment

Human soft-label training improves distributional alignment on ambiguous examples. On CIFAR-10H-Hard, ChaosNLI, and POPQUORN, “Soft-Label Training Preserves Epistemic Uncertainty” reports a 32% lower KL divergence from human annotations on average and a 61% stronger correlation between model entropy and annotation entropy (Singh et al., 18 Nov 2025). The principal improvement occurs in the model’s behavior on examples with substantial human disagreement.

The controlled audit in (Pavlovic et al., 18 May 2026) further shows that human soft labels reduce KL divergence and Brier score relative to both original hard labels and newly collected human majority labels. This indicates that improvements cannot be attributed solely to correcting modal labels; retaining the distribution itself contributes additional information.

Semi-supervised learning and search agents

EBPL demonstrates the value of calibrated confidence in pseudo-label selection, reporting lower ECE and improved recognition relative to curriculum-labeling baselines (Toba et al., 2024). SearchJev applies the same general principle to search control: soft targets and proper scoring losses are combined with temperature scaling, and calibrated confidence determines whether a short decision is accepted or delegated (Cao et al., 4 Oct 2026).

These applications differ in objective. EBPL uses calibration to improve pseudo-label selection, whereas SearchJev uses calibration to allocate computation between System 1 and System 2. In both cases, confidence is operational rather than merely descriptive.

Clinical decision support and preference systems

The clinical evaluation framework in (Flores et al., 17 Jun 2025) emphasizes that accuracy and AUC can conceal prevalence-dependent and cost-sensitive failures. The ranking-calibration framework in (Thies et al., 28 May 2026) similarly shows that full-ranking, pairwise, sub-ranking, and top-$(1,0)$0 calibration correspond to different downstream decisions. Together, these results extend SLCD beyond flat classification: a calibrated output must be evaluated at the level at which it will be consumed.

6. Limitations, controversies, and future directions

Soft targets are not automatically true probabilities

A learned target distribution describes what supervision supports or what improves optimization. It is not necessarily the true posterior $(1,0)$1, a Bayesian posterior over parameters, a probability of decision success, or an estimate of OOD uncertainty. Semantic similarity, annotator disagreement, model uncertainty, and probability of correctness are distinct concepts.

Human annotation distributions may encode legitimate pluralism, annotator noise, cultural differences, instruction effects, or sampling bias. Synthetic distributions may reflect a teacher or generator rather than human uncertainty. A target can be soft without being calibrated for the intended deployment population.

Meta-set and teacher dependence

Meta-learned labels require a clean and representative meta set. If it is biased, small, or noisy, the labels may optimize the wrong objective. Teacher-based LDL and distillation depend on teacher calibration and representation quality. EBPL depends on the EBM specification, SGLD quality, and the assumption that learned feature-space density is informative for confidence.

Calibration is target- and decision-dependent

Calibration against hard labels, human distributions, ranking events, clinical prevalence ranges, or search decisions answers different questions. Temperature scaling fitted on one-hot labels need not calibrate a human distribution. Pairwise calibration does not imply top-$(1,0)$2 calibration. Global calibration does not guarantee utility-optimal decisions under asymmetric costs or distribution shift.

A credible SLCD evaluation should therefore report the target distribution, the calibration object, the decision family, and the deployment assumptions explicitly. It should combine accuracy or ranking metrics with proper scoring rules, distributional divergence, calibration diagnostics, and decision-level outcomes.

Computational and statistical limitations

Instance-level soft labels introduce additional degrees of freedom and can overfit a meta set. One-step bilevel approximations provide only local estimates of label effects. Full ranking distributions scale factorially with the number of items. ECE depends on binning and can become misleading for sparse probabilities. Isotonic calibration requires order-preserving corruption for the stated Bayes-error guarantee. Energy-based calibration introduces sampling cost, while direct decision models such as SearchJev require enumerable legal option sets.

Directions for SLCD research

Prominent directions include:

• Distribution-aware calibration: fit temperature, isotonic, vector, matrix, or Dirichlet calibrators directly to soft targets rather than one-hot labels.
• Decision-aware objectives: optimize expected utility, asymmetric costs, abstention, selective risk, and risk-coverage performance.
• Hierarchical calibration: calibrate conditional probabilities and verify calibration after hierarchical recombination.
• Structured calibration: evaluate pairwise, sub-ranking, top-$(1,0)$3, and full-ranking probabilities according to the downstream action.
• Robust soft-label acquisition: model annotator identity, expertise, subgroup variation, and correlations among annotators.
• Shift-aware calibration: account for prevalence shift, label shift, domain shift, and changes in the annotation population.
• Few-annotation methods: estimate per-instance uncertainty from limited labels through shrinkage, hierarchical models, or pooled calibration.
• Decision-oriented auditing: use calibration decision loss relative to structured post-processing families rather than relying exclusively on unrestricted ECE.
• Explicit component ablations: separately measure the effects of soft targets, Brier terms, source weights, meta-learning, temperature scaling, and delegation.
• Multiclass and structured theory: extend binary decision-oriented calibration guarantees to simplex-valued, hierarchical, multilabel, and ranking distributions.

SLCD is therefore best understood as a family of methods and evaluation principles rather than a single universally specified algorithm. Its central requirement is that uncertain supervision, predictive distributions, calibration procedures, and downstream actions refer to the same uncertainty structure. Soft labels can improve regularization, preserve human disagreement, correct annotation errors, and support selective computation, but they produce calibrated decisions only when their target semantics and evaluation criteria are aligned with the decisions the system is intended to make.