Uncertain Similarity & Unlabeled Learning (USimUL)

• USimUL is a weakly supervised framework that replaces precise labels with uncertain relational information and unlabeled data to estimate classification risk.
• It employs unbiased risk estimation techniques by reformulating supervised risk through uncertain similarity triplets, pairwise confidence, and mixture-based formulations.
• USimUL enhances privacy and reduces label dependency while enabling iterative self-refinement and robust learning even in noisy, privacy-sensitive scenarios.

Uncertain Similarity and Unlabeled Learning (USimUL) denotes a family of weakly supervised learning settings in which direct instance labels are unavailable or severely limited, while supervision is supplied by uncertain relational information and unlabeled data. In the explicit USimUL formulation, the learner receives uncertain similarity triplets $(x,\{x',x''\})$ such that two of the three instances share the same class label but it is unknown which two, together with unlabeled samples, and learns by constructing an unbiased estimator of the ordinary classification risk (Wei et al., 15 Sep 2025). More broadly, recent work uses the same conceptual lens for pairwise similarity-confidence learning, two-unlabeled-mixture learning, bag-level proportion constraints with uncertain features, and LLM pseudo-label refinement, all of which replace trusted instance labels by uncertain similarity structure plus unlabeled data (Cao et al., 2021, Asano et al., 18 Feb 2025, Xiao et al., 2019).

1. Scope and defining settings

Across the literature, USimUL is not a single formalism but a cluster of related formulations in which supervision is indirect, relational, or aggregate. The common principle is that the learner does not trust pointwise labels and instead reconstructs class information from pairwise similarity, confidence scores, unlabeled mixtures, bag-level proportions, or shifted unlabeled distributions.

Supervision form	Core object	Representative paper
Uncertain similarity triplets	$(x,\{x',x''\})$ with two of three same-class, pair unknown	"Learning from Uncertain Similarity and Unlabeled Data" (Wei et al., 15 Sep 2025)
Similarity-confidence pairs	$(\mathbf{x},\mathbf{x}'), s$, with $s=p(y=y'\mid \mathbf{x},\mathbf{x}')$	"Learning from Similarity-Confidence Data" (Cao et al., 2021)
Two unlabeled mixtures	$$\widetilde{\mathcal{C}}_{\mathrm{p}}, \widetilde{\mathcal{C}}_{\mathrm{n}}$$ with different positive ratios	"Self Iterative Label Refinement via Robust Unlabeled Learning" (Asano et al., 18 Feb 2025)
Bags with proportions and uncertain features	Bags $B_I$, proportions $P_I$, perturbations $\Delta \mathbf{x}$	"Transfer Learning-Based Label Proportions Method with Data of Uncertainty" (Xiao et al., 2019)
Positive and imperfect unlabeled data	positives from truncation, unlabeled data under shift	"Learning with Positive and Imperfect Unlabeled Data" (Lee et al., 14 Apr 2025)

The explicit privacy motivation appears most clearly in the triplet-based USimUL formulation. Standard similarity-based weak supervision can reveal exact relational label information: if a pair is known to be similar and one label is later exposed, the other label is determined. USimUL weakens this by replacing a precise pairwise statement with the weaker condition that, in a triplet, two out of three share the same class but it is unknown which two, so disclosure of one label does not deterministically reveal the others (Wei et al., 15 Sep 2025).

A distinct but closely related line replaces similarity labels by relational uncertainty comparisons. In active classification with uncertainty comparison queries, uncertainty is defined as $u(x)=|\eta(x)-0.5|$, where $\eta(x)=p(Y=+1\mid X=x)$, and a noisy oracle answers which of two points is more uncertain. This does not identify labels directly, but it localizes the decision boundary and can be combined with positivity comparisons to label an unlabeled pool without explicit labeling queries (Cui et al., 2020).

2. Mathematical formulations

The common target is the standard binary classification risk

$(x,\{x',x''\})$0

or its class-conditional decomposition

$(x,\{x',x''\})$1

with $(x,\{x',x''\})$2 and $(x,\{x',x''\})$3. USimUL methods differ in how they rewrite this supervised risk using only weak signals.

In similarity-confidence learning, the basic observation is a pairwise probability

$(x,\{x',x''\})$4

with unlabeled pairs drawn from the product marginal $(x,\{x',x''\})$5. The empirical risk can then be written as a sum of two unbiased components,

$(x,\{x',x''\})$6

where the weights depend on $(x,\{x',x''\})$7 and the class priors, yielding an unbiased estimator of the ordinary classification risk from pairwise confidence alone (Cao et al., 2021).

In UU learning, the supervision is not pairwise but mixture-based. Two corpora

$(x,\{x',x''\})$8

are sampled from

$(x,\{x',x''\})$9

with $(\mathbf{x},\mathbf{x}'), s$0. Lu et al. showed that a linear combination of risks over these two unlabeled mixtures yields an unbiased estimator of the supervised PN risk, and supervised PN learning appears as the special case $(\mathbf{x},\mathbf{x}'), s$1 (Asano et al., 18 Feb 2025).

The explicit USimUL triplet model starts from uncertain similarity triplets $(\mathbf{x},\mathbf{x}'), s$2 satisfying the event that either $(\mathbf{x},\mathbf{x}'), s$3 or $(\mathbf{x},\mathbf{x}'), s$4 is same-class. This induces the marginal

$(\mathbf{x},\mathbf{x}'), s$5

and the supervised risk can be rewritten as

$(\mathbf{x},\mathbf{x}'), s$6

for suitable coefficients $(\mathbf{x},\mathbf{x}'), s$7 inside $(\mathbf{x},\mathbf{x}'), s$8 and $(\mathbf{x},\mathbf{x}'), s$9. The resulting empirical estimator is unbiased for the true classification risk, despite the fact that no exact pairwise similarity and no pointwise labels are observed (Wei et al., 15 Sep 2025).

A different route replaces instance labels by bag-level proportions. In learning with label proportions, a bag $s=p(y=y'\mid \mathbf{x},\mathbf{x}')$0 carries only

$s=p(y=y'\mid \mathbf{x},\mathbf{x}')$1

and the model enforces that the bag-average prediction matches a transformed proportion $s=p(y=y'\mid \mathbf{x},\mathbf{x}')$2. In the transfer-learning variant, both source and target bags contain uncertain data $s=p(y=y'\mid \mathbf{x},\mathbf{x}')$3 with $s=p(y=y'\mid \mathbf{x},\mathbf{x}')$4, so the uncertainty lies in feature geometry rather than in pairwise annotations (Xiao et al., 2019).

This suggests that USimUL is best viewed as a risk-reconstruction program: the weak signal may be pairwise probabilistic similarity, uncertain triplet structure, group proportions, or mixture proportions, but the mathematical objective remains the same—rewrite supervised risk in terms of expectations under observables.

3. Unbiased risk estimation and risk correction

Unbiasedness is the central statistical device in this literature. SU classification, Sconf learning, SconfConfDiff learning, and triplet-based USimUL all derive empirical objectives whose expectation equals the ordinary supervised risk. For SU classification, the estimator built from similar pairs and unlabeled points achieves the optimal parametric convergence rate, with excess risk controlled by

$s=p(y=y'\mid \mathbf{x},\mathbf{x}')$5

under standard Rademacher-complexity assumptions (Bao et al., 2018). For Sconf learning, the empirical minimizer under the unbiased similarity-confidence risk satisfies

$s=p(y=y'\mid \mathbf{x},\mathbf{x}')$6

and analogous optimal-rate bounds hold for the corrected estimator as well (Cao et al., 2021). The explicit USimUL triplet estimator likewise attains a parametric $s=p(y=y'\mid \mathbf{x},\mathbf{x}')$7 rate under boundedness and Rademacher assumptions (Wei et al., 15 Sep 2025). The same pattern reappears when similarity-confidence is combined with confidence-difference: both the convex-combination estimator and the interaction-based estimator achieve optimal convergence rates, and the symmetric $s=p(y=y'\mid \mathbf{x},\mathbf{x}')$8 form minimizes variance in the interaction-based family (Tate et al., 7 Aug 2025).

A recurring obstacle is that unbiased estimators often contain negative coefficients. In Sconf learning, the components $s=p(y=y'\mid \mathbf{x},\mathbf{x}')$9 and $$\widetilde{\mathcal{C}}_{\mathrm{p}}, \widetilde{\mathcal{C}}_{\mathrm{n}}$$0 can become negative, and this empirically correlates with severe overfitting in high-capacity models. The standard remedy is risk correction via a function

$$\widetilde{\mathcal{C}}_{\mathrm{p}}, \widetilde{\mathcal{C}}_{\mathrm{n}}$$1

yielding non-negative correction ($$\widetilde{\mathcal{C}}_{\mathrm{p}}, \widetilde{\mathcal{C}}_{\mathrm{n}}$$2) or ABS correction ($$\widetilde{\mathcal{C}}_{\mathrm{p}}, \widetilde{\mathcal{C}}_{\mathrm{n}}$$3); the bias introduced by correction decays exponentially in $$\widetilde{\mathcal{C}}_{\mathrm{p}}, \widetilde{\mathcal{C}}_{\mathrm{n}}$$4 (Cao et al., 2021). The same logic drives later work. Robust UU replaces negative normalized risks by a generalized leaky ReLU

$$\widetilde{\mathcal{C}}_{\mathrm{p}}, \widetilde{\mathcal{C}}_{\mathrm{n}}$$5

and the self-refinement paper fixes $$\widetilde{\mathcal{C}}_{\mathrm{p}}, \widetilde{\mathcal{C}}_{\mathrm{n}}$$6 to suppress negative-risk overfitting under noisy LLM pseudo-labels (Asano et al., 18 Feb 2025). SconfConfDiff learning uses ReLU- and ABS-corrected variants of the interaction-based estimator, while the triplet-based USimUL paper applies either $$\widetilde{\mathcal{C}}_{\mathrm{p}}, \widetilde{\mathcal{C}}_{\mathrm{n}}$$7 or $$\widetilde{\mathcal{C}}_{\mathrm{p}}, \widetilde{\mathcal{C}}_{\mathrm{n}}$$8, with the absolute-value correction reported as the stronger practical choice (Tate et al., 7 Aug 2025, Wei et al., 15 Sep 2025).

The methodological consensus is therefore narrow but clear: unbiased risk estimators make weak supervision statistically usable, but practical training usually requires a correction layer that prevents empirical risk from becoming artificially negative.

4. Algorithmic patterns and representative systems

One prominent algorithmic pattern is iterative self-refinement from uncertain pseudo-labels. In robust unlabeled learning for LLMs, the pipeline is: use an LLM to create uncertain pseudo-labels on an unlabeled corpus; derive two pseudo-unlabeled sets with different positive class ratios; train a robust UU classifier on these noisy corpora; then re-label the entire corpus and repeat. In the few-labeled variant, only 50 labeled examples are used to estimate $$\widetilde{\mathcal{C}}_{\mathrm{p}}, \widetilde{\mathcal{C}}_{\mathrm{n}}$$9 and $B_I$0; the classifier then alternates between robust UU training and corpus-wide re-labeling (Asano et al., 18 Feb 2025).

A second pattern is alternating optimization under feature uncertainty. In TL-LLP, two LLP tasks are tied by

$B_I$1

while each instance is uncertain within a bounded reachability area

$B_I$2

Training alternates between solving a convex QP for shared and task-specific parameters and updating $B_I$3 so that uncertain inputs move within their admissible balls in directions that reduce bag-level LLP error (Xiao et al., 2019). This suggests a USimUL interpretation in which similarity itself is not fixed but changes within a constrained geometric neighborhood.

A third pattern models uncertainty directly in learned representations. In uncertainty-aware few-shot classification, each query–prototype similarity is treated as a Gaussian random variable

$B_I$4

with $B_I$5 and $B_I$6 predicted by a graph-based uncertainty estimator over all query–prototype pairs of an episode; training then optimizes a Monte Carlo approximation of the expected softmax probability under these similarity distributions (Zhang et al., 2020). In SUMix, standard geometric mix ratios are replaced by a semantic ratio

$B_I$7

and uncertainty is introduced through an MLP-based scalar and the regularization weight

$B_I$8

which downweights semantically mismatched or uncertain mixed samples (Qin et al., 2024). In semi-supervised learning, a related uncertainty-aware objective combines aleatoric Gaussian pseudo-label likelihoods with epistemic orthogonal certificates, using

$B_I$9

to modulate how strongly unlabeled data is trusted (Kazemi, 2022).

These systems broaden USimUL beyond pairwise weak supervision. The same core idea—treat similarity or label evidence as uncertain, quantify that uncertainty, and feed it into the optimization objective—appears in LLM self-training, few-shot metric learning, mixup, and semi-supervised consistency regularization.

5. Empirical regimes and applications

The empirical literature consistently tests USimUL-like methods in domains where direct labels are scarce, expensive, noisy, or privacy-sensitive. In iterative robust UU refinement, six binary tasks are used: Fake News, Saroco, Safety, Corona Sentiment, Green Patent, and Protein Structure. Data are split 70%/10%/20% for train/validation/test, evaluation is held-out accuracy over 3 seeds, and the method shows monotonic or steady improvements over 5 iterations on the easier tasks while consistently improving across iterations on the harder tasks. The few-labeled variant, using only 50 labeled examples for prior estimation, initially lags Oracle but converges toward near-oracle performance; the method surpasses the initial GPT-4o or GPT-4o-mini pseudo-label quality and DeepSeek-R1 on the harder domains (Asano et al., 18 Feb 2025).

In LLP with uncertain data, TL-LLP is evaluated against MeanMap, InvCal, alter-$P_I$0SVM, and conv-$P_I$1SVM across nine datasets and bag sizes from 2 to 64. When synthetic Gaussian noise is added to features, with noise percentage increasing from 0% to 32%, all methods degrade, but TL-LLP consistently yields higher accuracy across all datasets. This is presented as empirical evidence that modeling reachability areas and iteratively adjusting $P_I$2 makes LLP less sensitive to feature noise (Xiao et al., 2019).

Pairwise-probabilistic approaches show a similar pattern. Sconf learning is evaluated on synthetic Gaussian mixtures and on binary-corrupted versions of MNIST, Fashion-MNIST, Kuzushiji-MNIST, EMNIST, CIFAR-10, and SVHN. Sconf-ABS and Sconf-NN outperform SD, Siamese, and contrastive baselines on all reported benchmarks, while the uncorrected unbiased estimator often overfits in deep settings (Cao et al., 2021). When similarity-confidence is combined with confidence-difference, the interaction-based estimator and its ABS-corrected variant achieve the strongest results across MNIST, Kuzushiji-MNIST, Fashion-MNIST, CIFAR-10, and several UCI datasets; the paper reports gains over the second-best method of $P_I$3 on Pendigits, $P_I$4 on Lost, $P_I$5 on MSRCv2, and $P_I$6 on BirdSong (Tate et al., 7 Aug 2025).

The explicit USimUL triplet framework is evaluated on benchmark datasets such as MNIST, Fashion-MNIST, Kuzushiji-MNIST, CIFAR-10, and SVHN, on weakly supervised datasets including Pendigits, Lost, MSRCv2, and BirdSong, and on privacy-sensitive medical datasets DDSM, PDMD, and PDSD. It consistently achieves the best accuracy across several class priors on the benchmark sets, improves over the second-best method by the margins above on the WSL datasets, and attains the strongest results on PDMD and PDSD at $P_I$7 while remaining competitive on DDSM (Wei et al., 15 Sep 2025).

Human-facing uncertainty supervision is feasible as well. In active classification with uncertainty comparison queries, experiments on Kuzushiji characters and car-preference data show that annotators can provide both positivity and uncertainty comparisons, and that these pairwise signals can replace explicit labels inside a boundary-localization and label-propagation pipeline (Cui et al., 2020). This suggests that USimUL is not only a statistical abstraction but also an annotation design strategy.

6. Assumptions, failure modes, and open directions

A persistent misconception is that any weak similarity signal can be turned into a classifier by simple sorting or self-training. The literature is more restrictive. Positivity comparison alone cannot recover labels without threshold information, because ranking by $P_I$8 does not reveal where $P_I$9 lies (Cui et al., 2020). UU learning requires two mixtures with sufficiently different class ratios; if LLM pseudo-labels are nearly random and $\Delta \mathbf{x}$0, the method breaks down (Asano et al., 18 Feb 2025). In Sconf learning, the factor $\Delta \mathbf{x}$1 in the estimation bounds implies that the problem becomes ill-conditioned as the class priors approach balance (Cao et al., 2021).

Class-prior estimation and distribution shift remain the dominant structural bottlenecks. Few-labeled prior estimation from 50 examples can fail when labeled samples are unrepresentative (Asano et al., 18 Feb 2025). The triplet-based USimUL estimator assumes known class priors and provides only a qualitative privacy argument rather than a formal differential-privacy guarantee (Wei et al., 15 Sep 2025). PIU learning addresses a more severe version of the same issue by allowing the unlabeled distribution to be shifted relative to the true feature distribution. Under generalized smoothness,

$\Delta \mathbf{x}$2

the sample complexity scales with a mismatch penalty $\Delta \mathbf{x}$3, and Denis’s classical reduction of PU learning to Massart noise fails under even slight shifts (Lee et al., 14 Apr 2025). This suggests that future USimUL theory will need to treat uncertainty in similarity and uncertainty in unlabeled-domain fidelity within one framework rather than as separate pathologies.

The scope of current methods is also narrow. Many formulations are binary only, including UU, Sconf, SU, triplet USimUL, and PIU. The LLM self-refinement framework explicitly identifies multi-class extensions, ranking, and RLHF/RLAIF post-training as future work (Asano et al., 18 Feb 2025). SconfConfDiff points to missing weak labels and joint use of supervised labels plus relational weak labels as natural next steps (Tate et al., 7 Aug 2025). SUMix notes that semi-supervised tasks could benefit from “more accurate pseudo-labels,” but does not itself provide an unlabeled-learning pipeline (Qin et al., 2024).

Taken together, these limitations define the current frontier of USimUL. The field has established that uncertain relational supervision can support unbiased risk estimation and optimal statistical rates. The unresolved problem is unification: combining privacy-preserving uncertain similarity, robust prior estimation, multi-class structure, and imperfect unlabeled distributions without losing either computational tractability or theoretical guarantees.