---
title: Positive-Unlabeled Contrastive Learning (PUCL)
url: https://www.emergentmind.com/topics/positive-unlabeled-contrastive-learning-pucl
type: topic
---

# Positive-Unlabeled Contrastive Learning (PUCL)

Positive-Unlabeled Contrastive Learning (PUCL) is a framework for self-supervised, semi-supervised, and weakly supervised representation learning in scenarios where only a subset of positive samples are labeled while the remaining dataset consists of unlabeled samples, which may contain both positive and negative instances. PUCL corrects for the negative-sampling bias inherent in standard contrastive learning with unlabeled negatives, enabling improved representation quality and downstream classification—especially under label scarcity, class imbalance, or noise conditions.

## 1. Motivation and Standard Contrastive Learning Bias

Contrastive learning algorithms such as SimCLR and MoCo typically train an encoder $f(\cdot)$ that projects data $x$ into normalized embeddings $z = f(x) \in \mathbb{S}^{d-1}$. Each anchor sample $x$ is paired with a positive sample $x^+$ (an augmentation), with similarity measured by $h(x, x') = \exp(z^\top z')$. Negatives are sampled from the remainder of the dataset, and the standard objective uses a denominator including these “negative” samples.

However, in realistic settings where negative labels are unavailable, negatives are drawn uniformly from the dataset (excluding $x$ and $x^+$). This introduces “negative-sampling bias” because the negative set is actually a positive-unlabeled (PU) mixture: it includes unlabeled positives. This contamination results in a biased contrastive loss, which can degrade the learned representations and downstream classifier accuracy [2401.08690].

## 2. Formalization: PUCL Loss and Estimator

PUCL treats the negative sampling problem as a PU learning problem. Under the SCAR (Selected-Completely-At-Random) assumption with positive prior $\alpha = \Pr(y=1)$ and labeling frequency $c$, the true negative distribution can be written as:

$$
p^-_x(x') = \frac{1-\alpha c}{1-\alpha} p^u_x(x') - \frac{\alpha(1-c)}{1-\alpha} p^+_x(x')
$$

where $p^u_x$ is the distribution of unlabeled samples (mixture) and $p^+_x$ is the distribution of positives (augmentations).

The debiased loss, denoted $L_{DeCL}$, replaces the noisy negative sum in the denominator with its expectation under the corrected negative distribution:

$$
L_{DeCL} = \mathbb{E}_{x, x^+}\left[-\log \frac{h(x, x^+)}{h(x, x^+) + N\mu_x}\right]
$$

with the corrected $\mu_x$ defined as

$$
\mu_x = \frac{1-\alpha c}{1-\alpha} \mathbb{E}_{x^u \sim p^u_x}[h(x, x^u)] - \frac{\alpha(1-c)}{1-\alpha} \mathbb{E}_{x^+ \sim p^+_x}[h(x, x^+)]
$$

In practice, Monte Carlo estimates of the expectations are used:

$$
\hat{\mu}_x = \max\Bigg\{ \frac{1-\alpha c}{1-\alpha}\frac{1}{M^u}\sum_{i=1}^{M^u} h(x, x^u_i) - \frac{\alpha(1-c)}{1-\alpha}\frac{1}{M^+}\sum_{j=1}^{M^+} h(x, x^+_j),\, e^{-1} \Bigg\}
$$

and the training objective becomes

$$
L_{PUCL} = \mathbb{E}_{x, x^+}\left[ -\log \frac{h(x, x^+)}{h(x, x^+) + N \hat{\mu}_x} \right]
$$

The lower clamp at $e^{-1}$ ensures numerical stability [2401.08690].

## 3. PU-Aware Contrastive Formulations and Extensions

Several variants and extensions of PUCL address different regimes and make varying assumptions about the availability of class prior $\pi$ or leverage explicit pseudo-labeling:

### 3.1 Unbiased Loss Without Known Priors

When the class prior is unknown, frameworks such as puCL [2402.06038] employ minibatch structures such that:

- For labeled positives, anchor views are pulled toward other labeled positives.
- For unlabeled anchors, each is pulled only toward its own augmentation.

The objective achieves unbiasedness and reduces variance compared to standard InfoNCE. Concretely, in a batch of $2b$ views:

$$
\mathcal{L}_{puCL} = -\frac{1}{2b} \sum_{i=1}^{2b} \Bigg[ 1_{x_i \in P} \frac{1}{|P \setminus \{i\}|} \sum_{j \in P \setminus \{i\}} \log P_{ij} + 1_{x_i \in U} \log P_{i,a(i)} \Bigg]
$$

where $P_{ij} = \exp(z_i \cdot z_j) / \sum_{k \neq i} \exp(z_i \cdot z_k)$ and $a(i)$ indexes the augmented pair [2402.06038].

### 3.2 Prior-Aware InfoNCE (puNCE)

If the positive prior $\pi$ is available or estimated, losses such as puNCE softly weight each unlabeled anchor’s positive and negative contributions:

- Each unlabeled anchor is treated as positive with weight $\pi$ and negative with $1-\pi$.
- The loss for unlabeled anchors interpolates between being a positive and a negative anchor by mixing the contributions accordingly [2206.01206][2402.06038].

### 3.3 Uncertainty-Weighted PUCL

To address highly noisy and imbalanced PU scenarios (e.g., cybersecurity, biomedical text), the Uncertainty Contrastive Framework (UCF) [2512.08969] incorporates:

- A per-sample uncertainty/confidence estimator
- Reweighting of each pair in the contrastive loss by uncertainty
- Positive-only anchor batches to stabilize gradients
- Adaptive, batch-level temperature scaling to further stabilize training

The resultant loss is:

$$
L^{PU} = \frac{1}{R} \sum_{i=1}^{R} \bigg[ I(x_i)\frac{1}{|B_1(x_i)|}\sum_{x_p \in B_1(x_i)} \ell(z_i, z_p) + (1 - I(x_i))\frac{1}{|B_0(x_i)|}\sum_{x_n \in B_0(x_i)} \ell(z_i, z_n) \bigg] \cdot u_i
$$

where $u_i$ is the per-sample uncertainty, $I$ is an (estimated) positive indicator, and $\ell$ is the standard InfoNCE kernel [2512.08969].

## 4. Algorithmic Implementation and Protocols

The general PUCL procedure can be implemented as a wrapper around standard contrastive pipelines:

- For each anchor, sample a positive view and multiple unlabeled samples.
- Estimate positive prior $\alpha$ (and $c$, if using the SCAR-based loss).
- Compute the correction term $\hat{\mu}_x$ and clamp it for stability.
- Standard augmentations, backbone choices, and big-batch protocols are used, e.g., SimCLR with batch size 512 ($N \approx 510$), MoCo with N=4096 [2401.08690].
- For UCF, dynamically sample batches containing both positives and unlabeled, use uncertainty-based weighting, positive anchors, and adaptive temperature [2512.08969].

Implementation is computationally efficient; the dominant costs are embedding evaluation and a scalar correction per anchor [2401.08690]. Hyperparameter sensitivity is low for the correction, with optimal positive prior typically matching the true class ratio.

## 5. Theoretical Analysis

PUCL algorithms provide the following guarantees:

- Negligible additional bias compared to the ideal contrastive loss; the deviation is upper bounded by $1/[2\sqrt{N}(e^2-1)]$ for population mean replacement [2401.08690].
- Unbiasedness for the “fully-labeled” contrastive objective in both puCL and puNCE when their assumptions hold [2402.06038][2206.01206].
- Variance and gradient-sampling bias are strictly smaller for puCL than for standard self-supervised contrastive losses, especially as labeled positive fraction increases [2402.06038].
- For downstream classification, theoretical results link cluster alignment, augmentation concentration, and linear probe error; optimizing the (PU) contrastive objective provably aligns clusters for k-means and enables vanishing classification error under natural conditions [2402.06038].
- Use of soft pseudo-labels (via weighting, not hard clustering) in puNCE recovers the expected supervised contrastive gradient in the asymptotic regime [2206.01206].

## 6. Empirical Results and Ablations

Extensive evaluation demonstrates that PUCL yields improvements across vision (CIFAR10/100, STL10, FMNIST), text (SST-2), and graph (InfoGraph) benchmarks:

- On CIFAR10, SimCLR with PUCL improves Top-1 accuracy from 90.44% (baseline) to 92.14%; similar gains hold for CMC and MoCo backbones [2401.08690].
- On graphs, PUCL consistently yields gains of +3–6 ppt over base InfoGraph, and +2–6 ppt versus strong baselines [2401.08690].
- UCF achieves over 93% accuracy and near-perfect recall in high-stakes malicious content detection, even with severe class imbalance [2512.08969].
- puNCE achieves dramatic downstream PU classification improvements, e.g., on PU-CIFAR10 achieving 97.6% with linear probe (vs. 88–92% for state-of-the-art baselines at $n_p=1k$) [2206.01206].
- Label and class prior ablations demonstrate robustness; small deviations have only minor effects on accuracy [2401.08690], and stable performance persists in the low-supervision regime [2206.01206][2402.06038].
- Empirical embedding visualizations (e.g., UMAP, t-SNE) confirm tighter clustering of positives and greater separation from negatives in PUCL-trained models [2401.08690][2512.08969].

## 7. Practical Considerations, Extensions, and Limitations

- PUCL is effective when a significant proportion of negatives are contaminated by unlabeled positives or under class imbalance.
- The main requirement is an estimate of positive class prior; performance is robust to moderate errors in estimation.
- Minimal architectural changes are needed: PUCL can be added as a wrapper to existing contrastive pipelines with low computational or sample overhead.
- Extensions such as UCF address highly noisy PU scenarios by including adaptivity (uncertainty, temperature, positive anchors) [2512.08969].
- PUCL methods are currently focused on binary PU settings but show promise for multi-modal, multi-class, and cross-domain applications [2512.08969]. Further theoretical analysis under higher noise and multimodal distributions is an active area of research.

## References

| Method             | Main Contribution                                                                         | arXiv id      |
|--------------------|------------------------------------------------------------------------------------------|---------------|
| PUCL (bias correction) | General bias-corrected loss and population-mean estimator for contrastive learning       | 2401.08690    |
| puCL, puNCE, puPL  | Unbiased/variance-reducing PU contrastive objectives, pseudo-label clustering             | 2402.06038    |
| UCF                | Uncertainty-weighted, adaptive, stabilized PU contrastive learning                        | 2512.08969    |
| puNCE              | Prior-weighted position for unlabeled views, leading to improved representations          | 2206.01206    |

Source: https://www.emergentmind.com/topics/positive-unlabeled-contrastive-learning-pucl