Supervised Contrastive Loss (SupCon Loss)

Updated 25 February 2026

Supervised Contrastive Loss (SupCon Loss) is a batch-wise, multi-positive contrastive learning objective that uses explicit labels to cluster embeddings effectively.
It aggregates all same-class samples as positives for each anchor, enhancing intra-class compactness and boosting feature discriminability against label imbalance and domain shifts.
The loss employs temperature scaling and simultaneous multi-positive and multi-negative formulations to optimize geometric separation in the embedding space.

Supervised Contrastive Loss (SupCon Loss) is a batch-wise, multi-positive, multi-negative objective for supervised representation learning, designed to leverage explicit label information to improve class-wise clustering and feature discriminability in deep neural networks. Unlike self-supervised contrastive objectives—which restrict positives to augmentations of the same instance—SupCon aggregates all same-class samples in a batch as positives for each anchor, encouraging compact, well-separated class clusters in embedding space. This fundamental design enables both superior linear separability and greater robustness to adverse scenarios such as label imbalance and domain shift.

1. Mathematical Definition and Loss Construction

Given a batch of $N$ labeled examples $\{(x_i, y_i)\}_{i=1}^N$ , each mapped (optionally via augmentation) to an embedding $z_i \in \mathbb{R}^d$ , the SupCon loss (Khosla et al., 2020) is defined as

$L_{\mathrm{SupCon}} = - \sum_{i=1}^N \;\sum_{p \in P(i)} \log \frac{ \exp\big(z_i^\top z_p / \tau\big) }{ \exp\big(z_i^\top z_p / \tau\big) + \sum_{n \in N(i)} \exp\big(z_i^\top z_n / \tau\big) }$

where:

$P(i)$ : Indices of all other embeddings in the batch (including augmentations) with the same label as $i$ (i.e., positives).
$N(i)$ : Indices of all embeddings in the batch with labels different from $y_i$ (negatives).
$\tau > 0$ : Temperature parameter scaling the cosine similarities.

Each anchor $z_i$ is encouraged to increase similarity to all positives ( $z_p$ with $y_p = y_i$ ) while decreasing similarity to all negatives. The averaging or summing over $p\in P(i)$ ensures that all positive pairs are explicitly “pulled together,” while negatives are implicitly “pushed apart” via the contrastive normalization.

If only one positive per anchor is used and other examples are treated as fixed negatives, the loss reduces to standard classification cross-entropy, thereby subsuming the conventional softmax framework (Khosla et al., 2020, Gauffre et al., 2024).

2. Algorithmic Skeleton and Implementation

SupCon is evaluated over embedded mini-batches. The canonical high-level pseudocode (Khosla et al., 2020, Seifi et al., 2023) is as follows:

for i, a in batch_indices:
    anchor = f(x_{i,a})
    create P(i): all indices of same-class (y_j == y_i), exclude (i,a)
    create N(i): all indices of different-class (y_j != y_i)
    for p in P(i):
        numerator = exp(anchor · z_p / tau)
        denominator = numerator + sum_{n in N(i)} exp(anchor · z_n / tau)
        loss_{i,a} -= log(numerator / denominator)
    loss_{i,a} /= |P(i)|  # Average over positives
total_loss = mean(loss_{i,a} over all anchors)

Embeddings are typically

\ell_2

-normalized to unit length. Large batch sizes or multiple augmentations per sample ensure that most classes are well represented within each mini-batch (Seifi et al., 2023).

3. Geometric, Theoretical, and Optimization Properties

SupCon fundamentally differs from both unsupervised InfoNCE and softmax cross-entropy in its geometrical and statistical behavior (Chen et al., 2022, Gill et al., 2023, Lee et al., 11 Mar 2025):

Multi-positive Contrasts: For any anchor, all same-class samples serve as positives, not just one (as in classic cross-entropy or triplet/margin losses). This direct multi-positive design sharpens intra-class compactness and enlarges inter-class separation.
Class Collapse Tendency: If implemented without auxiliary mechanisms, all within-class representations collapse to a single point (the “class-collapse” geometry), yielding a regular simplex configuration that is provably optimal under the pure SupCon loss (Chen et al., 2022, Lee et al., 11 Mar 2025). This maximizes linear separability but destroys intra-class variability, impairing fine-grained discrimination and transfer learning.
Hyperparameter Influences: The temperature $\tau$ regulates the "hardness" of softmax normalization, with low $\tau$ concentrating gradients on the most difficult (“hard”) positive and negative pairs. Theoretical results show within-class spread can be controlled by incorporating a weighted class-conditional term or by adjusting $\alpha$ in hybrid objectives, with exact collapse boundaries derivable as a function of $(\alpha, \tau, n, m)$ , where $n$ is per-class batch size and $m$ is the number of classes (Lee et al., 11 Mar 2025).

Relation to Mutual Information Bound

SupCon, when interpreted from the mutual information perspective, does not strictly retain the tight lower bound on $I(Z;Y)$ that InfoNCE provides in the unsupervised setting. Recent work (ProjNCE) generalizes SupCon to ensure a valid MI bound by adding an adjustment term and employing projections—e.g., class centroids, conditional expectations, medians—to refine the geometric structure of class clusters (Jeong et al., 11 Jun 2025).

4. Positive and Negative Pair Construction in Diverse Settings

The construction of $P(i)$ and $N(i)$ is central to the applicability of SupCon:

Multi-class classification: Positives are all same-class samples in the batch (excluding the anchor). Negatives are all other samples (Khosla et al., 2020, Seifi et al., 2023).
Multi-label classification: For anchor $i$ , positives are defined via label intersection (i.e., samples sharing at least one label), with Jaccard or similar weighting schemes often used to manage variable overlap. Negatives are samples with disjoint label sets (Audibert et al., 2024).
Continuous action or regression (imitation learning): Continuous action vectors are quantized into discrete bins and mapped to pseudo-class labels, allowing standard SupCon pairing logic to operate (Celemin et al., 15 Sep 2025).
Imbalanced and long-tail settings: Poorly represented (minority) classes are at risk of collapse—mitigated by class-conditional weighting, feature compaction, or architecture-level “prototype” priors (Mildenberger et al., 21 Mar 2025, Alvis et al., 2023).

For effective representation learning, batch construction should ensure sufficient positive pairs for rare classes, or specialized modifications must be applied.

5. Extensions, Regularizations, and Recent Developments

A wide spectrum of extensions and modifications to SupCon address its practical or theoretical limitations:

Prototype-guided contrastive losses: Incorporation of fixed or learnable class prototypes into batches anchors global geometry, achieving neural collapse even in the presence of imbalance (Gill et al., 2023, Gauffre et al., 2024).
Generalized and soft-label versions: GenSCL replaces binary label matching with cross-entropy between “soft” label similarities (e.g., as produced by CutMix or distillation), unlocking compatibility with modern regularizers (Kim et al., 2022).
Margin control and debiasing: The $\varepsilon$ -SupInfoNCE enforces a minimum margin between positives and negatives, improving robustness to dataset bias and ensuring fairer intra-class treatment (Barbano et al., 2022).
Hard negative/positive modulation: Tuned Contrastive Learning (TCL) introduces parameters that upweight challenging positives and negatives, allowing more aggressive mining without destabilizing gradients (Animesh et al., 2023).
Robustness to label noise: SupCon does not automatically guarantee robustness to label noise; modified objectives such as SymNCE are required to enforce population risk invariance under symmetric corruption (Cui et al., 2 Jan 2025).

A summary table of major variants is shown below:

Variant	Key Modification	Main Target
SupCon	Multi-positive loss	Standard supervised contrastive learning
ProjNCE	Projection term + MI adjustment	Tight mutual information bounds
GenSCL	Cross-entropy label similarity	Compatibility with soft labels, regularization
TCL	Tunable hard-mining parameters	Stronger gradients for hard samples
$\varepsilon$ -SupInfoNCE	Margin constraint	Debiasing, robustness to bias
PSupCon/Proto	Learnable/fixed prototypes	Improved geometry under imbalance
SymNCE	Symmetrized noise-robust loss	Label noise robustness

6. Applications and Impact in Modern ML

SupCon and its descendants have become foundational in supervised representation learning across multiple application domains:

Vision Benchmarks: SupCon pretraining yields increased linear probe accuracy on standard datasets such as CIFAR-10, CIFAR-100, and ImageNet, regularly outperforming cross-entropy baselines (Khosla et al., 2020, Kim et al., 2022). Prototype and balance-aware variants set state-of-the-art marks for long-tail and OOD-robust classification (Alvis et al., 2023, Seifi et al., 2023, Gill et al., 2023).
Domain Generalization/OOD Detection: Embeddings from SupCon exhibit superior class clustering and reduced overconfidence on out-of-distribution samples, and prototype-based extensions further strengthen OOD detection (Seifi et al., 2023, Gill et al., 2023).
Imbalanced/Binary Datasets: Naive SupCon fails under extreme class imbalance (majority cluster collapse); simple algorithmic adjustments restore performance and embedding structure (Mildenberger et al., 21 Mar 2025, Lee et al., 11 Mar 2025).
Imitation/RL and Multi-label Learning: Adapting SupCon to action quantization in imitation learning and to label overlaps in multi-label settings consistently yields gains in representation alignment and downstream policy learning (Celemin et al., 15 Sep 2025, Audibert et al., 2024).

SupCon’s flexibility and batchwise formulation make it broadly applicable in multi-task, semi-supervised, and data-poor training regimes (Gauffre et al., 2024).

7. Open Problems and Research Directions

Despite its practical successes, open research areas remain:

Theoretical Guarantees: The full characterization of SupCon’s minima, their geometry (neural collapse and variants), and its mutual information relationships (beyond ProjNCE) remain active areas (Jeong et al., 11 Jun 2025, Lee et al., 11 Mar 2025).
Optimizing Class Spread: Balancing class collapse with the need for intra-class variability and subclass discrimination, especially in transfer and robustness contexts, requires hybrid losses and architecture-level biases (Chen et al., 2022).
Intra-class Repulsion: The original SupCon formulation can encourage unintentional same-class repulsion; newer formulations such as SINCERE address this by excluding same-class points from denominators (Feeney et al., 2023).
Long-tail and Noisy Label Regimes: Further robustness, especially in real-world datasets with both long-tailed and noisy labels, motivates further developments in adaptive weighting and risk-consistent loss design (Cui et al., 2 Jan 2025, Alvis et al., 2023).
Generalization to Structured and Non-Euclidean Domains: Extension of SupCon’s pairwise constructions and geometric priors to structured objects (graphs, sequences) and manifolds is an open research axis.

SupCon and its ecosystem of extensions have solidified their place at the core of modern supervised representation learning. Ongoing work continues to address its limitations, adapt it to new domains, and refine its theoretical underpinnings.