---
title: 'SINCERE: Supervised InfoNCE Extensions'
url: https://www.emergentmind.com/topics/supervised-infonce-extensions-sincere
type: topic
---

# SINCERE: Supervised InfoNCE Extensions

Supervised InfoNCE Extensions (SINCERE) encompass a family of theoretically motivated modifications of the InfoNCE and supervised contrastive losses, developed to address shortcomings of prior approaches—most notably, intra-class repulsion in SupCon—and to provide robust, information-theoretic guarantees for supervised representation learning. This article details the motivations, formal definitions, theoretical properties, empirical findings, and practical recommendations of SINCERE and related methodologies, situating them within the broader context of robust and flexible supervised contrastive learning.

## 1. Theoretical Background and Motivation

The InfoNCE loss is foundational in self-supervised learning, encouraging the alignment of augmented views of the same instance (positives) while separating all other samples (negatives) in the representation space. With the introduction of class labels, SupCon extends InfoNCE to treat all same-class samples as positives. However, the design of SupCon can result in intra-class repulsion; each time a same-class positive appears only in the denominator of an InfoNCE-type term (but not the numerator), it is effectively treated as a negative, causing repulsion among same-class embeddings, especially as the number of positives in a batch grows [2309.14277].

This motivates the search for extensions of InfoNCE that maintain the theoretical properties of the original loss but properly utilize label information. SINCERE (Supervised InfoNCE REvisited) and other recent approaches formalize supervised contrastive learning as a probabilistic noise-contrastive estimation (NCE) problem, thus restoring the principled discrimination between positive and negative pairs. In parallel, frameworks such as ProjNCE generalize the projection/critic structure of supervised contrastive losses to unify supervised and self-supervised objectives under a mutual information (MI) bound [2506.09810].

## 2. Formal Definitions: SINCERE and ProjNCE

### SINCERE Loss

SINCERE reformulates the supervised noise-contrastive setting by drawing a single anchor from the set of positives (i.e., all batch samples of a given class), while the remaining class members act as known positives and all other samples are negatives. The SINCERE loss for an anchor $z_S$ and positive $z_p$ is:
\[
L_{\rm SINCERE}(z_S, z_p) = -\log \frac{\exp(z_S \cdot z_p/\tau)}{\exp(z_S \cdot z_p/\tau) + \sum_{n \in \mathcal N} \exp(z_n \cdot z_p/\tau)}
\]
where $\mathcal N$ is the set of negatives (samples not of the anchor's class) and $\tau$ is a temperature parameter. Crucially, every positive $p$ that appears in the denominator is always included in the numerator, thereby eliminating intra-class repulsion [2309.14277].

### ProjNCE

ProjNCE generalizes both InfoNCE and SupCon by introducing parametric projection mappings $g_+, g_-$ for class embeddings and an optional adjustment term for negative pairs:
\[
I_{\rm NCE}^{\rm proj} = I_{\rm NCE}^{\rm self\text{-}p} + R
\]
where $I_{\rm NCE}^{\rm self\text{-}p}$ follows:
\[
I_{\rm NCE}^{\rm self\text{-}p} = \frac{1}{N} \sum_{i=1}^N \left[ -\log \frac{ \exp(\psi(f(x_i), g_+(c_i))) }{ \sum_{j=1}^N \exp(\psi(f(x_i), g_-(c_j))) } \right]
\]
and $R$ is an expectation-based adjustment term over negative pairings. The critic $\psi$ is typically a scaled cosine similarity. Different choices of $g_+, g_-$ recover standard supervised (centroid-based SupCon), orthogonal projection (mean-based), kernel-smoothed, or median-based robust alignments [2506.09810].

## 3. Information-Theoretic Guarantees and Robustness

### Information-Theoretic Bound (SINCERE)

The expected SINCERE loss lower-bounds the symmetrized Kullback-Leibler divergence between class and noise distributions:
\[
L(\theta) \geq \log |\mathcal N| - [ D_{\rm KL}(p^- \| p^+) + D_{\rm KL}(p^+ \| p^-) ]
\]
where $p^+$ and $p^-$ denote the class and noise distributions, respectively. This relates SINCERE’s minimization directly to maximizing distributional separability in embedding space [2309.14277].

### Mutual Information Bound (ProjNCE)

ProjNCE is constructed so that $-I_{\rm NCE}^{\rm proj}$ is an explicit lower bound on $I(X;C)$ up to a batch-size dependent additive constant, thereby providing a mutual information maximization objective at the class level, not just at the instance level [2506.09810].

### Robustness to Label Noise

A unified theoretical framework shows that standard InfoNCE and SupCon are not robust to symmetric label noise since their risk decompositions have function-dependent bias terms. Losses satisfying $\Delta\mathcal R = {\rm const}$ (i.e., the “noise-bias” independence condition) are robust. Symmetric InfoNCE (SymNCE), created by averaging InfoNCE and a reverse-order variant (RevNCE), and RINCE (Ranking InfoNCE) both satisfy this noise-tolerance criterion and outperform standard losses under severe label corruption [2501.01130].

## 4. Extensions and Related Losses

| Loss Name      | Addressed Setting        | Key Mechanism                                         |
|----------------|-------------------------|-------------------------------------------------------|
| SINCERE        | Standard supervised     | Probabilistic NCE that always includes all positives  |
| ProjNCE        | Flexible supervised/self| Projection-based embedding, mutual information bound  |
| SymNCE         | Label noise robust      | Adds InfoNCE and RevNCE, ensuring constant noise bias |
| RINCE          | Noisy/graded positives  | InfoNCE loss over ranked positive tiers               |

ProjNCE encompasses centroid, mean, median, or kernel-based projections as special cases, with the adjustment term $R$ ensuring the MI bound applies. SymNCE provably cancels the function-dependent noise-bias term, yielding robustness to label noise under the symmetric noise model [2501.01130]. RINCE enables multi-tiered positive supervision, e.g., using superclasses or continuous similarity, and is beneficial where the binary positive/negative split is ill-defined [2201.11736].

## 5. Empirical Evaluation and Observed Benefits

Empirical findings across CIFAR-10, CIFAR-100, ImageNet-100, and noisy-label benchmarks demonstrate:

- SINCERE eliminates intra-class repulsion, yields tighter intra-class clusters, higher margin between classes in cosine similarity, and substantially improves transfer learning performance, especially for linear probing on Cars-196 and FGVC-Aircraft, with minimal change to standard classification accuracy [2309.14277].
- ProjNCE provides consistent improvements over SupCon and cross-entropy (e.g., 0.3–0.7% top-1 accuracy on CIFAR-10/100, tiny-ImageNet) and produces representations with higher estimated mutual information [2506.09810].
- SymNCE outperforms SupCon and RINCE under both synthetic and real-world label noise (e.g., Clothing1M dataset, CIFAR-10/CIFAR-100 at high corruption rates), achieving linear-probe accuracy gains from 2–5% [2501.01130].
- RINCE demonstrates benefits for hierarchical and graded similarity (e.g., superclasses), out-of-distribution detection, and video representation learning, yielding measurable accuracy and retrieval improvements over standard InfoNCE [2201.11736].

## 6. Practical Recommendations

- **Batch Size**: Larger batches (e.g., $N \geq 256$ or $512$) improve the tightness of information-theoretic bounds, especially for adjustment or kernel-based projection methods [2506.09810].
- **Temperature**: Typical values include $\tau=0.07$ (ProjNCE), $\tau=0.1$ (SINCERE); adjust per dataset and stability [2309.14277, 2506.09810].
- **Projection Function Choice**:
  - Centroid (SupCon): Suitable for clean, balanced data.
  - Mean (SoftSupCon), kernel-based: Improved robustness under label noise or class imbalance.
  - Median (MedSupCon): Robust to outliers and extreme noise [2506.09810].
- **Adjustment Term Weighting**: In very noisy conditions, scale the adjustment term $R$ by $\beta>1$ (e.g., $\beta \in [5, 10]$), trading off intra-class tightness for improved class separation [2506.09810].
- **Computational Complexity**: SINCERE, ProjNCE, and SupCon all have similar computational profiles—$O(N^2 D)$ for pairwise dot-product computations per batch [2309.14277].
- **Integration**: SINCERE and ProjNCE act as drop-in replacements for SupCon in existing pipelines; RINCE and SymNCE may require ranked positive construction or additional compositional logic [2201.11736, 2501.01130].

## 7. Future Directions and Open Questions

Several lines of investigation are proposed in the literature:

- Extending SINCERE to directly predict sets of all positives versus pairwise terms, potentially allowing higher-order modeling of class distributions at the expense of computational complexity [2309.14277].
- Adapting these frameworks for multi-modal, multi-view, or multi-modal contrastive learning.
- Developing further robust loss functions for structured or asymmetric label noise, and establishing tighter information-theoretic bounds incorporating finite mini-batch approximations [2501.01130].
- Generalizing projection functions in ProjNCE to more sophisticated kernel, graph, or optimal transport embeddings, aligning with domain-specific geometry or class structure [2506.09810].
- Incorporating margin-based or class-conditional weighting into SINCERE/ProjNCE for long-tailed or multi-modal data.

SINCERE and its related supervised InfoNCE extensions represent a rigorous convergence of geometric, probabilistic, and information-theoretic views on supervised representation learning, providing flexible, robust, and theoretically grounded objectives for modern contrastive pipelines [2309.14277, 2506.09810, 2501.01130].

Source: https://www.emergentmind.com/topics/supervised-infonce-extensions-sincere