---
title: 'SC-InfoNCE: Scaled Convergence in Contrastive Learning'
url: https://www.emergentmind.com/topics/scaled-convergence-infonce-sc-infonce
type: topic
---

# SC-InfoNCE: Scaled Convergence in Contrastive Learning

Scaled Convergence InfoNCE (SC-InfoNCE) is a contrastive learning objective that generalizes the InfoNCE loss by introducing a tunable convergence target, thereby enabling flexible control over feature similarity alignment. Unlike the standard InfoNCE, which promotes uniform clustering based on a constant target, SC-InfoNCE exploits a transition probability matrix (TPM) induced by data augmentations and scales it by a factor \( s \) to modulate the influence of augmented view dynamics on learned representations. This framework yields a principled mechanism for tuning alignment strength in accordance with data statistics and downstream requirements [2511.12180].

## 1. Recapitulation of the InfoNCE Objective and Transition Matrix Formalism

Let \( \mathcal{D} = \{ x_1, \ldots, x_n \} \) be an unlabeled dataset, and let an augmentation distribution \( T \) induce a finite feature space \( S \) with cardinality \( m = |S| \). The transition-probability matrix \( T \in [0,1]^{m \times m} \) is defined as
\[
T_{ij} = \Pr(\text{augmented view of feature } i \text{ equals feature } j).
\]
A parametric encoder \( f_\theta \) produces embeddings \( z_i = f_\theta(t_1(x_i)) \), \( z_j = f_\theta(t_2(x_i)) \) for independent augmentations \( t_1, t_2 \sim T \). Cosine similarity (typically after \( \ell_2 \)-normalization) is used: \( \mathrm{sim}(z_i, z_j) = z_i^\top z_j \), with a temperature \( \tau > 0 \).

The predicted pairwise probability is:
\[
P_{ij}(\theta) = \frac{\exp(\mathrm{sim}(z_i, z_j)/\tau)}{\sum_{k=1}^{B}\exp(\mathrm{sim}(z_i, z_k)/\tau)}, \quad i,j=1\ldots B,
\]
where \( B \) is the batch size. The InfoNCE loss is:
\[
\mathcal{L}_{\mathrm{InfoNCE}}(\theta)
= -\sum_{i=1}^{B}\sum_{j=1}^{B}\mathbf{1}\{j=i^+\} \log P_{ij}(\theta),
\]
with \( i^+ \) the positive index for anchor \( i \). In expectation, InfoNCE drives \( P_{ij} \) toward a constant determined by the statistics of \( T \), promoting uniform “clustering” in representation space [2511.12180].

## 2. SC-InfoNCE: Definition and Mathematical Formulation

SC-InfoNCE extends InfoNCE by replacing the uniform convergence target with a scaled, data-driven target. A scalar \( s > 0 \) is introduced to form a new target matrix:
\[
Q = s T, \quad Q_{ij} = sT_{ij}.
\]
The SC-InfoNCE objective is then:
\[
\mathcal{L}_{\mathrm{SC\text{-}InfoNCE}}(\theta)
= -\sum_{i=1}^{B}\sum_{j=1}^{B} Q_{ij} \log P_{ij}(\theta)
= -\sum_{i,j} sT_{ij}\log\frac{\exp(\mathrm{sim}(z_i, z_j)/\tau)}{\sum_{k} \exp(\mathrm{sim}(z_i,z_k)/\tau)}.
\]
This loss can equivalently be expressed as a cross-entropy between the predicted probability matrix and the scaled TPM.

The gradient with respect to the similarity matrix entry \( S_{ij} = \mathrm{sim}(z_i, z_j) \) is:
\[
\frac{\partial \mathcal{L}}{\partial S_{ij}}
= -\frac{1}{\tau}(Q_{ij} - P_{ij}(\theta)).
\]
At stationarity, \( P_{ij}(\theta) = Q_{ij} = sT_{ij} \).

## 3. Theoretical Properties and Feature Clustering

Under the assumption of a sufficiently expressive encoder and an infinite data stream, any stationary point of SC-InfoNCE satisfies \( P_{ij}(\theta) = sT_{ij} \) for all \( i, j \). Through the softmax link \( P_{ij} \propto \exp(S_{ij}/\tau) \), this yields \( S_{ij} \propto \tau \log[sT_{ij}] + \text{const} \). Feature pairs with large \( T_{ij} \) (frequent cross-augmentation) will attain higher similarity, naturally imposing a soft clustering structure where affinities are prescribed by \( sT \).

The scaling parameter \( s \) modulates the geometry: larger \( s \) amplifies log differences in transition probabilities, facilitating cluster separation but risking mode collapse when \( s \) is too large. Smaller \( s \) sharpens sensitivity to local differences but may reduce inter-cluster distinctness. In downstream scenarios matching the co-occurrence pattern of \( T \), proper normalization of \( Q \) aligns pretraining geometry to test-time statistics [2511.12180].

## 4. Algorithmic Implementation

The typical pipeline for SC-InfoNCE pretraining is:

1. Estimate the transition matrix \( T_{ij} \) via Monte-Carlo simulation over augmentations.
2. Form the target matrix \( Q = s T \).
3. For each epoch and mini-batch:
   - Sample two augmentations per anchor and encode to obtain \( z \) representations.
   - Compute all pairwise similarities \( S_{ip} \).
   - Compute softmax probabilities \( P_{ip} \).
   - Calculate the cross-entropy loss \( -\sum_{i,p} Q_{ip} \log P_{ip} \).
   - Backpropagate and update parameters \( \theta \).

Recommended hyperparameter ranges are:
- \( s \in [0.5, 2.0] \), with \( s \approx 1.0 \) often a strong default.
- \( \tau \in [0.05, 1.0] \), where smaller values sharpen the output distribution.
- Batch size: [256, 1024] (vision), [64, 256] (text/graph).
- Learning rate and weight decay as used in base InfoNCE protocols.

## 5. Empirical Evaluation Across Domains

Experiments were performed using vision (CIFAR-10, CIFAR-100, STL-10, ImageNet-100; ResNet-50 pretrained for 200 epochs), graph (COLLAB, DD, NCI1, PROTEINS; 3-layer GCN), and text (STS-B, SICK-R; BERT-base on 1M Wikipedia sentences). Baselines included SCL, InfoNCE, DCL, DHEL, and f-MICL.

Performance was assessed through linear-probe accuracy. Representative results:

| Dataset         | Std InfoNCE | SC-InfoNCE (best s) | Δ      |
|-----------------|-------------|---------------------|--------|
| CIFAR-10        | 90.53       | 91.49               | +0.96  |
| CIFAR-100       | 50.90       | 51.95               | +1.05  |
| ImageNet-100    | 74.62       | 75.62               | +1.00  |
| STL-10          | 84.07       | 85.54               | +1.47  |
| COLLAB          | 75.98       | 76.28               | +0.30  |
| DD              | 73.92       | 75.89               | +1.97  |
| NCI1            | 75.38       | 75.72               | +0.34  |
| PROTEINS        | 70.44       | 73.41               | +2.97  |
| STS-B           | 74.95       | 77.64               | +2.69  |
| SICK-R          | 73.87       | 75.48               | +1.61  |

Ablation over \( s \) on CIFAR-10 reveals:
| \( s \) | 0.5  | 1.0  | 1.5  |
|---------|------|------|------|
| Acc.    | 89.1 | 90.5 | 91.2 |

*This suggests* that moderate increases in \( s \) can consistently improve performance; however, excessive scaling risks instability.

## 6. Practical Considerations and Limitations

Choosing \( s \) should be guided by data characteristics:
- For mild augmentations and low inter-class separation, increase \( s > 1 \).
- To mitigate embedding collapse (few large embedding covariance eigenvalues), decrease \( s < 1 \).
- For fine-grained tasks (e.g., STS-B), moderate increases (\( s=1.2-1.5 \)) can enhance subtle representation fidelity.

Limitations include the necessity to estimate \( T \), which can be challenging in high-dimensional settings; one may approximate \( T \) among prototypes or clusters. The global scaling \( s \) may not correct for class imbalance, and row- or time-adaptive scaling may be advantageous—this remains an open direction. Excessive \( s \) may result in representation collapse if \( sT \) contains large entries [2511.12180].

SC-InfoNCE enables principled, tunable alignment to augmentation-induced feature affinities, trading off inter- versus intra-cluster structure and yielding competitive results across multiple domains.

Source: https://www.emergentmind.com/topics/scaled-convergence-infonce-sc-infonce