Scaled-Convergence InfoNCE

• The paper introduces SC-InfoNCE, which dynamically modulates convergence via analytic reparameterization, overcoming static temperature limitations in InfoNCE.
• It employs techniques like analytic logit mapping and convergence target scaling to enhance representational alignment and cluster separation.
• Empirical evaluations across vision, language, and graph tasks confirm that SC-InfoNCE delivers improved gradient behavior and robust performance over traditional InfoNCE.

Scaled-Convergence InfoNCE (SC-InfoNCE) designates a family of modifications to the canonical InfoNCE loss function for contrastive learning, aimed at overcoming intrinsic limitations of static temperature scaling. In place of a constant or hand-tuned temperature, SC-InfoNCE architectures dynamically modulate convergence behavior—either by analytic reparameterization of similarities, or by explicit rescaling driven by theoretical properties of the learning process or feature space. This approach provides a mechanism to robustly control representational alignment, cluster separation, and learning dynamics across domains, without imposing substantial tuning burden or altering downstream architectures.

1. InfoNCE Loss and Its Limitations

The InfoNCE loss underpins modern self-supervised contrastive learning in domains including vision, language, and graph learning. For a batch $\{x_i, \tilde{x}_i\}_{i=1}^N$ and a normalized encoder $f(\cdot)$, cosine similarities are computed as $s_{ij} = f(x_i)^\top f(\tilde{x}_j)$, resulting in the NT-Xent form: $$L = \sum_{i=1}^N L_i, \quad L_i = -\log\frac{\exp(s_{ii}/\tau)}{\sum_{j=1}^N \exp(s_{ij}/\tau)}.$$ Here $\tau > 0$ is a temperature hyperparameter, critically affecting optimization behavior and downstream performance.

The temperature's effect is twofold:

• Representational bottleneck: Division by $\tau$ constrains the softmax input to $[-1/\tau, 1/\tau]$, which fundamentally prevents perfect one-hot asymptotics.
• Gradient pathology: For small $\tau$ the loss landscape exhibits vanishing gradients except near optima, while large $\tau$ yields nonzero gradients even at optimal point. A single value of $\tau$ cannot balance descent speed and first-order optimality uniformly for all batch sizes or data distributions (Kim et al., 29 Jan 2025).

2. SC-InfoNCE via Analytic Logit Mapping

To eliminate the temperature hyperparameter, the “Temperature-Free Loss Function for Contrastive Learning” (Kim et al., 29 Jan 2025) replaces the division-by-$f(\cdot)$0 normalization with a surjective map from the bounded similarity $f(\cdot)$1 to the real line: $f(\cdot)$2 This yields the SC-InfoNCE loss: $f(\cdot)$3 For symmetric positive/negative similarity $f(\cdot)$4, this simplifies to

$f(\cdot)$5

This mapping restores an unbounded logit range, thus resolving the representational and gradient vanishing limitations of the original InfoNCE formulation (Kim et al., 29 Jan 2025).

3. SC-InfoNCE via Convergence Target Scaling

Alternatively, “Understanding InfoNCE: Transition Probability Matrix Induced Feature Clustering” (Cheng et al., 15 Nov 2025) conceptualizes InfoNCE optimization in terms of a transition probability matrix $f(\cdot)$6 governing the augmentation dynamics in explicit feature space. At convergence, the per-pair probability $f(\cdot)$7 of two augmented views sharing the same source approaches a dataset-specific constant.

SC-InfoNCE generalizes this by introducing scalars $f(\cdot)$8 and $f(\cdot)$9 that directly scale the positive and negative softmax terms. The modified objective takes the form: $s_{ij} = f(x_i)^\top f(\tilde{x}_j)$0 Explicitly, the positive and negative logits are reweighted as: $s_{ij} = f(x_i)^\top f(\tilde{x}_j)$1 This parameterization provides fine-grained control over the convergence target and feature clustering strength (Cheng et al., 15 Nov 2025).

4. Similarity-Adaptive Temperature Scaling

A related mechanism, as developed in “Dynamically Scaled Temperature in Self-Supervised Contrastive Learning” (Manna et al., 2023), replaces the static scalar $s_{ij} = f(x_i)^\top f(\tilde{x}_j)$2 with a similarity-dependent temperature $s_{ij} = f(x_i)^\top f(\tilde{x}_j)$3. The proposed form,

$s_{ij} = f(x_i)^\top f(\tilde{x}_j)$4

produces a “valley” at $s_{ij} = f(x_i)^\top f(\tilde{x}_j)$5 and “rims” at $s_{ij} = f(x_i)^\top f(\tilde{x}_j)$6: strongest penalties (lowest $s_{ij} = f(x_i)^\top f(\tilde{x}_j)$7) for mid-range similarity, with softer treatment for highly negative or positive pairs. This structure reduces over-penalization of false negatives (semantically similar pairs), preserving local manifold structure, and aligns optimization gradients with semantic hardness (Manna et al., 2023).

The resulting SC-InfoNCE loss is: $s_{ij} = f(x_i)^\top f(\tilde{x}_j)$8 This yields robustness across data imbalance and modality, and maintains tight alignment–uniformity trade-offs.

5. Gradient Analysis and Convergence Properties

Across all SC-InfoNCE variants, the critical improvement is well-conditioned, problem-adaptive gradient scaling:

• The analytic mapping in (Kim et al., 29 Jan 2025) provides a strict positivity for all $s_{ij} = f(x_i)^\top f(\tilde{x}_j)$9, vanishing exactly at the optimum, and is independent of extraneous hyperparameters. In contrast, standard InfoNCE gradients can vanish too early or persist at optimality due to static $$L = \sum_{i=1}^N L_i, \quad L_i = -\log\frac{\exp(s_{ii}/\tau)}{\sum_{j=1}^N \exp(s_{ij}/\tau)}.$$0.
• The adaptive temperature approach (Manna et al., 2023) yields per-pair gradients $$L = \sum_{i=1}^N L_i, \quad L_i = -\log\frac{\exp(s_{ii}/\tau)}{\sum_{j=1}^N \exp(s_{ij}/\tau)}.$$1, modulating penalty according to semantic hardness.
• The convergence target framework (Cheng et al., 15 Nov 2025) allows explicit trading of intra-class tightness for inter-class separation by setting the scale $$L = \sum_{i=1}^N L_i, \quad L_i = -\log\frac{\exp(s_{ii}/\tau)}{\sum_{j=1}^N \exp(s_{ij}/\tau)}.$$2 and bias $$L = \sum_{i=1}^N L_i, \quad L_i = -\log\frac{\exp(s_{ii}/\tau)}{\sum_{j=1}^N \exp(s_{ij}/\tau)}.$$3 in SC-InfoNCE, controlling cluster statistics and alignment behavior.

These analytical properties yield more reliable and stable convergence, and enable the loss to be flexibly adapted to the statistical invariances demanded by downstream tasks.

6. Empirical Evaluation and Benchmarks

Empirical results on vision, graph, text, recommendation, and anomaly detection tasks confirm that SC-InfoNCE methods provide consistent and sometimes substantial gains over standard InfoNCE with best-tuned $$L = \sum_{i=1}^N L_i, \quad L_i = -\log\frac{\exp(s_{ii}/\tau)}{\sum_{j=1}^N \exp(s_{ij}/\tau)}.$$4, while bypassing the need for costly hyperparameter search.

Representative Results

Domain & Task	Best $$L = \sum_{i=1}^N L_i, \quad L_i = -\log\frac{\exp(s_{ii}/\tau)}{\sum_{j=1}^N \exp(s_{ij}/\tau)}.$$5 Baseline	SC-InfoNCE Result
Imagenette, kNN Top-1 (%)	84.43 ± 0.20	84.65 ± 0.27
CiteSeer F1-Micro/Macro (%)	67.33 ± 3.02/60.47 ± 2.82	67.95 ± 2.10/60.56 ± 2.09
CIFAR-10 Anomaly (ROC-AUC %)	97.215	97.279
ImageNet-100 Top-1 (%)	75.54	78.78
CIFAR-10 Linear (%)	90.53	91.49
PROTEINS Graph (%)	70.44	73.41
STS-B (Spearman corr.)	74.95	77.64

Across these and additional downstream benchmarks (language bias probing, recommendation, sentence embedding), SC-InfoNCE outcomes are either robustly superior or closely match best-tuned static baselines (Kim et al., 29 Jan 2025, Cheng et al., 15 Nov 2025, Manna et al., 2023).

7. Hyperparameter Guidance and Practical Use

• Temperature-free variant (Kim et al., 29 Jan 2025): No hyperparameter search is required. Recommended as a drop-in replacement to InfoNCE, particularly in regimes where temperature selection is infeasible or fragile.
• Convergence scaling (Cheng et al., 15 Nov 2025): Default $$L = \sum_{i=1}^N L_i, \quad L_i = -\log\frac{\exp(s_{ii}/\tau)}{\sum_{j=1}^N \exp(s_{ij}/\tau)}.$$6. Adjust $$L = \sum_{i=1}^N L_i, \quad L_i = -\log\frac{\exp(s_{ii}/\tau)}{\sum_{j=1}^N \exp(s_{ij}/\tau)}.$$7 downward for fine-grained subclass discrimination, upward for cluster separation.
• Adaptive temperature (Manna et al., 2023): Recommended ranges on $$L = \sum_{i=1}^N L_i, \quad L_i = -\log\frac{\exp(s_{ii}/\tau)}{\sum_{j=1}^N \exp(s_{ij}/\tau)}.$$8, $$L = \sum_{i=1}^N L_i, \quad L_i = -\log\frac{\exp(s_{ii}/\tau)}{\sum_{j=1}^N \exp(s_{ij}/\tau)}.$$9 track standard practice; e.g., $\tau > 0$0 for ImageNet100. Performance is robust to the precise analytic form (cosine, linear, exponential), provided the valley/rim structure is retained.

These guidelines enable practitioners to deploy SC-InfoNCE variants across a wide range of domains without specialized search or tuning investment.

---

In summary, Scaled-Convergence InfoNCE provides a theoretically grounded suite of improvements to contrastive learning objectives. By replacing static temperature scaling with analytic mapping, convergence-target rescaling, or similarity-adaptive temperature, SC-InfoNCE methods offer stable optimization, improved feature clustering, and domain-robust performance benefits, verified across a broad empirical spectrum (Kim et al., 29 Jan 2025, Cheng et al., 15 Nov 2025, Manna et al., 2023).