Dual-Temperature InfoNCE

• Dual-Temperature InfoNCE is a contrastive loss that separates intra-anchor (vector) and inter-anchor (scalar) hardness using two distinct temperature parameters.
• It refines self-supervised learning by allowing stable updates even with limited negatives, eliminating the need for large queues or momentum encoders.
• Empirical evaluations show that this formulation improves accuracy across architectures like ResNet and ViT, proving robust to hyperparameter and batch size variations.

Dual-temperature InfoNCE refers to a class of contrastive loss formulations in self-supervised learning that decompose and separately control the contributions of intra-anchor (vector) and inter-anchor (scalar) hardness through distinct temperature parameters. This approach emerges from a fine-grained analysis of InfoNCE and addresses the limitations of conventional single-temperature methods, particularly in settings where the negative sample size (NSS) is limited by minibatch size and the use of large momentum-based dictionaries (as in MoCo) is impractical or undesirable (Zhang et al., 2022).

1. Standard InfoNCE Loss and Hardness Sensitivity

The InfoNCE loss, central to contrastive learning frameworks such as SimCLR and MoCo, is defined for an anchor $q$, its positive $k^+$, and negatives $\{ k_j \}$ as: $$L(q, k^+, \{k_j\}) = -\log \frac{\exp(q \cdot k^+ / \tau)}{\exp(q \cdot k^+ / \tau) + \sum_j \exp(q \cdot k_j / \tau)}$$ where $\tau$ is a scalar temperature parameter regulating the sharpness of the softmax. Lowering $\tau$ intensifies penalties for hard negatives and emphasizes uniformity in the learned feature space (Manna et al., 2023).

This loss intrinsically encodes two hardness-aware effects:

• Intra-anchor hardness, via differential weighting over negatives within the softmax;
• Inter-anchor hardness, by assigning each anchor an overall magnitude through the sum of its softmax weights (Zhang et al., 2022).

2. The Dual-Temperature Decomposition: Vector and Scalar Components

A detailed gradient analysis of InfoNCE reveals two distinct components:

• Vector component $v_i$: $v_i = k^+ - \sum_j \hat{p}_j^i k_j$, where $\hat{p}_j^i = p_j^i / \sum_j p_j^i$, controls alignment and the directional push against negatives.
• Scalar component $s_i = \sum_j p_j^i$, controls the magnitude of update per anchor, reflecting inter-anchor hardness due to negative sample statistics (Zhang et al., 2022).

Dual-temperature InfoNCE assigns separate temperatures to these effects:

• $k^+$0 (vector/intra-anchor): tunes alignment vs. uniformity, typically chosen small ($k^+$1).
• $k^+$2 (scalar/inter-anchor): flattens or sharpens anchor-wise weights, often set large ($k^+$3).

The dual-temperature loss is given as: $k^+$4 with $k^+$5, where $k^+$6 is the positive-pair probability computed with $k^+$7 (Zhang et al., 2022).

3. Algorithmic Simplifications: From MoCo to SimCo

Conventional MoCo architectures employ a momentum encoder and a large queue of negative samples to ensure stable scalar anchor weights $k^+$8. Dual-temperature InfoNCE enables the removal of both:

• SimMoCo: eliminates the queue, keeps the momentum encoder; negatives come from the current minibatch; uses dual temperatures.
• SimCo: eliminates both queue and momentum encoder; a single network is used symmetrically for all views and negatives derive exclusively from the batch (Zhang et al., 2022).

This simplification is enabled by decoupling scalar sensitivity through $k^+$9, allowing stable updates even with limited and freshly-sampled negatives.

4. Hyperparameter Selection and Practical Recommendations

Empirically, dual-temperature InfoNCE is robust to hyperparameter variations provided that $\{ k_j \}$0:

• Recommended settings: $\{ k_j \}$1, $\{ k_j \}$2.
• Performance stability: For $\{ k_j \}$3, accuracy remains stable; only very small $\{ k_j \}$4 ($\{ k_j \}$5) reintroduces instability in scalar weights (Zhang et al., 2022).
• Batch size: This approach exhibits robustness across batch sizes, outperforming MoCo v2 even at small minibatches.
• Architectural compatibility: Consistent empirical improvements are observed on ResNet-18, ResNet-50, ViT-tiny, and Swin-tiny (Zhang et al., 2022).

5. Empirical Evaluation and Comparisons

The dual-temperature InfoNCE formulation achieves superior or comparable accuracy to MoCo v2 with substantially smaller effective negative sets and without momentum encoders or memory queues:

• On CIFAR-100 with ResNet-18 (200 epochs, batch 256): MoCo v2 (single $\{ k_j \}$6, queue 65536) achieves 53.28% top-1, SimMoCo (dual-T, no queue) achieves 54.11%, and SimCo (dual-T, no queue, no momentum) achieves 58.35%.
• In further ablations, SimCo consistently outperforms MoCo v2 over longer training (200→800 epochs), varying batch sizes, and across architectures (Zhang et al., 2022).

6. Theoretical and Conceptual Significance

The dual decomposition clarifies that:

• Scalar anchor weights are unstable and sensitive to NSS when computed at low temperatures and with limited/unstable keys, thereby necessitating large queues in conventional MoCo. Raising $\{ k_j \}$7 smooths these weights, reducing variance and mitigating the need for a large negative dictionary.
• Vector components (alignment/uniformity) primarily require fresh negatives and preserve sensitivity to hard negatives with small $\{ k_j \}$8, even at modest batch sizes.
• Bridge to non-contrastive SSL: Many “negative-free” methods (BYOL, SimSiam) implicitly set anchor weights to unity, effectively neutralizing inter-anchor hardness—a property shown to often improve self-supervised learning (Zhang et al., 2022).

7. Broader Implications

Dual-temperature InfoNCE provides a principled decomposition of the contrastive loss, facilitating efficient and robust self-supervised learning without reliance on large, stale negative queues or specialized encoders. This yields empirical gains, broadens the design space for SSL losses, and bridges contrasts with non-contrastive methods. It serves as both a practical simplification and a conceptual advance in understanding the mechanics of hardness-awareness in InfoNCE and related frameworks (Zhang et al., 2022).