Similarity-Weighted Supervised Contrastive Loss

• The paper introduces a novel loss that leverages heterogeneous t-vMF similarities for positives and negatives, yielding an implicit angular margin for regularization.
• The methodology replaces cosine similarity with t-vMF similarity and uses distinct parameters for positive (κₚ) and negative (κₙ) pairs to optimize robustness under distribution shift.
• Empirical results demonstrate that this approach improves worst-group and out-of-distribution accuracy in settings like subpopulation shifts and domain generalization.

Similarity-Weighted Supervised Contrastive Loss denotes a family of supervised contrastive objectives in which the contribution of a pair is modulated by a similarity-dependent factor rather than being treated uniformly once it is declared positive or negative. In the most specific sense supported by current arXiv literature, the term is especially apt for the heterogeneous-similarity formulation introduced in "Supervised Contrastive Learning with Heterogeneous Similarity for Distribution Shifts" (Kutsuna, 2023), where the usual cosine similarity in supervised contrastive learning is replaced by the t-vMF similarity and, crucially, different similarity parameters are used for positive and negative pairs. In that formulation, the weighting is not an auxiliary heuristic layered onto the loss; it is built into the similarity function itself, yielding a margin-like regularization effect that is designed to mitigate performance degradation under distribution shift.

1. Conceptual placement within supervised contrastive learning

Supervised contrastive learning extends batch contrastive learning to the fully supervised setting by using label information to define positives and negatives: clusters of points belonging to the same class are pulled together in embedding space, while samples from different classes are pushed apart (Khosla et al., 2020). A standard supervised contrastive objective for anchor $x$ can be written as

$$\mathcal{L}_{\text{cl}(\theta)} := - \frac{1}{\left|\mathcal{S}_x^p\right|} \sum_{x_p \in \mathcal{S}_x^p} \log \frac{e^{ \text{sim}(f_\theta(x), f_\theta(x_p))/\tau }} {e^{ \text{sim}(f_\theta(x), f_\theta(x_p))/\tau } + \sum_{x_n \in \mathcal{S}_x^n} e^{ \text{sim}(f_\theta(x), f_\theta(x_n))/\tau }} .$$

Here $\mathcal{S}_x^p$ is the positive set, $\mathcal{S}_x^n$ is the negative set, and $\text{sim}(\cdot,\cdot)$ is usually cosine similarity.

Within this general template, a similarity-weighted formulation alters how pairwise affinity enters the numerator and denominator. In the heterogeneous-similarity setting, positives and negatives remain defined by supervision, but the scoring rule is no longer identical across pair types. The central claim of (Kutsuna, 2023) is that this asymmetry is the source of the useful regularization effect under subpopulation shift and domain generalization, not merely the replacement of cosine by a different similarity family.

This distinguishes the method from two adjacent but different lines of work. First, some papers reinterpret cross-entropy and label smoothing as contrastive objectives with class- or temperature-dependent weighting, rather than sample-sample similarity weighting (Khosla et al., 2020). Second, other papers introduce alternative nonlinear similarity measures or explicit relation weights, but do so in self-supervised, multi-label, multi-task, or weakly supervised settings rather than in the specific distribution-shift regime studied in (Kutsuna, 2023).

2. Heterogeneous similarity formulation

The defining modification in (Kutsuna, 2023) is the substitution of cosine similarity by the t-vMF similarity

$$\phi_\kappa(a, b) := \frac{1+ \cos(a,b)}{1+\kappa\left(1-\cos(a,b)\right)} - 1, \qquad \kappa \in (-1/2,\infty).$$

When $\kappa=0$, this reduces exactly to cosine similarity. The paper characterizes the t-vMF similarity as heavier-tailed than cosine in the sense that changing $\kappa$ changes how similarity behaves away from perfect alignment (Kutsuna, 2023).

The proposed supervised contrastive loss is

$$\tilde{\mathcal{L}_{\text{cl}(\theta)}} := - \frac{1}{\left|\mathcal{S}_x^p\right|} \sum_{x_p \in \mathcal{S}_x^p} \log \frac{e^{ \phi_{\kappa_p}(f_\theta(x), f_\theta(x_p))/\tau }} {e^{ \phi_{\kappa_p}(f_\theta(x), f_\theta(x_p))/\tau } + \sum_{x_n \in \mathcal{S}_x^n} e^{ \phi_{\kappa_n}(f_\theta(x), f_\theta(x_n))/\tau }} .$$

This is the paper’s similarity-weighted supervised contrastive loss in the narrow sense: positives are scored with $\kappa_p$, negatives with $$\mathcal{L}_{\text{cl}(\theta)} := - \frac{1}{\left|\mathcal{S}_x^p\right|} \sum_{x_p \in \mathcal{S}_x^p} \log \frac{e^{ \text{sim}(f_\theta(x), f_\theta(x_p))/\tau }} {e^{ \text{sim}(f_\theta(x), f_\theta(x_p))/\tau } + \sum_{x_n \in \mathcal{S}_x^n} e^{ \text{sim}(f_\theta(x), f_\theta(x_n))/\tau }} .$$0, and the useful regime is $$\mathcal{L}_{\text{cl}(\theta)} := - \frac{1}{\left|\mathcal{S}_x^p\right|} \sum_{x_p \in \mathcal{S}_x^p} \log \frac{e^{ \text{sim}(f_\theta(x), f_\theta(x_p))/\tau }} {e^{ \text{sim}(f_\theta(x), f_\theta(x_p))/\tau } + \sum_{x_n \in \mathcal{S}_x^n} e^{ \text{sim}(f_\theta(x), f_\theta(x_n))/\tau }} .$$1.

The paper also gives a one-parameter parameterization

$$\mathcal{L}_{\text{cl}(\theta)} := - \frac{1}{\left|\mathcal{S}_x^p\right|} \sum_{x_p \in \mathcal{S}_x^p} \log \frac{e^{ \text{sim}(f_\theta(x), f_\theta(x_p))/\tau }} {e^{ \text{sim}(f_\theta(x), f_\theta(x_p))/\tau } + \sum_{x_n \in \mathcal{S}_x^n} e^{ \text{sim}(f_\theta(x), f_\theta(x_n))/\tau }} .$$2

These values are chosen so that the similarity is symmetric at angle $$\mathcal{L}_{\text{cl}(\theta)} := - \frac{1}{\left|\mathcal{S}_x^p\right|} \sum_{x_p \in \mathcal{S}_x^p} \log \frac{e^{ \text{sim}(f_\theta(x), f_\theta(x_p))/\tau }} {e^{ \text{sim}(f_\theta(x), f_\theta(x_p))/\tau } + \sum_{x_n \in \mathcal{S}_x^n} e^{ \text{sim}(f_\theta(x), f_\theta(x_n))/\tau }} .$$3. Larger $$\mathcal{L}_{\text{cl}(\theta)} := - \frac{1}{\left|\mathcal{S}_x^p\right|} \sum_{x_p \in \mathcal{S}_x^p} \log \frac{e^{ \text{sim}(f_\theta(x), f_\theta(x_p))/\tau }} {e^{ \text{sim}(f_\theta(x), f_\theta(x_p))/\tau } + \sum_{x_n \in \mathcal{S}_x^n} e^{ \text{sim}(f_\theta(x), f_\theta(x_n))/\tau }} .$$4 increases $$\mathcal{L}_{\text{cl}(\theta)} := - \frac{1}{\left|\mathcal{S}_x^p\right|} \sum_{x_p \in \mathcal{S}_x^p} \log \frac{e^{ \text{sim}(f_\theta(x), f_\theta(x_p))/\tau }} {e^{ \text{sim}(f_\theta(x), f_\theta(x_p))/\tau } + \sum_{x_n \in \mathcal{S}_x^n} e^{ \text{sim}(f_\theta(x), f_\theta(x_n))/\tau }} .$$5, which strengthens the implicit margin and thus the regularization.

A notable negative result is equally important for the concept. The paper states that for equal $$\mathcal{L}_{\text{cl}(\theta)} := - \frac{1}{\left|\mathcal{S}_x^p\right|} \sum_{x_p \in \mathcal{S}_x^p} \log \frac{e^{ \text{sim}(f_\theta(x), f_\theta(x_p))/\tau }} {e^{ \text{sim}(f_\theta(x), f_\theta(x_p))/\tau } + \sum_{x_n \in \mathcal{S}_x^n} e^{ \text{sim}(f_\theta(x), f_\theta(x_n))/\tau }} .$$6, simply swapping cosine for t-vMF does not materially improve test accuracy. This establishes that the operative mechanism is not a generic nonlinear similarity replacement, but the heterogeneous parameterization $$\mathcal{L}_{\text{cl}(\theta)} := - \frac{1}{\left|\mathcal{S}_x^p\right|} \sum_{x_p \in \mathcal{S}_x^p} \log \frac{e^{ \text{sim}(f_\theta(x), f_\theta(x_p))/\tau }} {e^{ \text{sim}(f_\theta(x), f_\theta(x_p))/\tau } + \sum_{x_n \in \mathcal{S}_x^n} e^{ \text{sim}(f_\theta(x), f_\theta(x_n))/\tau }} .$$7.

3. Margin-like regularization mechanism

The mechanism is analyzed by rewriting the denominator after factoring out the positive term: $$\mathcal{L}_{\text{cl}(\theta)} := - \frac{1}{\left|\mathcal{S}_x^p\right|} \sum_{x_p \in \mathcal{S}_x^p} \log \frac{e^{ \text{sim}(f_\theta(x), f_\theta(x_p))/\tau }} {e^{ \text{sim}(f_\theta(x), f_\theta(x_p))/\tau } + \sum_{x_n \in \mathcal{S}_x^n} e^{ \text{sim}(f_\theta(x), f_\theta(x_n))/\tau }} .$$8 Defining

$$\mathcal{L}_{\text{cl}(\theta)} := - \frac{1}{\left|\mathcal{S}_x^p\right|} \sum_{x_p \in \mathcal{S}_x^p} \log \frac{e^{ \text{sim}(f_\theta(x), f_\theta(x_p))/\tau }} {e^{ \text{sim}(f_\theta(x), f_\theta(x_p))/\tau } + \sum_{x_n \in \mathcal{S}_x^n} e^{ \text{sim}(f_\theta(x), f_\theta(x_n))/\tau }} .$$9

reducing the loss means making $\mathcal{S}_x^p$0 smaller (Kutsuna, 2023).

Let $\mathcal{S}_x^p$1 and $\mathcal{S}_x^p$2 denote the angles between the anchor and the positive and negative embeddings. If $\mathcal{S}_x^p$3, the sign of $\mathcal{S}_x^p$4 depends simply on whether $\mathcal{S}_x^p$5, namely whether the positive is closer in angle than the negative. If $\mathcal{S}_x^p$6, the paper shows that there exists a positive margin $\mathcal{S}_x^p$7 such that

$\mathcal{S}_x^p$8

Ordinary cosine-based supervised contrastive learning therefore requires only that the positive be closer than the negative, whereas the heterogeneous-similarity version requires the positive to be closer by a margin. The margin is not inserted manually as a fixed additive constant; it emerges from the difference between $\mathcal{S}_x^p$9 and $\mathcal{S}_x^n$0. The paper explicitly describes this as a margin-like regularization, analogous in spirit to ArcFace or LDAM, but applied to contrastive learning rather than cross-entropy (Kutsuna, 2023).

This margin interpretation is the most precise way to understand the expression “similarity-weighted” in this setting. The similarity function itself changes the effective geometry of competition between positives and negatives. A plausible implication is that the loss penalizes small angular separations more aggressively when they are likely to encode spurious group- or domain-specific structure, although the paper formulates this operationally as a regularization effect rather than as a claim about a single universal mechanism.

4. Role under distribution shift

The motivating problem is distribution shift, where the distribution of data changes between training and testing and model performance can degrade substantially (Kutsuna, 2023). The paper emphasizes two concrete settings: subpopulation shift and domain generalization. In both, a nonstandard overfitting phenomenon can occur: as training proceeds, overall or in-distribution accuracy improves, but performance on minority groups or unseen domains degrades.

The proposed loss is used as a regularizer on top of the main classification loss. In experiments, the model is trained with CE or group-DRO for prediction, plus the contrastive loss on the embedding space. The intended effect is to pressure same-class samples to cluster more robustly while maintaining stronger separation from negatives. The paper argues that such structured representation learning prevents the model from becoming too specialized to majority groups or training domains (Kutsuna, 2023).

On CelebA subpopulation shift, the paper reports that worst-group accuracy degrades during training when only group-DRO is used, even though overall accuracy remains strong. Adding ordinary supervised contrastive learning with cosine similarity helps only a little, but the heterogeneous-similarity contrastive regularizer prevents the worst-group degradation much more effectively. Against weight decay, stochastic depth, and RandAugment, the method is described as competitive or better for worst-group performance.

On Camelyon17 domain generalization, CE training yields high ID accuracy while OOD accuracy on unseen hospitals degrades over training. The proposed loss mitigates this degradation, improving OOD accuracy while slightly reducing ID accuracy, which the paper interprets as consistent with a regularization effect that trades a bit of in-domain fitting for better robustness. PCA visualizations are also used to show that CE leads to representations organized mostly around label discrimination, while CE plus supervised contrastive learning or the proposed loss yields more class-aggregated embeddings that appear more robust across hospitals (Kutsuna, 2023).

In this literature, the term “similarity-weighted” is therefore tightly linked to robustness under shift, not merely to metric design. The paper’s central practical claim is that the asymmetric similarity parameterization functions as a regularizer precisely in settings where representation learning would otherwise drift toward majority-group or training-domain cues.

5. Empirical behavior, boundary conditions, and misconceptions

A central empirical finding is that the similarity family alone is insufficient. The paper states that simply using t-vMF with $\mathcal{S}_x^n$1 behaves similarly to cosine, confirming that the key factor is the positive/negative asymmetry, not the similarity family alone (Kutsuna, 2023). This directly addresses a common misconception that any generalized or heavier-tailed similarity should automatically improve robustness.

A second empirical boundary condition appears on iWildCam. When the baseline training does not exhibit noticeable OOD degradation, the proposed method provides only minor gains. The paper presents this as a limitation and as supporting evidence for its interpretation: the method is most useful when there is a genuine overfitting-like shift in representation toward training domains or majority groups.

These observations narrow the expected scope of the method. It is not presented as a universally superior replacement for standard supervised contrastive loss. Rather, it is a regularizer targeted at a specific failure mode. This suggests that selection criteria for similarity-weighted supervised contrastive objectives should depend on whether training dynamics actually display a separation between improving ID metrics and degrading minority-group or OOD metrics.

The same section also clarifies the relationship to the broader supervised contrastive literature. Standard supervised contrastive learning itself is already known to pull same-class points together and push apart different-class clusters, and it has been reported to show benefits for robustness to natural corruptions and to be more stable to hyperparameter settings such as optimizers and data augmentations (Khosla et al., 2020). The heterogeneous-similarity formulation refines that baseline by introducing an implicit margin in the pairwise similarity comparison rather than by redefining the supervised contrastive paradigm wholesale.

6. Related formulations of similarity weighting

The broader literature uses the idea of similarity weighting in several non-equivalent ways. The following comparison is descriptive rather than taxonomic; the cited methods all modulate contrastive learning by a richer notion of relation than a binary positive/negative indicator, but they differ in where the weighting enters the objective.

Paper	Setting	Weighting mechanism
(Kutsuna, 2023)	Distribution shift	Different t-vMF parameters for positives and negatives
(Huang et al., 2024)	Multi-label supervised contrastive learning	Per-positive factor $\mathcal{S}_x^n$2
(Mu et al., 2023)	Multi-similarity supervised contrastive learning	Uncertainty-based task weights $\mathcal{S}_x^n$3
(Zhou et al., 28 May 2025)	Weakly supervised contrastive learning	Graph weights from continuous semantic similarity
(Jiang et al., 2022)	Self-supervised imaging	Jaccard-style ratio-based similarity

In "Similarity-Dissimilarity Loss for Multi-label Supervised Contrastive Learning" (Huang et al., 2024), the weighting is explicitly relation-aware. The paper defines five relations $\mathcal{S}_x^n$4–$\mathcal{S}_x^n$5 over anchor and sample label sets, keeps overlapping samples as positives, and re-weights each positive by

$\mathcal{S}_x^n$6

so that exact matches contribute most, partial overlaps contribute less, and disjoint samples contribute nothing.

In "Multi-Similarity Contrastive Learning" (Mu et al., 2023), similarity weighting occurs across multiple label-defined relations rather than within a single pairwise similarity function. The method uses separate projection heads for distinct similarity metrics and learns uncertainty parameters $\mathcal{S}_x^n$7, giving the weighted objective

$\mathcal{S}_x^n$8

Uncertain tasks are down-weighted automatically.

In "Weakly-Supervised Contrastive Learning for Imprecise Class Labels" (Zhou et al., 28 May 2025), hard same-class supervision is replaced by continuous semantic similarity used as graph weights in a perturbation augmentation graph. The method is framed as a weakly-supervised spectral contrastive loss and is mathematically related to supervised contrastive learning when the recovery matrix accurately reconstructs the true label posterior.

By contrast, "Refining Self-Supervised Learning in Imaging: Beyond Linear Metric" (Jiang et al., 2022) replaces cosine similarity with a Jaccard-style ratio that combines similarity and dissimilarity through a bi-projector system. That paper is self-supervised rather than supervised, but it is relevant because it makes explicit that similarity replacement can be conceived as a ratio-based dependency objective rather than as a pure dot-product score.

Taken together, these works show that “Similarity-Weighted Supervised Contrastive Loss” is best understood as a broad design pattern rather than a single canonical formula. In one important strand, represented by (Kutsuna, 2023), weighting is realized by heterogeneous similarity parameters for positives and negatives, yielding an implicit angular margin that regularizes representation learning under distribution shift. In other strands, weighting is defined by multi-label overlap structure, uncertainty across similarity tasks, or continuous weakly supervised pairwise affinities. The common theme is that supervised contrastive learning is made more discriminating by refusing to treat all positives, all negatives, or all similarity relations as equivalent.