Multi-Similarity Contrastive Learning

• Multi-Similarity Contrastive Learning (MSCon) is a supervised framework that uses multiple categorical attributes to form diverse similarity metrics for representation learning.
• The method employs distinct projection heads and an uncertainty-based weighting mechanism to adaptively balance noisy or unreliable similarity signals.
• Empirical results demonstrate that MSCon significantly improves both in-domain and out-of-domain performance compared to traditional contrastive learning methods.

Multi-Similarity Contrastive Learning (MSCon) is a supervised representation learning framework designed to address the limitations of contrastive methods that optimize with respect to only a single similarity relation. In datasets where examples are annotated along multiple categorical attributes—each inducing a unique notion of similarity—MSCon leverages supervision from all available similarity metrics. The method introduces per-metric projection heads and integrates a principled uncertainty-based weighting mechanism, resulting in improved generalization, particularly for out-of-domain tasks and settings with noisy or unreliable similarity information (Mu et al., 2023).

1. Motivation and Problem Statement

Traditional contrastive learning frameworks such as SimCLR and SupCon assume a single notion of similarity (e.g., class membership) for forming positive and negative pairs in representation space. This approach is suboptimal in real-world datasets where each instance can simultaneously possess multiple attributes (e.g., category, style, gender), each defining a distinct relational structure among data points. Simply aggregating multiple supervised contrastive losses by summation assumes equal task reliability and can degrade generalization, especially when some metrics are noisy. MSCon resolves this by learning a separate projection for each metric and adaptively down-weighting uncertain similarity tasks, resisting overfitting due to corrupted or ambiguous attributes.

2. Formal Specification

Let the dataset $D = \{(x_1, y_1), ..., (x_M, y_M)\}$ consist of $M$ examples $x_i$ with multi-relational annotation $y_i = (y_i^1, ..., y_i^C)$, where each $y_i^c$ is a discrete label for attribute $c$ ($c=1,...,C$), inducing a distinct similarity metric. The architecture comprises a shared encoder $f(\cdot; \theta_e)$, mapping $x_i$ to $h_i \in \mathbb{R}^d$. For each metric $c$, a distinct projection head $$g_c(\cdot; \theta_c): \mathbb{R}^d \to \mathbb{R}^k$$ is followed by $l_2$-normalization to produce $v_i^c = g_c(h_i) / \|g_c(h_i)\|_2$. Positives under metric $c$ for anchor $i$ are $P^c(i) = \{j\neq i: y_j^c = y_i^c\}$; negatives, $N^c(i) = \{k\neq i: y_k^c \neq y_i^c\}$. The pairwise similarity is $s_c(x_i, x_j) = \langle v_i^c, v_j^c \rangle$.

3. Multi-Similarity Contrastive Loss Definition

For a given similarity metric $c$ and anchor $i$, the supervised contrastive loss is

$$L_{c,i} = -\frac{1}{|P^c(i)|} \sum_{p \in P^c(i)} \log \left[ \frac{\exp(s_c(x_i, x_p)/\tau)}{\sum_{a \in B \setminus \{i\}} \exp(s_c(x_i, x_a)/\tau)} \right],$$

where $\tau$ is the inverse temperature. The complete MSCon loss for a batch is a weighted sum over metrics: $$L_{\mathrm{MSCon}} = \sum_{c=1}^C w_c \sum_{i \in B} L_{c,i}$$ with learnable, nonnegative metric weights $w_c$. Alternatively, this is expressed over positive and negative index pairs to clarify contributions per metric.

4. Uncertainty-Based Weighting Mechanism

MSCon introduces an uncertainty parameter $\sigma_c > 0$ for each metric, controlling the effective temperature as $\tau \sigma_c^2$. The learning objective is justified via a pseudo-likelihood formulation for each metric: $$p_c(y_i^c | v_i^c, \tau) \propto \frac{1}{|P^c_{y_i^c}|} \sum_{p \in P^c_{y_i^c}} \exp(\langle v_i^c, v_p^c \rangle / \tau).$$ Maximizing this pseudo-likelihood yields, up to Jensen's inequality, the standard supervised contrastive loss. The batch negative log pseudo-likelihood for metric $c$ is then

$$-\sum_{i \in B} \log p_c(y_i^c) \propto \frac{1}{\sigma_c^2} \sum_{i \in B} L_{c,i} + 2 \log \sigma_c.$$

Hence, the joint MSCon objective to be minimized is

$$\min_{\theta_e, \{\theta_c\}, \{\sigma_c>0\}} \sum_{c=1}^C \left[ \frac{1}{\sigma_c^2} \sum_{i \in B} L_{c,i} + 2 \log \sigma_c \right],$$

with $w_c = 1/\sigma_c^2$. The $2 \log \sigma_c$ penalty prevents degenerate solutions causing weight collapse.

5. Optimization Procedure and Implementation

The training regime consists of the following steps:

• Initialize encoder parameters $\theta_e$, projection heads $\{\theta_c\}$, and uncertainties $\{\sigma_c=1.0\}$.
• For each epoch and minibatch:
  • Encode and project inputs per metric.
  • Determine positive and negative sets for each $c$.
  • Compute $L_c = \sum_{i=1}^{B} L_{c,i}$ per metric, using current $\sigma_c$ for temperature scaling.
  • Sum weighted losses and log-penalty: $$\mathrm{Loss} = \sum_{c=1}^C [(1/\sigma_c^2) \cdot L_c + 2 \log \sigma_c]$$.
  • Backpropagate and update $\theta_e$, $\{\theta_c\}$, and $\{\sigma_c\}$ (SGD or Adam).
• After training, discard $\{g_c\}$; use $f$ for downstream tasks.

Key implementation details include:

• Normalizing $g_c(\cdot)$ outputs onto the unit sphere.
• Initializing all $\sigma_c$ equally for stable early learning.
• Standard augmentations (random crop, flip, color jitter).
• Recommended optimizer and hyperparameters: SGD (momentum 0.9), learning rate 0.05, batch size 64, $\tau=0.1$, weight decay $1\mathrm{e}{-4}$, 200 epochs for learning, projection dimensions 32 (small datasets) or 64 (large).

6. Empirical Results and Comparative Analysis

MSCon has been empirically validated on multi-relational benchmarks:

• Zappos50k: 50K shoe images labeled by category (4), closure style (5), gender (4); held-out task: brand (20 classes). Encoder: ResNet-18, projection heads (32-dim).
• MEDIC: $\approx$71K disaster images annotated for damage severity (3), disaster type (7), humanitarian relevance (4), informativeness (2). Held-out: one metric at a time. Encoder: ResNet-50, projection heads (64-dim).
• All encoders pretrained on ImageNet and fine-tuned with MSCon; embeddings evaluated via frozen linear classifiers.

MSCon outperformed single-task and multi-task cross-entropy, SimCLR, SupCon, and Conditional Similarity Networks with triplet loss. In-domain top-1 accuracy (mean±std over 1,000 bootstrap trials), Zappos50k tasks: Category 97.17±0.27, Closure 94.37±0.35, Gender 85.98±0.56. Out-of-domain (Zappos brand): 42.62±1.52 vs. 32.10±1.48 for the best cross-entropy multi-task. On MEDIC hold-out, MSCon led or matched state-of-the-art except for the binary informativeness task (85.22±0.30 vs 86.18±0.30).

Ablation studies on label corruption show that, as a metric’s labels are increasingly corrupted (fraction $\rho$), the learned weight $w_c$ ($=1/\sigma_c^2$) for that metric decays toward zero, preserving performance except when all metrics are corrupted. Fixed-weight MSCon collapses under maximum corruption, indicating the efficacy of adaptive weighting.

7. Theoretical Foundations and Analysis

By supervising with respect to all available similarity metrics, MSCon drives the encoder to capture factors common to the relational structures present in the data. The uncertainty-based weighting mechanism is theoretically justified by pseudo-likelihood maximization under a task-specific noise model; weights $w_c = 1/\sigma_c^2$ correspond to maximum likelihood under Gaussian noise assumptions. The $2\log\sigma_c$ penalty ensures non-trivial uncertainty estimates and prevents trivial solutions where a metric’s uncertainty is collapsed. Empirically, the learned weights respond dynamically to signal quality, effectively rejecting noisy or less-informative relational labels.

Ablation studies indicate that introducing additional similarity metrics, even if some are noisy, does not degrade performance provided adaptive metric weights are learned. The optimal temperature parameter $\tau=0.1$ was found to be robust across tasks. Analysis of weight dynamics demonstrates that $w_c$ generally decays linearly with increasing corruption ratio $\rho$ in a synthetic task corruption setup.

8. Practical Considerations

MSCon is readily implemented atop standard deep metric learning pipelines. For each new similarity metric, a new projection head must be instantiated; however, only the encoder is retained for downstream applications after pretraining. Projected vectors should be $l_2$-normalized to the unit sphere. Batch size and the quality of augmentations critically affect contrastive sample diversity; larger batches are beneficial. Hyperparameter tuning for the temperature parameter $\tau$ and careful initialization of uncertainties are essential for stability. Linear probing is used for evaluation to isolate representation quality.

MSCon improves both in-domain and out-of-domain generalization, especially where true underlying tasks are not fully captured by any single similarity metric. Its uncertainty-based weighting renders it robust to overfitting from noisy relational information, making it suitable for complex, multi-relational datasets in vision and beyond (Mu et al., 2023).