---
title: Multi-Similarity Loss in Deep Metric Learning
url: https://www.emergentmind.com/topics/multi-similarity-loss
type: topic
---

# Multi-Similarity Loss in Deep Metric Learning

Multi-Similarity Loss is a class of loss functions central to modern deep metric learning and contrastive representation learning. It regularizes embedding models by leveraging information from multiple notions of similarity, outperforming traditional pair/triplet-based approaches in image retrieval, cross-modal retrieval, and robust representation learning. Integrating the General Pair Weighting (GPW) framework, multi-similarity loss enables principled and efficient mining and weighting of training pairs, and extensions such as Multi-Similarity Contrastive Loss (MSCon) and Symmetric Multi-Similarity Loss (SMS) exploit multiple metrics or soft-label information for enhanced performance and generalization.

## 1. Motivation and Historical Background

In classical deep metric learning, most methods relied on fixed rules for positive and negative mining, such as “contrastive,” “triplet,” and “lifted-structure” losses. These approaches were fundamentally limited by redundant pair sampling and coarse, uniform weighting schemes. They typically only exploited a single “signal” per pair: either the raw similarity (“self”), the relative ranking among positives, or the separation from negatives in the batch.

Multi-Similarity Loss (MS Loss), introduced by Wang et al. [1904.06627], addressed these limitations by supporting three distinct similarity signals—self, positive-relative, and negative-relative—within a unified, differentiable formulation. This principled weighting allows broader exploitation of batch information and enables larger, more informative gradients per batch step. Subsequent variants, such as Multi-Similarity Contrastive Loss (MSCon) and Symmetric Multi-Similarity Loss (SMS), further extended this framework to settings with multiple, possibly uncertain, notions of similarity [2307.02712][2406.12256]. Such scenarios are prevalent in real-world data, where objects are annotated with multiple categorical or soft affiliations.

## 2. The General Pair Weighting and Multi-Similarity Loss Formulation

Multi-Similarity Loss is rooted in the General Pair Weighting (GPW) view, where the gradient of any pair-based metric learning loss decomposes as a sum of pairwise weights:
$$
\frac{\partial \mathcal{L}}{\partial \theta} = \sum_{i,j} w_{ij} \frac{\partial S_{ij}}{\partial \theta}
$$
where $w_{ij} = \left| \frac{\partial \mathcal{L}}{\partial S_{ij}} \right|$, $S_{ij}$ is the cosine similarity between $f(x_i)$ and $f(x_j)$, and $f$ is a unit-normalizing embedding function.

### Original Multi-Similarity Loss

Given a batch $\{x_i\}$ with labels $y_i$, positives $P_i$, and negatives $N_i$, the loss is formulated as:
$$
\mathcal{L}_{MS} = \frac{1}{N}\sum_{i=1}^N \Bigg\{
\frac{1}{\alpha}\log\Big[1 + \sum_{j\in P_i} e^{-\alpha(S_{ij}-\gamma)}\Big]
+\frac{1}{\beta}\log\Big[1 + \sum_{k\in N_i} e^{\beta(S_{ik}-\gamma)}\Big]
\Bigg\}
$$
with sharpness parameters $\alpha,\beta > 0$ and margin $\gamma$ [1904.06627]. The loss uses an explicit mining step to focus on “informative” positives/negatives based on relative similarity, followed by a soft weighting based on both $S_{ij}$ and its hardness compared to other pairs.

### Mining and Weighting Mechanism

- Informative positive set: $\mathcal{P}_i = \{ j: y_j=y_i, S_{ij} < \text{hardest\_neg}_i + \epsilon \}$
- Informative negative set: $\mathcal{N}_i = \{ j: y_j\neq y_i, S_{ij} > \text{hardest\_pos}_i - \epsilon \}$
- Pairs are then exponentially weighted and combined in the loss.

This design ensures that only the most “violating” or “hard” pairs contribute significant gradient signal, improving both retrieval precision and training efficiency.

## 3. Extensions: Multi-Similarity Contrastive and Symmetric Multi-Similarity Losses

### Multi-Similarity Contrastive Loss (MSCon)

When data carries multiple categorical or semantic attributes (e.g., category, closure, gender for images), each attribute induces a distinct similarity relation. MSCon, as introduced by Mu et al. [2307.02712], learns one projection head per metric and forms a multi-similarity objective by summing a supervised contrastive (SupCon) loss per metric:
$$
L^{mscon} = \sum_{c=1}^C \sum_{i=1}^{2N} L_{c,i}^{mscon}
$$
where $L_{c,i}^{mscon}$ is a SupCon loss over the $c^\text{th}$ relational head. 

### Uncertainty-based Task Weighting

MSCon incorporates a learnable task-specific uncertainty $\sigma_c > 0$, yielding the regularized objective:
$$
\min_{f, {g^c}, {\sigma_c}} \sum_{c=1}^C \left[ \frac{1}{\sigma_c^2} \sum_{i=1}^{2N} L_{c,i}^{mscon} + 2\log \sigma_c \right]
$$
This weighting down-scales the contribution of “uncertain” or noisy similarity tasks, leading to better out-of-domain (OOD) generalization and more robust multi-attribute representations [2307.02712].

### Symmetric Multi-Similarity Loss

For cross-modal or soft-label scenarios (e.g., video–text with soft correlation matrices), the Symmetric Multi-Similarity Loss (SMS) employs the difference between soft correlation scores $R_{ijk} = c_{ij} - c_{ik}$ as the margin, enforcing a symmetric ordering via hinge-style triplet loss:
$$
\mathcal{L}_{SMS} = \sum_{(i,j,k)}
\begin{cases}
\left[R_{ijk}\gamma - S_{ij} + S_{ik}\right]_+, & R_{ijk}>0 \\
\left[-R_{ijk}\gamma + S_{ij} - S_{ik}\right]_+, & R_{ijk}<0 \\
\left[|S_{ij} - S_{ik}| - \tau\right]_+, & R_{ijk}=0 \\
\end{cases}
$$
where $\gamma$ controls the margin and $\tau$ is a relaxation factor to prevent degenerate updates when $R_{ijk}=0$ [2406.12256].

## 4. Algorithmic and Implementation Details

Multi-Similarity Loss and its derivatives are implemented via efficient matrix operations within deep learning frameworks:

- **Batch construction:** Use multiple samples per class to enable informative positive and negative mining.
- **Pairwise similarity matrix computation:** Compute all cosine similarities in the batch ($B\times B$); efficient masking is used to select anchor–positive and anchor–negative pairs.
- **Mining step:** For each anchor, vectorized reduction is used to extract hardest positives/negatives and construct the sets $\mathcal{P}_i$, $\mathcal{N}_i$.
- **Weighting step:** Exponential (softmax-like) weighting over the mined pairs for greater gradient selectivity.
- **Stabilization:** Care is taken to avoid numerical overflow in exponentials by judicious parameter selection (e.g., $\alpha=2$, $\beta=50$).
- **Batch size:** Empirically, robust estimation requires batch sizes of at least $60$–$120$ for effective mining.
- **Final update:** Fully vectorized gradient calculation is supported, with no need for custom backward passes.

For multi-task cases (MSCon), each metric’s loss is weighted by the inverse variance $\sigma_c^2$ and self-regularized by $2\log \sigma_c$; gradients are accumulated over all tasks before joint optimization [2307.02712].

## 5. Empirical Performance and Ablation Studies

Multi-Similarity Loss and its generalizations deliver state-of-the-art performance on multiple benchmarks:

| Dataset           | Loss/Method           | Recall@1 (%) or Top-1 (%) | Key Setting/Attribute         |
|-------------------|----------------------|---------------------------|-------------------------------|
| CUB-200           | MS Loss              | 65.7                      | d=512, vs. 60.6 (ABE)         |
| Cars-196          | MS Loss              | 84.1                      | vs. 81.4 (HTL)                |
| In-Shop Clothes   | MS Loss              | 89.7                      | vs. 80.9 (prior)              |
| SOP               | MS Loss              | 78.2                      | vs. 74.8 (ABE)                |
| Zappos50k         | MSCon                | 97.17/94.37/85.98         | Category/Closure/Gender       |
| MEDIC             | MSCon                | 81.00/79.14/81.69/85.15   | Multi-attribute, in-domain    |
| EK-100            | SMS                  | 57.0/69.2, 62.1/73.0      | ViT-B, ViT-L (mAP/nDCG)       |

Ablation studies reveal that:
- Incorporating all three signals (P+S+N) yields stronger performance than using any single mining or weighting component [1904.06627].
- Learned uncertainty weighting (in MSCon) significantly improves out-of-domain accuracy, especially when certain similarity metrics are noisy or intentionally corrupted [2307.02712].
- Introducing relaxation factor $\tau$ in SMS yields notable boosts in mAP [2406.12256].
- SMS outperforms adaptive MI-MM variants by explicit utilization of soft-label differences and symmetric loss structure [2406.12256].

## 6. Comparative Analysis and Practical Implications

Multi-Similarity Loss unifies and extends traditional pair-based and triplet-based losses:
- **Contrastive loss:** Only exploits self-similarity, with all mined pairs weighted equally.
- **Triplet/Historam/Lifted structure:** Partially exploit positive- or negative- relative signals, but lack joint mining and weighting.
- **MS Loss:** Combines strict mining (positive-relative) with soft, differentiable weighting (self and negative-relative), yielding sharper gradient focus and better utilization of informative pairs [1904.06627].

Extensions such as MSCon and SMS are directly suited to multi-task and soft-label settings:
- MSCon dynamically balances contributions from multiple relations by uncertainty-based weights, leading to generalizable and robust embedding models [2307.02712].
- SMS generalizes the batch mining and weighting approach to soft, real-valued label relations, appropriate for complex retrieval benchmarks [2406.12256].

These methods are suited for retrieval, classification with multiple labels, and scenarios with heterogeneous or noisy supervision.

## 7. Limitations and Best Practices

Key limitations and recommendations include:
- Mining margin $\epsilon$ and weighting sharpness parameters must be selected appropriately to ensure the presence of informative pairs and avoid degenerate gradients.
- Batch size must be sufficient to supply positives/negatives per anchor.
- Large values of $\alpha$ or $\beta$ may cause numerical overflow; parameter tuning or use of mixed precision is advised.
- In multi-task or attribute-rich settings, uncertainty-based weighting is critical to prevent noisy tasks from degrading overall representations.
- For soft-label scenarios, the relaxation term $\tau$ prevents wasted model capacity on near-duplicate label pairs.

Adhering to these guidelines, Multi-Similarity Loss remains a robust and adaptable family for high-performance metric learning across domains [1904.06627][2307.02712][2406.12256].

Source: https://www.emergentmind.com/topics/multi-similarity-loss