---
title: Multiple Negatives Ranking in Dense Retrieval
url: https://www.emergentmind.com/topics/multiple-negatives-ranking-mnr
type: topic
---

# Multiple Negatives Ranking in Dense Retrieval

Multiple Negatives Ranking (MNR) is a batch-wise contrastive loss framework for fine-tuning deep encoders, particularly Sentence-BERT (SBERT), for dense retrieval tasks such as question answering over long text passages. MNR leverages in-batch negatives, using every other positive instance in a mini-batch as “hard” negatives for each anchor, obviating the need for costly negative mining. The technique is core to SPBERTQA—a two-stage retrieval system combining statistical and neural models for medical question answering in Vietnamese, where MNR demonstrates substantial empirical benefits over both bag-of-words and vanilla transformer baselines [2206.09600]. 

## 1. Mathematical Definition

Given a batch of $K$ question–passage pairs $\{(x_i, y_i)\}_{i=1}^K$ where $x_i$ represents the $i$-th question and $y_i$ the gold answer-passage, the training framework utilizes cosine similarity $S(x, y)$ between question and passage representations from a SBERT encoder. The MNR loss is defined:

\[
\mathcal{L}_{\rm MNR}
\;=\;
-\,\frac{1}{N}\,\frac{1}{K}\,
\sum_{i=1}^K
\Bigl[
  S(x_i, y_i)
  \;-\;
  \log\!\bigl(\sum_{j=1}^K e^{S(x_i, y_j)}\bigr)
\Bigr]
\]

Here, $N$ is the total count of training pairs (often omitted in per-batch settings), $K$ is batch size, $S(x, y)$ is the cosine similarity of mean-pooled SBERT outputs, and $y_j$ (for $j\neq i$) acts as a negative for anchor $x_i$. This loss incentivizes the encoder to assign higher cosine similarity to the correct question–answer pairs relative to all other in-batch answers.

## 2. Anchor, Positive, and Negative Construction

For each element in a batch:

- **Anchor $(x_i)$**: the $i$-th question in the current mini-batch.
- **Positive $(y_i)$**: the gold answer-passage corresponding to $x_i$.
- **Negatives $\{y_j \mid j \neq i\}$**: the remaining $K-1$ answer-passages in the batch, each of which is the gold passage for some other batch question.

This in-batch scheme exploits natural “hard” negatives without explicit negative mining and scales linearly with batch size, thus efficiently increasing the hardness of negative samples as batch size increases.

## 3. Hyperparameters and Implementation Details

The SPBERTQA implementation employs the following parameters and choices:

- **Encoder Backbone**: PhoBERT (monolingual Vietnamese RoBERTa).
- **Fine-tuning Configuration**:
  - Epochs: 15
  - Batch size ($K$): 32
  - Learning rate: $2 \times 10^{-5}$
  - Max sequence length: 256 tokens
  - Regularization: AdamW with default HuggingFace settings (weight decay)
- **Similarity Metric**: Cosine similarity of mean-pooled final layer embeddings.
- **Temperature Scaling**: No extra scaling beyond the implicit $\tau = 1$ in the exponential terms.
- **Negative Mining**: Fully in-batch; no external negative example mining is performed.

## 4. Integration into Two-Stage Retrieval

SPBERTQA structures retrieval in two distinct phases:

A. **BM25-Based Filtering**:
   - Each corpus passage is split into sentences.
   - BM25 indexes all sentences and, given a question, retrieves the top $K_s=5$ most relevant sentences per passage as candidate blocks, effectively narrowing candidate search space.

B. **SBERT Reranking with MNR Fine-Tuning**:
   - In training, questions and sentence blocks are encoded via SBERT, and MNR loss is optimized to maximize correct pair similarity relative to in-batch negatives.
   - At inference, cosine similarity from the fine-tuned SBERT encoder re-ranks the candidate sentences.
   - BM25 is used solely as a filter; SBERT reranking with MNR constitutes the only learned, gradient-driven step.

## 5. Empirical Impact of MNR Fine-Tuning

Ablation studies on the ViHealthQA test set demonstrate the impact of different retrieval models and MNR fine-tuning. Representative retrieval metric values (in %) are summarized below.

| Model                                      | mAP   | P@1   | P@10  |
|--------------------------------------------|-------|-------|-------|
| BM25                                       | 56.93 | 44.96 | 70.09 |
| PhoBERT (no fine-tune)                     | 12.45 | 6.95  | 23.10 |
| BM25 + XLM-R (MNR fine-tuned)              | 53.85 | 46.05 | 79.04 |
| BM25 + mBERT (MNR fine-tuned)              | 55.52 | 44.91 | 75.71 |
| BM25 + PhoBERT (MNR fine-tuned, SPBERTQA)  | 62.25 | 50.92 | 83.76 |

Key findings:

- Fine-tuning PhoBERT with MNR raises mAP from 56.93 (BM25 alone) to 62.25, and P@1 from 44.96 to 50.92.
- A pure PhoBERT embedder without MNR fine-tuning or BM25 suffers from drastically degraded retrieval (e.g., P@1 ≈ 7%).
- Domain- and language-matched backbones (PhoBERT) with MNR outperform multilingual models (XLM-R, mBERT) in this applied context.

## 6. Practical Reproducibility in SBERT-Style Retrieval

Implementation of MNR within an SBERT-style retrieval pipeline based on the SPBERTQA design involves:

1. Forming mini-batches with $K$ question–answer pairs.
2. Treating the batch’s other gold answers as in-batch negatives for each anchor.
3. Fine-tuning with the MNR loss over cosine similarities between question and passage representations.
4. Using BM25 as a lightweight sentence-level filter for long passages before neural reranking.

Expectation is an improvement of approximately 5–6 percentage points in mean average precision over BM25-only retrieval with this method [2206.09600].

## 7. Significance in Domain-Adapted Dense Retrieval

The empirical advantage of MNR arises from its ability to supply a diverse set of semantically rich, difficult negatives at every update, accelerating the training of discriminative representations for dense retrieval. The improvement is contingent upon a strong, domain-matched encoder backbone and benefits from the efficient, mining-free composition of negatives present naturally within question-answering batch construction. The lack of improvement from multilingual encoders in comparison to PhoBERT plus MNR in the ViHealthQA setting suggests strong synergy between corpus/domain adaptation and hard-negative-driven fine-tuning [2206.09600].

Source: https://www.emergentmind.com/topics/multiple-negatives-ranking-mnr