---
title: Pairwise Relevance Distillation (PairDistill)
url: https://www.emergentmind.com/topics/pairwise-relevance-distillation-pairdistill
type: topic
---

# Pairwise Relevance Distillation (PairDistill)

Searching arXiv for the cited papers and closely related work on PairDistill.
Pairwise Relevance Distillation, usually abbreviated **PairDistill**, is a dense retrieval distillation method that transfers **relative relevance** supervision from a **pairwise reranker** into a dense retriever, rather than relying only on **absolute relevance scores** from a pointwise reranker [2410.01383]. In the formulation introduced in "PairDistill: Pairwise Relevance Distillation for Dense Retrieval" [2410.01383], the central claim is that pointwise rerankers assign **absolute relevance scores** to documents independently and therefore face **challenges related to inconsistent comparisons**, whereas pairwise rerankers provide **fine-grained distinctions between similarly relevant documents**. PairDistill therefore combines standard dense retrieval training with both pointwise and pairwise distillation losses, so that the student retriever learns not only which passages are relevant, but also which passage should rank above another for the same query.

## 1. Conceptual motivation and problem formulation

PairDistill is situated in the standard dense retrieval setting in which a query \(q\) is matched against passages or documents \(d \in \mathcal{D}\), and the retriever assigns a score \(s(q,d)\) used for ranking [2410.01383]. Dense retrievers are efficient because they encode queries and documents separately and retrieve by vector similarity, with common choices including dot product, cosine similarity, or ColBERT-style MaxSim. The paper keeps this general setup and identifies a specific weakness in prevailing distillation pipelines: dense retrievers are often improved by distillation from a stronger reranker, usually a cross-encoder, but the transferred supervision is usually **pointwise**.

The paper’s critique of pointwise distillation is precise. A pointwise reranker assigns an **absolute relevance score** to each candidate independently. Those scores are described as **poorly calibrated across documents**, so comparing them directly can be noisy or inconsistent. PairDistill treats this as a supervision problem: if dense retriever training attempts to imitate such scores or the distributions derived from them, then the student inherits the teacher’s calibration issues. The proposed remedy is to use a **pairwise reranker** as teacher, so that the supervision is phrased as a direct comparison—whether \(d_i\) is more relevant than \(d_j\) for \(q\)—instead of a comparison between absolute numbers.

In this sense, PairDistill moves the distillation target from scalar score imitation to **ranking structure**. The paper’s motivating intuition is that pairwise rerankers model **relative relevance** directly and therefore provide a finer-grained signal, particularly among closely competing passages near the top of a ranking. A plausible implication is that PairDistill is not merely a stronger teacher-student recipe, but a change in what is considered the transferable unit of knowledge: not an isolated score, but an ordered preference.

## 2. Mathematical formulation of PairDistill

The student retriever defines a relevance score
\[
s(q,d) = \mathrm{Sim}(q,d)
\]
and standard dense retriever training uses a contrastive objective such as InfoNCE:
\[
\mathcal{L}_{\mathrm{CL}} = - \log \frac{\exp(s(q,d^+))}{\sum_{d \in \mathcal{D}'} \exp(s(q,d))}.
\]
Here \(d^+\) is a positive passage and \(\mathcal{D}'\) includes positives and negatives [2410.01383].

PairDistill retains that contrastive term but augments it with two distillation components. The first is **pointwise knowledge distillation**. A pointwise teacher assigns a score \(S_{\text{point}}(q,d)\), which is converted into a probability distribution over candidates:
\[
P_{\text{point}}(d \mid q) = \frac{\exp(S_{\text{point}}(q,d)/T)}{\sum_{d' \in \mathcal{D}'} \exp(S_{\text{point}}(q,d')/T)}.
\]
The student similarly defines
\[
P(d \mid q) = \frac{\exp(s(q,d))}{\sum_{d' \in \mathcal{D}'} \exp(s(q,d'))},
\]
and the pointwise distillation loss is
\[
\mathcal{L}_{\mathrm{KD}} = \frac{1}{|\mathcal{B}|} \mathrm{KL}\!\left(P_{\text{point}}(d \mid q)\,\|\,P(d \mid q)\right).
\]

The second, and defining, component is **pairwise distillation**. Instead of supervising single documents independently, the teacher provides
\[
P_{\text{pair}}(d_i > d_j \mid q),
\]
the probability that \(d_i\) is more relevant than \(d_j\) for query \(q\). The student converts its own scores into a pairwise preference probability:
\[
P(d_i > d_j \mid q) = \frac{\exp(s(q,d_i))}{\exp(s(q,d_i)) + \exp(s(q,d_j))}.
\]
The pairwise distillation loss is then
\[
\mathcal{L}_{\mathrm{pair}} = \frac{1}{B} \sum_{(d_i,d_j)\sim \mathcal{D}_{\text{pair}}} \mathrm{KL}\!\left( P_{\text{pair}}(d_i > d_j \mid q)\,\|\,P(d_i > d_j \mid q) \right).
\]

The full PairDistill objective combines all three components:
\[
\mathcal{L} = \mathcal{L}_{\mathrm{CL}} + \lambda_{\mathrm{KD}} \mathcal{L}_{\mathrm{KD}} + \lambda_{\mathrm{pair}} \mathcal{L}_{\mathrm{pair}}.
\]
In the zero-shot setting, where labeled positives are unavailable, the contrastive term is dropped:
\[
\mathcal{L}_{\mathrm{zs}} = \mathcal{L}_{\mathrm{KD}} + \lambda_{\mathrm{pair}} \mathcal{L}_{\mathrm{pair}}.
\]

The methodological distinction is explicit. PairDistill does not replace dense retrieval with pairwise reranking at inference time; instead, it uses pairwise teacher signals so that the dense retriever’s own similarity scores satisfy the implied ranking relation. If the teacher prefers \(d_i\) over \(d_j\), the student is pushed toward
\[
s(q,d_i) > s(q,d_j).
\]
This suggests that the retrieval model is being trained to internalize reranking judgments as geometry in its embedding or late-interaction score space.

## 3. Teacher models, candidate generation, and pair selection

PairDistill uses a two-stage teacher process on retrieved candidates [2410.01383]. First, for each query, the current dense retriever retrieves top-\(k\) passages. Second, those candidates are reranked in two ways. A **pointwise reranking teacher** reranks each passage independently and provides \(S_{\text{point}}(q,d)\); in the main experiments this teacher is **MiniLM cross-encoder**. A **pairwise reranking teacher** compares two passages jointly and returns \(P_{\text{pair}}(d_i > d_j \mid q)\); in the main experiments this teacher is **duoT5-3B**. The paper also notes that pairwise reranking can be done in a **supervised/classification-based** way using labeled triplets, or in an **instruction-based** way using an LLM in zero-shot mode.

Pairwise supervision is more expensive than pointwise scoring, so PairDistill does not enumerate all possible pairs. Instead, it defines the pair set with a heuristic:
\[
\mathcal{D}_{\text{pair}} = \{(d_i,d_j) \mid d_i,d_j \in \mathrm{ret}_k(q),\, i\neq j,\, |i-j|<\delta\}.
\]
The intuition given in the paper is threefold: documents very far apart in the ranking are already easy to distinguish; pairs near each other provide more useful training signal; and pairwise supervision is computationally expensive, so restricting the pair set reduces cost. In experiments, the paper uses this heuristic and reports that removing it **slightly hurts performance**.

This pair-selection mechanism is a defining practical feature rather than a minor optimization. PairDistill’s claim is not that every possible teacher comparison is necessary, but that **local ranking ambiguities** are where pairwise teachers are most informative. A plausible implication is that PairDistill concentrates computation on the part of the ranking where calibration errors and fine-grained preference reversals are most consequential.

## 4. Iterative training and optimization procedure

PairDistill includes an **iterative refinement loop** rather than a one-shot distillation pass [2410.01383]. The procedure is: train a retriever; use it to retrieve top-\(k\) passages; rerank them with pointwise and pairwise teachers; fine-tune the retriever using \(\mathcal{L}\); then use the updated retriever for the next iteration. The paper’s rationale is that if distillation uses only a fixed retrieved set, the model can overfit to those passages, whereas refreshing the candidate set allows the retriever to improve over time.

The paper gives a simplified algorithmic view:
1. retrieve candidates for query \(q\) with the current dense retriever;
2. pointwise rerank the top-\(k\) candidates using a cross-encoder;
3. pairwise rerank sampled pairs among the top candidates using a pairwise teacher;
4. compute losses—contrastive loss for labeled positives, KL distillation to the pointwise teacher, and KL distillation to the pairwise teacher;
5. update retriever parameters;
6. optionally refresh retrieval and repeat.

In the reported experiments, PairDistill uses **ColBERTv2** as the main retriever initialization, and also tests **DPR** to show architecture-agnostic gains. The retrieval budget is to retrieve **top-100 passages**, sample **50 pairs** per query for pairwise reranking, with \(\lambda_{\text{pair}} = 3.0\), \(\lambda_{\text{KD}} = 1.0\), and temperature \(T=1.0\). Training is conducted on **4 V100 GPUs, 32GB each**, and inference uses the **PLAID engine for efficient late-interaction retrieval**. The paper also specifies **max document length 300 for BEIR/LoTTE** and mentions **specific query length settings for some datasets**.

The iterative structure matters conceptually because PairDistill is not only transferring teacher judgments onto a static candidate pool. It is repeatedly redefining the candidate space seen during training. The paper reports **improvement after the first iteration**, **best or near-best after two iterations**, and **convergence after that**. This suggests that pairwise distillation is coupled to retrieval quality itself: better retrieval produces more informative candidate comparisons, which in turn produce a better retriever.

## 5. Experimental settings and empirical results

PairDistill is evaluated in several regimes [2410.01383]. For supervised training, the paper uses the **MS MARCO** training set with **about 502K labeled queries** and **about 8.8M passages**, and also uses **306K unlabeled queries** for distillation. The primary in-domain benchmarks are **MS MARCO dev**, measured by **MRR@10**, and **TREC DL19 / DL20**, measured by **NDCG@10**. Out-of-domain and zero-shot evaluation uses **BEIR** with **18 datasets** and metric **NDCG@10**, and **LoTTE** with **Recall@5 / Success@5-style reporting in the paper’s summary**. Open-domain QA retrieval uses **Natural Questions**, **TriviaQA**, and **SQuAD**, with **Recall@5**.

The paper reports that PairDistill achieves **state-of-the-art or best-in-class performance** across several settings. On **MS MARCO dev**, **PairDistill: 40.7 MRR@10**, compared with **ColBERTv2 baseline: 39.7**. On **TREC DL19 / DL20**, it is reported as **Best on DL19** and **Near-best / second-best on DL20**. On **BEIR**, it achieves **Best overall average across BEIR subsets among comparable models**, outperforming strong baselines such as **ColBERTv2**, **DRAGON+**, **SPLADE++**, **ABEL-FT**, and **GTR-XXL**. The paper emphasizes that this is notable because it is **zero-shot** and often beats even domain-adapted models. On **LoTTE**, it reports **state-of-the-art on both search and forum subsets**. In open-domain QA retrieval, Recall@5 is reported as **NQ: 71.8**, **TriviaQA: 77.4**, and **SQuAD: 66.9**, all better than the **ColBERTv2** and **BM25** baselines reported in the paper.

The ablations sharpen the interpretation of these results. Removing \(\mathcal{L}_{\mathrm{pair}}\) hurts performance, indicating that the pairwise teacher contributes beyond pointwise distillation. Using only \(\mathcal{L}_{\mathrm{pair}}\) **slightly hurts compared to full PairDistill**, so the best configuration combines pointwise and pairwise KD. Removing the pair sampling heuristic causes performance to **drop slightly**. The method works not only with **ColBERTv2** but also with **bert-base-uncased** and **DPR**, which the paper presents as evidence of broad applicability.

One particularly important analysis reports that pointwise reranking disagrees with the pairwise teacher on a substantial fraction of sampled pairs—**about 31% disagreement** in one analysis. The paper uses this to support the claim that pairwise and pointwise teachers provide meaningfully different supervision, and that pairwise supervision can capture more accurate relative preferences. In the zero-shot setting, when using **flan-t5-xl** for instruction-based reranking, pairwise distillation still improves over pointwise KD alone on target-domain datasets such as **FiQA**, **BioASQ**, and **Climate-FEVER**.

## 6. Relation to adjacent pairwise distillation frameworks, misconceptions, and limitations

A recurring source of confusion is that **PairDistill** names a specific dense retrieval method, but the broader idea of pairwise distillation appears in several adjacent literatures. The following comparison reflects only the distinctions explicitly described in the cited papers.

| Method | Domain | Core transfer signal |
|---|---|---|
| PairDistill [2410.01383] | Dense retrieval | Pairwise relevance preferences |
| GRKD [2504.20482] | Class output distillation | Ordered class pairs |
| PRD [2507.04820] | Document ranking | Pairwise teacher labels from PRP |
| DiPair [2010.03099] | Text-pair modeling | Teacher-generated soft labels |
| PDD [2407.07302] | Real-world image SR | Pairwise distances |

PairDistill is closest in spirit to **Group Relative Knowledge Distillation (GRKD)**, which also shifts supervision from absolute outputs to relative structure [2504.20482]. GRKD constructs ordered class pairs
\[
P = \{(i,j)\mid s_i > s_j\}
\]
and trains the student to preserve the teacher’s preference orderings using a pairwise ranking-style loss. The shared principle is the same: preserve **ranking consistency rather than exact probability regression**. The difference is that GRKD is formulated for **class output distributions** and combines a group relative loss with a standard soft-label term, whereas PairDistill is formulated for **dense retrieval** and combines contrastive learning, pointwise KD, and pairwise KD.

It is also closely related to **Pairwise Ranking Distillation (PRD)**, which distills a **pointwise student ranker** from **Pairwise Ranking Prompting (PRP)** teacher labels [2507.04820]. PRD uses a **Pairwise Logistic Ranking Loss** and emphasizes that with only **2% of pairs**, it can match the performance of using all pairs for teacher supervision. The relation is conceptual rather than terminological: PRD is specifically about transferring the ranking behavior of a pairwise LLM teacher into a cheaper pointwise ranker, while PairDistill is about distilling pairwise reranking supervision into a dense retriever.

By contrast, **DiPair** is not a pairwise preference distillation method in the same sense [2010.03099]. It is a **dual-encoder + head** framework for text pairs or n-ary tuples, distilled from a BERT teacher with teacher-generated soft labels. Its main concern is scalable interaction modeling for pair inputs rather than pairwise teacher judgments between documents. Likewise, **Pairwise Distance Distillation (PDD)** uses pairwise distances between model outputs in **unsupervised real-world image super-resolution**, not relevance preferences [2407.07302]. These distinctions matter because “pairwise distillation” is a family resemblance, not a single method class.

The main practical limitation identified for PairDistill is computational. Pairwise distillation may require **more training pairs**, pairwise reranking is **more expensive than pointwise scoring**, and the method depends on a strong teacher, a candidate retrieval stage, and efficient pair generation and reranking infrastructure [2410.01383]. The paper mitigates this with pair sampling, but cost remains a consideration. A common misconception is therefore that PairDistill eliminates reranking expense; the paper does not make that claim. Its contribution is to use expensive pairwise supervision selectively during training so that the final system is still a dense retriever. Another misconception is that PairDistill rejects pointwise knowledge entirely; in fact, the reported best-performing formulation combines **contrastive learning**, **pointwise KD**, and **pairwise KD**, indicating that relative supervision complements rather than wholly displaces absolute teacher signals.

Source: https://www.emergentmind.com/topics/pairwise-relevance-distillation-pairdistill