Pairwise Relevance Distillation (PairDistill)

• The paper introduces PairDistill, transferring relative relevance via pairwise supervision rather than absolute scores.
• It combines contrastive learning with pointwise and pairwise distillation losses to improve dense retriever training.
• Empirical results on MS MARCO, TREC, and BEIR show enhanced ranking performance and robustness in both supervised and zero-shot settings.

Searching arXiv for the cited papers and closely related work on PairDistill. Pairwise Relevance Distillation, usually abbreviated PairDistill, is a dense retrieval distillation method that transfers relative relevance supervision from a pairwise reranker into a dense retriever, rather than relying only on absolute relevance scores from a pointwise reranker (Huang et al., 2024). In the formulation introduced in "PairDistill: Pairwise Relevance Distillation for Dense Retrieval" (Huang et al., 2024), the central claim is that pointwise rerankers assign absolute relevance scores to documents independently and therefore face challenges related to inconsistent comparisons, whereas pairwise rerankers provide fine-grained distinctions between similarly relevant documents. PairDistill therefore combines standard dense retrieval training with both pointwise and pairwise distillation losses, so that the student retriever learns not only which passages are relevant, but also which passage should rank above another for the same query.

1. Conceptual motivation and problem formulation

PairDistill is situated in the standard dense retrieval setting in which a query $q$ is matched against passages or documents $d \in \mathcal{D}$, and the retriever assigns a score $s(q,d)$ used for ranking (Huang et al., 2024). Dense retrievers are efficient because they encode queries and documents separately and retrieve by vector similarity, with common choices including dot product, cosine similarity, or ColBERT-style MaxSim. The paper keeps this general setup and identifies a specific weakness in prevailing distillation pipelines: dense retrievers are often improved by distillation from a stronger reranker, usually a cross-encoder, but the transferred supervision is usually pointwise.

The paper’s critique of pointwise distillation is precise. A pointwise reranker assigns an absolute relevance score to each candidate independently. Those scores are described as poorly calibrated across documents, so comparing them directly can be noisy or inconsistent. PairDistill treats this as a supervision problem: if dense retriever training attempts to imitate such scores or the distributions derived from them, then the student inherits the teacher’s calibration issues. The proposed remedy is to use a pairwise reranker as teacher, so that the supervision is phrased as a direct comparison—whether $d_i$ is more relevant than $d_j$ for $q$—instead of a comparison between absolute numbers.

In this sense, PairDistill moves the distillation target from scalar score imitation to ranking structure. The paper’s motivating intuition is that pairwise rerankers model relative relevance directly and therefore provide a finer-grained signal, particularly among closely competing passages near the top of a ranking. A plausible implication is that PairDistill is not merely a stronger teacher-student recipe, but a change in what is considered the transferable unit of knowledge: not an isolated score, but an ordered preference.

2. Mathematical formulation of PairDistill

The student retriever defines a relevance score

$s(q,d) = \mathrm{Sim}(q,d)$

and standard dense retriever training uses a contrastive objective such as InfoNCE: $$\mathcal{L}_{\mathrm{CL}} = - \log \frac{\exp(s(q,d^+))}{\sum_{d \in \mathcal{D}'} \exp(s(q,d))}.$$ Here $d^+$ is a positive passage and $\mathcal{D}'$ includes positives and negatives (Huang et al., 2024).

PairDistill retains that contrastive term but augments it with two distillation components. The first is pointwise knowledge distillation. A pointwise teacher assigns a score $d \in \mathcal{D}$0, which is converted into a probability distribution over candidates: $d \in \mathcal{D}$1 The student similarly defines

$d \in \mathcal{D}$2

and the pointwise distillation loss is

$d \in \mathcal{D}$3

The second, and defining, component is pairwise distillation. Instead of supervising single documents independently, the teacher provides

$d \in \mathcal{D}$4

the probability that $d \in \mathcal{D}$5 is more relevant than $d \in \mathcal{D}$6 for query $d \in \mathcal{D}$7. The student converts its own scores into a pairwise preference probability: $d \in \mathcal{D}$8 The pairwise distillation loss is then

$d \in \mathcal{D}$9

The full PairDistill objective combines all three components: $s(q,d)$0 In the zero-shot setting, where labeled positives are unavailable, the contrastive term is dropped: $s(q,d)$1

The methodological distinction is explicit. PairDistill does not replace dense retrieval with pairwise reranking at inference time; instead, it uses pairwise teacher signals so that the dense retriever’s own similarity scores satisfy the implied ranking relation. If the teacher prefers $s(q,d)$2 over $s(q,d)$3, the student is pushed toward

$s(q,d)$4

This suggests that the retrieval model is being trained to internalize reranking judgments as geometry in its embedding or late-interaction score space.

3. Teacher models, candidate generation, and pair selection

PairDistill uses a two-stage teacher process on retrieved candidates (Huang et al., 2024). First, for each query, the current dense retriever retrieves top-$s(q,d)$5 passages. Second, those candidates are reranked in two ways. A pointwise reranking teacher reranks each passage independently and provides $s(q,d)$6; in the main experiments this teacher is MiniLM cross-encoder. A pairwise reranking teacher compares two passages jointly and returns $s(q,d)$7; in the main experiments this teacher is duoT5-3B. The paper also notes that pairwise reranking can be done in a supervised/classification-based way using labeled triplets, or in an instruction-based way using an LLM in zero-shot mode.

Pairwise supervision is more expensive than pointwise scoring, so PairDistill does not enumerate all possible pairs. Instead, it defines the pair set with a heuristic: $s(q,d)$8 The intuition given in the paper is threefold: documents very far apart in the ranking are already easy to distinguish; pairs near each other provide more useful training signal; and pairwise supervision is computationally expensive, so restricting the pair set reduces cost. In experiments, the paper uses this heuristic and reports that removing it slightly hurts performance.

This pair-selection mechanism is a defining practical feature rather than a minor optimization. PairDistill’s claim is not that every possible teacher comparison is necessary, but that local ranking ambiguities are where pairwise teachers are most informative. A plausible implication is that PairDistill concentrates computation on the part of the ranking where calibration errors and fine-grained preference reversals are most consequential.

4. Iterative training and optimization procedure

PairDistill includes an iterative refinement loop rather than a one-shot distillation pass (Huang et al., 2024). The procedure is: train a retriever; use it to retrieve top-$s(q,d)$9 passages; rerank them with pointwise and pairwise teachers; fine-tune the retriever using $d_i$0; then use the updated retriever for the next iteration. The paper’s rationale is that if distillation uses only a fixed retrieved set, the model can overfit to those passages, whereas refreshing the candidate set allows the retriever to improve over time.

The paper gives a simplified algorithmic view:

1. retrieve candidates for query $d_i$1 with the current dense retriever;
2. pointwise rerank the top-$d_i$2 candidates using a cross-encoder;
3. pairwise rerank sampled pairs among the top candidates using a pairwise teacher;
4. compute losses—contrastive loss for labeled positives, KL distillation to the pointwise teacher, and KL distillation to the pairwise teacher;
5. update retriever parameters;
6. optionally refresh retrieval and repeat.

In the reported experiments, PairDistill uses ColBERTv2 as the main retriever initialization, and also tests DPR to show architecture-agnostic gains. The retrieval budget is to retrieve top-100 passages, sample 50 pairs per query for pairwise reranking, with $d_i$3, $d_i$4, and temperature $d_i$5. Training is conducted on 4 V100 GPUs, 32GB each, and inference uses the PLAID engine for efficient late-interaction retrieval. The paper also specifies max document length 300 for BEIR/LoTTE and mentions specific query length settings for some datasets.

The iterative structure matters conceptually because PairDistill is not only transferring teacher judgments onto a static candidate pool. It is repeatedly redefining the candidate space seen during training. The paper reports improvement after the first iteration, best or near-best after two iterations, and convergence after that. This suggests that pairwise distillation is coupled to retrieval quality itself: better retrieval produces more informative candidate comparisons, which in turn produce a better retriever.

5. Experimental settings and empirical results

PairDistill is evaluated in several regimes (Huang et al., 2024). For supervised training, the paper uses the MS MARCO training set with about 502K labeled queries and about 8.8M passages, and also uses 306K unlabeled queries for distillation. The primary in-domain benchmarks are MS MARCO dev, measured by MRR@10, and TREC DL19 / DL20, measured by NDCG@10. Out-of-domain and zero-shot evaluation uses BEIR with 18 datasets and metric NDCG@10, and LoTTE with Recall@5 / Success@5-style reporting in the paper’s summary. Open-domain QA retrieval uses Natural Questions, TriviaQA, and SQuAD, with Recall@5.

The paper reports that PairDistill achieves state-of-the-art or best-in-class performance across several settings. On MS MARCO dev, PairDistill: 40.7 MRR@10, compared with ColBERTv2 baseline: 39.7. On TREC DL19 / DL20, it is reported as Best on DL19 and Near-best / second-best on DL20. On BEIR, it achieves Best overall average across BEIR subsets among comparable models, outperforming strong baselines such as ColBERTv2, DRAGON+, SPLADE++, ABEL-FT, and GTR-XXL. The paper emphasizes that this is notable because it is zero-shot and often beats even domain-adapted models. On LoTTE, it reports state-of-the-art on both search and forum subsets. In open-domain QA retrieval, Recall@5 is reported as NQ: 71.8, TriviaQA: 77.4, and SQuAD: 66.9, all better than the ColBERTv2 and BM25 baselines reported in the paper.

The ablations sharpen the interpretation of these results. Removing $d_i$6 hurts performance, indicating that the pairwise teacher contributes beyond pointwise distillation. Using only $d_i$7 slightly hurts compared to full PairDistill, so the best configuration combines pointwise and pairwise KD. Removing the pair sampling heuristic causes performance to drop slightly. The method works not only with ColBERTv2 but also with bert-base-uncased and DPR, which the paper presents as evidence of broad applicability.

One particularly important analysis reports that pointwise reranking disagrees with the pairwise teacher on a substantial fraction of sampled pairs—about 31% disagreement in one analysis. The paper uses this to support the claim that pairwise and pointwise teachers provide meaningfully different supervision, and that pairwise supervision can capture more accurate relative preferences. In the zero-shot setting, when using flan-t5-xl for instruction-based reranking, pairwise distillation still improves over pointwise KD alone on target-domain datasets such as FiQA, BioASQ, and Climate-FEVER.

6. Relation to adjacent pairwise distillation frameworks, misconceptions, and limitations

A recurring source of confusion is that PairDistill names a specific dense retrieval method, but the broader idea of pairwise distillation appears in several adjacent literatures. The following comparison reflects only the distinctions explicitly described in the cited papers.

Method	Domain	Core transfer signal
PairDistill (Huang et al., 2024)	Dense retrieval	Pairwise relevance preferences
GRKD (Li et al., 29 Apr 2025)	Class output distillation	Ordered class pairs
PRD (Wu et al., 7 Jul 2025)	Document ranking	Pairwise teacher labels from PRP
DiPair (Chen et al., 2020)	Text-pair modeling	Teacher-generated soft labels
PDD (Zhang et al., 2024)	Real-world image SR	Pairwise distances

PairDistill is closest in spirit to Group Relative Knowledge Distillation (GRKD), which also shifts supervision from absolute outputs to relative structure (Li et al., 29 Apr 2025). GRKD constructs ordered class pairs

$d_i$8

and trains the student to preserve the teacher’s preference orderings using a pairwise ranking-style loss. The shared principle is the same: preserve ranking consistency rather than exact probability regression. The difference is that GRKD is formulated for class output distributions and combines a group relative loss with a standard soft-label term, whereas PairDistill is formulated for dense retrieval and combines contrastive learning, pointwise KD, and pairwise KD.

It is also closely related to Pairwise Ranking Distillation (PRD), which distills a pointwise student ranker from Pairwise Ranking Prompting (PRP) teacher labels (Wu et al., 7 Jul 2025). PRD uses a Pairwise Logistic Ranking Loss and emphasizes that with only 2% of pairs, it can match the performance of using all pairs for teacher supervision. The relation is conceptual rather than terminological: PRD is specifically about transferring the ranking behavior of a pairwise LLM teacher into a cheaper pointwise ranker, while PairDistill is about distilling pairwise reranking supervision into a dense retriever.

By contrast, DiPair is not a pairwise preference distillation method in the same sense (Chen et al., 2020). It is a dual-encoder + head framework for text pairs or n-ary tuples, distilled from a BERT teacher with teacher-generated soft labels. Its main concern is scalable interaction modeling for pair inputs rather than pairwise teacher judgments between documents. Likewise, Pairwise Distance Distillation (PDD) uses pairwise distances between model outputs in unsupervised real-world image super-resolution, not relevance preferences (Zhang et al., 2024). These distinctions matter because “pairwise distillation” is a family resemblance, not a single method class.

The main practical limitation identified for PairDistill is computational. Pairwise distillation may require more training pairs, pairwise reranking is more expensive than pointwise scoring, and the method depends on a strong teacher, a candidate retrieval stage, and efficient pair generation and reranking infrastructure (Huang et al., 2024). The paper mitigates this with pair sampling, but cost remains a consideration. A common misconception is therefore that PairDistill eliminates reranking expense; the paper does not make that claim. Its contribution is to use expensive pairwise supervision selectively during training so that the final system is still a dense retriever. Another misconception is that PairDistill rejects pointwise knowledge entirely; in fact, the reported best-performing formulation combines contrastive learning, pointwise KD, and pairwise KD, indicating that relative supervision complements rather than wholly displaces absolute teacher signals.