Relation-Based Pairwise Similarity Distillation

• Relation-Based Pairwise Similarity Distillation is a method that transfers structured inter-instance and inter-class relationships from a teacher model to a student model.
• It employs techniques like cosine similarity, normalization, and KL or L2 losses to explicitly align relational information, enhancing discrimination and robustness.
• The approach is applied across domains—vision, graphs, retrieval, and language—improving model compression, fine-grained recognition, and cross-modal tasks.

Relation-Based Pairwise Similarity Distillation is a paradigm in knowledge distillation and representation learning that aims to transfer structural or relational information from a high-capacity teacher model to a student model by explicitly aligning pairwise relationships—such as similarities or relative orderings—rather than, or in addition to, aligning unstructured outputs or individual embeddings. The motivation arises from limitations observed in classical distillation losses—which focus on pointwise outputs, such as logits or feature activations—where informative inter-instance or inter-class relations are lost, compromising discrimination, robustness, and transfer. The last several years have seen a proliferation of methods that encode, preserve, and distill pairwise relational knowledge in a range of domains, including vision, graphs, retrieval, language, and speech.

1. Core Principles of Relation-Based Pairwise Similarity Distillation

At its core, relation-based pairwise similarity distillation constructs, from internal representations or outputs, a similarity or affinity measure between pairs of samples (or classes, or features), possibly within a batch, graph, or memory bank. The distilled quantity may be:

• A full pairwise similarity matrix $S \in \mathbb{R}^{b \times b}$ within a batch (e.g., cosine, inner product, Euclidean distance) (Tung et al., 2019, Li et al., 2022).
• Neighborhood-based similarities in structured data, e.g., nodes and their 1-hop neighbors in a graph (Chen et al., 2024).
• Cross-modal image-text relations (Chung et al., 26 Mar 2026).
• Logit-level pairwise comparison signals (e.g., margin-based class logit differences) (Li et al., 29 Apr 2025, Xu et al., 21 Jul 2025).
• Pairwise probabilities or preference scores (e.g., in ranking, comparison, or analogy tasks) (Wu et al., 7 Jul 2025, Li et al., 29 Apr 2025, Ushio et al., 2021).

The principle guiding these objectives is that a student should not merely mimic the teacher's pointwise behavior, but should also preserve the topology, margin structure, and discriminability encoded by the teacher's relations between instances or classes. Various methods convert raw similarities to probability distributions (e.g., with temperature-softmax) and minimize a distance (usually KL divergence or $L_2$) between teacher and student similarity distributions. Others formulate a margin-based, contrastive, or triplet ranking loss to preserve the teacher’s preference ordering.

2. Methodological Frameworks and Loss Formulations

2.1. Affinity Construction, Normalization, and Matching

As systematized in (Li et al., 2022), the process can be modularized into three components:

• Affinity Construction: Selecting the similarity measure $g(\cdot, \cdot)$, typically cosine or dot product (which empirically outperform distance metrics).
• Normalization: Applying row-wise or matrix-based normalization (e.g., row-$L_2$) to remove magnitude bias and standardize scale.
• Loss Function: Computing a divergence (e.g., Frobenius, KL, smooth-$L_1$) between normalized teacher and student affinity matrices.

For instance, Similarity-Preserving KD (Tung et al., 2019) computes normalized batchwise cosine similarity matrices for both teacher and student, then matches them via an $L_2$ (Frobenius) loss:

$L_\text{SP} = \frac{1}{b^2} \| G_T - G_S \|_F^2$

Alternatively, temperature-softmax and KL divergence are employed to form "soft" similarity distributions, as in graph node neighborhoods (Chen et al., 2024) and relational representation distillation (Giakoumoglou et al., 2024).

2.2. Neighborhood-, Memory-, and Class-Structure

Certain approaches restrict relational matching to structured localities—such as 1-hop node neighborhoods in graphs (Chen et al., 2024), memory bank retrievals in embeddings (Mishra et al., 15 Aug 2025, Giakoumoglou et al., 2024), or class-centric centroids for imbalanced data (Xing et al., 2021). Passage-centric loss (Ren et al., 2021) and class-wise triplet mining (Ye et al., 2022) instantiate structure- or task-aware tuple formation.

2.3. Margin-Based and Ranking Losses

Rank-based distillation, as in Group Relative Knowledge Distillation (GRKD) (Li et al., 29 Apr 2025) and LDRLD (Xu et al., 21 Jul 2025), computes for every teacher output the set of class pairs $(i, j)$ such that $s_i > s_j$ and imposes on the student a margin-style objective (e.g., log-sigmoid on differences between student logits), focusing on preserving teacher-derived orderings or margins. In retrieval and document ranking, preference-based pairwise logistic losses are used (Wu et al., 7 Jul 2025).

3. Applications Across Domains

3.1. Graph Representation and Autoencoding

ClearGAE (Chen et al., 2024) introduces pairwise similarity distillation for graph autoencoders, whereby node feature neighborhoods' softmaxed cosine similarity patterns are distilled from the encoder (teacher) to the decoder (student). The objective supplements the classic MSE with a KL constraint between soft neighborhoods, restoring node distinctness and preventing representation collapse in reconstructed graphs. ClearGAE achieves consistent gains across node classification, link prediction, and graph classification tasks.

3.2. Vision: Classification, Fine-Grained Recognition, and Face Identification

In image classification, methods such as Similarity-Preserving KD (Tung et al., 2019), mAKD (Li et al., 2022), and RPSD (Mishra et al., 15 Aug 2025) demonstrate that transferring affinity structure via pairwise similarity matching consistently boosts student accuracy over standard KD or attention transfer. Local dense logit distillation (Xu et al., 21 Jul 2025) recursively extracts top logit pairs, adaptively weights them, and distills local pairwise KL divergences, resulting in additional gains on fine-grained datasets. Categorical contrastive frameworks (Xing et al., 2021) employ supervised memory banks and sample-to-class centroids to handle class imbalance and high intra-class variance in medical images.

3.3. Cross-Modal and Multi-Task Transfer

CLIP-RD (Chung et al., 26 Mar 2026) introduces vertical and cross relational distillation to preserve the geometry and symmetry of cross-modal embedding relations across teacher and student models, not merely matching aligned pairs but the full interaction distributions, confirmed by improved performance in zero-shot classification and retrieval.

3.4. Retrieval, Ranking, NLP, and Multimodal

PAIR (Ren et al., 2021) in passage retrieval and PRP-based ranking distillation (Wu et al., 7 Jul 2025) both adapt relation-based objectives to dual-encoder architectures and pointwise rankers, respectively, demonstrating that harvesting and distilling pairwise preference signals or similarity margins leads to significant improvements over pure pointwise supervision, even with sparse pair sampling.

In NLP, the distillation of relation embeddings (Ushio et al., 2021) leverages a prompt-based architecture and enforces a triplet margin and pairwise classification loss over word-pair relations, setting new standards in analogy and relation-classification accuracy.

4. Empirical Performance and Practical Deployment

Extensive evaluations show that relation-based pairwise similarity distillation methods deliver consistent, often substantial improvements across:

• Compression settings, where students are much smaller than teachers (Tung et al., 2019, Li et al., 2022, Mishra et al., 15 Aug 2025).
• Transfer learning, few-shot, or cross-task distillation when label sets are partially or totally non-overlapping (Ye et al., 2022, Ushio et al., 2021).
• Fine-grained, imbalanced, or high intra-class variance datasets (e.g., medical images, bird species) where conventional KD fails to separate classes (Xing et al., 2021, Li et al., 29 Apr 2025).
• Resource-constrained settings, as relational losses often impose modest computational and memory overhead (e.g., ClearGAE’s $O(|E| F)$ neighbor-only scaling (Chen et al., 2024), RRD's $L_2$0 FLOPs (Giakoumoglou et al., 2024)).

A recurring outcome is improved Top-5 or Top-10 classification retrieval accuracy, as the student model better reproduces the teacher's inter-class proximity structure, with especially dramatic gains in retrieval, ranking, or analogical reasoning tasks (Wu et al., 7 Jul 2025, Ushio et al., 2021).

Representative Method	Relational Quantity	Key Loss/Matching	Domain
ClearGAE (Chen et al., 2024)	Node neighborhood sim	KL(P_{ij}
SP KD (Tung et al., 2019)	Batchwise affinity	$L_2$1 (Frobenius)	Vision
mAKD (Li et al., 2022)	Batchwise affinity	Flexible (KL, SL1, $L_2$2)	Vision
RRD (Giakoumoglou et al., 2024)	Memorybank similarity	KL(Tᵢ
CLIP-RD (Chung et al., 26 Mar 2026)	Intra/cross-modal sim	InfoNCE + KL, multi-dir	Multi-modal
LDRLD (Xu et al., 21 Jul 2025)	Logit pairwise rel.	Weighted pairwise KL	Vision (fine-grained)
GRKD (Li et al., 29 Apr 2025)	Logit orderings	Margin, pairwise log-sigmoid	Classification/LLM

5. Variations, Limitations, and Theoretical Considerations

Relation-based distillation methods differ in choice of:

• Affinity metrics (cosine, dot product, distance), with cosine similarity outperforming in most cases (Li et al., 2022, Tung et al., 2019).
• Normalization scheme (row-wise, matrix-wise), influencing stability and convergence (Li et al., 2022).
• Loss function (KL, $L_2$3, smooth $L_2$4), with KL and smooth $L_2$5 observed to offer the most robust and generalizable behavior (Li et al., 2022).
• Sampling regime: full batch, memorybank, graph neighborhood, or adaptively mined "hard" tuples (Mishra et al., 15 Aug 2025, Ye et al., 2022).

Potential limitations include scaling to extremely large graphs or corpora due to $L_2$6 pairwise similarity. This is often mitigated via local neighborhoods, memory banks, or sparse sampling (Chen et al., 2024, Giakoumoglou et al., 2024, Wu et al., 7 Jul 2025). Another caveat is that relation-based objectives presuppose that the teacher’s similarity structure is meaningful; poor or insufficiently trained teachers can propagate undesirable bias or noise in relations (Tung et al., 2019).

On the theoretical front, methods such as RRD (Giakoumoglou et al., 2024) interpolate between InfoNCE (instance discrimination) and KL-based soft supervision by relaxing the rigid one-hot target to a structured, temperature-controlled distribution. GRKD (Li et al., 29 Apr 2025) articulates why the preservation of pairwise orderings (inductive bias) is more robust to calibration errors than absolute matching.

6. Extensions, Generalizations, and Outlook

Relation-based pairwise similarity distillation has been generalized across numerous tasks and data modalities, including:

• Graphs (node, link, and graph-level prediction) (Chen et al., 2024);
• Visual representation learning, face recognition (Mishra et al., 15 Aug 2025), and fine-grained image recognition (Xu et al., 21 Jul 2025);
• Medical image and imbalanced data (Xing et al., 2021);
• Large-scale (trillion-pair) text pair modeling and retrieval (Chen et al., 2020, Ren et al., 2021);
• Embedding geometric structure preservation in CLIP-like multi-modal models (Chung et al., 26 Mar 2026);
• Most recently, LLM ranking and preference-based distillation (Li et al., 29 Apr 2025, Wu et al., 7 Jul 2025).

Open research directions include interleaving higher-order relational knowledge (triplets, clusters), non-Euclidean geometry, dynamic affinity metrics, and applications in open-set recognition, continual learning, complex retrieval, and structured prediction (e.g., NER, parsing). Dynamic balancing and instance-adaptive weighting, as in GNoRP (Li et al., 2022, Mishra et al., 15 Aug 2025), have the potential to make such approaches more robust and less sensitive to hyperparameters.

7. Summary Table of Key Approaches

Method	Distilled Relation	Distillation Objective	Primary Domain	arXiv ID
ClearGAE	Node–neighbor similarities	KL between softmaxed cosines	Graph autoencoders	(Chen et al., 2024)
SP KD	Full batch similarities	Batchwise normalized $L_2$7	Image classification	(Tung et al., 2019)
mAKD	Modular affinity + loss	(CS, row-$L_2$8, smooth $L_2$9/KL)	Deep nets (generic)	(Li et al., 2022)
RRD	Memory bank pairwise sim.	Dual-temp KL over similarity dists	Vision, representation	(Giakoumoglou et al., 2024)
CLIP-RD	Intra/cross-modal sim.	VRD+XRD (KL + InfoNCE)	Multimodal, retrieval	(Chung et al., 26 Mar 2026)
LDRLD	Dense logit pair relations	Weighted pairwise KL	Fine-grained image	(Xu et al., 21 Jul 2025)
CRCKD	Class-guided sample and centroid	CCD + CRP (contrastive+KL)	Medical, imbalanced	(Xing et al., 2021)
DistillHash	Pairwise similarity on data	Bayesian KL with distilled pairs	Unsupervised hashing	(Yang et al., 2019)
Pairwise Preference Distillation	Pairwise doc rankings	Pairwise logistic regression	Document ranking	(Wu et al., 7 Jul 2025)
GRKD	Pairwise class orderings	Margin log-sigmoid ordering loss	LLMs, fine-grained classes	(Li et al., 29 Apr 2025)

The development and empirical validation of relation-based pairwise similarity distillation has established it as a robust, flexible, and theoretically grounded complement to classical distillation, capable of recovering both discriminative and geometric structure in student models across a wide range of architectures and modalities.