Relational Distillation: Structural Knowledge Transfer

• Relational distillation is a technique that transfers not only individual predictions but also the inter-sample relationships, such as distances and angles, from the teacher to the student.
• It utilizes loss functions like Huber and KL divergence to align pairwise distances, angular similarities, and logit-space relationships between model embeddings.
• Empirical evidence shows that relational distillation improves performance in fine-grained, heterogeneous, and multi-modal tasks by capturing complex inductive biases.

Relational distillation refers to a family of knowledge distillation techniques in which the student model is trained to match not only pointwise outputs or activations from the teacher, but also the relations—such as distances, angles, or pairwise similarities—among samples in the feature or logit space. This approach generalizes classical knowledge distillation by encoding structural, geometric, or semantic relationships, with evidence that it yields superior transfer of complex inductive biases, especially in domains requiring fine-grained structure or robust generalization.

1. Conceptual Foundations of Relational Distillation

Classical knowledge distillation (KD) focuses on instance-level knowledge by matching teacher and student soft-target distributions or intermediate activations. Relational distillation, in contrast, introduces objectives that align mutual relations among data points as embedded by the models. The canonical formulation was presented as Relational Knowledge Distillation (RKD) (Park et al., 2019), which for a mini-batch $\{x_i\}$ computes teacher and student embeddings $t_i = f_T(x_i)$, $s_i = f_S(x_i)$, and forms losses to align structural relationships such as scaled Euclidean distances and angle-wise similarities between embedding triplets.

Variants expand these ideas to other spaces (logit, semantic tokens, graph nodes) and settings (heterogeneous architectures, semi-supervised learning, multi-modality, and even quantum feature spaces). The central tenet is that by preserving the teacher’s relational geometry, the student can capture not just “what is likely,” but “how samples and concepts relate”—yielding better manifold structure and often superior generalization.

2. Mathematical Formalisms and Loss Functions

Relational distillation methods operationalize the notion of “structure transfer” via several technical mechanisms:

A. Pairwise Distance Alignment

$$\mathcal{L}_{\mathrm{dist}} = \sum_{i<j} \mathrm{Huber}\Big( \frac{\|t_i-t_j\|_2}{\mu_T}, \frac{\|s_i-s_j\|_2}{\mu_S} \Big)$$

where $\mu_*$ denotes mean batchwise teacher or student distance. (RKD (Park et al., 2019))

B. Angle/Triplet Similarity Alignment

$$\mathcal{L}_{\mathrm{angle}} = \sum_{i,j,k} \mathrm{Huber}\big( \langle \mathbf{e}^{ij}_T, \mathbf{e}^{kj}_T \rangle,\, \langle \mathbf{e}^{ij}_S, \mathbf{e}^{kj}_S \rangle \big)$$

with $\mathbf{e}^{ij} = (x_i - x_j)/\|x_i - x_j\|$.

C. Similarity Distribution Matching

$$\mathcal{L}_{\mathrm{rel}} = \sum_i \mathrm{KL}\Big(p^T_i(j),\,p^S_i(j)\Big)$$

where $p^T_i(j)\propto \exp(\phi(t_i,t_j)/\tau_T)$, and analogously for the student, over a memory bank or batch ($\phi$: similarity function) (Giakoumoglou et al., 2024).

D. Logit-Space and Class-wise Relations

Decoupled logit relational losses align both class-conditional and sample-wise affinity matrices, for example: $$\mathcal{L}_{\rm class} = \frac{1}{B}\sum_i \mathrm{KL}(R^s_{\mathrm{class},i} \| R^t_{\mathrm{class},i}),$$ with $R_{\mathrm{class},i}$ the softmax-normalized pairwise logit similarity matrix for sample $i$ (Yang et al., 10 Feb 2025).

E. Graph and High-Order Relational Losses

In graph domains, one aligns relational graph structures (e.g., adjacency or spectral embeddings) (Wang et al., 2024), or MetaCorr matrices of node-type means (Liu et al., 2022).

F. Quantum Kernel Alignment

Features are mapped to quantum Hilbert space, and the student matches quantum kernel similarities computed as pairwise fidelities (Liu et al., 18 Aug 2025): $$\mathcal{L}_{\mathrm{qr}} = \sum_{i<j} \mathrm{Huber}(k_s(x_i,x_j),\,k_t(x_i,x_j))$$

3. Principal Algorithmic Templates

Relational distillation mechanisms are architected along several motifs:

• Batchwise Structure Alignment: Compute all pairwise (and potentially triplet) relations in a batch for both teacher and student, minimizing the divergence under a robust loss (Huber or KL).
• Local Pairwise Logit Decoupling: Decompose the softmax logit vector into micro-distributions over top-d classes, recursively decoupling and recombining logits, then aligning these local 2-class (or d-class) distributions (Xu et al., 21 Jul 2025).
• Feature/Activation Graph Construction: For CNNs, channels at each layer are nodes, with edges encoding channel–channel similarity; for sample sets, nodes are examples, and edges quantify affinity (cosine, Pearson, quantum kernels) (Wang et al., 2024, Xu et al., 2024).
• Memory Banks and Hard Mining: Memory banks stabilize relational targets across mini-batches, allowing efficient mining or weighting of hard relational pairs (Mishra et al., 15 Aug 2025).
• Auxiliary Class-Oriented Networks: Trainable relation networks are used to extract and reinforce class-discriminative relations, supplementing handcrafted metrics (Yu et al., 2023).
• Dynamic Multi-Scale Fusion: Multi-stage features are dynamically fused and their aggregate relations are matched to the teacher, particularly in heterogeneous (cross-architecture) scenarios (Yang et al., 10 Feb 2025).

4. Application Domains and Empirical Gains

Relational distillation has been adapted to a broad spectrum of modalities and tasks.

Metric Learning & Face Recognition: Aligning relational geometry enables superior recall/verification, with students matching or even exceeding teachers on CUB, Cars-196, and LFW benchmarks (Park et al., 2019, Mishra et al., 15 Aug 2025).

Vision Transformers and CNNs: Semantic relation distillation via superpixels (SeRKD) improves transfer and generalization, especially in compressing ViTs for ImageNet and downstream tasks (Yan et al., 27 Mar 2025).

Graph Learning: In heterogeneous graphs, multi-type relational distillation improves classification, clustering, and node embedding structure (Liu et al., 2022). For graph data distillation, aligning relational entity graphs is crucial (Gao et al., 8 Oct 2025).

LLMs and Analogical Reasoning: Fine-tuned function vectors capturing inter-concept relations boost LLMs’ analogy performance, far-analogy tests, and align with human relational judgments (Kang et al., 13 Jan 2026).

Logit-based KD: Local dense relational logit distillation (LDRLD) explicitly models all critical inter-class logit pairs, improving distillation across CNN, ViT, and hybrid architectures on CIFAR-100, ImageNet, and Tiny-ImageNet (Xu et al., 21 Jul 2025).

Self-Supervised & Unlabeled Settings: Relational distillation via compact descriptors and queuing allows unlabeled compression for copyright detection, closing the gap to large self-supervised teachers (Kim et al., 2024); prompt-based relational graph distillation enables annotation-free extraction of task-relevant structure from foundation models (Xu et al., 2024).

5. Theoretical Insights, Guarantees, and Analysis

There is now rigorous theoretical grounding for the clustering and generalization properties of relational KD. Modeling the teacher’s feature-induced similarity as a weighted population graph, minimizing the expected squared error between teacher–student similarities is equivalent to spectral clustering with label-efficient learning guarantees (Dong et al., 2023). Under reasonable partition and margin assumptions, population-level relational distillation provably induces clusterings close to ground truth, with finite-sample guarantees scaling as inverse square root of the unlabeled sample size.

In semi-supervised learning, the “global” structure induced by relational loss complements the “local” regularizations from consistency objectives. Combining both strengthens generalization, especially for weak augmentations or scarce unlabeled data.

6. Extensions: Multi-Modality, Quantum, and Heterogeneous Structures

Recent work expands relational distillation to:

• Multi-modal Distillation: Image-to-3D/LiDAR relational distillation aligns the structure of representations across 2D–3D domains, improving performance for zero-/few-shot 3D segmentation and reducing mode mismatch (Mahmoud et al., 2024).
• Quantum Relational Distillation: By embedding feature vectors as quantum states and aligning quantum kernel similarities, quantum-enhanced RKD achieves consistent but modest improvements in both vision and language tasks, even though all inference remains classical (Liu et al., 18 Aug 2025).
• Relational Database Compression: Relational Database Distillation (RDD) leverages graph-construction, clustering, and kernel ridge regression–guided objectives to distill multi-table RDBs into highly compressed heterogeneous graphs for scalable GNN training, preserving both fidelity and inter-table relational structure (Gao et al., 8 Oct 2025).

7. Comparative Analysis, Limitations, and Best Practices

Empirical ablations show that:

• Relational alignment (distance, angle, affinity, or logit relations) consistently outperforms instance-only KD (Park et al., 2019, Giakoumoglou et al., 2024, Xu et al., 21 Jul 2025).
• Dynamic/adaptive weighting (e.g., hard-mining, adaptive decay) further improves transfer, emphasizing hard-to-distinguish pairs (Mishra et al., 15 Aug 2025, Xu et al., 21 Jul 2025).
• Heterogeneous architectures benefit from decoupled relation alignment and dynamic fusion, which are more robust to architectural mismatches than fixed-layer or fixed-relation approaches (Yang et al., 10 Feb 2025).
• Limiting computation/scaling with batch size or local superpixel/cluster sampling is necessary to maintain tractability, especially for angle losses ($O(B^3)$) (Yan et al., 27 Mar 2025).

Notable open challenges include scaling quantum relational methods, developing adaptive structural thresholds for graph distillation, and extending part-wise or cross-image semantic relation alignment. For best results, batchwise normalization, soft teacher distributions, memory banks, and a combination of distance/angle metrics with KL-based logit matching are commonly recommended.

---

In summary, relational distillation unifies and advances classical, graph-based, semantic, logit, and multi-modal knowledge-transfer techniques by explicitly modeling and transferring the teacher’s inductive geometry—yielding consistent, often state-of-the-art gains across architectures and modalities (Park et al., 2019, Xu et al., 21 Jul 2025, Yang et al., 10 Feb 2025, Giakoumoglou et al., 2024, Liu et al., 2022, Wang et al., 2024, Yan et al., 27 Mar 2025, Xu et al., 2024, Gao et al., 8 Oct 2025, Mahmoud et al., 2024, Kang et al., 13 Jan 2026, Mishra et al., 15 Aug 2025, Yu et al., 2023).