Local Dense Relational Logit Distillation

• The paper introduces LDRLD, a method that improves knowledge distillation by recursively selecting top student logits and forming dense, pairwise relations.
• It leverages Local Logit Knowledge Integrity and Remaining Non-Target Knowledge to balance fine-grained discriminability with overall transfer accuracy.
• Empirical results on benchmarks like CIFAR-100 and ImageNet demonstrate that LDRLD consistently outperforms traditional KD methods with modest computational overhead.

Searching arXiv for the primary paper and closely related distillation work to ground the article in cited literature. to=arxiv_search.search 天天中json code syntax not sure. to=arxiv_search.search 彩神争霸输钱json code {"query":"(Xu et al., 21 Jul 2025)"}【อ่านข้อความเต็มjson code to=arxiv_search.search ￣奇米影视json code {"query":"(Xu et al., 21 Jul 2025)"} күңелjson code Searching for related arXiv papers on local dense relational/logit distillation and relational distillation variants. Local Dense Relational Logit Distillation (LDRLD) is a logit-based knowledge distillation method introduced in 2025 to transfer fine-grained inter-class knowledge from a teacher to a student by recursively selecting a ranked local subset of logits, recombining that subset into dense pairwise relations, weighting those relations according to rank structure, and distilling the remaining non-target logits to preserve knowledge completeness. In the formulation of “Local Dense Logit Relations for Enhanced Knowledge Distillation” (Xu et al., 21 Jul 2025), LDRLD addresses a limitation of vanilla logit distillation: global softened-softmax alignment over all $C$ classes tends to emphasize high-probability classes, suppress differences among low-probability classes, and weaken fine-grained inter-class discriminability, particularly for semantically similar classes.

1. Conceptual basis and motivation

The starting point for LDRLD is standard logit-based KD. Let teacher and student logits be $Z^t \in \mathbb{R}^{1\times C}$ and $Z^s \in \mathbb{R}^{1\times C}$, with softened probabilities $p^t$ and $p^s$ obtained by softmax with temperature. Vanilla KD minimizes $\mathrm{KL}(P^t\Vert P^s)$ globally over all classes. The paper argues that this global coupling induces information redundancy and weakens discriminability among confusing classes, because unrelated classes participate in the same normalization and dilute local contrasts (Xu et al., 21 Jul 2025).

LDRLD replaces that globally coupled view with a local dense relational view. Rather than treating the logit vector only as a distribution over all classes, it identifies a small ranked subset of classes from the student’s current hypotheses, constructs all pairwise relations inside that subset, and distills those relations with adaptive importance weights. The method then distills the remaining non-target logits outside the selected subset so that the student is not trained only on a truncated local view.

A central intuition is given by pairwise normalization. When the relation between two categories is computed through a two-class softmax, the inter-class contrast is sharper than under global softmax; the paper notes that $|p_1^t-p_2^t|$ under pairwise normalization is larger than under global normalization. This local contrast is used as the basic carrier of fine-grained knowledge in LDRLD (Xu et al., 21 Jul 2025).

2. Recursive local subset construction and dense pair formation

For a given sample, LDRLD sorts the student logits $Z^s$ in descending order and denotes by $\pi(d)$ the index of the $d$-th largest logit. The recursion depth $Z^t \in \mathbb{R}^{1\times C}$0 determines the size of the local subset. At iteration $Z^t \in \mathbb{R}^{1\times C}$1, the current top-1 unmasked student logit is extracted, with

$Z^t \in \mathbb{R}^{1\times C}$2

and a mask $Z^t \in \mathbb{R}^{1\times C}$3 is applied so that

$Z^t \in \mathbb{R}^{1\times C}$4

The teacher and student logits are then updated by Hadamard masking,

$Z^t \in \mathbb{R}^{1\times C}$5

which prevents reselection of the same logit. After $Z^t \in \mathbb{R}^{1\times C}$6 iterations, the selected teacher and student logits form

$Z^t \in \mathbb{R}^{1\times C}$7

The selected subset is then recombined into all pairwise class relations. For every pair $Z^t \in \mathbb{R}^{1\times C}$8 with $Z^t \in \mathbb{R}^{1\times C}$9, LDRLD concatenates the two logits into a two-dimensional relation using $Z^s \in \mathbb{R}^{1\times C}$0. The paper writes this recursively as

$Z^s \in \mathbb{R}^{1\times C}$1

with the analogous definition for $Z^s \in \mathbb{R}^{1\times C}$2.

Each pair is normalized independently by a two-class softmax with temperature $Z^s \in \mathbb{R}^{1\times C}$3. For a pair $Z^s \in \mathbb{R}^{1\times C}$4,

$Z^s \in \mathbb{R}^{1\times C}$5

and

$Z^s \in \mathbb{R}^{1\times C}$6

The resulting unweighted relational loss over all pairs among the $Z^s \in \mathbb{R}^{1\times C}$7 selected logits is

$Z^s \in \mathbb{R}^{1\times C}$8

This construction is dense in the sense that the selected local subset generates $Z^s \in \mathbb{R}^{1\times C}$9 pairwise relations, but local in the sense that it is restricted to the student’s current top-$p^t$0 hypotheses rather than all $p^t$1 class pairs (Xu et al., 21 Jul 2025).

3. Objective function: ADW, LLKI, and remaining non-target knowledge

LDRLD supplements pairwise relational transfer with two additional components. The first is Local Logit Knowledge Integrity (LLKI), which preserves the sequential arrangement of the selected logits. LLKI is defined as

$p^t$2

where $p^t$3 and $p^t$4, and $p^t$5 is softmax over the $p^t$6 selected logits.

The second is the Adaptive Decay Weight (ADW), which assigns different weights to different class pairs. ADW combines Inverse Rank Weighting (IRW) and Exponential Rank Decay (ERD). If $p^t$7 and $p^t$8 are the rank positions of two classes in the local subset, then

$p^t$9

and

$p^s$0

The combined weight is

$p^s$1

IRW emphasizes pairs whose ranks are closer, while ERD reduces weights for pairs lower in the ranked list. The weighted local relational loss is therefore

$p^s$2

The local objective becomes

$p^s$3

After the top-$p^s$4 subset is removed, LDRLD distills the remaining non-target logits,

$p^s$5

using

$p^s$6

where the softmax is restricted to indices $p^s$7. The paper distinguishes this “remaining non-target knowledge” from DKD’s non-target KD: RNTK excludes the entire top-$p^s$8 subset rather than excluding only the true target class, thereby reducing reliance on common or high-confidence categories and activating low-confidence classes.

With task loss

$p^s$9

the final objective is

$\mathrm{KL}(P^t\Vert P^s)$0

where $\mathrm{KL}(P^t\Vert P^s)$1 and $\mathrm{KL}(P^t\Vert P^s)$2 balance local relational transfer and remaining non-target transfer (Xu et al., 21 Jul 2025).

4. Optimization procedure, hyperparameters, and computational profile

The training pipeline is explicitly algorithmic. For each mini-batch, the teacher and student logits are computed; a local subset of size $\mathrm{KL}(P^t\Vert P^s)$3 is built by repeatedly extracting the current top-1 logit of the student and masking it out; all class pairs inside that subset are generated; pairwise probabilities are computed via two-class softmax; the local relation loss with ADW and the LLKI loss are evaluated; the remaining non-target KL loss is computed on indices $\mathrm{KL}(P^t\Vert P^s)$4; and the total loss is backpropagated through the student (Xu et al., 21 Jul 2025).

The paper reports experiments on CIFAR-100, ImageNet-1K, Tiny-ImageNet, Market-1501, and MS-COCO2017. For CIFAR-100 and Tiny-ImageNet, training uses SGD with momentum $\mathrm{KL}(P^t\Vert P^s)$5, weight decay $\mathrm{KL}(P^t\Vert P^s)$6, 240 epochs, learning-rate drops by $\mathrm{KL}(P^t\Vert P^s)$7 at epochs 150, 180, and 210, linear warm-up for 20 epochs, and temperature $\mathrm{KL}(P^t\Vert P^s)$8. Batch sizes are 64 for CIFAR-100 and 128 for Tiny-ImageNet. The default recursion depth is $\mathrm{KL}(P^t\Vert P^s)$9, except for ResNet32%%%%60$p^s$61%%%%ShuffleNetV1 where $|p_1^t-p_2^t|$2. Typical CIFAR-100 values are $|p_1^t-p_2^t|$3 and $|p_1^t-p_2^t|$4, with examples such as ResNet56$|p_1^t-p_2^t|$5ResNet20 using $|p_1^t-p_2^t|$6, ResNet32%%%%67$p^s$68%%%%ShuffleNetV2 using $|p_1^t-p_2^t|$9, and WRN-40-2$Z^s$0WRN-16-2 using $Z^s$1. For ImageNet-1K, the reported setup uses SGD, weight decay $Z^s$2, 100 epochs, learning rate 0.1, warm-up for 10 epochs, batch size 256, and temperature $Z^s$3, with examples ResNet34$Z^s$4ResNet18 using $Z^s$5 and ResNet50$Z^s$6MobileNetV1 using $Z^s$7.

Sensitivity analyses identify recursion depth as a critical hyperparameter. Too small a $Z^s$8 under-covers relations, whereas too large a $Z^s$9 introduces noise and redundancy; best performance typically occurs at $\pi(d)$0. The addition of $\pi(d)$1 consistently improves over uniform pair weighting, with an example on CIFAR-100 VGG13$\pi(d)$2MobileNetV2 of $\pi(d)$3 relative to $\pi(d)$4 without ADW. The paper also reports that $\pi(d)$5 and $\pi(d)$6 each help individually and that their combination is strongest, with examples of $\pi(d)$7 over baseline for ResNet32%%%%88$p^s$89%%%%ResNet8$d$04 and $d$1 for WRN-40-2$d$2WRN-40-1 (Xu et al., 21 Jul 2025).

Computationally, pairwise relations among the top-$d$3 subset cost $d$4 per sample, which is cheaper than modeling all $d$5 class pairs when $d$6. The paper states that LDRLD uses no extra memory compared to KD and DKD, and reports the same GPU memory, 2052 MB on RTX 3090 in the reported setup. On CIFAR-100 with ResNet32%%%%97$p^s$98%%%%ResNet8$d$94 on RTX 3090, average per-epoch runtime is 11.89 s for KD, 12.04 s for DKD, and 13.65 s for LDRLD, compared with 17.86 s for CRD and 23.44 s for ReviewKD. The method therefore adds modest overhead relative to KD and DKD while remaining substantially more efficient than feature-based distillation (Xu et al., 21 Jul 2025).

5. Empirical performance across classification, fine-grained recognition, detection, and re-identification

On CIFAR-100 with same-architecture transfers, LDRLD improves ResNet32%%%%100$p^s$101%%%%ResNet8$Z^t \in \mathbb{R}^{1\times C}$024 from 73.33% under KD to 77.20% under LDRLD, WRN-40-2$Z^t \in \mathbb{R}^{1\times C}$03WRN-16-2 from 74.92% to 76.35%, WRN-40-2$Z^t \in \mathbb{R}^{1\times C}$04WRN-40-1 from 73.54% to 74.98%, and VGG13$Z^t \in \mathbb{R}^{1\times C}$05VGG8 from 72.98% to 75.06%. For different architectures on CIFAR-100, it improves ResNet50$Z^t \in \mathbb{R}^{1\times C}$06MobileNetV2 from 67.35% to 70.74%, ResNet32%%%%107$p^s$108%%%%ShuffleNetV2 from 74.45% to 77.33%, ResNet32%%%%109$p^s$110%%%%ShuffleNetV1 from 74.07% to 76.46%, and VGG13$Z^t \in \mathbb{R}^{1\times C}$11MobileNetV2 from 67.37% to 70.11% (Xu et al., 21 Jul 2025).

On ImageNet-1K, ResNet34$Z^t \in \mathbb{R}^{1\times C}$12ResNet18 increases from 70.66% top-1 and 89.88% top-5 under KD to 71.88% top-1 and 90.58% top-5 under LDRLD. For the heterogeneous transfer ResNet50$Z^t \in \mathbb{R}^{1\times C}$13MobileNetV1, KD yields 70.49% top-1 and 89.92% top-5, while LDRLD yields 73.12% top-1 and 91.43% top-5. The paper also notes that WTTM is slightly higher in same-architecture top-1, at 72.19 versus 71.88. On Tiny-ImageNet, VGG13$Z^t \in \mathbb{R}^{1\times C}$14VGG8 increases from 57.33% to 60.91%, WRN-40-2$Z^t \in \mathbb{R}^{1\times C}$15WRN-16-2 from 59.16% to 60.67%, and ResNet50$Z^t \in \mathbb{R}^{1\times C}$16MobileNetV2 from 60.02% to 60.31% (Xu et al., 21 Jul 2025).

The reported gains are larger in heterogeneous vision-transformer-to-CNN settings on CIFAR-100. Swin-T$Z^t \in \mathbb{R}^{1\times C}$17ResNet18 improves from 78.74% to 82.17%, ViT-S$Z^t \in \mathbb{R}^{1\times C}$18ResNet18 from 77.26% to 80.36%, Mixer-B/16$Z^t \in \mathbb{R}^{1\times C}$19ResNet18 from 77.79% to 80.69%, Swin-T$Z^t \in \mathbb{R}^{1\times C}$20MobileNetV2 from 74.68% to 81.64%, ViT-S$Z^t \in \mathbb{R}^{1\times C}$21MobileNetV2 from 72.77% to 79.21%, and Mixer-B/16$Z^t \in \mathbb{R}^{1\times C}$22MobileNetV2 from 73.33% to 80.64%. On fine-grained CUB-200, ResNet32%%%%123$p^s$124%%%%MobileNetV2 improves from 56.09% to 60.99%, ResNet32%%%%125$p^s$126%%%%ShuffleNetV1 from 61.68% to 65.19%, VGG13$Z^t \in \mathbb{R}^{1\times C}$27MobileNetV2 from 53.98% to 59.73%, and VGG13$Z^t \in \mathbb{R}^{1\times C}$28VGG8 from 64.18% to 68.27%. These results are consistent with the claim that denser local relations are especially useful for visually confusable categories (Xu et al., 21 Jul 2025).

The method is also evaluated beyond image classification. On MS-COCO2017 detection with Faster R-CNN FPN, R-101$Z^t \in \mathbb{R}^{1\times C}$29R-18 achieves AP 35.12 under LDRLD, compared with 33.97 for KD and 34.88 for DKD; R-101$Z^t \in \mathbb{R}^{1\times C}$30R-50 yields AP 39.31 under LDRLD, compared with 38.35 for KD and 39.01 for DKD. On Market-1501 re-identification, a ResNet18 student baseline has Rank-1 85.04 and mAP 65.30, while LDRLD improves Rank-1 by 3.27, Rank-5 by 0.99, Rank-10 by 0.80, and mAP by 6.83 over baseline. The overall picture reported in the paper is that LDRLD compares favorably with state-of-the-art logit-based distillation approaches across diverse tasks (Xu et al., 21 Jul 2025).

6. Relation to adjacent formulations, misconceptions, and scope

Within the KD literature, LDRLD is positioned as a logit-based alternative to both globally normalized KD and more costly feature-level relational methods. Relative to classical KD, the distinguishing operation is pairwise softmax over selected class pairs rather than a single global softmax. Relative to RKD and CRD, the method remains logit-based rather than feature-based, which the paper presents as the reason it retains simplicity and efficiency. Relative to DKD and NKD, the novelty lies in recursive decoupling and dense pairwise recombination of a top-$Z^t \in \mathbb{R}^{1\times C}$31 local subset, combined with ADW and RNTK. Relative to dynamic-temperature approaches such as CTKD, LSKD, and WTTM, the novelty is not temperature control but local dense relational modeling with recursive selection and adaptive pair weighting (Xu et al., 21 Jul 2025).

Several nearby papers use related language but do not define the same method. “Linkless Link Prediction via Relational Distillation” formulates an anchor-centered relational KD framework for link prediction, using local dense candidate sets $Z^t \in \mathbb{R}^{1\times C}$32, a distribution-based matching loss, and a rank-based matching loss, but it operates over link targets around an anchor node rather than over class logits in a multiclass classifier (Guo et al., 2022). “Logit Distance Bounds Representational Similarity” does not introduce the name LDRLD, yet it proves that matching all pairwise logit gaps is equivalent to logit-distance distillation and yields explicit guarantees for mean CCA and a representation dissimilarity $Z^t \in \mathbb{R}^{1\times C}$33; in that sense, it supplies a theoretical interpretation of dense relational logit alignment, though not the recursive top-$Z^t \in \mathbb{R}^{1\times C}$34 procedure or ADW of LDRLD (Nielsen et al., 17 Feb 2026). “Localization Distillation for Dense Object Detection” is local, dense, and logit-level over FPN locations, but the paper explicitly states that it has no relational term in the usual sense; its mechanism is localization-logit KL with valuable localization region masking rather than pairwise class-relation modeling (Zheng et al., 2021).

A recurring misconception is therefore that any local logit KD method qualifies as LDRLD. In the strict sense of the 2025 formulation, LDRLD refers to the specific combination of recursive top-$Z^t \in \mathbb{R}^{1\times C}$35 extraction on student logits, dense pairwise recombination within the selected subset, ADW through IRW and ERD, LLKI over the selected logits, and RNTK over the excluded remainder. The paper also notes several limitations and failure modes: the recursion depth $Z^t \in \mathbb{R}^{1\times C}$36 must be tuned manually; gains can be smaller in some large-scale settings with limited students, as illustrated by the ImageNet same-architecture case where WTTM slightly exceeds LDRLD in top-1; and when the student is capacity-limited, reducing $Z^t \in \mathbb{R}^{1\times C}$37 or increasing $Z^t \in \mathbb{R}^{1\times C}$38 may be necessary to avoid overwhelming the student with too many local pairs. The authors state that the code will be made publicly available (Xu et al., 21 Jul 2025).