Spatial Attention-Based Distillation

• Spatial attention-based distillation involves transferring spatial weighting from teacher models to students via attention maps, enabling better localization in feature prediction.
• It employs methods like direct attention-map matching, attention-refined feature alignment, and probability distribution matching to capture salient spatial information.
• Empirical studies demonstrate that this approach significantly improves performance in semantic segmentation, video action recognition, and object detection.

Searching arXiv for recent and foundational papers on spatial attention-based distillation. Spatial attention-based distillation is a class of knowledge distillation methods in which the transfer signal is not limited to logits or raw intermediate features, but explicitly encodes where a teacher model allocates importance over spatial, or spatiotemporal, feature structures. Across dense prediction, video understanding, multi-modal medical imaging, visual reasoning, and generative modeling, the central premise is that student models benefit when they inherit the teacher’s localization priors, salient-region weighting, or spatial relational structure rather than merely matching outputs. In the literature, this idea appears in several forms: direct attention-map matching, attention-refined feature matching, channel-wise spatial probability matching, one-to-all spatial matching, cross-modal or cross-instance masking, and generative constraints that preserve the feature distribution under attention (Wang et al., 2023, Mansourian et al., 2024, Shu et al., 2020, Lin et al., 2022).

1. Definition and conceptual scope

Spatial attention-based distillation denotes a KD regime in which the teacher supplies an explicit or implicit spatial weighting over feature maps, and the student is optimized to reproduce that weighting, to match attention-refined features, or to align distributions induced by attention. The “spatial” component refers to importance over image regions, voxels, or per-frame spatial locations; in video, several works treat attention as spatiotemporal in practice because the spatial weighting varies over time (Wang et al., 2023, Liu et al., 2019). The “distillation” component refers to training a compact or otherwise constrained student with supervision derived from a stronger teacher or privileged auxiliary stream (Mansourian et al., 2024, Liu et al., 2019, Wu et al., 2023).

A recurring motivation is that naïve one-to-one feature regression is often misaligned with the semantics of dense or structured prediction. In semantic segmentation, raw MSE between unrefined features can be suboptimal because features may contain activation noise or emphasize irrelevant regions, while logits-only KD is insufficient for capturing intra-image relations (Mansourian et al., 2024). In dense prediction more broadly, point-wise and pair-wise spatial matching can treat all pixels equally and dilute learning in salient regions (Shu et al., 2020). In classification and segmentation, the “Knowledge Distillation via the Target-aware Transformer” formulation argues that the semantic information at the same spatial location usually varies between teacher and student because of architecture differences, which undermines one-to-one spatial matching (Lin et al., 2022). In video recognition, attention distillation is motivated by the need to transfer motion-aware localization from an optical-flow teacher to an RGB student without overwriting RGB features with flow features (Liu et al., 2019).

The field therefore spans several operational definitions of attention. Some methods construct explicit spatial masks from intermediate activations by aggregating over channels (López-Cifuentes et al., 2022, Zhang et al., 2020). Others employ modular attention blocks such as CBAM to produce channel and spatial gates and then distill the resulting refined features with MSE (Mansourian et al., 2024). Still others convert each feature channel into a spatial probability distribution via softmax and minimize a teacher-to-student KL divergence, effectively treating each channel as a spatial attention map (Shu et al., 2020). In video action recognition, attention can be defined over a 3D feature tensor $F \in \mathbb{R}^{T \times C \times HW}$, so that attention maps encode the distribution of semantic importance over space and time (Wang et al., 2023).

2. Core formulations of spatial attention transfer

A basic spatial attention construction aggregates channel responses at each spatial location. In the DCT-driven scene recognition framework, if $F_{t,l}$ and $F_{s,l}$ denote teacher and student features at layer $l$, the attention map is defined by mean squared channel aggregation:

$$A^t_l(x,y) = \frac{1}{C_t} \sum_{c=1}^{C_t} F_{t,l}(c,x,y)^2, \qquad A^s_l(x,y) = \frac{1}{C_s} \sum_{c=1}^{C_s} F_{s,l}(c,x,y)^2,$$

followed by min–max normalization and 2D DCT-II before computing the distillation loss (López-Cifuentes et al., 2022). This formulation interprets high activation energy as a spatial focus signal and then aligns global spatial structure in the frequency domain rather than via pixel-wise L2.

A second canonical formulation refines features through an attention module and then matches the refined features directly. In AttnFD for semantic segmentation, CBAM applies channel attention

$M_c = \sigma(MLP(AvgPool(F)) + MLP(MaxPool(F)))$

to obtain $F' = M_c \odot F$, then spatial attention

$$M_s = \sigma\big(f^{7\times 7}([AvgPool_c(F'), MaxPool_c(F')])\big)$$

to obtain $F'' = M_s \odot F'$, and the distillation objective is an MSE between channel-normalized, attention-refined teacher and student features across selected layers (Mansourian et al., 2024). Here, attention transfer is not a separate supervision on $M_s$ or $F_{t,l}$0 alone; the teacher’s “where to look” and “what to emphasize” are embedded in the refined features themselves.

A third formulation turns attention into a probability distribution. In “Channel-wise Knowledge Distillation for Dense Prediction,” each channel of a feature map is normalized over spatial positions via softmax with temperature $F_{t,l}$1:

$F_{t,l}$2

and the KD loss is

$F_{t,l}$3

This treats each channel as a spatial attention map and uses asymmetric KL to emphasize salient teacher regions (Shu et al., 2020).

A fourth formulation abandons one-to-one correspondence entirely. In TaT, teacher pixel $F_{t,l}$4 attends to all student positions through

$F_{t,l}$5

or in the parametric form

$F_{t,l}$6

followed by

$F_{t,l}$7

This one-to-all spatial matching is explicitly designed to mitigate semantic misalignment between teacher and student architectures (Lin et al., 2022).

A fifth formulation is attention-conditioned generation. In video action recognition, “Generative Model-based Feature Knowledge Distillation” defines attention on 3D feature maps as

$F_{t,l}$8

with

$F_{t,l}$9

and constrains the attention module through a conditional variational autoencoder:

$F_{s,l}$0

with $F_{s,l}$1 (Wang et al., 2023). This is not simple attention matching; the attention is required to preserve the feature distribution when applied.

3. Architectural patterns and training regimes

Several architectural patterns recur across the literature. One pattern inserts attention modules into both teacher and student and distills at corresponding semantic stages. AttnFD applies CBAM to backbone features, encoder output, and decoder feature in DeepLabV3+ with a ResNet-101 teacher and ResNet-18 or MobileNetV2 students, using student-side alignment with a $F_{s,l}$2 convolution and bilinear interpolation where needed (Mansourian et al., 2024). The teacher network and its CBAM parameters are pre-trained and fixed during student training, while the student network and student-side CBAM are optimized jointly.

A second pattern uses a frozen privileged teacher or teacher modality to supply attention supervision. In video attention distillation, a frozen optical-flow I3D teacher provides motion attention maps for an RGB I3D student with two attention heads, one for motion and one for appearance (Liu et al., 2019). In ADU-Depth, the teacher consumes concatenated left-view and right-view images through a Swin-Transformer-based stereo model, while the student is monocular and incorporates self-attention adaptation layers plus an uncertainty estimation module (Wu et al., 2023). In visual reasoning, the teacher may receive an externally constructed spatial mask derived from Probabilistic Soft Logic, or predict an in-network attention mask conditioned on the question, while the student learns only from labels and teacher predictions (Aditya et al., 2018).

A third pattern is intra-network or cross-modal self-distillation. In prostate cancer segmentation, CSAD does not employ an external teacher; rather, two modality-specific streams for T2W and ADC images distill activation-derived spatial attention maps across modalities and consecutive depths via a symmetric KL divergence (Zhang et al., 2020). Attention maps are generated after every encoding stage and connected in an interlaced manner, such as T2W layer $F_{s,l}$3 to ADC layer $F_{s,l}$4 and vice versa, while a Spatial Correlated Feature Fusion module models pairwise spatial correlation across modalities (Zhang et al., 2020).

A fourth pattern uses adaptive masks derived from teacher–student interactions. ACAM-KD introduces Student–Teacher Cross-Attention Feature Fusion, where teacher queries attend over student keys and values to form a fused representation, and Adaptive Spatial–Channel Masking generates masks from that fused representation for masked feature distillation (Lan et al., 8 Mar 2025). LIAF-KD similarly uses learnable instance selectors trained on ROI-aligned teacher features, then evaluates both teacher and student ROI features during distillation to produce teacher-driven and student-driven instance weights that are stamped back into spatial masks over ROIs (Liu et al., 27 Mar 2026). In both cases, masking is dynamic rather than static and teacher-only.

Training schedules also vary substantially. AttnFD uses a standard single-stage student optimization with

$F_{s,l}$5

where $F_{s,l}$6 is denoted $F_{s,l}$7 and set to $F_{s,l}$8 for PASCAL VOC and $F_{s,l}$9 for Cityscapes (Mansourian et al., 2024). Generative video KD alternates between a generative phase

$l$0

with $l$1, and an attention phase

$l$2

with $l$3 (Wang et al., 2023). PCD uses only a pixel-wise contrastive objective during self-supervised pretraining, with a SpatialAdaptor reshaping the teacher head and an MHSA module inserted after the student projection head, both of which are removed before downstream finetuning (Huang et al., 2022). This suggests that spatial attention-based distillation is not tied to a single optimization template; it can appear in supervised, self-supervised, offline alternating, and training-only auxiliary regimes.

4. Task-specific instantiations

Recognition and video understanding

In video action recognition, spatial attention-based distillation has been used both for cross-modal motion transfer and for spatiotemporal feature semantics. The 2019 “Attention Distillation for Learning Video Representations” framework generates attention maps from the output of the 4th convolutional block of I3D, normalizes them within each temporal slice by a softmax over $l$4, and uses them to pool features from Inception-5b (Liu et al., 2019). The motion attention of a flow teacher is distilled into the student through

$l$5

The final objective combines classification, motion-attention distillation, and a KL regularizer that keeps appearance attention close to a uniform prior (Liu et al., 2019).

The later generative framework for action recognition extends this by arguing that prior video KD methods overlook the spatial-temporal semantics inherent in intermediate 3D feature maps (Wang et al., 2023). Teacher and student backbones are I3D and Top-I3D, respectively, with generalization experiments on Bottom-I3D and I2D, and AFSD-based teacher–student variants for action detection (Wang et al., 2023). Attention is computed on $l$6 with Conv1D along the temporal axis, group normalization, and sigmoid gating, and the resulting attention map is applied multiplicatively after channel alignment. The authors describe this as attention-based feature semantics, where attention maps summarize spatiotemporal importance over the 3D feature volume (Wang et al., 2023).

Semantic segmentation and dense prediction

In semantic segmentation, AttnFD is a minimalistic yet high-performing formulation. It uses only CBAM-refined features and a basic MSE loss, with pre-ReLU features and channel-wise normalization before the loss (Mansourian et al., 2024). Distillation across Backbone, Encoder, and Decoder stages yields the best results, with the Decoder contributing the largest single-stage boost (Mansourian et al., 2024). Channel and spatial attention are treated as complementary: the former conveys “what to emphasize,” the latter “where to look” (Mansourian et al., 2024).

Channel-wise KD takes a different stance by claiming that in semantic segmentation, some high-level channels tend to encode category-wise saliency masks analogous to class activation maps (Shu et al., 2020). It therefore aligns per-channel spatial probability distributions rather than collapsed spatial attention maps. This approach is explicitly positioned against point-wise, local, pair-wise, and holistic spatial KD methods, with lower computational cost than pair-wise affinity approaches and stronger accuracy (Shu et al., 2020).

Object detection

In object detection, several methods distinguish between local, instance-centric, and global attention transfer. AFD for remote sensing introduces a multi-instance attention mechanism over local patches plus attention global distillation through GCNet to reconstruct relationships between pixels (Shamsolmoali et al., 2023). Its final feature-level loss applies both spatial and channel masks:

$l$7

and it also distills classification and localization heads with spatial masking (Shamsolmoali et al., 2023).

ACAM-KD and LIAF-KD move beyond static teacher-driven masking. ACAM-KD uses cross-attention fusion

$l$8

followed by adaptive spatial and channel masks generated from pooled or flattened fused features (Lan et al., 8 Mar 2025). LIAF-KD instead learns an ensemble of instance selectors over ROI-aligned teacher features, with per-instance scores

$l$9

and later uses the same selectors on teacher and student ROI features to build teacher and student masks over ROIs (Liu et al., 27 Mar 2026). A plausible implication is that spatial attention-based distillation in detection is increasingly shifting from fixed teacher saliency to student-aware, instance-adaptive weighting.

Multi-modal and reasoning tasks

Spatial attention-based distillation also appears in multi-modal segmentation and visual reasoning. In ADU-Depth, attention-adapted feature distillation applies self-attention to the student’s encoded and decoded features before L2 distillation to the stereo teacher’s features, and uncertainty-aware feature and response distillation weight residuals by inverse variance (Wu et al., 2023). In VQA, privileged spatial knowledge is encoded as a mask derived from scene-graph annotations and question parsing, or predicted internally by an attention module, and the student learns from the teacher’s soft predictions while never seeing the privileged masks at inference (Aditya et al., 2018). These cases show that the teacher’s advantage can come from extra modalities, extra annotations, or extra reasoning machinery rather than only higher capacity.

5. Empirical trends and comparative findings

Across tasks, the empirical record consistently indicates that transferring spatial focus can improve students beyond logits-only or raw-feature baselines. In semantic segmentation, AttnFD reports on Pascal VOC 2012 that a ResNet-18 student improves from $$A^t_l(x,y) = \frac{1}{C_t} \sum_{c=1}^{C_t} F_{t,l}(c,x,y)^2, \qquad A^s_l(x,y) = \frac{1}{C_s} \sum_{c=1}^{C_s} F_{s,l}(c,x,y)^2,$$0 mIoU to $$A^t_l(x,y) = \frac{1}{C_t} \sum_{c=1}^{C_t} F_{t,l}(c,x,y)^2, \qquad A^s_l(x,y) = \frac{1}{C_s} \sum_{c=1}^{C_s} F_{s,l}(c,x,y)^2,$$1, and a MobileNetV2 student from $$A^t_l(x,y) = \frac{1}{C_t} \sum_{c=1}^{C_t} F_{t,l}(c,x,y)^2, \qquad A^s_l(x,y) = \frac{1}{C_s} \sum_{c=1}^{C_s} F_{s,l}(c,x,y)^2,$$2 to $$A^t_l(x,y) = \frac{1}{C_t} \sum_{c=1}^{C_t} F_{t,l}(c,x,y)^2, \qquad A^s_l(x,y) = \frac{1}{C_s} \sum_{c=1}^{C_s} F_{s,l}(c,x,y)^2,$$3 (Mansourian et al., 2024). On Cityscapes, the same method improves a ResNet-18 student from $$A^t_l(x,y) = \frac{1}{C_t} \sum_{c=1}^{C_t} F_{t,l}(c,x,y)^2, \qquad A^s_l(x,y) = \frac{1}{C_s} \sum_{c=1}^{C_s} F_{s,l}(c,x,y)^2,$$4 to $$A^t_l(x,y) = \frac{1}{C_t} \sum_{c=1}^{C_t} F_{t,l}(c,x,y)^2, \qquad A^s_l(x,y) = \frac{1}{C_s} \sum_{c=1}^{C_s} F_{s,l}(c,x,y)^2,$$5 mIoU and a MobileNetV2 student from $$A^t_l(x,y) = \frac{1}{C_t} \sum_{c=1}^{C_t} F_{t,l}(c,x,y)^2, \qquad A^s_l(x,y) = \frac{1}{C_s} \sum_{c=1}^{C_s} F_{s,l}(c,x,y)^2,$$6 to $$A^t_l(x,y) = \frac{1}{C_t} \sum_{c=1}^{C_t} F_{t,l}(c,x,y)^2, \qquad A^s_l(x,y) = \frac{1}{C_s} \sum_{c=1}^{C_s} F_{s,l}(c,x,y)^2,$$7 (Mansourian et al., 2024). The paper also states that AttnFD outperforms several baselines, including KD, AT, SP, ICKD, Overhaul, DistKD, CIRKD, and LAD on the reported settings (Mansourian et al., 2024).

Channel-wise KD reports that on Cityscapes val with a PSPNet-R101 teacher and PSPNet-R18 student, its channel-softmax plus asymmetric KL reaches $$A^t_l(x,y) = \frac{1}{C_t} \sum_{c=1}^{C_t} F_{t,l}(c,x,y)^2, \qquad A^s_l(x,y) = \frac{1}{C_s} \sum_{c=1}^{C_s} F_{s,l}(c,x,y)^2,$$8 mIoU on logits distillation, compared with $$A^t_l(x,y) = \frac{1}{C_t} \sum_{c=1}^{C_t} F_{t,l}(c,x,y)^2, \qquad A^s_l(x,y) = \frac{1}{C_s} \sum_{c=1}^{C_s} F_{s,l}(c,x,y)^2,$$9 for Attention Transfer and $M_c = \sigma(MLP(AvgPool(F)) + MLP(MaxPool(F)))$0 for a spatial KL baseline (Shu et al., 2020). The same paper reports strong gains across semantic segmentation and object detection, including COCO results such as Faster R-CNN improving from $M_c = \sigma(MLP(AvgPool(F)) + MLP(MaxPool(F)))$1 AP to $M_c = \sigma(MLP(AvgPool(F)) + MLP(MaxPool(F)))$2 AP and RetinaNet from $M_c = \sigma(MLP(AvgPool(F)) + MLP(MaxPool(F)))$3 to $M_c = \sigma(MLP(AvgPool(F)) + MLP(MaxPool(F)))$4 AP under channel-wise distillation (Shu et al., 2020).

In video action recognition, the generative spatiotemporal framework reports a Top-I3D student improving on UCF101 from Top-1/Top-5 $M_c = \sigma(MLP(AvgPool(F)) + MLP(MaxPool(F)))$5 to $M_c = \sigma(MLP(AvgPool(F)) + MLP(MaxPool(F)))$6, and on HMDB51 from $M_c = \sigma(MLP(AvgPool(F)) + MLP(MaxPool(F)))$7 to $M_c = \sigma(MLP(AvgPool(F)) + MLP(MaxPool(F)))$8, with FLOPS increasing from $M_c = \sigma(MLP(AvgPool(F)) + MLP(MaxPool(F)))$9 to $F' = M_c \odot F$0 (Wang et al., 2023). The same method reports consistent gains in THUMOS14 action detection, with average mAP increasing from $F' = M_c \odot F$1 to $F' = M_c \odot F$2 (Wang et al., 2023). Earlier video attention distillation likewise reports RGB-only improvements on UCF101 from $F' = M_c \odot F$3 to $F' = M_c \odot F$4, on HMDB51 from $F' = M_c \odot F$5 to $F' = M_c \odot F$6, and on Something-Something V2 from $F' = M_c \odot F$7 top-1/top-5 to $F' = M_c \odot F$8 (Liu et al., 2019).

In dense detector distillation, ACAM-KD reports improvements on COCO2017 such as RetinaNet reaching $F' = M_c \odot F$9 mAP versus $$M_s = \sigma\big(f^{7\times 7}([AvgPool_c(F'), MaxPool_c(F')])\big)$$0 for FreeKD under a ResNet-101 to ResNet-50 setting, and RepPoints reaching $$M_s = \sigma\big(f^{7\times 7}([AvgPool_c(F'), MaxPool_c(F')])\big)$$1 versus $$M_s = \sigma\big(f^{7\times 7}([AvgPool_c(F'), MaxPool_c(F')])\big)$$2 for MasKD (Lan et al., 8 Mar 2025). LIAF-KD reports gains such as a GFL ResNet-50 student on COCO improving from $$M_s = \sigma\big(f^{7\times 7}([AvgPool_c(F'), MaxPool_c(F')])\big)$$3 to $$M_s = \sigma\big(f^{7\times 7}([AvgPool_c(F'), MaxPool_c(F')])\big)$$4 mAP and on KITTI from $$M_s = \sigma\big(f^{7\times 7}([AvgPool_c(F'), MaxPool_c(F')])\big)$$5 to $$M_s = \sigma\big(f^{7\times 7}([AvgPool_c(F'), MaxPool_c(F')])\big)$$6 mAP (Liu et al., 27 Mar 2026). In remote sensing, AFD reports a RetinaNet student on DOTA improving from $$M_s = \sigma\big(f^{7\times 7}([AvgPool_c(F'), MaxPool_c(F')])\big)$$7 to $$M_s = \sigma\big(f^{7\times 7}([AvgPool_c(F'), MaxPool_c(F')])\big)$$8 mAP and a Cascade R-CNN student from $$M_s = \sigma\big(f^{7\times 7}([AvgPool_c(F'), MaxPool_c(F')])\big)$$9 to $F'' = M_s \odot F'$0 (Shamsolmoali et al., 2023).

The literature also contains task-specific evidence that spatial attention transfer is especially helpful when the student is weak, the task is spatially structured, or the teacher has access to privileged information. In the generative video KD paper, distillation from 3D to 2D benefits the most: I2D improves from $F'' = M_s \odot F'$1 to $F'' = M_s \odot F'$2 on UCF101 (Wang et al., 2023). In ADU-Depth, adding attention, uncertainty, and focal-depth terms progressively improves a baseline student from Sq Rel/Abs Rel/RMSE $F'' = M_s \odot F'$3 to $F'' = M_s \odot F'$4 (Wu et al., 2023). In VQA, a student distilled from an external-mask teacher improves from an RN baseline of $F'' = M_s \odot F'$5 to $F'' = M_s \odot F'$6 on Sort-of-CLEVR, while the teacher reaches $F'' = M_s \odot F'$7 (Aditya et al., 2018). These patterns suggest that attention-based distillation is most advantageous when plain output alignment fails to convey the structured inductive bias that the teacher possesses.

6. Limitations, controversies, and open directions

A common misconception is that spatial attention-based distillation is equivalent to matching a single 2D saliency map. The surveyed methods contradict this. Some operate with both channel and spatial attention (Mansourian et al., 2024, Lan et al., 8 Mar 2025); some interpret each channel as its own spatial distribution (Shu et al., 2020); some employ one-to-all spatial correspondences rather than locationwise matching (Lin et al., 2022); and some constrain attention through generation or uncertainty modeling (Wang et al., 2023, Wu et al., 2023). In video, attention may be spatial-only per voxel but vary over time, making it effectively spatiotemporal (Wang et al., 2023). In detection, instance-level scores can be lifted back to spatial masks over ROIs (Liu et al., 27 Mar 2026).

Another misconception is that attention transfer is necessarily lightweight and straightforward. While several methods emphasize simplicity, such as AttnFD’s CBAM-plus-MSE formulation (Mansourian et al., 2024), others introduce substantial training complexity. Generative spatiotemporal KD requires alternating optimization phases and a CVAE that is removed only at test time (Wang et al., 2023). ACAM-KD adds cross-attention with an $F'' = M_s \odot F'$8 affinity per level during training (Lan et al., 8 Mar 2025). LIAF-KD requires a selector-learning pre-distillation stage and ROI extraction (Liu et al., 27 Mar 2026). PCD uses a MoCo-style negative queue of capacity $F'' = M_s \odot F'$9 and a SpatialAdaptor to preserve the teacher head’s distribution under a 2D reshaping (Huang et al., 2022). The claim that spatial attention distillation is always a drop-in replacement for feature KD would therefore be inaccurate.

Bias transfer and teacher quality remain persistent concerns. AttnFD notes that if the teacher’s attention is biased, the student may inherit these biases (Mansourian et al., 2024). The generative video KD framework states that if teacher attention is noisy, attention matching may be less effective (Wang et al., 2023). In ADU-Depth, domain shift between stereo teacher and monocular student is significant enough to require attention adaptation and uncertainty weighting (Wu et al., 2023). In VQA, the quality of privileged masks depends on scene-graph quality and parsing accuracy (Aditya et al., 2018). This suggests that attention-based supervision is only as reliable as the localization prior that produces it.

Several open directions are already articulated in the literature. AttnFD explicitly points to detection and instance segmentation as plausible extensions, suggesting distillation of CBAM-refined features at pyramid levels and ROI features (Mansourian et al., 2024). ACAM-KD and LIAF-KD point toward increasingly student-aware and instance-adaptive masking (Lan et al., 8 Mar 2025, Liu et al., 27 Mar 2026). The generative video KD paper indicates that attention-based feature semantics can transfer especially well to weaker temporal models (Wang et al., 2023). PCD suggests that spatially precise, pixel-level contrastive correspondence is a promising route for dense prediction pretraining (Huang et al., 2022). A plausible implication is that future work will continue to unify attention transfer with relational modeling, uncertainty estimation, instance adaptivity, and generative reconstruction rather than treating attention as an isolated map-matching primitive.

Overall, spatial attention-based distillation has evolved from direct attention-map regression into a broader family of localization-aware transfer mechanisms. The central unifying idea remains stable: for tasks in which spatial arrangement, salient regions, and structured context are essential, compressing the teacher’s focus can be as important as compressing its outputs (Wang et al., 2023, Mansourian et al., 2024, Shu et al., 2020, Lin et al., 2022).