Input-Aware Distillation Loss

• Input-Aware Distillation Loss is a methodology that modulates teacher-student supervision by integrating input-dependent signals such as SNR, pose keypoints, and attention maps.
• It spans multiple modalities including diffusion models, video synthesis, and language modeling, utilizing diverse conditioning signals to target the most informative regions of the input.
• The approach combines conventional training objectives with corrective, adaptive weighting mechanisms to address issues like uneven supervision, low-fidelity predictions, and noisy labels.

Searching arXiv for papers on input-aware distillation loss and closely related formulations across modalities. Input-aware distillation loss denotes a family of distillation objectives in which the supervisory signal or its weighting is conditioned on the actual input, rather than being fixed uniformly across samples, timesteps, tokens, pixels, or regions. In the literature, this conditioning has been instantiated through signal-to-noise ratio in diffusion distillation, pose-derived regions in co-speech video generation, teacher self-attention in token embedding initialization, photometric evidence in self-supervised depth estimation, attention-derived spatial maps in visual recognition, input gradients in adversarial training, uncertainty-weighted pixel relations in incremental segmentation, and Jacobian-based local geometry in generative distillation. Across these settings, the common principle is that the distillation target, its weight, or both are selected from input-dependent structure so that the student matches the teacher where the input indicates that supervision is most informative or most reliable (Liu et al., 2023, Dong et al., 2024, Dobler et al., 26 May 2025, Huang et al., 1 Jun 2026).

1. Concept and scope

Input-aware distillation differs from conventional knowledge distillation in that it does not treat all supervisory elements as equally informative. Instead, it modulates the objective using input-conditioned signals such as timestep-dependent SNR, pose keypoints, attention scores, region masks, per-pixel photometric residuals, or gradients with respect to the current input. This makes the supervision adaptive to the specific sample and to the specific internal structure through which the sample is processed.

Several papers make this explicit. In text-to-audio diffusion distillation, the Balanced SNR-Aware method uses a timestep-dependent weight defined by the diffusion state, with the student loss written as

$L_{\theta}=w(\gamma_{t})\mathbb{E}_{\epsilon,t,\Tilde{z}_{0}\Vert \Tilde{z}_{0}-\hat{x}_{\theta}(z_{t},\tau)\Vert^{2}_{2}$

and the weighting interpreted as

$w(t) = \min\!\big(\text{SNR}(t) + 1,\ \gamma\big)$

with

$\text{SNR}(t)=\frac{\alpha_{t}^{2}{\sigma_{t}^{2}}$

(Liu et al., 2023). In co-speech video generation, the loss is conditioned on pose-defined spatial regions:

$\mathcal{L}_{\mathrm{region}=\sum_{r\in \mathcal{R}\lambda_r\,\mathbb{E}_{t}\Big[\mathcal{L}_r\big(m_{r,t}\odot I_t,\ m_{r,t}\odot \hat{I}_t\big)\Big],$

where the masks are computed from pose keypoints (Lu et al., 2 Oct 2025). In token embedding distillation for LLMs, the supervision is restricted to positions selected by teacher self-attention: $\mathcal{L}_{\text{AweDist} \;=\; \min_{\mathbf{e_\tau} \in \mathbb{R}^d} \;\mathbb{E}_{s \sim \mathcal{S}} \left[\frac{1}{\left| \mathcal{M}(s_\tau, s_{\hat{\tau}}) \right|} \sum_{(i,j) \in \mathcal{M}(s_\tau, s_{\hat{\tau}})} \left\| \mathcal{H}^{(l)}_{\mathbf{e_\tau}}(s_{\hat{\tau}})_i - \mathcal{H}^{(l)}(s_\tau)_j \right\|_2^2 \right].$ Here, the set $\mathcal{M}(s_\tau, s_{\hat{\tau}})$ depends on the actual input sequence and on which positions attend to the token span under the teacher (Dobler et al., 26 May 2025).

This suggests that “input-aware distillation loss” is best understood not as a single formula, but as a design pattern spanning teacher–student regression, contrastive distillation, self-distillation, distribution matching, and post-training weighting.

2. Loss construction mechanisms

The defining mechanism is input-conditioned selection or weighting. The conditioning variable may be scalar, spatial, temporal, token-level, or differential.

A scalar weighting example appears in diffusion distillation. The Balanced SNR-Aware formulation replaces unbounded or vanishing timestep weights with \begin{gather} L_{\theta} = min(\frac{\alpha_{t}^{2}{\sigma_{t}^{2}+1,\gamma)\Vert x-\hat{x}{t}\Vert^{2}{2}. \end{gather} The surrounding explanation states that the added $1$ prevents low-SNR weights from vanishing, while the cap $\gamma$ prevents high-SNR overweighting (Liu et al., 2023). The same paper states that $\gamma$ is set to $5$ empirically and that three distillation iterations reduce the number of steps from $w(t) = \min\!\big(\text{SNR}(t) + 1,\ \gamma\big)$0 to $w(t) = \min\!\big(\text{SNR}(t) + 1,\ \gamma\big)$1 (Liu et al., 2023).

A spatial conditioning example appears in pose-conditioned video diffusion. There, the masks are constructed from pose keypoints using

$w(t) = \min\!\big(\text{SNR}(t) + 1,\ \gamma\big)$2

followed by morphological dilation and normalization to $w(t) = \min\!\big(\text{SNR}(t) + 1,\ \gamma\big)$3 (Lu et al., 2 Oct 2025). The region-aware loss then combines ArcFace distance for faces and LPIPS for hands, arms, shoulders, and upper body (Lu et al., 2 Oct 2025). The paper states that this input-aware component uses ground-truth-vs-student reconstruction and perceptual terms with pose-defined masks, combined with DMD reverse-KL for distribution matching (Lu et al., 2 Oct 2025).

A token-level conditioning example appears in AweDist. The teacher attention aggregation

$w(t) = \min\!\big(\text{SNR}(t) + 1,\ \gamma\big)$4

determines whether an aligned pair $w(t) = \min\!\big(\text{SNR}(t) + 1,\ \gamma\big)$5 enters $w(t) = \min\!\big(\text{SNR}(t) + 1,\ \gamma\big)$6 (Dobler et al., 26 May 2025). The loss therefore supervises only positions that “would attend” to the new token’s original span.

A per-pixel selection mechanism appears in MAL for self-supervised depth estimation. The distillation target is defined by

$w(t) = \min\!\big(\text{SNR}(t) + 1,\ \gamma\big)$7

and

$w(t) = \min\!\big(\text{SNR}(t) + 1,\ \gamma\big)$8

with

$w(t) = \min\!\big(\text{SNR}(t) + 1,\ \gamma\big)$9

The selector depends on which depth better explains the input photometry at that pixel (Dong et al., 2024).

Other formulations replace weighting by directional alignment. Input Gradient Distillation aligns teacher and student input gradients through cosine similarity: $\text{SNR}(t)=\frac{\alpha_{t}^{2}{\sigma_{t}^{2}}$0 where

$\text{SNR}(t)=\frac{\alpha_{t}^{2}{\sigma_{t}^{2}}$1

The full objective combines adversarial training and gradient alignment (Chen et al., 2023). Geometry-Aware Distillation instead aligns finite-difference responses to perturbed input noise: $\text{SNR}(t)=\frac{\alpha_{t}^{2}{\sigma_{t}^{2}}$2

$\text{SNR}(t)=\frac{\alpha_{t}^{2}{\sigma_{t}^{2}}$3

This loss is input-aware because it is defined by the local response of the specific input noise sample $\text{SNR}(t)=\frac{\alpha_{t}^{2}{\sigma_{t}^{2}}$4 (Huang et al., 1 Jun 2026).

3. Feature- and attention-conditioned formulations

A major branch of input-aware distillation uses intermediate representations rather than logits. These methods condition the target on attention maps, feature geometry, or uncertainty-aware pixel relations.

“Attention Distillation” defines an input-aware feature-matching loss on self-attention outputs of a pretrained diffusion U-Net. For a current latent $\text{SNR}(t)=\frac{\alpha_{t}^{2}{\sigma_{t}^{2}}$5 and reference latent $\text{SNR}(t)=\frac{\alpha_{t}^{2}{\sigma_{t}^{2}}$6, it matches

$\text{SNR}(t)=\frac{\alpha_{t}^{2}{\sigma_{t}^{2}}$7

to the reference-guided target

$\text{SNR}(t)=\frac{\alpha_{t}^{2}{\sigma_{t}^{2}}$8

The loss is

$\text{SNR}(t)=\frac{\alpha_{t}^{2}{\sigma_{t}^{2}}$9

The method is input-aware because the target itself depends jointly on the current sample’s queries and the specific reference’s keys and values (Zhou et al., 27 Feb 2025).

Angular Margin-based Distillation uses teacher attention maps to define per-input positive and negative regions. The normalized positive and negative maps are

$\mathcal{L}_{\mathrm{region}=\sum_{r\in \mathcal{R}\lambda_r\,\mathbb{E}_{t}\Big[\mathcal{L}_r\big(m_{r,t}\odot I_t,\ m_{r,t}\odot \hat{I}_t\big)\Big],$0

with analogous student quantities, and the angular probability map is

$\mathcal{L}_{\mathrm{region}=\sum_{r\in \mathcal{R}\lambda_r\,\mathbb{E}_{t}\Big[\mathcal{L}_r\big(m_{r,t}\odot I_t,\ m_{r,t}\odot \hat{I}_t\big)\Big],$1

The AMD loss matches $\mathcal{L}_{\mathrm{region}=\sum_{r\in \mathcal{R}\lambda_r\,\mathbb{E}_{t}\Big[\mathcal{L}_r\big(m_{r,t}\odot I_t,\ m_{r,t}\odot \hat{I}_t\big)\Big],$2, $\mathcal{L}_{\mathrm{region}=\sum_{r\in \mathcal{R}\lambda_r\,\mathbb{E}_{t}\Big[\mathcal{L}_r\big(m_{r,t}\odot I_t,\ m_{r,t}\odot \hat{I}_t\big)\Big],$3, and $\mathcal{L}_{\mathrm{region}=\sum_{r\in \mathcal{R}\lambda_r\,\mathbb{E}_{t}\Big[\mathcal{L}_r\big(m_{r,t}\odot I_t,\ m_{r,t}\odot \hat{I}_t\big)\Big],$4 between teacher and student across layer pairs (Jeon et al., 2023). The paper states that the process is input-aware because the positive and negative maps are produced from the teacher’s features for that specific image (Jeon et al., 2023).

Uncertainty-aware Contrastive Distillation for incremental segmentation constructs positives and negatives from the current mini-batch’s extended semantic maps. For an anchor pixel, the uncertainty-aware loss weights each anchor–positive pair by

$\mathcal{L}_{\mathrm{region}=\sum_{r\in \mathcal{R}\lambda_r\,\mathbb{E}_{t}\Big[\mathcal{L}_r\big(m_{r,t}\odot I_t,\ m_{r,t}\odot \hat{I}_t\big)\Big],$5

and optimizes

$\mathcal{L}_{\mathrm{region}=\sum_{r\in \mathcal{R}\lambda_r\,\mathbb{E}_{t}\Big[\mathcal{L}_r\big(m_{r,t}\odot I_t,\ m_{r,t}\odot \hat{I}_t\big)\Big],$6

This combines input-awareness, because positives and negatives are derived from the current batch, and uncertainty-awareness, because weights come from the current teacher probability maps and ground truth (Yang et al., 2022).

4. Generative diffusion distillation

Generative diffusion models provide several of the clearest instances of input-aware distillation loss because the input includes timestep, noise state, and conditioning signals, all of which can be used to modulate the student objective.

In text-to-audio diffusion, the student predicts $\mathcal{L}_{\mathrm{region}=\sum_{r\in \mathcal{R}\lambda_r\,\mathbb{E}_{t}\Big[\mathcal{L}_r\big(m_{r,t}\odot I_t,\ m_{r,t}\odot \hat{I}_t\big)\Big],$7 and matches a deterministic two-step DDIM teacher target $\mathcal{L}_{\mathrm{region}=\sum_{r\in \mathcal{R}\lambda_r\,\mathbb{E}_{t}\Big[\mathcal{L}_r\big(m_{r,t}\odot I_t,\ m_{r,t}\odot \hat{I}_t\big)\Big],$8 in one step. The weighting

$\mathcal{L}_{\mathrm{region}=\sum_{r\in \mathcal{R}\lambda_r\,\mathbb{E}_{t}\Big[\mathcal{L}_r\big(m_{r,t}\odot I_t,\ m_{r,t}\odot \hat{I}_t\big)\Big],$9

is introduced because unbounded SNR weighting biases learning toward low-noise timesteps and Min-SNR-$\mathcal{L}_{\text{AweDist} \;=\; \min_{\mathbf{e_\tau} \in \mathbb{R}^d} \;\mathbb{E}_{s \sim \mathcal{S}} \left[\frac{1}{\left| \mathcal{M}(s_\tau, s_{\hat{\tau}}) \right|} \sum_{(i,j) \in \mathcal{M}(s_\tau, s_{\hat{\tau}})} \left\| \mathcal{H}^{(l)}_{\mathbf{e_\tau}}(s_{\hat{\tau}})_i - \mathcal{H}^{(l)}(s_\tau)_j \right\|_2^2 \right].$0 weighting can allow weights to go to zero at low SNR (Liu et al., 2023). The reported results on AudioCaps include Teacher 200 steps with FAD $\mathcal{L}_{\text{AweDist} \;=\; \min_{\mathbf{e_\tau} \in \mathbb{R}^d} \;\mathbb{E}_{s \sim \mathcal{S}} \left[\frac{1}{\left| \mathcal{M}(s_\tau, s_{\hat{\tau}}) \right|} \sum_{(i,j) \in \mathcal{M}(s_\tau, s_{\hat{\tau}})} \left\| \mathcal{H}^{(l)}_{\mathbf{e_\tau}}(s_{\hat{\tau}})_i - \mathcal{H}^{(l)}(s_\tau)_j \right\|_2^2 \right].$1, Teacher 25 steps with FAD $\mathcal{L}_{\text{AweDist} \;=\; \min_{\mathbf{e_\tau} \in \mathbb{R}^d} \;\mathbb{E}_{s \sim \mathcal{S}} \left[\frac{1}{\left| \mathcal{M}(s_\tau, s_{\hat{\tau}}) \right|} \sum_{(i,j) \in \mathcal{M}(s_\tau, s_{\hat{\tau}})} \left\| \mathcal{H}^{(l)}_{\mathbf{e_\tau}}(s_{\hat{\tau}})_i - \mathcal{H}^{(l)}(s_\tau)_j \right\|_2^2 \right].$2, Student (Salimans, 25 steps) with FAD $\mathcal{L}_{\text{AweDist} \;=\; \min_{\mathbf{e_\tau} \in \mathbb{R}^d} \;\mathbb{E}_{s \sim \mathcal{S}} \left[\frac{1}{\left| \mathcal{M}(s_\tau, s_{\hat{\tau}}) \right|} \sum_{(i,j) \in \mathcal{M}(s_\tau, s_{\hat{\tau}})} \left\| \mathcal{H}^{(l)}_{\mathbf{e_\tau}}(s_{\hat{\tau}})_i - \mathcal{H}^{(l)}(s_\tau)_j \right\|_2^2 \right].$3, Student (Hang Min-SNR-$\mathcal{L}_{\text{AweDist} \;=\; \min_{\mathbf{e_\tau} \in \mathbb{R}^d} \;\mathbb{E}_{s \sim \mathcal{S}} \left[\frac{1}{\left| \mathcal{M}(s_\tau, s_{\hat{\tau}}) \right|} \sum_{(i,j) \in \mathcal{M}(s_\tau, s_{\hat{\tau}})} \left\| \mathcal{H}^{(l)}_{\mathbf{e_\tau}}(s_{\hat{\tau}})_i - \mathcal{H}^{(l)}(s_\tau)_j \right\|_2^2 \right].$4, 25 steps) with FAD $\mathcal{L}_{\text{AweDist} \;=\; \min_{\mathbf{e_\tau} \in \mathbb{R}^d} \;\mathbb{E}_{s \sim \mathcal{S}} \left[\frac{1}{\left| \mathcal{M}(s_\tau, s_{\hat{\tau}}) \right|} \sum_{(i,j) \in \mathcal{M}(s_\tau, s_{\hat{\tau}})} \left\| \mathcal{H}^{(l)}_{\mathbf{e_\tau}}(s_{\hat{\tau}})_i - \mathcal{H}^{(l)}(s_\tau)_j \right\|_2^2 \right].$5, and Student (BSA, ours, 25 steps) with FAD $\mathcal{L}_{\text{AweDist} \;=\; \min_{\mathbf{e_\tau} \in \mathbb{R}^d} \;\mathbb{E}_{s \sim \mathcal{S}} \left[\frac{1}{\left| \mathcal{M}(s_\tau, s_{\hat{\tau}}) \right|} \sum_{(i,j) \in \mathcal{M}(s_\tau, s_{\hat{\tau}})} \left\| \mathcal{H}^{(l)}_{\mathbf{e_\tau}}(s_{\hat{\tau}})_i - \mathcal{H}^{(l)}(s_\tau)_j \right\|_2^2 \right].$6 (Liu et al., 2023). The same section reports that the number of sampling steps is reduced from $\mathcal{L}_{\text{AweDist} \;=\; \min_{\mathbf{e_\tau} \in \mathbb{R}^d} \;\mathbb{E}_{s \sim \mathcal{S}} \left[\frac{1}{\left| \mathcal{M}(s_\tau, s_{\hat{\tau}}) \right|} \sum_{(i,j) \in \mathcal{M}(s_\tau, s_{\hat{\tau}})} \left\| \mathcal{H}^{(l)}_{\mathbf{e_\tau}}(s_{\hat{\tau}})_i - \mathcal{H}^{(l)}(s_\tau)_j \right\|_2^2 \right].$7 to $\mathcal{L}_{\text{AweDist} \;=\; \min_{\mathbf{e_\tau} \in \mathbb{R}^d} \;\mathbb{E}_{s \sim \mathcal{S}} \left[\frac{1}{\left| \mathcal{M}(s_\tau, s_{\hat{\tau}}) \right|} \sum_{(i,j) \in \mathcal{M}(s_\tau, s_{\hat{\tau}})} \left\| \mathcal{H}^{(l)}_{\mathbf{e_\tau}}(s_{\hat{\tau}})_i - \mathcal{H}^{(l)}(s_\tau)_j \right\|_2^2 \right].$8 with minimal degradation (Liu et al., 2023).

In co-speech video generation, the baseline distillation signal is DMD reverse-KL, but few-step distillation degrades face and hand quality. The proposed input-aware component is a pose-masked region loss: $\mathcal{L}_{\text{AweDist} \;=\; \min_{\mathbf{e_\tau} \in \mathbb{R}^d} \;\mathbb{E}_{s \sim \mathcal{S}} \left[\frac{1}{\left| \mathcal{M}(s_\tau, s_{\hat{\tau}}) \right|} \sum_{(i,j) \in \mathcal{M}(s_\tau, s_{\hat{\tau}})} \left\| \mathcal{H}^{(l)}_{\mathbf{e_\tau}}(s_{\hat{\tau}})_i - \mathcal{H}^{(l)}(s_\tau)_j \right\|_2^2 \right].$9 The paper reports that DMD distillation alone achieves $\mathcal{M}(s_\tau, s_{\hat{\tau}})$0 FPS but visual quality drops to SSIM $\mathcal{M}(s_\tau, s_{\hat{\tau}})$1 and PSNR $\mathcal{M}(s_\tau, s_{\hat{\tau}})$2, whereas adding the input-aware region loss recovers quality to SSIM $\mathcal{M}(s_\tau, s_{\hat{\tau}})$3 and PSNR $\mathcal{M}(s_\tau, s_{\hat{\tau}})$4, and improves Sync-C to $\mathcal{M}(s_\tau, s_{\hat{\tau}})$5 and HKC to $\mathcal{M}(s_\tau, s_{\hat{\tau}})$6 at $\mathcal{M}(s_\tau, s_{\hat{\tau}})$7 FPS (Lu et al., 2 Oct 2025).

Geometry-Aware Distillation addresses a different failure mode: loss of sensitivity to initial noise in text-to-image distillation. The core JVP objective is

$\mathcal{M}(s_\tau, s_{\hat{\tau}})$8

implemented by finite differences through $\mathcal{M}(s_\tau, s_{\hat{\tau}})$9 (Huang et al., 1 Jun 2026). The paper reports training-time overheads of $1$0 time and $1$1 memory for LADD on SD2.1 UNet, $1$2 time and $1$3 memory for TDM on PixArt-$1$4 DiT, and $1$5 time and $1$6 memory for SiD on SANA Flow-DiT (Huang et al., 1 Jun 2026). It also reports seed-sensitivity gains, including LADD baseline self-identifiability $1$7 versus LADD+GAD $1$8, compared with teacher $1$9 (Huang et al., 1 Jun 2026).

These diffusion examples show three distinct forms of input-awareness: weighting by the diffusion state, masking by structured conditioning such as pose, and matching local differential behavior with respect to the actual input noise.

5. Language-model and reasoning-oriented variants

Input-aware distillation in language modeling appears both as representation matching conditioned on attention and as output sanitization or data reweighting conditioned on the current input’s influence profile.

AweDist learns embeddings for new input tokens by distilling teacher hidden states from the original tokenization into a frozen transformer augmented with new token embeddings. The target positions are selected by teacher self-attention to the original subtoken span, making the loss sequence-specific (Dobler et al., 26 May 2025). The paper reports that AweDist evaluates on Mistral-7B-v0.1, OLMo-2-7B-1124-Instruct, Llama-3-8B, Llama-3-8B-Instruct, Llama-3.1-8B, Llama-3.1-8B-Instruct, Llama-3.2-3B, and Llama-3.2-3B-Instruct, with approximately $\gamma$0 domain words, up to $\gamma$1 contexts per token, length $\gamma$2, and about $\gamma$3 minutes for about $\gamma$4 tokens on a single H100 80GB GPU (Dobler et al., 26 May 2025).

A different notion of input-awareness appears in anti-distillation for black-box LLM APIs. The paper “Towards Distillation-Resistant LLMs” characterizes distillation-relevant information by

$\gamma$5

where $\gamma$6 denotes the input query, $\gamma$7 the ground-truth next token, and $\gamma$8 the teacher logits (Fang et al., 3 Feb 2026). The proposed transformation $\gamma$9 is optimized with

$\gamma$0

where $\gamma$1 and $\gamma$2 are gradients of the student KD objective under original and transformed logits (Fang et al., 3 Feb 2026). The paper states that this method is input-aware because it computes $\gamma$3 and $\gamma$4 on the current batch and directly suppresses the per-input guidance signal embedded in logits (Fang et al., 3 Feb 2026). Reported teacher utility on GSM8K changes from $\gamma$5 to $\gamma$6 for Qwen2.5-7B and from $\gamma$7 to $\gamma$8 for Llama-3.1-8B (Fang et al., 3 Feb 2026).

AIR applies input-aware weighting at the level of reasoning steps rather than teacher features. It computes a token-level loss divergence

$\gamma$9

aggregates step-level scores

$5$0

and defines step weights

$5$1

with normalized

$5$2

The weighted SFT loss is

$5$3

The paper reports that step-level AIR weighting raises average accuracy from $5$4 to $5$5, and that AIR-Step reaches average $5$6 versus $5$7 for entropy-based step selection (Liu et al., 15 Dec 2025).

These examples suggest a broader interpretation in language modeling: the input-aware component may target internal states, logits, or even training data weights, provided that the supervision is conditioned on signals extracted from the specific input instance.

6. Reliability, robustness, and common design trade-offs

A recurring theme is that input-aware distillation is often introduced to correct a failure of uniform supervision. The failure may be imbalance across timesteps, low fidelity in task-critical regions, noisy pseudo-labels, or overemphasis on unreliable teacher predictions.

The MAL framework states this explicitly. Standard multi-frame distillation in ManyDepth applies consistency only on unreliable pixels $5$8 and uses the teacher depth as the sole target; MAL extends distillation to the full image and adaptively chooses between teacher depth and cost-volume depth according to which yields the lower reprojection error (Dong et al., 2024). On CityScapes, the paper reports that ManyDepth baseline Abs Rel $5$9 improves to $w(t) = \min\!\big(\text{SNR}(t) + 1,\ \gamma\big)$00 with MAL, and DualRefine baseline Abs Rel $w(t) = \min\!\big(\text{SNR}(t) + 1,\ \gamma\big)$01 improves to $w(t) = \min\!\big(\text{SNR}(t) + 1,\ \gamma\big)$02 with MAL (Dong et al., 2024). The ablation reports Original Abs Rel $w(t) = \min\!\big(\text{SNR}(t) + 1,\ \gamma\big)$03, Temporal hints only $w(t) = \min\!\big(\text{SNR}(t) + 1,\ \gamma\big)$04, Distillation hints only $w(t) = \min\!\big(\text{SNR}(t) + 1,\ \gamma\big)$05, and both with MLRA $w(t) = \min\!\big(\text{SNR}(t) + 1,\ \gamma\big)$06 (Dong et al., 2024).

In adversarial training, IGD is motivated by the “inequality phenomenon,” namely that $w(t) = \min\!\big(\text{SNR}(t) + 1,\ \gamma\big)$07-AT increases the Gini coefficient of input-gradient attribution maps and thus concentrates sensitivity on fewer pixels (Chen et al., 2023). The loss aligns the direction of clean-input gradients to a standard-trained teacher while keeping adversarial training on perturbed inputs. On ImageNet-100, the paper reports that global Gini drops from $w(t) = \min\!\big(\text{SNR}(t) + 1,\ \gamma\big)$08 for PGDAT to $w(t) = \min\!\big(\text{SNR}(t) + 1,\ \gamma\big)$09 for IGD with $w(t) = \min\!\big(\text{SNR}(t) + 1,\ \gamma\big)$10, while adversarial accuracy decreases from $w(t) = \min\!\big(\text{SNR}(t) + 1,\ \gamma\big)$11 to $w(t) = \min\!\big(\text{SNR}(t) + 1,\ \gamma\big)$12 (Chen et al., 2023). It also reports up to $w(t) = \min\!\big(\text{SNR}(t) + 1,\ \gamma\big)$13 reduction against inductive noise, $w(t) = \min\!\big(\text{SNR}(t) + 1,\ \gamma\big)$14 against inductive occlusion, $w(t) = \min\!\big(\text{SNR}(t) + 1,\ \gamma\big)$15 against random noise, and $w(t) = \min\!\big(\text{SNR}(t) + 1,\ \gamma\big)$16 on ImageNet-C relative to vanilla $w(t) = \min\!\big(\text{SNR}(t) + 1,\ \gamma\big)$17-AT (Chen et al., 2023).

UCD addresses reliability through uncertainty weighting. The dot-product weight $w(t) = \min\!\big(\text{SNR}(t) + 1,\ \gamma\big)$18 suppresses uncertain teacher pseudo-labels without requiring explicit clipping or calibration (Yang et al., 2022). The paper reports best temperature $w(t) = \min\!\big(\text{SNR}(t) + 1,\ \gamma\big)$19 and best $w(t) = \min\!\big(\text{SNR}(t) + 1,\ \gamma\big)$20, with mIoU gains over MiB and PLOP across Pascal VOC, ADE20K, and Cityscapes protocols (Yang et al., 2022).

From these cases, a plausible implication is that input-aware distillation is frequently adopted when a global teacher target is known to be locally unreliable, locally uninformative, or locally misaligned with the student’s learning dynamics.

7. Historical pattern and generalization across modalities

The surveyed papers indicate that input-aware distillation has expanded from classical supervised settings into diffusion models, video synthesis, language modeling, adversarial robustness, self-supervised geometry, and incremental dense prediction. The specific implementations differ, but they can be grouped by what aspect of the input they condition on.

Conditioning signal	Representative formulation	Example paper
Diffusion state	$w(t) = \min\!\big(\text{SNR}(t) + 1,\ \gamma\big)$21	(Liu et al., 2023)
Spatial structure	Pose-defined masks $w(t) = \min\!\big(\text{SNR}(t) + 1,\ \gamma\big)$22	(Lu et al., 2 Oct 2025)
Attention structure	Attention-selected set $w(t) = \min\!\big(\text{SNR}(t) + 1,\ \gamma\big)$23	(Dobler et al., 26 May 2025)
Photometric evidence	$w(t) = \min\!\big(\text{SNR}(t) + 1,\ \gamma\big)$24 chosen by $w(t) = \min\!\big(\text{SNR}(t) + 1,\ \gamma\big)$25 and $w(t) = \min\!\big(\text{SNR}(t) + 1,\ \gamma\big)$26	(Dong et al., 2024)
Representation geometry	JVP matching $w(t) = \min\!\big(\text{SNR}(t) + 1,\ \gamma\big)$27	(Huang et al., 1 Jun 2026)
Pixel uncertainty	Weight $w(t) = \min\!\big(\text{SNR}(t) + 1,\ \gamma\big)$28	(Yang et al., 2022)

Despite the diversity, two structural motifs recur. First, the teacher target is often no longer static: it may be recomputed under perturbation, restricted to attention-selected positions, or selected between multiple candidate targets. Second, the input-aware component is often combined with a conventional base objective rather than replacing it entirely. Examples include $w(t) = \min\!\big(\text{SNR}(t) + 1,\ \gamma\big)$29 in co-speech video (Lu et al., 2 Oct 2025), $w(t) = \min\!\big(\text{SNR}(t) + 1,\ \gamma\big)$30 in text-to-image distillation (Huang et al., 1 Jun 2026), $w(t) = \min\!\big(\text{SNR}(t) + 1,\ \gamma\big)$31 in incremental segmentation (Yang et al., 2022), and adversarial training plus gradient alignment in IGD (Chen et al., 2023).

This suggests that input-aware distillation is usually framed as a corrective term that restores information lost by uniform averaging, rather than as a wholesale replacement for the main training criterion. A related misconception is that “input-aware” always means spatial masking. The cited literature shows that the notion is broader: it also includes timestep-aware weighting, token-attention gating, per-step reasoning importance, and finite-difference sensitivity alignment.

In aggregate, the topic refers to a general methodology for distillation in which the teacher–student coupling depends on input-conditioned evidence about importance, confidence, locality, or geometry. The continued appearance of this pattern across modalities suggests that the central problem is not merely transferring outputs, but deciding which parts of the teacher’s behavior should be transferred for each input and with what strength (Liu et al., 2023, Lu et al., 2 Oct 2025, Dobler et al., 26 May 2025, Dong et al., 2024, Yang et al., 2022, Huang et al., 1 Jun 2026).