Source-wise Distillation

• Source-wise distillation is a method that decomposes knowledge transfer by clearly delineating the origin of supervision, such as modality-specific teachers.
• It leverages paired data and targeted teacher signals—like RGB, textual, or acoustic sources—to replace manual annotation and tailor student learning.
• The approach adapts distinct supervision types to address specific mismatches and failure modes, thereby optimizing performance across diverse applications.

Searching arXiv for recent and foundational papers relevant to source-wise distillation. First, I’ll look for papers explicitly using the term "source-wise distillation" or closely related formulations. Source-wise distillation denotes a family of knowledge-distillation procedures in which the central design variable is the source of supervision rather than only the student architecture or the final loss. In different subfields, the “source” may be an ISP-processed RGB pipeline supervising a RAW model, a textual teacher and an acoustic teacher jointly supervising an audio-LLM, multiple alternative reasoning teachers generating parallel datasets for the same query corpus, a visual foundation model supervising a lidar backbone, or a role-specific augmentation distribution exposing the teacher’s knowledge differently to teacher and student (Schwartz et al., 2021, Yang et al., 23 Sep 2025, Tian et al., 20 May 2025, Wu et al., 20 Apr 2026, Fu et al., 2020). The common objective is to transfer task-relevant structure while avoiding expensive target-domain annotation, preserving source-specific competence, or controlling what kind of teacher signal the student actually learns from.

1. Conceptual scope and recurring definitions

The literature uses source-wise distillation in several closely related senses. In one line of work, it refers to modality-specific sources of supervision: an RGB teacher for a RAW student, or a textual teacher plus an acoustic teacher for an audio-language student (Schwartz et al., 2021, Yang et al., 23 Sep 2025). In another, it refers to teacher-specific distilled datasets, where the query set is fixed and only the teacher source changes (Tian et al., 20 May 2025). In cross-modal 2D-to-3D transfer, it denotes supervision flowing from a visual source modality to a lidar target modality at the frame level (Wu et al., 20 Apr 2026). A broader interpretation also includes adaptation of the data source or augmentation distribution used to expose teacher knowledge to the student (Fu et al., 2020).

Setting	Source of supervision	Student or target
ISP Distillation	Processed RGB teacher pipeline	RAW classifier or segmenter
Audio reasoning distillation	Textual teacher and acoustic teacher	Large audio-LLM
Indoor lidar distillation	Visual foundation model on RGB images	WaffleIron lidar backbone

Across these formulations, the student does not merely imitate a teacher’s final prediction. It may align with soft probabilities, hidden states, feature descriptors, pseudo-labels, or source-conditioned data distributions. This suggests that source-wise distillation is best understood as a decomposition of supervision by origin: who or what modality provides the signal, what representation is transferred, and which failure mode that source is meant to correct.

2. Paired-data and cross-modal source transfer

In "ISP Distillation" (Schwartz et al., 2021), the problem is that a camera ISP transforms sensor RAW into human-friendly RGB, but the ISP is tuned for visual appearance, not for downstream machine tasks such as classification or segmentation. High-level models assume a canonical RGB space, yet dropping the ISP naively causes a performance drop because RAW is noisy, mosaiced, low-level, and camera-specific. The proposed solution is to use RAW-RGB paired data with pixel alignment and an off-the-shelf pretrained RGB model as teacher. The student operates directly on RAW, inherits the teacher target from the paired RGB image, and thereby learns to produce RGB-pipeline-like predictions without manual RAW annotation.

The teacher–student formulation is explicit: $M_{RGB}(RGB), \qquad M_{RAW}(RAW).$ The student is initialized from RGB model weights, and RAW is converted into a 3-channel tensor by bilinear interpolation over the Bayer mosaic while keeping original sensor samples unchanged. The basic distillation loss is

$L_{KD} = -\sum_i p_i \log(q_i),$

with $p=M_{RGB}(RGB)$ and $q=M_{RAW}(RAW)$, both softened with temperature $T$. The paper also adds an $\ell_2$ feature-alignment term before the classifier and, when labels are available, combines supervised and distilled objectives as

$L = \alpha L_{CE} + (1-\alpha)L_{KD},$

using $T=4$ and $\alpha=0.9$ (Schwartz et al., 2021).

The same source-wise logic appears in indoor frame-wise lidar semantic segmentation (Wu et al., 20 Apr 2026), but now the source is a visual foundation model rather than an RGB task network. Each lidar scan is paired with the temporally closest camera image, intrinsic/extrinsic calibration is used to project each lidar point $p_i$ to a pixel $L_{KD} = -\sum_i p_i \log(q_i),$0, and the projected teacher signal supervises the 3D student. The core objective is feature alignment: $L_{KD} = -\sum_i p_i \log(q_i),$1 where both $L_{KD} = -\sum_i p_i \log(q_i),$2 and $L_{KD} = -\sum_i p_i \log(q_i),$3 are $L_{KD} = -\sum_i p_i \log(q_i),$4-normalized. Here the source-wise dimension is strictly cross-modal: RGB image features from a VFM supervise lidar point descriptors.

Empirically, the paired-source setup in ISP Distillation substantially narrows the gap to RGB models. On synthetic noisy mosaiced ImageNet with ResNet18, the reported top-1 values are 69.76 for clean RGB, 29.23 for a pretrained RGB model applied directly to RAW, 57.21 for training on RAW with ground-truth labels, 56.59 for training on RAW with predicted labels, and 62.46 for ISP Distillation; for MobileNetV2, ground-truth and predicted-label training give 56.31 and 56.73, while ISP Distillation gives 62.92 (Schwartz et al., 2021). In indoor lidar, the distilled model reaches up to 56.5 mIoU against pseudo labels and 35.8 mIoU against real labels on ITC, while the supervised cross-domain RandLA-Net baseline trained on S3DIS achieves 10.7 mIoU, 21.4 mAcc, and 26.8 oAcc (Wu et al., 20 Apr 2026).

These results establish a core property of source-wise distillation: paired source data can replace target-domain manual annotation when the source model already encodes the transformations that make the downstream task tractable.

3. Multi-source teachers and modality-preserving distillation

"Teaching Audio Models to Reason: A Unified Framework for Source- and Layer-wise Distillation" (Yang et al., 23 Sep 2025) formalizes source-wise distillation as the use of both textual and acoustic teachers to provide complementary modality-specific supervision. The student is a large audio-LLM (LALM) initialized from a pretrained Qwen2.5-Omni-7B thinker model. The textual teacher is Qwen3-8B, which is strong at symbolic reasoning but cannot process raw audio. The acoustic teacher is a frozen snapshot of the pre-distillation student, denoted $L_{KD} = -\sum_i p_i \log(q_i),$5, whose purpose is to preserve the original audio-processing capability.

The source-wise split is operationalized as two losses: a textual distillation loss $L_{KD} = -\sum_i p_i \log(q_i),$6 and an acoustic distillation loss $L_{KD} = -\sum_i p_i \log(q_i),$7, combined with supervised fine-tuning: $L_{KD} = -\sum_i p_i \log(q_i),$8 Because the textual teacher cannot consume raw audio, each sample is converted into a textualized audio description $L_{KD} = -\sum_i p_i \log(q_i),$9. The teacher then supervises the output sequence $p=M_{RGB}(RGB)$0, consisting of a reasoning trace and final answer. The acoustic teacher, by contrast, does not provide logits; it anchors the student’s hidden states at audio-token positions: $p=M_{RGB}(RGB)$1

This formulation separates two roles that a single teacher cannot provide well in this setting. The textual teacher transfers symbolic reasoning, commonsense inference, structured intermediate reasoning traces, and answer-generation patterns. The acoustic teacher transfers raw audio representation stability, preservation of perceptual/audio grounding, and resistance to catastrophic forgetting. Source-wise distillation is therefore not simply teacher ensembling. It is a structured decomposition of the supervision origin according to the failure modes of the student.

The empirical findings isolate the effect of adding the acoustic source. The key comparison is Layer-wise txt KD + SFT versus Layer-wise txt KD + ac KD + SFT. The combined model reaches 73.30% on AQA average, and adding acoustic KD improves sound AQA by +4.51% and SER by +6.38% (Yang et al., 23 Sep 2025). The reported interpretation is that reasoning-oriented distillation alone can improve reasoning-heavy AQA while harming SER, whereas acoustic distillation restores the ability to perceive and analyze low-level acoustic features.

A recurring consequence follows: in source-wise distillation, different sources are not interchangeable even when they improve the same task. One source may increase symbolic competence while another protects modality-specific representations that would otherwise be erased.

4. Teacher source as dataset generator

A distinct usage of source-wise distillation appears in reasoning-data construction. "Not All Correct Answers Are Equal: Why Your Distillation Source Matters" (Tian et al., 20 May 2025) fixes a shared corpus of 1.89 million queries and changes only the teacher that generates distilled responses. The three teachers are AM-Thinking-v1, Qwen3-235B-A22B, and DeepSeek-R1. Each query is independently sent to each teacher, generation is repeated until the output passes verification, and the result is three parallel datasets whose differences can be attributed to the teacher source rather than the prompt pool.

The verification rule is

$p=M_{RGB}(RGB)$2

Verification is task dependent: math uses Math-Verify and then Qwen2.5-7B-Instruct if needed; code uses sandbox execution; scientific reasoning uses similarity to reference answer scored by Qwen2.5-7B-Instruct; instruction following uses ifeval validator with Qwen2.5-72B-Instruct filling missing constraints; and multi-turn or general reasoning uses Decision-Tree-Reward-Llama-3.1-8B. Students are trained from the Qwen2.5-32B base model with learning rate 8e-5, max sequence length 32k, global batch size 64, and 2 epochs (Tian et al., 20 May 2025).

The central empirical finding is that changing only the teacher source changes student quality substantially. The student distilled from AM-Thinking-v1 achieves 84.3 on AIME2024, 72.2 on AIME2025, 98.4 on MATH500, and 65.9 on LiveCodeBench. The corresponding Qwen3-distilled scores are 79.4, 62.2, 93.9, and 59.6; the DeepSeek-R1-distilled scores are 70.9, 52.8, 95.8, and 57.0 (Tian et al., 20 May 2025). The paper further reports mean perplexity values of 2.5 for AM-Thinking-v1, 2.9 for DeepSeek-R1, and 3.0 for Qwen3-235B-A22B, and emphasizes that AM-Thinking-v1 exhibits greater token-length diversity, including many short sequences and many very long sequences for math.

This usage of source-wise distillation turns the teacher into a data generator rather than only a source of online logits or features. The source now determines the distribution of verified reasoning traces, the output-length policy the student learns, and the benchmark behavior of the final model. The claim that “not all correct answers are equal” is therefore a statement about supervision provenance: correctness alone does not define distillation quality if different sources induce different reasoning styles and trainability.

5. Adjacent granularities: sample-wise, dataset-wise, and role-wise sources

A broader view of the literature extends the source question below the level of the teacher model. "CKD: Contrastive Knowledge Distillation from A Sample-wise Perspective" (Zhu et al., 2024) treats distillation as a sample-wise alignment problem. Given a minibatch $p=M_{RGB}(RGB)$3, the teacher and student produce per-sample logits $p=M_{RGB}(RGB)$4. CKD defines an intra-sample term

$p=M_{RGB}(RGB)$5

an inter-sample term

$p=M_{RGB}(RGB)$6

and the final distillation loss

$p=M_{RGB}(RGB)$7

The positive pair is $p=M_{RGB}(RGB)$8, and the negative pairs are cross-sample combinations. The paper frames this as relevant to source-wise / sample-wise knowledge distillation, since the supervision structure is defined at the granularity of individual samples rather than classes or tapped features. On ImageNet-1K, CKD improves vanilla KD by +1.58% Top-1 / +0.93% Top-5 for ResNet-34 $p=M_{RGB}(RGB)$9 ResNet-18 and +4.39% Top-1 / +2.38% Top-5 for ResNet-50 $q=M_{RAW}(RAW)$0 MobileNetV2 (Zhu et al., 2024).

At the dataset level, "What is Dataset Distillation Learning?" (Yang et al., 2024) analyzes distilled data as a form of source-wise information compression. The paper’s main result is that distilled data is recognizable by real-trained models, but mixing it with real data during training can hurt performance. It further argues that distilled data preserves the information a model learns from real data early in training, and uses exact leave-one-out influence functions,

$q=M_{RAW}(RAW)$1

to show that individual synthetic points can encode semantically meaningful attributes such as “yellow car,” “parking lot,” or “plane on runway” (Yang et al., 2024). In this interpretation, the source is not a teacher network but the original training trajectory and the semantic substructure it compresses.

At the data-pipeline level, "Role-Wise Data Augmentation for Knowledge Distillation" (Fu et al., 2020) argues that teacher and student should not necessarily see the same training examples. The method uses Population-Based Augmentation (PBA) to learn an epoch-wise augmentation schedule $q=M_{RAW}(RAW)$2 for the teacher in Stage-$q=M_{RAW}(RAW)$3, and a separate schedule $q=M_{RAW}(RAW)$4 for the student under KD in Stage-$q=M_{RAW}(RAW)$5. The crucial empirical result is that the teacher’s best augmentation policy is not the student’s best policy. On CIFAR-100 with 4-bit ResNet18, the paper reports 69.81 for vanilla training, 70.24 for Stage-$q=M_{RAW}(RAW)$6, 73.85 for only II-KD, and 74.21 for Stage-$q=M_{RAW}(RAW)$7; with the teacher schedule reused for the student, the score is 73.46, below the 74.21 obtained by the student-specific schedule (Fu et al., 2020).

Taken together, these works indicate that the “source” in distillation can be localized at multiple levels: teacher identity, modality, sample correspondence, dataset construction, or augmentation distribution. This suggests that source-wise distillation is not a single algorithmic template but a family of design choices about which origin of information should be matched, preserved, or reweighted.

6. Empirical regularities, assumptions, and limitations

Several regularities recur across source-wise formulations. First, source-wise distillation frequently reduces annotation cost by replacing manual labels with supervision inherited from a source model. ISP Distillation uses paired RAW-RGB data and pretrained RGB predictions, so no manual RAW labels are required in the main setup (Schwartz et al., 2021). Indoor lidar distillation likewise uses VFM-derived pseudo labels because frame-wise indoor lidar semantic annotations are scarce (Wu et al., 20 Apr 2026). A plausible implication is that source-wise distillation is especially attractive when the target representation is difficult for humans to annotate directly.

Second, the choice of source can improve one capability while degrading another if not balanced. In audio reasoning, SFT-only can degrade sound-related AQA and SER, Top-layer txt KD helps only slightly, Layer-wise txt KD improves reasoning-heavy AQA but harms SER, and adding acoustic KD restores acoustic competence while preserving reasoning gains (Yang et al., 23 Sep 2025). In reasoning-trace distillation, the best-performing source is not the source with the longest outputs or the largest math share, but AM-Thinking-v1, whose distilled data has lower perplexity and broader token-length diversity (Tian et al., 20 May 2025). In ISP Distillation, the paper reports that feature loss helps, stronger teacher helps, gradual blending between RGB and RAW did not help, and localized distillation for segmentation did not improve over global distillation (Schwartz et al., 2021).

Third, pseudo-label or distilled-data evaluation must be treated cautiously. In indoor lidar segmentation, performance on ITC is 56.5 mIoU against pseudo labels but only 35.8 mIoU against real labels, showing that pseudo labels are optimistic and can overestimate performance (Wu et al., 20 Apr 2026). In dataset distillation, distilled data is not a drop-in replacement for real data during arbitrary training: it works in the standard train-on-distilled/test-on-real regime, but adding real data to distilled data can reduce performance (Yang et al., 2024). These findings argue against a common misconception that source-wise distillation simply substitutes for ordinary supervision without altering the geometry of training.

The practical assumptions are similarly consistent. Paired or projected supervision requires reliable alignment: RAW-RGB pairs must be aligned; lidar–camera pairs must be sufficiently synchronized and calibrated; image teacher outputs must be semantically meaningful; and point–pixel correspondences must be reliable (Schwartz et al., 2021, Wu et al., 20 Apr 2026). Multi-source teacher setups require that the sources contribute nonredundant information rather than duplicate the same signal (Yang et al., 23 Sep 2025). Teacher-generated dataset approaches require a verification mechanism that filters raw generations into usable supervision (Tian et al., 20 May 2025).

The literature therefore suggests a narrow but powerful conclusion. Source-wise distillation is most effective when the supervision source is chosen to compensate for a known mismatch: RAW versus RGB preprocessing, audio perception versus text reasoning, image semantics versus unlabeled lidar, or student capacity versus teacher-oriented augmentation. It is less a single method than a principle of distillation design: specify the origin of the transferable information, separate it from the information that should be preserved, and optimize the student against that decomposition rather than against an undifferentiated teacher signal.