---
title: Source-wise Distillation
url: https://www.emergentmind.com/topics/source-wise-distillation
type: topic
---

# Source-wise Distillation

Searching arXiv for recent and foundational papers relevant to source-wise distillation.
First, I’ll look for papers explicitly using the term "source-wise distillation" or closely related formulations.
Source-wise distillation denotes a family of knowledge-distillation procedures in which the central design variable is the **source of supervision** rather than only the student architecture or the final loss. In different subfields, the “source” may be an ISP-processed RGB pipeline supervising a RAW model, a textual teacher and an acoustic teacher jointly supervising an audio-language model, multiple alternative reasoning teachers generating parallel datasets for the same query corpus, a visual foundation model supervising a lidar backbone, or a role-specific augmentation distribution exposing the teacher’s knowledge differently to teacher and student [2101.10203] [2509.18579] [2505.14464] [2604.18831] [2004.08861]. The common objective is to transfer task-relevant structure while avoiding expensive target-domain annotation, preserving source-specific competence, or controlling what kind of teacher signal the student actually learns from.

## 1. Conceptual scope and recurring definitions

The literature uses *source-wise distillation* in several closely related senses. In one line of work, it refers to **modality-specific sources of supervision**: an RGB teacher for a RAW student, or a textual teacher plus an acoustic teacher for an audio-language student [2101.10203] [2509.18579]. In another, it refers to **teacher-specific distilled datasets**, where the query set is fixed and only the teacher source changes [2505.14464]. In cross-modal 2D-to-3D transfer, it denotes supervision flowing from a **visual source modality** to a lidar target modality at the frame level [2604.18831]. A broader interpretation also includes adaptation of the **data source or augmentation distribution** used to expose teacher knowledge to the student [2004.08861].

| Setting | Source of supervision | Student or target |
|---|---|---|
| ISP Distillation | Processed RGB teacher pipeline | RAW classifier or segmenter |
| Audio reasoning distillation | Textual teacher and acoustic teacher | Large audio-language model |
| Indoor lidar distillation | Visual foundation model on RGB images | WaffleIron lidar backbone |

Across these formulations, the student does not merely imitate a teacher’s final prediction. It may align with soft probabilities, hidden states, feature descriptors, pseudo-labels, or source-conditioned data distributions. This suggests that source-wise distillation is best understood as a **decomposition of supervision by origin**: *who* or *what modality* provides the signal, *what representation* is transferred, and *which failure mode* that source is meant to correct.

## 2. Paired-data and cross-modal source transfer

In "ISP Distillation" [2101.10203], the problem is that a camera ISP transforms sensor RAW into human-friendly RGB, but the ISP is tuned for **visual appearance**, not for downstream machine tasks such as classification or segmentation. High-level models assume a canonical RGB space, yet dropping the ISP naively causes a **performance drop** because RAW is noisy, mosaiced, low-level, and camera-specific. The proposed solution is to use **RAW-RGB paired data** with pixel alignment and an **off-the-shelf pretrained RGB model** as teacher. The student operates directly on RAW, inherits the teacher target from the paired RGB image, and thereby learns to produce RGB-pipeline-like predictions without manual RAW annotation.

The teacher–student formulation is explicit:
\[
M_{RGB}(RGB), \qquad M_{RAW}(RAW).
\]
The student is initialized from RGB model weights, and RAW is converted into a 3-channel tensor by **bilinear interpolation** over the Bayer mosaic while keeping original sensor samples unchanged. The basic distillation loss is
\[
L_{KD} = -\sum_i p_i \log(q_i),
\]
with \(p=M_{RGB}(RGB)\) and \(q=M_{RAW}(RAW)\), both softened with temperature \(T\). The paper also adds an \(\ell_2\) feature-alignment term before the classifier and, when labels are available, combines supervised and distilled objectives as
\[
L = \alpha L_{CE} + (1-\alpha)L_{KD},
\]
using \(T=4\) and \(\alpha=0.9\) [2101.10203].

The same source-wise logic appears in indoor frame-wise lidar semantic segmentation [2604.18831], but now the source is a **visual foundation model** rather than an RGB task network. Each lidar scan is paired with the temporally closest camera image, intrinsic/extrinsic calibration is used to project each lidar point \(p_i\) to a pixel \(\rho(i)\), and the projected teacher signal supervises the 3D student. The core objective is feature alignment:
\[
\mathcal{L}_{\mathrm{sim}} = \frac{1}{N}\sum_{i=1}^{N} \left\| \tilde{f}_i - g_{\rho(i)} \right\|_2,
\]
where both \(\tilde{f}_i\) and \(g_{\rho(i)}\) are \(\ell_2\)-normalized. Here the source-wise dimension is strictly cross-modal: **RGB image features from a VFM** supervise **lidar point descriptors**.

Empirically, the paired-source setup in ISP Distillation substantially narrows the gap to RGB models. On synthetic noisy mosaiced ImageNet with ResNet18, the reported top-1 values are **69.76** for clean RGB, **29.23** for a pretrained RGB model applied directly to RAW, **57.21** for training on RAW with ground-truth labels, **56.59** for training on RAW with predicted labels, and **62.46** for ISP Distillation; for MobileNetV2, ground-truth and predicted-label training give **56.31** and **56.73**, while ISP Distillation gives **62.92** [2101.10203]. In indoor lidar, the distilled model reaches up to **56.5 mIoU** against pseudo labels and **35.8 mIoU** against real labels on ITC, while the supervised cross-domain RandLA-Net baseline trained on S3DIS achieves **10.7 mIoU**, **21.4 mAcc**, and **26.8 oAcc** [2604.18831].

These results establish a core property of source-wise distillation: paired source data can replace target-domain manual annotation when the source model already encodes the transformations that make the downstream task tractable.

## 3. Multi-source teachers and modality-preserving distillation

"Teaching Audio Models to Reason: A Unified Framework for Source- and Layer-wise Distillation" [2509.18579] formalizes source-wise distillation as the use of **both textual and acoustic teachers to provide complementary modality-specific supervision**. The student is a **large audio-language model (LALM)** initialized from a pretrained Qwen2.5-Omni-7B thinker model. The textual teacher is **Qwen3-8B**, which is strong at symbolic reasoning but cannot process raw audio. The acoustic teacher is a frozen snapshot of the pre-distillation student, denoted \(S0\), whose purpose is to preserve the original audio-processing capability.

The source-wise split is operationalized as two losses: a **textual distillation loss** \(\mathcal{L}_{\rm txt}\) and an **acoustic distillation loss** \(\mathcal{L}_{\rm ac}\), combined with supervised fine-tuning:
\[
\mathcal{L}_{\rm joint} = \mathcal{L}_{\rm txt} + \alpha_{\rm ac} \mathcal{L}_{\rm ac} + \alpha_{\rm SFT}\mathcal{L}_{\rm SFT}.
\]
Because the textual teacher cannot consume raw audio, each sample is converted into a **textualized audio description** \(d\). The teacher then supervises the output sequence \(y=\{r,a\}\), consisting of a reasoning trace and final answer. The acoustic teacher, by contrast, does not provide logits; it anchors the student’s hidden states at audio-token positions:
\[
\mathcal{L}_{\rm ac} = \sum_{t \in \mathcal{T}_{x}} \sum_{i=1}^{L_S} \mathrm{KD}\Big( h^{S0}_{i,t} \;\Big\|\; h^{S}_{i,t} \Big).
\]

This formulation separates two roles that a single teacher cannot provide well in this setting. The **textual teacher** transfers symbolic reasoning, commonsense inference, structured intermediate reasoning traces, and answer-generation patterns. The **acoustic teacher** transfers raw audio representation stability, preservation of perceptual/audio grounding, and resistance to catastrophic forgetting. Source-wise distillation is therefore not simply teacher ensembling. It is a structured decomposition of the supervision origin according to the failure modes of the student.

The empirical findings isolate the effect of adding the acoustic source. The key comparison is **Layer-wise txt KD + SFT** versus **Layer-wise txt KD + ac KD + SFT**. The combined model reaches **73.30%** on AQA average, and adding acoustic KD improves **sound AQA by +4.51%** and **SER by +6.38%** [2509.18579]. The reported interpretation is that reasoning-oriented distillation alone can improve reasoning-heavy AQA while harming SER, whereas acoustic distillation restores the ability to perceive and analyze low-level acoustic features.

A recurring consequence follows: in source-wise distillation, different sources are not interchangeable even when they improve the same task. One source may increase symbolic competence while another protects modality-specific representations that would otherwise be erased.

## 4. Teacher source as dataset generator

A distinct usage of source-wise distillation appears in reasoning-data construction. "Not All Correct Answers Are Equal: Why Your Distillation Source Matters" [2505.14464] fixes a **shared corpus of 1.89 million queries** and changes only the teacher that generates distilled responses. The three teachers are **AM-Thinking-v1**, **Qwen3-235B-A22B**, and **DeepSeek-R1**. Each query is independently sent to each teacher, generation is repeated until the output passes verification, and the result is **three parallel datasets** whose differences can be attributed to the teacher source rather than the prompt pool.

The verification rule is
\[
\text{verification score} \geq 0.9.
\]
Verification is task dependent: math uses Math-Verify and then Qwen2.5-7B-Instruct if needed; code uses sandbox execution; scientific reasoning uses similarity to reference answer scored by Qwen2.5-7B-Instruct; instruction following uses ifeval validator with Qwen2.5-72B-Instruct filling missing constraints; and multi-turn or general reasoning uses Decision-Tree-Reward-Llama-3.1-8B. Students are trained from the **Qwen2.5-32B base model** with learning rate **8e-5**, max sequence length **32k**, global batch size **64**, and **2** epochs [2505.14464].

The central empirical finding is that changing only the teacher source changes student quality substantially. The student distilled from **AM-Thinking-v1** achieves **84.3** on AIME2024, **72.2** on AIME2025, **98.4** on MATH500, and **65.9** on LiveCodeBench. The corresponding Qwen3-distilled scores are **79.4**, **62.2**, **93.9**, and **59.6**; the DeepSeek-R1-distilled scores are **70.9**, **52.8**, **95.8**, and **57.0** [2505.14464]. The paper further reports mean perplexity values of **2.5** for AM-Thinking-v1, **2.9** for DeepSeek-R1, and **3.0** for Qwen3-235B-A22B, and emphasizes that AM-Thinking-v1 exhibits greater token-length diversity, including many **short sequences** and many **very long sequences** for math.

This usage of source-wise distillation turns the teacher into a **data generator** rather than only a source of online logits or features. The source now determines the distribution of verified reasoning traces, the output-length policy the student learns, and the benchmark behavior of the final model. The claim that “not all correct answers are equal” is therefore a statement about supervision provenance: correctness alone does not define distillation quality if different sources induce different reasoning styles and trainability.

## 5. Adjacent granularities: sample-wise, dataset-wise, and role-wise sources

A broader view of the literature extends the source question below the level of the teacher model. "CKD: Contrastive Knowledge Distillation from A Sample-wise Perspective" [2404.14109] treats distillation as a **sample-wise alignment problem**. Given a minibatch \(\mathcal{X}=[x_0,x_1,\ldots,x_{n-1}]\), the teacher and student produce per-sample logits \(\mathbf{t}_i,\mathbf{s}_i\in\mathbb{R}^c\). CKD defines an intra-sample term
\[
\mathcal{L}_{\text{intra}} = \frac{1}{n}\sum_i d(\mathbf{t}_i,\mathbf{s}_i),
\]
an inter-sample term
\[
\mathcal{L}_{\text{inter}} = -\frac{1}{n(n-1)}\sum_{i}\sum_{j\neq i} d(\mathbf{s}_i,\mathbf{s}_j),
\]
and the final distillation loss
\[
\mathcal{L}_{\text{KD}}=\mathcal{L}_{\text{intra}}+\beta \mathcal{L}_{\text{inter}}.
\]
The positive pair is \((\mathbf{t}_i,\mathbf{s}_i)\), and the negative pairs are cross-sample combinations. The paper frames this as relevant to **source-wise / sample-wise knowledge distillation**, since the supervision structure is defined at the granularity of individual samples rather than classes or tapped features. On ImageNet-1K, CKD improves vanilla KD by **+1.58% Top-1 / +0.93% Top-5** for ResNet-34 \(\to\) ResNet-18 and **+4.39% Top-1 / +2.38% Top-5** for ResNet-50 \(\to\) MobileNetV2 [2404.14109].

At the dataset level, "What is Dataset Distillation Learning?" [2406.04284] analyzes distilled data as a form of **source-wise information compression**. The paper’s main result is that distilled data is recognizable by real-trained models, but mixing it with real data during training can *hurt* performance. It further argues that distilled data preserves the information a model learns from real data **early in training**, and uses exact leave-one-out influence functions,
\[
I_{x_d \rightarrow x_t} = L(x_t; \hat\theta_{-x_d}) - L(x_t; \hat\theta),
\]
to show that individual synthetic points can encode semantically meaningful attributes such as “yellow car,” “parking lot,” or “plane on runway” [2406.04284]. In this interpretation, the source is not a teacher network but the original training trajectory and the semantic substructure it compresses.

At the data-pipeline level, "Role-Wise Data Augmentation for Knowledge Distillation" [2004.08861] argues that teacher and student should not necessarily see the same training examples. The method uses **Population-Based Augmentation (PBA)** to learn an epoch-wise augmentation schedule \(\mathcal{A}_T\) for the teacher in Stage-\(\alpha\), and a separate schedule \(\mathcal{A}_S\) for the student under KD in Stage-\(\beta\). The crucial empirical result is that the teacher’s best augmentation policy is not the student’s best policy. On CIFAR-100 with 4-bit ResNet18, the paper reports **69.81** for vanilla training, **70.24** for Stage-\(\alpha\), **73.85** for only II-KD, and **74.21** for Stage-\(\beta\); with the teacher schedule reused for the student, the score is **73.46**, below the **74.21** obtained by the student-specific schedule [2004.08861].

Taken together, these works indicate that the “source” in distillation can be localized at multiple levels: teacher identity, modality, sample correspondence, dataset construction, or augmentation distribution. This suggests that source-wise distillation is not a single algorithmic template but a family of design choices about **which origin of information should be matched, preserved, or reweighted**.

## 6. Empirical regularities, assumptions, and limitations

Several regularities recur across source-wise formulations. First, source-wise distillation frequently reduces annotation cost by replacing manual labels with supervision inherited from a source model. ISP Distillation uses paired RAW-RGB data and pretrained RGB predictions, so no manual RAW labels are required in the main setup [2101.10203]. Indoor lidar distillation likewise uses VFM-derived pseudo labels because frame-wise indoor lidar semantic annotations are scarce [2604.18831]. A plausible implication is that source-wise distillation is especially attractive when the target representation is difficult for humans to annotate directly.

Second, the choice of source can improve one capability while degrading another if not balanced. In audio reasoning, **SFT-only** can degrade sound-related AQA and SER, **Top-layer txt KD** helps only slightly, **Layer-wise txt KD** improves reasoning-heavy AQA but harms SER, and **adding acoustic KD** restores acoustic competence while preserving reasoning gains [2509.18579]. In reasoning-trace distillation, the best-performing source is not the source with the longest outputs or the largest math share, but **AM-Thinking-v1**, whose distilled data has lower perplexity and broader token-length diversity [2505.14464]. In ISP Distillation, the paper reports that **feature loss helps**, **stronger teacher helps**, **gradual blending between RGB and RAW did not help**, and **localized distillation for segmentation did not improve over global distillation** [2101.10203].

Third, pseudo-label or distilled-data evaluation must be treated cautiously. In indoor lidar segmentation, performance on ITC is **56.5 mIoU** against pseudo labels but only **35.8 mIoU** against real labels, showing that pseudo labels are **optimistic** and can overestimate performance [2604.18831]. In dataset distillation, distilled data is not a drop-in replacement for real data during arbitrary training: it works in the standard train-on-distilled/test-on-real regime, but adding real data to distilled data can reduce performance [2406.04284]. These findings argue against a common misconception that source-wise distillation simply substitutes for ordinary supervision without altering the geometry of training.

The practical assumptions are similarly consistent. Paired or projected supervision requires reliable alignment: RAW-RGB pairs must be aligned; lidar–camera pairs must be sufficiently synchronized and calibrated; image teacher outputs must be semantically meaningful; and point–pixel correspondences must be reliable [2101.10203] [2604.18831]. Multi-source teacher setups require that the sources contribute nonredundant information rather than duplicate the same signal [2509.18579]. Teacher-generated dataset approaches require a verification mechanism that filters raw generations into usable supervision [2505.14464].

The literature therefore suggests a narrow but powerful conclusion. Source-wise distillation is most effective when the supervision source is chosen to compensate for a known mismatch: RAW versus RGB preprocessing, audio perception versus text reasoning, image semantics versus unlabeled lidar, or student capacity versus teacher-oriented augmentation. It is less a single method than a principle of distillation design: **specify the origin of the transferable information, separate it from the information that should be preserved, and optimize the student against that decomposition rather than against an undifferentiated teacher signal**.

Source: https://www.emergentmind.com/topics/source-wise-distillation