Delta-KD: Preserving Finetuning Shifts

• Delta-KD is a knowledge distillation method that captures the teacher’s finetuning-induced distribution shift to generate a synthetic student target distribution.
• It reframes distillation as behavior shift imitation, addressing representational mismatches and outperforming conventional token-level and sequence-level methods.
• Empirical evaluations on Ultrachat-200k and OpenMathReasoning show improved ROUGE scores, validating its effectiveness in instruction tuning and reasoning tasks.

Searching arXiv for the specified papers and closely related delta-distillation work. arXiv query: (Cao et al., 18 Sep 2025) Delta Knowledge Distillation (Delta-KD) is a knowledge distillation method for LLMs that extends token-level KD by preserving the distributional shift introduced during the teacher’s supervised finetuning (SFT), rather than directly forcing the student to match the teacher’s fully finetuned output distribution. In the formulation reported in "Delta Knowledge Distillation for LLMs" (Cao et al., 18 Sep 2025), the central object is a token-level shift operator, denoted $\Delta$, that captures how the teacher’s output probabilities change from a pretrained state to a finetuned state. The student is then trained against a synthetic target distribution obtained by applying that shift in the student’s own output space. Empirical results on ROUGE-1, ROUGE-2, and ROUGE-L for Ultrachat-200k and OpenMathReasoning show that Delta-KD outperforms supervised finetuning, forward-KL and reverse-KL token-level KD, and the sequence-level baselines SeqKD and MiniLLM (Cao et al., 18 Sep 2025).

1. Problem formulation and departure from conventional token-level KD

In standard token-level knowledge distillation for LLMs, the student is trained by minimizing the Kullback–Leibler divergence between the teacher’s output distribution and the student’s output distribution:

$$L_{\text{KD}}=\mathbb{E}_{x \sim D}\left[\mathrm{KL}\left(\pi_t(\cdot \mid x)\,\|\,\pi_s(\cdot \mid x)\right)\right].$$

This formulation, as summarized in (Cao et al., 18 Sep 2025), assumes that the teacher and student share the same optimal representation space and that the student should directly match the teacher’s final finetuned behavior.

The paper identifies two limitations in that assumption set. First, student models may lack the capacity to represent the teacher’s outputs, especially when the size disparity is large. Second, direct alignment to the teacher’s final outputs ignores how much the teacher changed during supervised finetuning, and therefore discards distributional shift information. Delta-KD is proposed precisely to address that mismatch.

The key conceptual change is that Delta-KD does not directly align the student to the teacher’s finetuned distribution. Instead, it asks the student to mimic the teacher’s distributional shift, or “Delta,” while remaining anchored in the student’s own pretrained output space. The resulting objective is not simple output imitation. It is behavior-shift imitation: the student learns to change its outputs analogously to how the teacher changed its own during supervised finetuning (Cao et al., 18 Sep 2025).

2. Distributional shift as the transferred object

The core object in Delta-KD is the distributional shift induced by finetuning. Given a pretrained teacher distribution $\pi_t^{\text{raw}}$, a finetuned teacher distribution $\pi_t^{\text{ft}}$, and a pretrained student distribution $\pi_s^{\text{raw}}$, the teacher’s shift is defined as

$$\Delta_t^{\text{raw}\to \text{ft}}(y\mid x)=\frac{\pi_t^{\text{ft}}(y\mid x)}{\pi_t^{\text{raw}}(y\mid x)}.$$

This quantity measures how much the teacher’s output probability for a token changes as a consequence of supervised finetuning (Cao et al., 18 Sep 2025).

Delta-KD preserves that shift by constructing a synthetic student target distribution. Rather than treating the teacher’s finetuned distribution as universally attainable, it applies the teacher’s shift to the student’s own pretrained distribution. The synthetic target is

$$\pi_s^*(y\mid x)\propto \pi_t^{\text{ft}}(y\mid x)\left(\frac{\pi_s^{\text{raw}}(y\mid x)}{\pi_t^{\text{raw}}(y\mid x)}\right)^\alpha,$$

with $\alpha \in [0,1]$ controlling how strongly the delta is followed. The construction is normalized over the vocabulary (Cao et al., 18 Sep 2025).

The intuition stated in the source is operational rather than metaphorical: if finetuning made the teacher much more likely to choose certain tokens, the student should likewise become more likely to choose those tokens relative to its own base likelihoods. This reframes distillation as preservation of a finetuning-induced directional change, not replication of an endpoint distribution. A plausible implication is that the method is better matched to teacher–student pairs with pronounced representational asymmetry, because the transferred object is a relative shift rather than an absolute target.

3. Mathematical formulation and optimization objectives

The paper defines a generic delta operator

$$\Delta(p_1,p_2)(y\mid x):=\frac{p_1(y\mid x)}{p_2(y\mid x)}.$$

Using this notation, the alignment principle is expressed as

$$\Delta(\pi_s^*,\pi_t^{\text{ft}})\propto \Delta(\pi_s^{\text{raw}},\pi_t^{\text{raw}})^\alpha.$$

Solving for the synthetic target yields

$$L_{\text{KD}}=\mathbb{E}_{x \sim D}\left[\mathrm{KL}\left(\pi_t(\cdot \mid x)\,\|\,\pi_s(\cdot \mid x)\right)\right].$$0

where $$L_{\text{KD}}=\mathbb{E}_{x \sim D}\left[\mathrm{KL}\left(\pi_t(\cdot \mid x)\,\|\,\pi_s(\cdot \mid x)\right)\right].$$1 is a normalization constant (Cao et al., 18 Sep 2025).

The distillation loss then becomes

$$L_{\text{KD}}=\mathbb{E}_{x \sim D}\left[\mathrm{KL}\left(\pi_t(\cdot \mid x)\,\|\,\pi_s(\cdot \mid x)\right)\right].$$2

This differs from forward-KL token-level KD in the target of the divergence. The student is not optimized against $$L_{\text{KD}}=\mathbb{E}_{x \sim D}\left[\mathrm{KL}\left(\pi_t(\cdot \mid x)\,\|\,\pi_s(\cdot \mid x)\right)\right].$$3 directly, but against the delta-shifted synthetic distribution $$L_{\text{KD}}=\mathbb{E}_{x \sim D}\left[\mathrm{KL}\left(\pi_t(\cdot \mid x)\,\|\,\pi_s(\cdot \mid x)\right)\right].$$4.

When labeled responses are available, the paper combines Delta-KD with supervised loss:

$$L_{\text{KD}}=\mathbb{E}_{x \sim D}\left[\mathrm{KL}\left(\pi_t(\cdot \mid x)\,\|\,\pi_s(\cdot \mid x)\right)\right].$$5

where $$L_{\text{KD}}=\mathbb{E}_{x \sim D}\left[\mathrm{KL}\left(\pi_t(\cdot \mid x)\,\|\,\pi_s(\cdot \mid x)\right)\right].$$6 is the standard cross-entropy loss on available labeled data (Cao et al., 18 Sep 2025). This combined objective places Delta-KD in the familiar SFT-plus-distillation regime while altering only the construction of the token-level target.

The paper also introduces more general “Parallelogram” variants that explore other combinations of base and shift terms. The summary explicitly notes practical restrictions due to stability, which indicates that the broader algebra of shift composition is not uniformly tractable in training (Cao et al., 18 Sep 2025).

4. Training workflow, systems design, and implementation considerations

The reported workflow requires access to three distributions in the relevant regimes: the teacher before supervised finetuning, the teacher after supervised finetuning, and the student before student-side adaptation. These distributions are used to construct the per-token synthetic target $$L_{\text{KD}}=\mathbb{E}_{x \sim D}\left[\mathrm{KL}\left(\pi_t(\cdot \mid x)\,\|\,\pi_s(\cdot \mid x)\right)\right].$$7, after which the student is optimized via KL divergence to that target, optionally together with SFT loss (Cao et al., 18 Sep 2025).

To scale the method to large models and long sequences, the paper states that teacher and student inferences are decoupled during training using a ZeroMQ-based distributed architecture and shared memory, explicitly to avoid GPU memory bottlenecks. That systems detail is significant because the method requires multiple model states and distributions, making naive joint execution expensive. The decoupled architecture is therefore part of the practical realization of Delta-KD, not an incidental engineering convenience (Cao et al., 18 Sep 2025).

The presence of the hyperparameter $$L_{\text{KD}}=\mathbb{E}_{x \sim D}\left[\mathrm{KL}\left(\pi_t(\cdot \mid x)\,\|\,\pi_s(\cdot \mid x)\right)\right].$$8 is another implementation-relevant feature. In the summarized formulation, $$L_{\text{KD}}=\mathbb{E}_{x \sim D}\left[\mathrm{KL}\left(\pi_t(\cdot \mid x)\,\|\,\pi_s(\cdot \mid x)\right)\right].$$9 corresponds to full behavior shift, whereas $\pi_t^{\text{raw}}$0 ignores the delta. This makes Delta-KD a tunable interpolation between direct reliance on the student’s own pretrained space and stronger preservation of the teacher’s finetuning-induced change. The source does not report a broader theoretical schedule for $\pi_t^{\text{raw}}$1, but its explicit inclusion means that the method can represent a family of objectives rather than a single fixed target.

5. Empirical evaluation

The evaluation reported in (Cao et al., 18 Sep 2025) uses two datasets, Ultrachat-200k for instruction tuning and OpenMathReasoning for math reasoning; a Qwen2.5-7B teacher; a Qwen2.5-1.5B student; and ROUGE-1, ROUGE-2, and ROUGE-L as the metrics. Baselines include supervised finetuning (SFT), token-level Forward KL (FKL) and Reverse KL (RKL) KD, and the sequence-level methods SeqKD and MiniLLM.

Dataset	Method	ROUGE-1 / ROUGE-2 / ROUGE-L
Ultrachat	SFT	0.4049 / 0.1830 / 0.2478
Ultrachat	FKL	0.5146 / 0.2477 / 0.3325
Ultrachat	RKL	0.5070 / 0.2397 / 0.3301
Ultrachat	SeqKD	0.5150 / 0.2502 / 0.3348
Ultrachat	MiniLLM	0.5211 / 0.2491 / 0.3347
Ultrachat	Delta-KD	0.5407 / 0.2599 / 0.3425
OpenMath	SFT	0.4531 / 0.2137 / 0.2305
OpenMath	FKL	0.4768 / 0.2327 / 0.2386
OpenMath	RKL	0.4757 / 0.2314 / 0.2377
OpenMath	SeqKD	0.4829 / 0.2389 / 0.2482
OpenMath	MiniLLM	0.4197 / 0.1843 / 0.2240
OpenMath	Delta-KD	0.5125 / 0.2664 / 0.2465

The summary states that Delta-KD achieves the highest scores across all metrics and datasets, outperforming all baselines (Cao et al., 18 Sep 2025). On Ultrachat, the method improves over the best non-Delta baseline on all three ROUGE metrics. On OpenMathReasoning, it improves ROUGE-1 and ROUGE-2 over all listed baselines, while the highest ROUGE-L among the shown numbers belongs to SeqKD rather than Delta-KD. The source nevertheless characterizes the results as highest across all metrics and datasets, and that is the formulation used in the supplied material.

The reported interpretation is that the gains are especially pronounced on ROUGE-1 and ROUGE-2, reflecting better capture of surface content and bigram coherence, and that improvements appear on both instruction-following and reasoning tasks (Cao et al., 18 Sep 2025). This suggests domain robustness across the two evaluated settings, though the reported evidence is confined to those datasets, model sizes, and metrics.

6. Relation to other delta-based distillation paradigms

The term “delta” has appeared in multiple distillation contexts, but the transferred object differs across papers. In Residual Knowledge Distillation, the residual is the feature discrepancy between teacher and student. The student $\pi_t^{\text{raw}}$2 is first trained to mimic the teacher feature map, and an assistant model $\pi_t^{\text{raw}}$3 is then trained on the residual error:

$\pi_t^{\text{raw}}$4

After training, the combined feature map is $\pi_t^{\text{raw}}$5, with the goal that $\pi_t^{\text{raw}}$6 (Gao et al., 2020). The paper also introduces progressive and integrated residual transfer across multiple blocks. In the supplied summary, RKD is presented as establishing a paradigm in which teacher–student difference signals are explicitly learned rather than only directly matched.

A second line appears in Delta Distillation for Efficient Video Processing, where the “delta” is temporal rather than teacher–student residual. If $\pi_t^{\text{raw}}$7 is the teacher feature at time $\pi_t^{\text{raw}}$8, the delta is

$\pi_t^{\text{raw}}$9

and the student is trained to predict that temporal variation from $\pi_t^{\text{ft}}$0 rather than to reproduce $\pi_t^{\text{ft}}$1 itself (Habibian et al., 2022). At inference, the teacher computes key-frame representations, and the student iteratively predicts deltas for successive frames; the method is explicitly positioned as exploiting temporal redundancy without optical-flow-based motion alignment.

Taken together, these papers suggest that “delta distillation” is best understood as a family of methods in which the distilled signal is a change quantity rather than a terminal representation. In RKD the change quantity is the residual feature error; in video delta distillation it is temporal feature variation; in Delta-KD for LLMs it is the output-distribution shift induced by supervised finetuning (Gao et al., 2020, Habibian et al., 2022, Cao et al., 18 Sep 2025). The common pattern is not a shared architecture but a shared premise: the difference signal may be easier, more targeted, or more faithful to the relevant adaptation process than direct imitation.

7. Interpretation, scope, and common misconceptions

A common misunderstanding is to treat Delta-KD as ordinary token-level KD with a reweighted loss. The defining feature is stronger than that: the target distribution itself is changed. The student is trained against a synthetic target that preserves the teacher’s finetuning-induced shift while remaining grounded in the student’s own pretrained output space (Cao et al., 18 Sep 2025).

A second misconception is that the method simply relaxes KD for smaller students. The paper’s critique is more specific: prior token-level KD assumes that teacher and student share the same optimal representation space, and Delta-KD is introduced precisely because that premise may not hold in many cases (Cao et al., 18 Sep 2025). The method therefore addresses representational mismatch at the level of the target distribution, not merely by weakening optimization pressure.

The scope of the reported evidence is also specific. The experiments use Qwen2.5-7B as teacher and Qwen2.5-1.5B as student, and the empirical comparisons are reported on Ultrachat-200k and OpenMathReasoning with ROUGE metrics (Cao et al., 18 Sep 2025). This suggests effectiveness for instruction tuning and math reasoning under the evaluated setup. A plausible implication is that the method is particularly relevant when the student cannot be expected to reproduce the teacher’s fully finetuned distribution directly, but the supplied material does not report broader cross-architecture or cross-metric validation beyond those experiments.

The Parallelogram variants underscore another point of interpretation. Delta-KD is not merely a heuristic recipe; it arises from an explicit algebra over base distributions and shifts. At the same time, the note about practical restrictions due to stability indicates that not every formally conceivable shift composition is equally usable in training (Cao et al., 18 Sep 2025). In that sense, Delta-KD occupies a middle ground between a theoretical reframing of KD and a practical recipe for large-scale LLM distillation.