Boomerang Distillation in LLMs

• Boomerang distillation is a model size interpolation technique that constructs a continuum of LLM sizes by systematically reinserting teacher layers into a distilled student model.
• It leverages a two-phase process combining knowledge distillation with cosine alignment to ensure smooth, monotonic performance scaling across intermediate models.
• The approach dramatically reduces computation by zero-shot synthesizing multiple model sizes through block replacement, cutting training cost and time compared to independent model training.

Boomerang distillation is a model size interpolation technique for LLMs that facilitates the construction of model families spanning a continuum of intermediate sizes. Rather than training each model size independently or relying on lossy pruning methods, boomerang distillation enables zero-shot synthesis of interpolated models by systematically distilling a large-capacity teacher model to a smaller student and then reconstructing intermediate-size networks by reinserting contiguous blocks of teacher layers into the student. This approach achieves smooth and monotonic scaling of model performance with size, allowing practical and computationally efficient adaptation of LLMs to diverse deployment constraints (Kangaslahti et al., 6 Oct 2025).

1. Formal Framework

The boomerang distillation procedure operates in two phases: distillation and interpolation. The process begins with a pretrained transformer-based teacher model $T$ comprising $L_T$ blocks with parameters $\theta_T$. A student model $S_0$ is initialized with $L_S < L_T$ blocks and parameters $\theta_{S_0}$, matching the teacher’s embedding dimension and vocabulary so that both produce comparable next-token logits: $$T(x_{<j})\in\mathbb{R}^V,\quad S_0(x_{<j})\in\mathbb{R}^V$$ The distillation objective is a convex combination of cross-entropy (CE) loss and Kullback–Leibler (KL) divergence based knowledge distillation (KD) loss: $$\mathcal L_{\mathrm{distill}} = \alpha\,\mathcal L_{\mathrm{CE}}(S_0,T) + (1-\alpha)\,\mathcal L_{\mathrm{KD}}(S_0,T)$$ where

• $$\mathcal L_{\mathrm{CE}} = \mathbb{E}_{x \sim \mathcal{X}} \sum_j -\log \mathrm{softmax}(S_0(x_{<j}))_{x_j}$$,
• $$\mathcal L_{\mathrm{KD}} = \tau^2\,\mathrm{KL}\Bigl(\mathrm{softmax}(T(x_{<j})/\tau)\,\|\,\mathrm{softmax}(S_0(x_{<j})/\tau)\Bigr)$$, with $L_T$0 denoting the distillation temperature and $L_T$1 controlling the trade-off.

In practice, a per-layer alignment term (cosine distance between corresponding teacher and student hidden states) is added: $L_T$2 Yielding the full loss: $L_T$3

2. Boomerang Distillation Algorithm

After distilling the student model $L_T$4, a sequence of intermediate models $L_T$5 is generated by progressively replacing, in a predetermined order, the student’s blocks with the corresponding teacher blocks. For $L_T$6 interpolation steps, each intermediate model $L_T$7 contains: $L_T$8 layers, where $L_T$9 increases monotonically from $\theta_T$0 to $\theta_T$1. The block reinsertion process does not require any further training, owing to the student-teacher alignment achieved during distillation. The replacement is performed either in contiguous blocks or incrementally by block, with each $\theta_T$2 created by substituting $\theta_T$3 teacher blocks into the preceding model. All intermediate models are thus constructed in a zero-shot fashion via weight copying.

Algorithm outline (pseudocode, as given in the original work):

$S_0$2 In the canonical setup, the student is partitioned into blocks $\theta_T$4, and block replacement swaps each $\theta_T$5 with its teacher predecessor.

3. Alignment and Interpolation Success

Effective boomerang distillation hinges on explicit alignment between student and teacher layer representations during student training. The layerwise cosine alignment term ensures that when a teacher block replaces its student counterpart, the adjacent layers are mutually compatible in distribution. Blocks with high hidden-state cosine similarity yield smooth transitions upon reinsertion; mismatched blocks cause sharp performance drops. This suggests the necessity of enforcing local representational compatibility to guarantee monotonic interpolation performance.

Successful interpolation is empirically linked to the quality of alignment as measured by average cosine similarity. High-quality aligned students facilitate nearly linear scaling of performance as more teacher blocks are restored.

4. Empirical Benchmarks and Model Scaling Behavior

Boomerang-distilled models were evaluated across average classification accuracy on 10 tasks and language modeling perplexity on WikiText. The following table summarizes classification accuracy (%) over varying parameter counts (billions):

Size (B)	2.7	3.1	3.5	3.9	4.4	4.8
Boomerang $\theta_T$6	67.2%	70.1%	72.8%	74.5%	76.0%	77.3%
Independent distill	67.2%	69.5%	72.3%	73.4%	75.6%	76.8%
Naive layer-drop	67.2%	68.0%	69.4%	70.1%	71.0%	71.5%

Boomerang interpolation produces a smooth, nearly linear increase in accuracy between the distilled student and full teacher, closely tracking (and sometimes surpassing) independently distilled models at small sizes, but outperforming at larger scales. Naive layer-dropping exhibits a large performance collapse with moderate layer removal. For perplexity, boomerang models achieve $\theta_T$7 of approximately 15 (student) to 9 (teacher), whereas naive pruning immediately raises perplexity to above 50. This smoothly decaying behavior suggests effective transfer of representational structure during the zero-shot interpolation process.

5. Computational Efficiency and Resource Implications

The computational advantage of boomerang distillation is pronounced. Training a single student $\theta_T$8 requires approximately 2.1 billion tokens over four H100 GPUs, typically translating to 12–72 hours wall-clock, depending on student capacity. Standard approaches would require running this process independently for each of $\theta_T$9 model sizes, incurring $S_0$0 the cost. In contrast, boomerang distillation permits all $S_0$1 intermediate models to be synthesized in seconds by weight copying, amortizing the cost to that of training a single student. For example, in Qwen3-4B experiments, producing 10 intermediate sizes via boomerang distillation yields ≈90% savings in both distillation FLOPs and wall-clock time relative to training 10 separate models. A plausible implication is a substantial drop in both the monetary and carbon footprint of LLM deployment across memory-constrained and scaling-sensitive environments.

6. Practical Considerations and Scope

Boomerang distillation is enabled by architectures in which subblocks of a deep transformer can be cleanly swapped between student and teacher. The alignment regularization is crucial to avoid degradation at swap boundaries. The technique provides fine-grained model family construction with negligible marginal cost for each size. A limitation is the requirement for careful initial block construction and alignment; failure in these stages can yield non-monotonicity or accuracy dips during interpolation. The method is agnostic to student size and can be instantiated for arbitrary depth reductions, provided architectural constraints and compatibility are met (Kangaslahti et al., 6 Oct 2025). The authors provide open-source code and models, further facilitating adoption and reproducibility.

7. Broader Impact and Extensions

Boomerang distillation addresses an important challenge in scalable model deployment by obviating the need for training model families or severely pruning large networks. The approach yields new opportunities for dynamic model adaptation and resource-efficient inference across heterogeneous compute environments. A plausible implication is simplified continuous scaling to match hardware or latency budgets without retraining. Further research may explore its applicability to non-transformer architectures and the boundaries of layer-wise compatibility for more diverse model designs.