Understanding Layer Patching in Model Size Interpolation

Abstract: Zero-shot model size interpolation aims to create new models of intermediate target sizes by combining existing models without additional training. Recent work on boomerang distillation [Kangaslahti et al., 2026] shows that a student LLM distilled from a larger teacher can be expanded by iteratively patching its layers, replacing student layers with contiguous blocks of teacher layers to obtain models whose size and performance interpolate between the student and the teacher. In this work, we provide the first systematic study of student-layer selection for model size interpolation. We cast finding the optimal layer subset for each model size as an optimization problem and prove it can be viewed as a shortest-path problem in a certain acyclic graph. In experiments, we show that patching strongly shapes interpolation behavior, with effects that vary substantially across model families. We find that simple sequential strategies--patching either from the first layer to the last or from the last to the first--often achieve surprisingly strong performance in practice. We further introduce KLPatch, a greedy patching algorithm based on KL divergence, which often improves over last-to-first patching and approximately solves the optimization problem. Together, our results provide a principled understanding of how layer patching affects model size interpolation and offer practical guidance for constructing near-optimal interpolated models.

• The paper demonstrates that layer patching order critically influences interpolation quality, directly impacting metrics like log-perplexity.
• It introduces the KLPatch algorithm, which leverages greedy KL divergence minimization to efficiently approximate optimal patching orders.
• Empirical results on models such as DistilBERT and DistilGPT2 show that strategic patching outperforms naive random and sequential approaches.

Systematic Analysis of Layer Patching in Model Size Interpolation

Introduction

Zero-shot model size interpolation is an emergent technique for deriving intermediate-sized LLMs by combining pre-existing models without further training. The foundational procedure, boomerang distillation, patches a distilled student by replacing its layers with corresponding teacher blocks, yielding interpolated models that traverse a continuous performance spectrum. This paper presents a comprehensive formal, empirical, and algorithmic study of student patching order as a critical parameter in interpolation, demonstrating its substantial influence on resulting performance, challenging established scaling laws, and proposing a principled optimization framework.

Formalization and Problem Structure

The patching process is framed as a combinatorial optimization over subset selection (which layers to patch for each model size) and ordering permutations (in what sequence to patch layers). The authors prove that both approaches can be algorithmically collapsed via greedy sequential construction. More significantly, the permutation formulation is mapped to a shortest-path problem on a KL-weighted Boolean lattice, formally connecting interpolation quality to path length in this acyclic graph.

Figure 1: Illustration of boomerang distillation and patching order trajectories, where patching sequence determines intermediate model performance.

Optimization is carried out over evaluation metrics such as area under perplexity interpolation curve (AUPIC), which aggregates performance across model sizes. The authors rigorously demonstrate that, under equidistant parameter increments, the shortest KL path between student and teacher yields the optimal patching order in log-perplexity.

Empirical Interpolation Landscape

The paper conducts exhaustive sweeps over all patching orders for DistilBERT and DistilGPT2 (6 layers each, permitting enumeration of all 720 orderings). Results reveal that patching order strongly determines interpolation trajectories, with last-to-first patching attaining global optimum for DistilBERT and near-optimality for DistilGPT2. Conversely, naive random patching may degrade performance, contradicting monotonic scaling laws. The landscape is locally smooth: orderings near the optimum (measured via permutation footrule distance) maintain near-optimal AUPIC, enabling efficient local search.

Figure 2: Performance curves across all patching order permutations for DistilBERT and DistilGPT2; optimal orderings yield near-linear perplexity interpolation, while random orders may underperform.

In higher parameter regimes (Qwen3-4B, Qwen3-8B, Pythia-6.9B), the design space expands combinatorially. By subsampling 200 patching orders per model, the authors show random orderings yield a narrow interquartile range, but optimal trajectories retain significant performance gap. Sequential patching remains competitive—but not optimal—implicating patching order as a primary lever for interpolation quality.

Figure 3: Downstream accuracy interpolation curves for large LLMs across sampled patching orders; sequential baselines do not always match best-sampled ordering.

KL-Guided Patching Algorithm (KLPatch)

To circumvent intractable enumeration, the authors derive KLPatch: a greedy $O(N^2)$ algorithm that iteratively patches the student layer whose replacement minimizes KL divergence to teacher output. This efficiently parameterizes the patching trajectory as a path in the KL-weighted interpolation graph, approximating the global optimum. Empirical evaluation demonstrates that KLPatch matches or outperforms sequential baselines and nearly matches the best of sampled orderings on both downstream classification/generation and validation perplexity.

Figure 4: KLPatch interpolation trajectories versus sequential and random patching for Qwen, Pythia; KLPatch tracks or outperforms baselines.

KLPatch performance is robust across calibration set choices, metrics, and layer initialization strategies, although certain family-specific initializations (e.g., Llama-3.2-3B, where layer one must be patched first) require minor adjustments for recovery of optimal interpolation.

Theoretical and Practical Implications

Formally, this analysis exposes the non-triviality of scaling laws under patching—model size alone is an insufficient predictor. Layer selection and order exert strong influence, and optimality is attainable only via careful combinatorial optimization. Practically, this enables compute-efficient construction of fine-grained model families, allowing practitioners to interpolate between student and teacher without retraining.

The utilization of KL divergence as a dense, task-agnostic signal ensures algorithmic generality, and empirical correlation with downstream metrics supports its use as a proxy criterion. The optimization landscape's local structure suggests potential for reinforcement learning or global search variants, further amortizing compute.

Figures Corroborating Key Results

• Figure 5: KL divergence on Wikitext correlates strongly with data perplexity, validating KL as a surrogate metric.
• Figure 6: KL path length versus AUPIC demonstrates alignment between graph-theoretic optimality and empirical interpolation quality.
• Figure 7: Visual depiction that KLPatch finds patching orders proximate to global optimum across permutation/trajectory space.

Figure 5: Strong correlation of KL divergence between interpolated models and the teacher and downstream perplexity in DistilBERT and DistilGPT2.

Figure 6: KL path length correlates robustly with AUPIC, supporting shortest-path formulation for optimal patching.

Figure 7: KLPatch recovers patching orders that closely follow the best possible interpolation trajectory.

Conclusions

Layer patching in zero-shot model interpolation is a decisive factor for performance, not a minor detail. The authors establish a principled optimization-theoretic view, confirm its empirical reality, and provide an efficient greedy algorithm. KLPatch reduces computational burden from intractable enumeration to practical scale, making fine-grained LLM interpolation feasible across model families. The results invite future investigation into global path search, amortized criteria, and adaptive methods for trajectory construction—advancing the theoretical and practical toolkit for scalable model adaptation.