Dual-Model Weight Selection

• The paper introduces dual-model weight selection as a novel framework that blends two model-derived weight sources to optimize downstream criteria like RMSE and validation accuracy.
• Dual-model weight selection is defined as combining weights via convex mixtures, direct averaging, or reversible basis constructions to leverage complementary model strengths.
• Practical implementations demonstrate actionable gains, such as a 5% RMSE reduction in regression tasks and improved ROC-AUC in drug activity and medical imaging benchmarks.

Dual-model weight selection denotes a class of procedures in which two model-derived weight sources are combined, selected, or reconstructed to optimize a downstream criterion. In current arXiv usage, the phrase encompasses convex mixtures of two regression predictors, direct averaging of two neural networks, data-free interpolation between two expert models, complementary partitioning of a teacher’s weights into two students, and reversible basis constructions for two low-rank updates (Echtenbruck et al., 2022, Rojas et al., 2024, Chaves et al., 15 Oct 2025, Tsutsumi et al., 28 Aug 2025, Alipour et al., 15 Oct 2025). The common structure is an optimization over a low-dimensional selection variable—typically a scalar weight, a model pair, a basis, or a pair of complementary masks—while the performance criterion ranges from RMSE and validation accuracy to exact reconstruction and data-free robustness. This suggests that dual-model weight selection is better understood as a family of two-source parameter-allocation problems than as a single canonical algorithm.

1. Scope of the term and representative formulations

The literature uses closely related language for several technically distinct constructions. In each case, two models, two updates, or two complementary weight sets are the primitive objects, but the selected quantity differs. The table summarizes representative formulations (Echtenbruck et al., 2022, Rojas et al., 2024, Chaves et al., 15 Oct 2025, Tsutsumi et al., 28 Aug 2025, Alipour et al., 15 Oct 2025, Wang, 15 Mar 2025).

Setting	Selected object	Representative formulation
Two-regressor ensemble	Convex scalar weight	$$\hat f_w(\mathbf{x}) = w\,\hat f_1(\mathbf{x}) + (1-w)\,\hat f_2(\mathbf{x})$$
Two-network weight-ensemble	Model pair $(i,j)$	$w_{\mathrm{avg}} := \tfrac12(w_i+w_j)$
Data-free model merge	Global scaling $\lambda$	$$\theta_{\mathrm{merged}}(\lambda) = (1-\lambda)\,\theta_A + \lambda\,\theta_B$$
Medical transfer initialization	Complementary masks	$$W_l^S \leftarrow M_l^S \odot \bar W_l^T,\; W_l^{S'} \leftarrow M_l^{S'} \odot \bar W_l^T$$
Reversible low-rank merge	Basis and coefficients	$x_i \approx W c_i^T$
Dual-weight learning	Excitatory/inhibitory components	$W = W_1 - W_2$

A central distinction is whether the procedure produces a single merged object or retains the ability to recover two specialized objects. Convex ensembling and direct weight averaging collapse two sources into one predictor or one parameter vector. By contrast, reversible model merging explicitly constructs a compact basis from which the original task-specific models can be recovered by linear combination (Alipour et al., 15 Oct 2025). Complementary teacher-weight partitioning occupies a third category: two lightweight models are initialized from disjoint subsets of a larger pretrained model, after which one model is trained and the other is maintained by EMA in a self-knowledge-distillation pipeline (Tsutsumi et al., 28 Aug 2025).

This diversity of usage matters because “selection” can mean at least four different operations: choosing a scalar coefficient, choosing a second model to average with an anchor model, choosing a basis dimension or subspace, or choosing complementary subsets of pretrained weights. Confusion between these regimes is a common source of category error.

2. Exact convex selection for two regression models

In the most classical formulation, dual-model weight selection is the problem of finding a convex mixture of two base regressors that minimizes mean squared error. Given training data $\{(\mathbf{x}_i,y_i)\}_{i=1}^n$, two regressors $\hat f_1,\hat f_2$, prediction vectors $(i,j)$0 and $(i,j)$1 with $(i,j)$2 and $(i,j)$3, and response vector $(i,j)$4, the objective is

$(i,j)$5

Equivalently, with

$(i,j)$6

the loss reduces to the univariate quadratic

$(i,j)$7

Differentiation yields the unconstrained minimizer

$(i,j)$8

and the constrained solution is

$(i,j)$9

Because the leading coefficient is $w_{\mathrm{avg}} := \tfrac12(w_i+w_j)$0 and the feasible set $w_{\mathrm{avg}} := \tfrac12(w_i+w_j)$1 is convex, the problem is a convex program with a unique global minimizer (Echtenbruck et al., 2022).

For exactly two models, no general QP solver is required. A single pass over the data is sufficient to accumulate $w_{\mathrm{avg}} := \tfrac12(w_i+w_j)$2 and $w_{\mathrm{avg}} := \tfrac12(w_i+w_j)$3, leading to time complexity $w_{\mathrm{avg}} := \tfrac12(w_i+w_j)$4 and memory $w_{\mathrm{avg}} := \tfrac12(w_i+w_j)$5 extra. The same framework extends to weighted data points by replacing

$w_{\mathrm{avg}} := \tfrac12(w_i+w_j)$6

with convexity preserved. If desired, an $w_{\mathrm{avg}} := \tfrac12(w_i+w_j)$7 penalty $w_{\mathrm{avg}} := \tfrac12(w_i+w_j)$8 shifts the solution to

$w_{\mathrm{avg}} := \tfrac12(w_i+w_j)$9

again clipped to $\lambda$0 (Echtenbruck et al., 2022).

Empirically, the paper reports that on a 4-D “MSG” test function with $\lambda$1 Latin-Hypercube points and leave-one-out CV, the best single Gaussian-kernel Kriging model achieved CV-RMSE $\lambda$2, the spline-kernel model achieved $\lambda$3, and the dual-model QP ensemble achieved $\lambda$4, a reduction of approximately $\lambda$5 versus the best single model. On the CHEMBL4159 drug-activity dataset ($\lambda$6), combining a random-forest and a LASSO regression increased ROC-AUC from $\lambda$7 for the random forest alone to $\lambda$8 for the two-model ensemble in stratified CV (Echtenbruck et al., 2022).

3. Direct weight averaging, diversity, and data-free interpolation

A second major usage arises in neural network weight-ensembling. Here one starts from a pool of pre-trained or fine-tuned networks of identical architecture,

$\lambda$9

and forms a dual-model weight-ensemble

$$\theta_{\mathrm{merged}}(\lambda) = (1-\lambda)\,\theta_A + \lambda\,\theta_B$$0

Selection of the pair $$\theta_{\mathrm{merged}}(\lambda) = (1-\lambda)\,\theta_A + \lambda\,\theta_B$$1 can be posed as maximizing

$$\theta_{\mathrm{merged}}(\lambda) = (1-\lambda)\,\theta_A + \lambda\,\theta_B$$2

or simply maximizing $$\theta_{\mathrm{merged}}(\lambda) = (1-\lambda)\,\theta_A + \lambda\,\theta_B$$3 when only in-distribution labels are available. To guide pair selection, the paper defines functional diversity by the ratio-error metric

$$\theta_{\mathrm{merged}}(\lambda) = (1-\lambda)\,\theta_A + \lambda\,\theta_B$$4

where $$\theta_{\mathrm{merged}}(\lambda) = (1-\lambda)\,\theta_A + \lambda\,\theta_B$$5 is the number of examples both models get wrong and $$\theta_{\mathrm{merged}}(\lambda) = (1-\lambda)\,\theta_A + \lambda\,\theta_B$$6 is the number of examples exactly one of them gets wrong, and weight-space diversity by

$$\theta_{\mathrm{merged}}(\lambda) = (1-\lambda)\,\theta_A + \lambda\,\theta_B$$7

For two-model soups, the “greedy” and “greedier” procedures coincide; a ranked-diversity heuristic first sorts candidates by descending $$\theta_{\mathrm{merged}}(\lambda) = (1-\lambda)\,\theta_A + \lambda\,\theta_B$$8 or $$\theta_{\mathrm{merged}}(\lambda) = (1-\lambda)\,\theta_A + \lambda\,\theta_B$$9 and evaluates them in that order. In OfficeHome experiments with ResNet-50 and 40 fine-tuned models per trial across 10 trials, greedier’s first addition yielded approximately $$W_l^S \leftarrow M_l^S \odot \bar W_l^T,\; W_l^{S'} \leftarrow M_l^{S'} \odot \bar W_l^T$$0–$$W_l^S \leftarrow M_l^S \odot \bar W_l^T,\; W_l^{S'} \leftarrow M_l^{S'} \odot \bar W_l^T$$1 ID accuracy gain over greedy’s choice and up to approximately $$W_l^S \leftarrow M_l^S \odot \bar W_l^T,\; W_l^{S'} \leftarrow M_l^{S'} \odot \bar W_l^T$$2–$$W_l^S \leftarrow M_l^S \odot \bar W_l^T,\; W_l^{S'} \leftarrow M_l^{S'} \odot \bar W_l^T$$3 OOD gain. Euclidean-ranked behaved similarly to ratio-error-ranked, and the authors conclude that high diversity enhances weight-ensembling while qualifying the extent to which diversity alone improves accuracy (Rojas et al., 2024).

A related but distinct problem appears in data-free model merging. Given two fine-tuned expert networks $$W_l^S \leftarrow M_l^S \odot \bar W_l^T,\; W_l^{S'} \leftarrow M_l^{S'} \odot \bar W_l^T$$4 and $$W_l^S \leftarrow M_l^S \odot \bar W_l^T,\; W_l^{S'} \leftarrow M_l^{S'} \odot \bar W_l^T$$5 with parameters $$W_l^S \leftarrow M_l^S \odot \bar W_l^T,\; W_l^{S'} \leftarrow M_l^{S'} \odot \bar W_l^T$$6 and $$W_l^S \leftarrow M_l^S \odot \bar W_l^T,\; W_l^{S'} \leftarrow M_l^{S'} \odot \bar W_l^T$$7, a standard linear merge introduces a global scaling parameter $$W_l^S \leftarrow M_l^S \odot \bar W_l^T,\; W_l^{S'} \leftarrow M_l^{S'} \odot \bar W_l^T$$8:

$$W_l^S \leftarrow M_l^S \odot \bar W_l^T,\; W_l^{S'} \leftarrow M_l^{S'} \odot \bar W_l^T$$9

Weight Weaving replaces one-shot selection of a single $x_i \approx W c_i^T$0 with pooling over a search grid $x_i \approx W c_i^T$1. For each $x_i \approx W c_i^T$2, one forms a candidate merge and then applies a pooling function to the resulting set of weight vectors. The paper gives three exemplar pooling operators: arithmetic mean, random uniform per-parameter selection, and max-magnitude pooling (MagMax). In the reported three-scenario benchmarks, Weight Weaving improved several state-of-the-art merges, including Breadcrumbs from $x_i \approx W c_i^T$3 to $x_i \approx W c_i^T$4, MagMax from $x_i \approx W c_i^T$5 to $x_i \approx W c_i^T$6, TIES from $x_i \approx W c_i^T$7 to $x_i \approx W c_i^T$8, PCB from $x_i \approx W c_i^T$9 to $W = W_1 - W_2$0, TSV from $W = W_1 - W_2$1 to $W = W_1 - W_2$2, and ISO-C from $W = W_1 - W_2$3 to $W = W_1 - W_2$4. The recommended default is $W = W_1 - W_2$5 in steps of $W = W_1 - W_2$6, extending to $W = W_1 - W_2$7 for methods such as PCB; arithmetic mean is the default pooling choice, and $W = W_1 - W_2$8–$W = W_1 - W_2$9 is reported as sufficient (Chaves et al., 15 Oct 2025).

These two lines of work share the same outer problem—how to exploit two pretrained models without additional inference cost at runtime—but they optimize different objects. Weight-ensembling selects a partner model, whereas Weight Weaving marginalizes over interpolation coefficients.

4. Complementary teacher-weight partitioning and self-knowledge distillation

In medical image classification, dual-model weight selection has been used as an initialization strategy for lightweight students derived from a larger pretrained teacher. Let the teacher be

$\{(\mathbf{x}_i,y_i)\}_{i=1}^n$0

Two smaller student models $\{(\mathbf{x}_i,y_i)\}_{i=1}^n$1 and $\{(\mathbf{x}_i,y_i)\}_{i=1}^n$2 with identical architectures but fewer channels are initialized from disjoint subsets of the mapped teacher tensor $\{(\mathbf{x}_i,y_i)\}_{i=1}^n$3. For each layer $\{(\mathbf{x}_i,y_i)\}_{i=1}^n$4, index sets

$\{(\mathbf{x}_i,y_i)\}_{i=1}^n$5

define binary masks $\{(\mathbf{x}_i,y_i)\}_{i=1}^n$6 and $\{(\mathbf{x}_i,y_i)\}_{i=1}^n$7, and the initialization is

$\{(\mathbf{x}_i,y_i)\}_{i=1}^n$8

By construction, every weight of the teacher is used exactly once across $\{(\mathbf{x}_i,y_i)\}_{i=1}^n$9 and $\hat f_1,\hat f_2$0. In the reported experiments, the index sets were chosen by a uniform-slice strategy, and no explicit loss is introduced at the selection stage beyond the structural coverage constraint $\hat f_1,\hat f_2$1 (Tsutsumi et al., 28 Aug 2025).

After initialization, the method freezes $\hat f_1,\hat f_2$2 as an auxiliary model updated only by EMA and trains $\hat f_1,\hat f_2$3 using classification and self-knowledge-distillation losses. With logits $\hat f_1,\hat f_2$4, temperature-scaled softmax outputs are

$\hat f_1,\hat f_2$5

The distillation loss is

$\hat f_1,\hat f_2$6

the classification loss is standard cross-entropy,

$\hat f_1,\hat f_2$7

and the total loss is

$\hat f_1,\hat f_2$8

The main student is updated by SGD or Adam, while the auxiliary model follows

$\hat f_1,\hat f_2$9

with $(i,j)$00 and stop-gradient on the auxiliary branch (Tsutsumi et al., 28 Aug 2025).

The paper evaluates the approach on chest X-ray images, lung computed tomography scans, and brain magnetic resonance imaging scans. The reported datasets are Chest X-ray with $(i,j)$01 images and $(i,j)$02 classes, Lung CT with $(i,j)$03 images and $(i,j)$04 classes, and Brain MRI with $(i,j)$05 images and $(i,j)$06 classes. Teacher-to-student configurations include ViT-S to ViT-T and ConvNeXt-T to ConvNeXt-F. With $(i,j)$07 training data, reported test-accuracy gains include Chest X-ray with ViT-T from $(i,j)$08 for CM1 to $(i,j)$09 for the proposed method, Chest X-ray with ConvNeXt-F from $(i,j)$10 to $(i,j)$11, Lung CT with ViT-T from $(i,j)$12 to $(i,j)$13, and Brain MRI with ConvNeXt-F from $(i,j)$14 to $(i,j)$15. The reported overhead is approximately $(i,j)$16–$(i,j)$17 additional GPU memory and approximately $(i,j)$18 additional training time (Tsutsumi et al., 28 Aug 2025).

5. Bi-level and reversible formulations

Dual-model weight selection also appears in formulations where one model learns how to weight inputs for another model. In “LLM Data Selection and Utilization via Dynamic Bi-level Optimization,” the two learned objects are the LLM parameters $(i,j)$19 and the Data Weighting Model parameters $(i,j)$20. The lower-level objective is

$(i,j)$21

and the upper-level objective is

$(i,j)$22

The DWM takes a mini-batch of $(i,j)$23 token sequences, embeds each sequence using a frozen or shallow-finetuned LM encoder, applies one self-attention block over the batch embedding matrix $(i,j)$24, and maps the resulting rows to raw weights $(i,j)$25 via two feed-forward layers. A softmax or ReLU normalization produces non-negative weights summing to $(i,j)$26. Training alternates between updating $(i,j)$27 with $(i,j)$28 frozen and updating $(i,j)$29 with $(i,j)$30 frozen, over $(i,j)$31 stages in the reported experiments. On a $(i,j)$32M model trained on $(i,j)$33B tokens, zero-shot average accuracy rose from $(i,j)$34 to $(i,j)$35 and two-shot average accuracy from $(i,j)$36 to $(i,j)$37 for RANDOM versus RANDOM + DWM. The learned weighting model also transferred from RANDOM to DSIR and QuRating, and on a $(i,j)$38B model imposed only approximately $(i,j)$39 extra FLOPs because large-model training used only forward passes through $(i,j)$40 (Yu et al., 22 Jul 2025).

A different generalization is reversible merging for low-rank weights. Let two fine-tuned low-rank updates be $(i,j)$41 and $(i,j)$42, with $(i,j)$43 and $(i,j)$44. For one row or column position, define task vectors $(i,j)$45 and stack them into

$(i,j)$46

Rather than storing both vectors separately, Reversible Model Merging seeks a basis $(i,j)$47 and coefficients $(i,j)$48 minimizing

$(i,j)$49

For the two-model case, choosing $(i,j)$50 allows exact reconstruction. The optimal basis is given by the top-$(i,j)$51 right singular vectors of $(i,j)$52, equivalently the top-$(i,j)$53 eigenvectors of $(i,j)$54, and the coefficients are orthogonal projections

$(i,j)$55

Because $(i,j)$56, one has $(i,j)$57 exactly. The reported computational cost is $(i,j)$58 per row or column position and $(i,j)$59 per layer, with storage $(i,j)$60 instead of $(i,j)$61. The paper’s central claim is that exact reconstruction implies no drop in task performance because the recovered low-rank updates coincide with the originals (Alipour et al., 15 Oct 2025).

Taken together, these works broaden weight selection beyond static scalar interpolation. One line learns weights dynamically by validation-aligned bi-level optimization; another replaces irreversible collapse with basis selection that preserves recoverability.

6. Limitations, misconceptions, and related directions

Several limitations recur across the literature. In the convex two-regressor setting, dual-model mixing cannot capture higher-order interactions beyond two predictors, and if $(i,j)$62 then $(i,j)$63 can become very small, leading to numerical instability; the paper notes that one may add a small ridge term $(i,j)$64 in practice (Echtenbruck et al., 2022). In neural weight-ensembling, high diversity is beneficial but not sufficient by itself to guarantee the best pair, and ranked-diversity often underperforms greedier even when it matches or slightly outperforms greedy on OOD evaluation (Rojas et al., 2024). In reversible merging, the argument is explicitly not that one merged model can dominate all specialized models; rather, the paper states that no merged model can consistently outperform one specialized for its task, motivating recoverability rather than unconditional collapse (Alipour et al., 15 Oct 2025).

A related misconception is that “weight selection” always refers to model coefficients. Earlier statistical work on fractionally-supervised classification instead used a single weight parameter $(i,j)$65 to balance labelled and unlabelled likelihood terms,

$(i,j)$66

and proposed selecting

$(i,j)$67

The same paper reports that BIC and ICL are monotone in $(i,j)$68 and pick a boundary, whereas $(i,j)$69 is the most stable of the criteria examined (Gallaugher et al., 2017). At the opposite end of the design space, the dual-weight learning framework of Wang and Shimazaki decomposes each neuron’s parameters into nonnegative $(i,j)$70 and $(i,j)$71 with effective weight $(i,j)$72 and reports that inference complexity remains the same as in the standard $(i,j)$73 setup, because one may collapse the representation after training (Wang, 15 Mar 2025).

These adjacent formulations indicate that the phrase “dual-model weight selection” sits within a broader methodological landscape of two-source allocation, decomposition, and interpolation. What unifies the area is not a single algorithmic template but the recurring attempt to exploit complementarity between two parameterized objects while controlling optimization cost, inference cost, or reconstruction error.