---
title: Dual-Model Weight Selection
url: https://www.emergentmind.com/topics/dual-model-weight-selection
type: topic
---

# Dual-Model Weight Selection

Dual-model weight selection denotes a class of procedures in which two model-derived weight sources are combined, selected, or reconstructed to optimize a downstream criterion. In current arXiv usage, the phrase encompasses convex mixtures of two regression predictors, direct averaging of two neural networks, data-free interpolation between two expert models, complementary partitioning of a teacher’s weights into two students, and reversible basis constructions for two low-rank updates [2206.11263][2409.02347][2510.13921][2508.20461][2510.14163]. The common structure is an optimization over a low-dimensional selection variable—typically a scalar weight, a model pair, a basis, or a pair of complementary masks—while the performance criterion ranges from RMSE and validation accuracy to exact reconstruction and data-free robustness. This suggests that dual-model weight selection is better understood as a family of two-source parameter-allocation problems than as a single canonical algorithm.

## 1. Scope of the term and representative formulations

The literature uses closely related language for several technically distinct constructions. In each case, two models, two updates, or two complementary weight sets are the primitive objects, but the selected quantity differs. The table summarizes representative formulations [2206.11263][2409.02347][2510.13921][2508.20461][2510.14163][2503.11965].

| Setting | Selected object | Representative formulation |
| --- | --- | --- |
| Two-regressor ensemble | Convex scalar weight | $\hat f_w(\mathbf{x}) = w\,\hat f_1(\mathbf{x}) + (1-w)\,\hat f_2(\mathbf{x})$ |
| Two-network weight-ensemble | Model pair $(i,j)$ | $w_{\mathrm{avg}} := \tfrac12(w_i+w_j)$ |
| Data-free model merge | Global scaling $\lambda$ | $\theta_{\mathrm{merged}}(\lambda) = (1-\lambda)\,\theta_A + \lambda\,\theta_B$ |
| Medical transfer initialization | Complementary masks | $W_l^S \leftarrow M_l^S \odot \bar W_l^T,\; W_l^{S'} \leftarrow M_l^{S'} \odot \bar W_l^T$ |
| Reversible low-rank merge | Basis and coefficients | $x_i \approx W c_i^T$ |
| Dual-weight learning | Excitatory/inhibitory components | $W = W_1 - W_2$ |

A central distinction is whether the procedure produces a **single merged object** or retains the ability to recover two specialized objects. Convex ensembling and direct weight averaging collapse two sources into one predictor or one parameter vector. By contrast, reversible model merging explicitly constructs a compact basis from which the original task-specific models can be recovered by linear combination [2510.14163]. Complementary teacher-weight partitioning occupies a third category: two lightweight models are initialized from disjoint subsets of a larger pretrained model, after which one model is trained and the other is maintained by EMA in a self-knowledge-distillation pipeline [2508.20461].

This diversity of usage matters because “selection” can mean at least four different operations: choosing a scalar coefficient, choosing a second model to average with an anchor model, choosing a basis dimension or subspace, or choosing complementary subsets of pretrained weights. Confusion between these regimes is a common source of category error.

## 2. Exact convex selection for two regression models

In the most classical formulation, dual-model weight selection is the problem of finding a convex mixture of two base regressors that minimizes mean squared error. Given training data $\{(\mathbf{x}_i,y_i)\}_{i=1}^n$, two regressors $\hat f_1,\hat f_2$, prediction vectors $\mathbf{a}$ and $\mathbf{b}$ with $a_i=\hat f_1(\mathbf{x}_i)$ and $b_i=\hat f_2(\mathbf{x}_i)$, and response vector $\mathbf{y}$, the objective is
$$
\min_{0\le w\le 1}\;\frac1n\sum_{i=1}^n\bigl(y_i-(w\,a_i+(1-w)b_i)\bigr)^2.
$$
Equivalently, with
$$
\mathbf{d}=\mathbf{a}-\mathbf{b},\qquad \mathbf{c}=\mathbf{b}-\mathbf{y},
$$
the loss reduces to the univariate quadratic
$$
L(w)=\frac1n\|w\mathbf{d}+\mathbf{c}\|_2^2
= \frac1n\Bigl(w^2\,\mathbf{d}^T\mathbf{d}+2w\,\mathbf{c}^T\mathbf{d}+\mathbf{c}^T\mathbf{c}\Bigr).
$$
Differentiation yields the unconstrained minimizer
$$
w^{\rm raw}
= -\,\frac{\mathbf{c}^T\mathbf{d}}{\mathbf{d}^T\mathbf{d}}
= \frac{(\mathbf{y}-\mathbf{b})^T(\mathbf{a}-\mathbf{b})}{\|\mathbf{a}-\mathbf{b}\|^2},
$$
and the constrained solution is
$$
w^*=\min\{1,\max\{0,w^{\rm raw}\}\}.
$$
Because the leading coefficient is $\tfrac1n\,\mathbf{d}^T\mathbf{d}\ge 0$ and the feasible set $[0,1]$ is convex, the problem is a convex program with a unique global minimizer [2206.11263].

For exactly two models, no general QP solver is required. A single pass over the data is sufficient to accumulate $\sum d_i^2$ and $\sum c_i d_i$, leading to time complexity $O(n)$ and memory $O(1)$ extra. The same framework extends to weighted data points by replacing
$$
\mathbf{d}\leftarrow (\beta_i(a_i-b_i))_{i=1}^n,\qquad
\mathbf{c}\leftarrow (\beta_i(b_i-y_i))_{i=1}^n,
$$
with convexity preserved. If desired, an $\ell_2$ penalty $\lambda w^2$ shifts the solution to
$$
w^*= \bigl(-\mathbf{c}^T\mathbf{d}\bigr)\big/\bigl(\mathbf{d}^T\mathbf{d}+\lambda n\bigr),
$$
again clipped to $[0,1]$ [2206.11263].

Empirically, the paper reports that on a 4-D “MSG” test function with $n=160$ Latin-Hypercube points and leave-one-out CV, the best single Gaussian-kernel Kriging model achieved CV-RMSE $\approx 0.152$, the spline-kernel model achieved $\approx 0.160$, and the dual-model QP ensemble achieved $\approx 0.144$, a reduction of approximately $5\%$ versus the best single model. On the CHEMBL4159 drug-activity dataset ($n\approx 18{,}000$), combining a random-forest and a LASSO regression increased ROC-AUC from $\approx 0.84$ for the random forest alone to $\approx 0.88$ for the two-model ensemble in stratified CV [2206.11263].

## 3. Direct weight averaging, diversity, and data-free interpolation

A second major usage arises in neural network weight-ensembling. Here one starts from a pool of pre-trained or fine-tuned networks of identical architecture,
$$
W=\{w_1,\dots,w_k\},\qquad w\in\mathbb{R}^n,
$$
and forms a dual-model weight-ensemble
$$
w_{\mathrm{avg}} := \tfrac12(w_i+w_j).
$$
Selection of the pair $(i,j)$ can be posed as maximizing
$$
F(i,j) := \alpha\cdot A_{\mathrm{ID}}(w_{\mathrm{avg}})+(1-\alpha)\cdot A_{\mathrm{OOD}}(w_{\mathrm{avg}}),
$$
or simply maximizing $A_{\mathrm{ID}}(w_{\mathrm{avg}})$ when only in-distribution labels are available. To guide pair selection, the paper defines functional diversity by the ratio-error metric
$$
d_D(w_i,w_j):=\frac{N_{\mathrm{uns}}}{N_{\mathrm{sha}}},
$$
where $N_{\mathrm{sha}}$ is the number of examples both models get wrong and $N_{\mathrm{uns}}$ is the number of examples exactly one of them gets wrong, and weight-space diversity by
$$
d_E(w_i,w_j):=\|w_i-w_j\|_2^2.
$$
For two-model soups, the “greedy” and “greedier” procedures coincide; a ranked-diversity heuristic first sorts candidates by descending $d_D$ or $d_E$ and evaluates them in that order. In OfficeHome experiments with ResNet-50 and 40 fine-tuned models per trial across 10 trials, greedier’s first addition yielded approximately $1$–$2\%$ ID accuracy gain over greedy’s choice and up to approximately $3$–$4\%$ OOD gain. Euclidean-ranked behaved similarly to ratio-error-ranked, and the authors conclude that high diversity enhances weight-ensembling while qualifying the extent to which diversity alone improves accuracy [2409.02347].

A related but distinct problem appears in data-free model merging. Given two fine-tuned expert networks $A$ and $B$ with parameters $\theta_A$ and $\theta_B$, a standard linear merge introduces a global scaling parameter $\lambda$:
$$
\theta_{\mathrm{merged}}(\lambda)
= (1-\lambda)\,\theta_A+\lambda\,\theta_B.
$$
Weight Weaving replaces one-shot selection of a single $\lambda$ with pooling over a search grid $\Lambda=\{\lambda_1,\dots,\lambda_K\}$. For each $\lambda_i$, one forms a candidate merge and then applies a pooling function to the resulting set of weight vectors. The paper gives three exemplar pooling operators: arithmetic mean, random uniform per-parameter selection, and max-magnitude pooling (MagMax). In the reported three-scenario benchmarks, Weight Weaving improved several state-of-the-art merges, including Breadcrumbs from $52.17$ to $68.11$, MagMax from $60.14$ to $69.77$, TIES from $68.39$ to $71.21$, PCB from $71.41$ to $72.10$, TSV from $73.11$ to $74.01$, and ISO-C from $72.38$ to $73.78$. The recommended default is $\lambda\in[0.1,1.0]$ in steps of $0.1$, extending to $[0.1,2.5]$ for methods such as PCB; arithmetic mean is the default pooling choice, and $K\approx 10$–$20$ is reported as sufficient [2510.13921].

These two lines of work share the same outer problem—how to exploit two pretrained models without additional inference cost at runtime—but they optimize different objects. Weight-ensembling selects a partner model, whereas Weight Weaving marginalizes over interpolation coefficients.

## 4. Complementary teacher-weight partitioning and self-knowledge distillation

In medical image classification, dual-model weight selection has been used as an initialization strategy for lightweight students derived from a larger pretrained teacher. Let the teacher be
$$
\theta_T=\{W_1^T,W_2^T,\dots,W_L^T\}.
$$
Two smaller student models $S$ and $S'$ with identical architectures but fewer channels are initialized from disjoint subsets of the mapped teacher tensor $\bar W_l^T$. For each layer $l$, index sets
$$
I_l^S \subset \{1,\dots,\dim(\bar W_l^T)\},\qquad
I_l^{S'}=\{1,\dots,\dim(\bar W_l^T)\}\setminus I_l^S
$$
define binary masks $M_l^S$ and $M_l^{S'}=1-M_l^S$, and the initialization is
$$
W_l^S \leftarrow M_l^S\odot \bar W_l^T,\qquad
W_l^{S'} \leftarrow M_l^{S'}\odot \bar W_l^T.
$$
By construction, every weight of the teacher is used exactly once across $S$ and $S'$. In the reported experiments, the index sets were chosen by a uniform-slice strategy, and no explicit loss is introduced at the selection stage beyond the structural coverage constraint $M_l^S+M_l^{S'}=1$ [2508.20461].

After initialization, the method freezes $S'$ as an auxiliary model updated only by EMA and trains $S$ using classification and self-knowledge-distillation losses. With logits $z^S(x),z^{S'}(x)\in\mathbb{R}^K$, temperature-scaled softmax outputs are
$$
p_k^S(x)=\frac{\exp(z_k^S/\tau)}{\sum_{j=1}^K \exp(z_j^S/\tau)},\qquad
p_k^{S'}(x)=\frac{\exp(z_k^{S'}/\tau)}{\sum_{j=1}^K \exp(z_j^{S'}/\tau)}.
$$
The distillation loss is
$$
L_{\mathrm{skd}}(x)=\mathrm{KL}\bigl(p^{S'}(x)\,\|\,p^S(x)\bigr),
$$
the classification loss is standard cross-entropy,
$$
L_{\mathrm{ce}}(x)= -\sum_{k=1}^K y_k\log p_k^S(x),
$$
and the total loss is
$$
L_{\mathrm{total}}(x)=\alpha\,L_{\mathrm{ce}}(x)+(1-\alpha)\tau^2 L_{\mathrm{skd}}(x).
$$
The main student is updated by SGD or Adam, while the auxiliary model follows
$$
\theta_{S'}\leftarrow \beta\,\theta_{S'}+(1-\beta)\,\theta_S,
$$
with $\beta\approx 0.9$ and stop-gradient on the auxiliary branch [2508.20461].

The paper evaluates the approach on chest X-ray images, lung computed tomography scans, and brain magnetic resonance imaging scans. The reported datasets are Chest X-ray with $21\,165$ images and $4$ classes, Lung CT with $17\,104$ images and $3$ classes, and Brain MRI with $7\,023$ images and $4$ classes. Teacher-to-student configurations include ViT-S to ViT-T and ConvNeXt-T to ConvNeXt-F. With $10\%$ training data, reported test-accuracy gains include Chest X-ray with ViT-T from $0.828$ for CM1 to $0.869$ for the proposed method, Chest X-ray with ConvNeXt-F from $0.850$ to $0.906$, Lung CT with ViT-T from $0.870$ to $0.889$, and Brain MRI with ConvNeXt-F from $0.837$ to $0.910$. The reported overhead is approximately $10$–$15\%$ additional GPU memory and approximately $20\%$ additional training time [2508.20461].

## 5. Bi-level and reversible formulations

Dual-model weight selection also appears in formulations where one model learns how to weight inputs for another model. In “LLM Data Selection and Utilization via Dynamic Bi-level Optimization,” the two learned objects are the LLM parameters $\theta$ and the Data Weighting Model parameters $\phi$. The lower-level objective is
$$
\theta^*(\phi)=\arg\min_{\theta} L_{\mathrm{train}}(\theta;\phi),\qquad
L_{\mathrm{train}}(\theta;\phi)=\sum_{i=1}^B w_i(\phi;x_{1\ldots B})\,\ell(\theta;x_i),
$$
and the upper-level objective is
$$
\phi^*=\arg\max_{\phi} R_{\mathrm{val}}(\theta^*(\phi)).
$$
The DWM takes a mini-batch of $B=8$ token sequences, embeds each sequence using a frozen or shallow-finetuned LM encoder, applies one self-attention block over the batch embedding matrix $E\in\mathbb{R}^{B\times d}$, and maps the resulting rows to raw weights $u_i$ via two feed-forward layers. A softmax or ReLU normalization produces non-negative weights summing to $1$. Training alternates between updating $\phi$ with $\theta$ frozen and updating $\theta$ with $\phi$ frozen, over $T=5$ stages in the reported experiments. On a $370$M model trained on $30$B tokens, zero-shot average accuracy rose from $42.4\%$ to $45.0\%$ and two-shot average accuracy from $45.1\%$ to $46.4\%$ for RANDOM versus RANDOM + DWM. The learned weighting model also transferred from RANDOM to DSIR and QuRating, and on a $1.3$B model imposed only approximately $9\%$ extra FLOPs because large-model training used only forward passes through $\phi$ [2507.16178].

A different generalization is reversible merging for low-rank weights. Let two fine-tuned low-rank updates be $\hat\Delta_1=A_1B_1$ and $\hat\Delta_2=A_2B_2$, with $A_i\in\mathbb{R}^{m\times r}$ and $B_i\in\mathbb{R}^{r\times d}$. For one row or column position, define task vectors $x_1,x_2\in\mathbb{R}^r$ and stack them into
$$
X=\begin{bmatrix}x_1^T\\ x_2^T\end{bmatrix}\in\mathbb{R}^{2\times r}.
$$
Rather than storing both vectors separately, Reversible Model Merging seeks a basis $W\in\mathbb{R}^{r\times p}$ and coefficients $C\in\mathbb{R}^{2\times p}$ minimizing
$$
\min_{W,C}\;\|X-CW^T\|_F^2
\quad\text{subject to}\quad W^TW=I_p.
$$
For the two-model case, choosing $p=2$ allows exact reconstruction. The optimal basis is given by the top-$2$ right singular vectors of $X$, equivalently the top-$2$ eigenvectors of $X^TX$, and the coefficients are orthogonal projections
$$
\alpha_i \equiv c_i = W^{*T}x_i.
$$
Because $\dim(\mathrm{span}\{x_1,x_2\})\le 2$, one has $x_i=W^*\alpha_i$ exactly. The reported computational cost is $O(r^2)$ per row or column position and $O((m+d)r^2)$ per layer, with storage $(m+d)(2p+r)$ instead of $(m+d)\cdot 2r$. The paper’s central claim is that exact reconstruction implies no drop in task performance because the recovered low-rank updates coincide with the originals [2510.14163].

Taken together, these works broaden weight selection beyond static scalar interpolation. One line learns weights dynamically by validation-aligned bi-level optimization; another replaces irreversible collapse with basis selection that preserves recoverability.

## 6. Limitations, misconceptions, and related directions

Several limitations recur across the literature. In the convex two-regressor setting, dual-model mixing cannot capture higher-order interactions beyond two predictors, and if $\mathbf{a}\approx\mathbf{b}$ then $\mathbf{d}^T\mathbf{d}$ can become very small, leading to numerical instability; the paper notes that one may add a small ridge term $\epsilon$ in practice [2206.11263]. In neural weight-ensembling, high diversity is beneficial but not sufficient by itself to guarantee the best pair, and ranked-diversity often underperforms greedier even when it matches or slightly outperforms greedy on OOD evaluation [2409.02347]. In reversible merging, the argument is explicitly not that one merged model can dominate all specialized models; rather, the paper states that no merged model can consistently outperform one specialized for its task, motivating recoverability rather than unconditional collapse [2510.14163].

A related misconception is that “weight selection” always refers to model coefficients. Earlier statistical work on fractionally-supervised classification instead used a single weight parameter $\alpha\in[0,1]$ to balance labelled and unlabelled likelihood terms,
$$
L_{\mathrm{FSC}}(\vartheta\mid D_o,\alpha)
=\bigl[L_{\mathrm{DA}}(\vartheta\mid D_L)\bigr]^\alpha
\bigl[L_{\mathrm{clust}}(\vartheta\mid D_U)\bigr]^{1-\alpha},
$$
and proposed selecting
$$
\alpha^*=\arg\min_{\alpha\in[0,1]}\det[W(\alpha)].
$$
The same paper reports that BIC and ICL are monotone in $\alpha$ and pick a boundary, whereas $\det W(\alpha)$ is the most stable of the criteria examined [1709.08258]. At the opposite end of the design space, the dual-weight learning framework of Wang and Shimazaki decomposes each neuron’s parameters into nonnegative $W_1$ and $W_2$ with effective weight $W=W_1-W_2$ and reports that inference complexity remains the same as in the standard $WX+\text{bias}$ setup, because one may collapse the representation after training [2503.11965].

These adjacent formulations indicate that the phrase “dual-model weight selection” sits within a broader methodological landscape of two-source allocation, decomposition, and interpolation. What unifies the area is not a single algorithmic template but the recurring attempt to exploit complementarity between two parameterized objects while controlling optimization cost, inference cost, or reconstruction error.

Source: https://www.emergentmind.com/topics/dual-model-weight-selection