---
title: Black-Box Model Merging
url: https://www.emergentmind.com/topics/black-box-model-merging-bmm
type: topic
---

# Black-Box Model Merging

Black-Box Model Merging (BMM) denotes a family of techniques for combining multiple models when the merger operates through externally accessible behavior, merge configurations, or checkpoint-level summaries rather than full joint retraining. Across the literature, the term is used in several technically distinct ways: as input-dependent mixing of black-box simulators, as inference-time coupling of a proprietary service with a local model, and as derivative-free optimization over merge configurations for large model repositories and large language models [2301.02296] [2006.13519] [2509.12951]. Taken together, these works suggest that BMM is best regarded as an umbrella concept rather than a single standardized algorithm.

## 1. Scope and terminological boundaries

In "Model Mixing Using Bayesian Additive Regression Trees" [2301.02296], Bayesian model mixing is a Bayesian mean-stacking model with input-dependent weights. The merger treats each simulator as a black box because it only uses the outputs \(\hat f_k(x_i)\) at the training inputs, never uses gradients of \(f_k\), and never accesses internal parameters of the simulators. The mixed predictor is
\[
\hat f_{\text{mix}}(x)=\sum_{k=1}^K w_k(x)\,\hat f_k(x).
\]

In "Black-box Adaptation of ASR for Accented Speech" [2006.13519], BMM is instantiated as black-box accent adaptation: a proprietary cloud ASR is available only through input-output access, while a local accent-tuned ASR is white-box. The merger is not a word-level ensemble; it is a fine-grained, frame-level coupling in which the service transcript guides frame-level inference in the local model.

In the LLM-merging literature, BMM is often framed as black-box optimization over merge configurations. "It’s Morphing Time: Unleashing the Potential of Multiple LLMs via Multi-objective Optimization" [2407.00487] formalizes merging as a multi-objective optimization problem over a configuration vector \(\boldsymbol{x}\). "Black-Box Optimization of Mixed Binary-Continuous Variables: Challenges and Opportunities in Evolutionary Model Merging" [2605.12326] formalizes Data Flow Space merging as a mixed binary–continuous black-box optimization problem. "Black-box Model Merging for Language-Model-as-a-Service with Massive Model Repositories" [2509.12951] defines BMM as fusing many closed-source LLMs, accessible only via APIs, into a single, stronger “virtual model.”

A neighboring line of work analyzes black-box operations without always using the term itself. "Rethinking Weight-Averaged Model-merging" [2411.09263] explicitly studies averaging on features and ensemble on logits/features, which are black-box operations because they only use outputs or feature activations, not weight access.

| Formulation | Representative papers | Accessible objects |
|---|---|---|
| Input-dependent output mixing | [2301.02296] | Simulator predictions \(\hat f_k(x)\) |
| Guided inference | [2006.13519] | Service transcript, confidences, local frame distributions |
| Merge-configuration search | [2407.00487], [2605.12326], [2509.12951], [2606.25761] | Validation metrics from candidate merges |

## 2. Core mathematical formulations

A canonical output-space formulation appears in [2301.02296]. Given observations
\[
Y_i = f_\dagger(x_i) + \epsilon_i,\quad \epsilon_i \stackrel{iid}{\sim} N(0,\sigma^2),
\]
and candidate simulators \(f_1(x),\dots,f_K(x)\), the paper models
\[
Y_i \mid \hat f(x_i), w(x_i),\sigma^2 \overset{ind}{\sim} N\Big(\hat f(x_i)^\top w(x_i),\,\sigma^2\Big),
\]
with \(\hat f(x_i)=(\hat f_1(x_i),\dots,\hat f_K(x_i))^\top\). Here, black-box merging means that the meta-model learns \(w_k(x)\) from data and simulator predictions only.

A different formulation appears in ASR. In [2006.13519], the local CTC model produces frame-level distributions
\[
\mathbf{P}=(P_1,\dots,P_T),\quad P_t(c)=p_{\mathcal{C}}(c\mid x_t),
\]
while the cloud service returns a transcript \(\mathbf{s}\) with word-level confidences. After Viterbi-like alignment of service characters to frames, FineMerge revises the local frame distribution by
\[
P_t^s(c) = (1 - \omega_t) P_t(c) + \omega_t \cdot \mathbf{1}[c = S_t],
\]
when the aligned service character is acoustically supported but not dominant. The merged transcript is then obtained by standard beam decoding with the local LM.

Configuration-space formulations dominate recent LLM work. In [2605.12326], DFS merging is formalized as
\[
\min_{\mathbf{x}, \mathbf{z}} \ f(\mathbf{x}, \mathbf{z})
\quad \text{subject to} \quad
\mathbf{x} \in \mathbb{R}^{n}, \ 
\mathbf{z} \in \{0,1\}^{m},
\]
where \(\mathbf{z}\) encodes layer selection and \(\mathbf{x}\) the scaling weights of selected layers. In [2407.00487], multi-task merging is written as
\[
\min_{\boldsymbol{x} \in \mathcal{X}} \; \boldsymbol{f}(\boldsymbol{x})
= \big(f_1(\boldsymbol{x}), f_2(\boldsymbol{x}), \ldots, f_M(\boldsymbol{x})\big)^T,
\]
with objectives derived from task performance. In [2509.12951], the LMaaS setting is abstracted as
\[
M^*(x) = \Phi\big(M_1(x), M_2(x), \dots, M_N(x); \theta\big),
\]
where only API queries are available.

A further optimizer-centric formulation appears in [2606.25761], where language-model merging is treated as optimization over merge weights
\[
\phi_{\mathrm{Merge}}
= \phi_{\mathrm{Base}}
+ \sum_i x_i \bigl(\phi_{\mathrm{FT},i} - \phi_{\mathrm{Base}}\bigr),
\]
with black-box optimizers operating on function evaluations of the merged model.

## 3. Mechanisms for combining models

One major BMM family uses localized weighting functions. In [2301.02296], the weight vector is modeled by a Bayesian Additive Regression Trees sum-of-trees representation,
\[
w(x_i) = \sum_{j=1}^m g(x_i, T_j, M_j),
\]
where each tree partitions input space and each terminal node contributes a vector in \(\mathbb{R}^K\). This yields nonparametric, input-dependent weights that can discover regions where one simulator should be up-weighted and another down-weighted. A notable design choice is that no explicit simplex constraint is imposed: the weights need not sum to \(1\) and need not be strictly non-negative, although priors are calibrated so individual weights tend to lie in \([0,1]\) and the sum tends to be near \(1\) where a convex combination is appropriate. The paper explicitly notes that, without constraints, interpretability of \(w_k(x)\) as probabilities diminishes; they are best seen as local scaling factors.

A second family uses structured guidance rather than direct averaging. FineMerge in [2006.13519] aligns the service transcript to local model frames, uses a threshold \(\psi\) to decide whether a service character is acoustically “heard,” and modulates the update strength by the confidence of the parent service word. This allows the merger to preserve local acoustic evidence when the service transcript is likely a language-model artifact, while still using the service output as a dynamic prior over alignments.

A third family operates on features, logits, or probabilities. [2411.09263] distinguishes weight averaging from black-box operations such as logit averaging,
\[
\ell_{\text{merged}}(x) = \frac{1}{K} \sum_{i=1}^K \ell_i(x),
\]
feature averaging,
\[
f_{\text{merged}}(x) = \frac{1}{K} \sum_{i=1}^K f_i(x),
\]
and probability averaging. In purely linear models, feature-level averaging is mathematically equivalent to weight-level averaging. In nonlinear networks, the equivalence breaks because weight averaging is “average then activate,” whereas feature averaging is “activate then average.”

Search-based BMM adds a further mechanism: the models themselves may remain unchanged while an optimizer tunes how they are combined. In [2509.12951], Evo-Merging uses a two-stage CMA-ES pipeline with sparsity-based denoising and sign-aware scaling. In [2606.25761], the same perspective is cast in a unified spherical black-box optimization framework in which different optimizers are distinguished by fitness aggregation and consensus scope.

## 4. Optimization-centric BMM

The multi-objective line treats merge design as a black-box search problem over expensive evaluations. MM-MO [2407.00487] uses qEHVI to optimize merging configurations defined over DARE+TIES controls. Hypervolume improvement is estimated under a surrogate posterior, and the search iterates by fitting a surrogate, selecting a batch of promising candidates, evaluating them on tasks such as C-EVAL and GSM8K, and updating the observed Pareto set. The paper reports that MM-MO reaches \(71.4\) on C-EVAL, \(66.56\) on GSM8K, and \(56.09\) on HumanEval, outperforming single models and handcrafted merges.

DFS merging introduces a mixed-variable structure. [2605.12326] emphasizes the conditional dependency between binary architecture variables and continuous scaling variables: if \(z_{i,j}=0\), then \(x_{i,j}\) is irrelevant. The structured approach first selects active layers via \(\mathbf{z}\) and then optimizes only the active scales \(\mathbf{x}_{A(\mathbf{z})}\). In the reported experiment, Structured DFS achieved \(26.7\%\) accuracy, Unstructured DFS achieved \(20.0\%\), and the effective search space was reduced by \(51.4\%\).

Evo-Merging [2509.12951] addresses massive repositories. Stage 1 optimizes per-model sparsity ratios \(\bm{\alpha}\) with uniform weights, while Stage 2 optimizes signed scaling weights \(\bm{\beta}\). The denoising objective is
\[
\mathcal{F}(\bm{\alpha}) = \mathcal{L}_{\text{CE}}\big(\mathcal{D}_{\text{val}}, \mathcal{M}_{\text{adapter}}(\bm{\alpha})\big) + \lambda_1 \|\bm{\alpha}\|_1,
\]
and the scaling objective is
\[
\mathcal{G}(\bm{\beta}) = \mathcal{L}_{\text{CE}}\big(\mathcal{D}_{\text{val}}, \mathcal{M}_{\text{adapter}}(\bm{\beta})\big) + \lambda_2 \|\bm{\beta}\|_1.
\]
A distinctive claim is that \(\beta_i\) can be positive or negative, so some models are added and others effectively subtracted. The paper also provides an error bound for asymmetric sparsification of LoRA \(A\) matrices,
\[
\|\Delta W - \Delta W'(\bm{\alpha})\|_F
\le \|B\|_F \cdot \|A - \mathbf{S}_{\bm{\alpha}}(A)\|_F.
\]

[2606.25761] generalizes several black-box optimizers through a master update,
\[
x_{t+1}^i = \mu_t x_t^i + \lambda_t m_t^i + \sigma_t\, s(x_t^i - m_t^i)\, \epsilon_t^i,
\]
with consensus terms determined by fitness aggregation and interaction kernels. Within this view, ES-OVI interpolates between flatness-preferring and sharpness-preferring search, while AdaPol and SchedPol combine OVI-like global search with cCBO-like multimodal refinement. The paper reports that the CBO-OVI hybrids achieve competitive results on language model merging under limited evaluation budgets.

## 5. Empirical regimes and application domains

Simulator mixing in nuclear and toy physics is one of the earliest explicit BMM settings in this set of works. In [2301.02296], Example 1a mixes \(f_s^{(2)}(x)\) and \(f_l^{(4)}(x)\): BART-BMM attains RMSE \(\approx 0.0053\), whereas HS yields RMSE \(\approx 1.9460\). In Example 1b, where both simulators overestimate the truth in the intermediate region, BART-BMM attains RMSE \(\approx 0.0057\) versus HS RMSE \(\approx 0.1141\), and the sum of weights drops below \(1\) in the problematic region. In a 2D trigonometric Taylor-expansion example, BART-BMM with \(30\) trees attains RMSE \(\approx 0.2575\).

Accent adaptation provides a structured-prediction instance of BMM. [2006.13519] reports, for Google US as service, Indian-accent WER/CER of \(18.45/10.65\) for FineMerge versus \(22.32/11.96\) for the service and \(27.99/16.98\) for the local model. For Australian accent, FineMerge attains \(16.90/9.33\) versus \(23.52/13.27\) for the service and \(24.41/14.55\) for the local model. For British accent, FineMerge attains WER \(16.47\) and CER \(9.79\). The paper states that the method achieves as much as \(28\%\) relative reduction in WER over both the local and service models.

Large-model merging is evaluated under both small and massive search spaces. MM-MO [2407.00487] outperforms task arithmetic, TIES, and default DARE+TIES on C-EVAL, GSM8K, and HumanEval. Evo-Merging [2509.12951] reports average out-of-domain Prec \(53.80\) and F1 \(52.13\), versus LoRaHub Prec \(42.72\) and F1 \(40.94\), while operating on more than \(100\) LoRA-based models. In in-domain multi-task merging across eight NER datasets, Evo-Merging reports F1 \(38.03\) versus LoRaHub F1 \(34.80\). The same paper shows that adding five noisy tasks raises Evo-Merging from F1 \(38.03\) to \(42.20\), whereas several baselines drop sharply.

Optimizer studies also report that the merging landscape can be multimodal. In [2606.25761], training and test accuracy over a 2D slice of merge-parameter space show multiple local maxima, and the paper reports that cCBO is beneficial under good initialization, OVI performs better under bad initialization, and SchedPol and AdaPol are competitive in both scenarios.

## 6. Misconceptions, boundary cases, and open problems

A recurrent misconception is to identify BMM with ordinary weight averaging. [2411.09263] makes the distinction explicit: weight-averaged soups require white-box parameter access, whereas logit averaging, feature averaging, and probability averaging are black-box operations. The paper further reports that logit ensembles almost always give the best accuracy across ResNet, DenseNet, VGG, ViT, and DeiT, while uniform soups are consistently worst and greedy soups improve but usually still trail logit ensembles. This does not imply that black-box averaging always dominates; the same study argues that weight merging can be more robust to increases in weights’ magnitudes and variances because it directly shrinks parameter norms and variances.

A second misconception is that BMM weights are always probabilities. In the BART-based simulator framework, no simplex constraint is imposed, and the weights are not posterior model probabilities [2301.02296]. This is central to the method’s ability to let \(\sum_k w_k(x) < 1\) in regions where all simulators collectively overshoot the truth.

A third boundary concerns what counts as “black-box.” BD-Merging [2603.03920] is described as fitting a partially black-box paradigm: it does not use fine-tuning data from individual tasks, but it assumes access to the pretrained backbone, task-specific checkpoints, task vectors, an unlabeled auxiliary dataset, and new training of an evidential head and router. Bayesian Model Merging [2605.12843] is plug-and-play and black-box in the sense of post-hoc fusion around anchor and expert checkpoints, yet it requires access to weights, shared architecture, and, in the data-assisted variant, activations; the paper explicitly notes that it is not applicable when only black-box outputs of experts are provided.

Open problems are consistent across the literature. [2605.12326] highlights scaling in \(K\), mixed-variable conditional dependencies, and the need for specialized optimizers such as CatCMA and ICatCMA. [2509.12951] emphasizes query efficiency, extension to multi-modal black-box models, and more sophisticated fusion functions \(\Phi\). [2006.13519] points to multi-service fusion, dialect and domain adaptation beyond accents, and more expressive but better-regularized fusion models. [2301.02296] notes data demands in high dimensions, limitations when all black-box models share systematic biases, and the difficulty of extending from mean mixing to density mixing. These directions suggest that the main unresolved issue is not whether BMM is feasible, but how to control granularity, search structure, and uncertainty when the accessible signal is limited to outputs, validation metrics, or lightweight checkpoint summaries.

Source: https://www.emergentmind.com/topics/black-box-model-merging-bmm