Black-Box Model Merging

• Black-Box Model Merging is an umbrella concept for fusing multiple models using only their outputs or external metrics.
• Techniques include input-dependent mixing, guided inference, and derivative-free optimization over merge configurations, with applications in speech recognition and language models.
• Empirical studies show improved performance metrics such as RMSE, WER, and F1 across domains like simulation, ASR, and LLM merging.

Black-Box Model Merging (BMM) denotes a family of techniques for combining multiple models when the merger operates through externally accessible behavior, merge configurations, or checkpoint-level summaries rather than full joint retraining. Across the literature, the term is used in several technically distinct ways: as input-dependent mixing of black-box simulators, as inference-time coupling of a proprietary service with a local model, and as derivative-free optimization over merge configurations for large model repositories and LLMs (Yannotty et al., 2023, Khandelwal et al., 2020, Chen et al., 16 Sep 2025). Taken together, these works suggest that BMM is best regarded as an umbrella concept rather than a single standardized algorithm.

1. Scope and terminological boundaries

In "Model Mixing Using Bayesian Additive Regression Trees" (Yannotty et al., 2023), Bayesian model mixing is a Bayesian mean-stacking model with input-dependent weights. The merger treats each simulator as a black box because it only uses the outputs $\hat f_k(x_i)$ at the training inputs, never uses gradients of $f_k$, and never accesses internal parameters of the simulators. The mixed predictor is

$$\hat f_{\text{mix}}(x)=\sum_{k=1}^K w_k(x)\,\hat f_k(x).$$

In "Black-box Adaptation of ASR for Accented Speech" (Khandelwal et al., 2020), BMM is instantiated as black-box accent adaptation: a proprietary cloud ASR is available only through input-output access, while a local accent-tuned ASR is white-box. The merger is not a word-level ensemble; it is a fine-grained, frame-level coupling in which the service transcript guides frame-level inference in the local model.

In the LLM-merging literature, BMM is often framed as black-box optimization over merge configurations. "It’s Morphing Time: Unleashing the Potential of Multiple LLMs via Multi-objective Optimization" (Li et al., 2024) formalizes merging as a multi-objective optimization problem over a configuration vector $\boldsymbol{x}$. "Black-Box Optimization of Mixed Binary-Continuous Variables: Challenges and Opportunities in Evolutionary Model Merging" (Niloy, 12 May 2026) formalizes Data Flow Space merging as a mixed binary–continuous black-box optimization problem. "Black-box Model Merging for Language-Model-as-a-Service with Massive Model Repositories" (Chen et al., 16 Sep 2025) defines BMM as fusing many closed-source LLMs, accessible only via APIs, into a single, stronger “virtual model.”

A neighboring line of work analyzes black-box operations without always using the term itself. "Rethinking Weight-Averaged Model-merging" (Wang et al., 2024) explicitly studies averaging on features and ensemble on logits/features, which are black-box operations because they only use outputs or feature activations, not weight access.

Formulation	Representative papers	Accessible objects
Input-dependent output mixing	(Yannotty et al., 2023)	Simulator predictions $\hat f_k(x)$
Guided inference	(Khandelwal et al., 2020)	Service transcript, confidences, local frame distributions
Merge-configuration search	(Li et al., 2024, Niloy, 12 May 2026, Chen et al., 16 Sep 2025, Ackermann et al., 24 Jun 2026)	Validation metrics from candidate merges

2. Core mathematical formulations

A canonical output-space formulation appears in (Yannotty et al., 2023). Given observations

$$Y_i = f_\dagger(x_i) + \epsilon_i,\quad \epsilon_i \stackrel{iid}{\sim} N(0,\sigma^2),$$

and candidate simulators $f_1(x),\dots,f_K(x)$, the paper models

$$Y_i \mid \hat f(x_i), w(x_i),\sigma^2 \overset{ind}{\sim} N\Big(\hat f(x_i)^\top w(x_i),\,\sigma^2\Big),$$

with $$\hat f(x_i)=(\hat f_1(x_i),\dots,\hat f_K(x_i))^\top$$. Here, black-box merging means that the meta-model learns $w_k(x)$ from data and simulator predictions only.

A different formulation appears in ASR. In (Khandelwal et al., 2020), the local CTC model produces frame-level distributions

$f_k$0

while the cloud service returns a transcript $f_k$1 with word-level confidences. After Viterbi-like alignment of service characters to frames, FineMerge revises the local frame distribution by

$f_k$2

when the aligned service character is acoustically supported but not dominant. The merged transcript is then obtained by standard beam decoding with the local LM.

Configuration-space formulations dominate recent LLM work. In (Niloy, 12 May 2026), DFS merging is formalized as

$f_k$3

where $f_k$4 encodes layer selection and $f_k$5 the scaling weights of selected layers. In (Li et al., 2024), multi-task merging is written as

$f_k$6

with objectives derived from task performance. In (Chen et al., 16 Sep 2025), the LMaaS setting is abstracted as

$f_k$7

where only API queries are available.

A further optimizer-centric formulation appears in (Ackermann et al., 24 Jun 2026), where language-model merging is treated as optimization over merge weights

$f_k$8

with black-box optimizers operating on function evaluations of the merged model.

3. Mechanisms for combining models

One major BMM family uses localized weighting functions. In (Yannotty et al., 2023), the weight vector is modeled by a Bayesian Additive Regression Trees sum-of-trees representation,

$f_k$9

where each tree partitions input space and each terminal node contributes a vector in $$\hat f_{\text{mix}}(x)=\sum_{k=1}^K w_k(x)\,\hat f_k(x).$$0. This yields nonparametric, input-dependent weights that can discover regions where one simulator should be up-weighted and another down-weighted. A notable design choice is that no explicit simplex constraint is imposed: the weights need not sum to $$\hat f_{\text{mix}}(x)=\sum_{k=1}^K w_k(x)\,\hat f_k(x).$$1 and need not be strictly non-negative, although priors are calibrated so individual weights tend to lie in $$\hat f_{\text{mix}}(x)=\sum_{k=1}^K w_k(x)\,\hat f_k(x).$$2 and the sum tends to be near $$\hat f_{\text{mix}}(x)=\sum_{k=1}^K w_k(x)\,\hat f_k(x).$$3 where a convex combination is appropriate. The paper explicitly notes that, without constraints, interpretability of $$\hat f_{\text{mix}}(x)=\sum_{k=1}^K w_k(x)\,\hat f_k(x).$$4 as probabilities diminishes; they are best seen as local scaling factors.

A second family uses structured guidance rather than direct averaging. FineMerge in (Khandelwal et al., 2020) aligns the service transcript to local model frames, uses a threshold $$\hat f_{\text{mix}}(x)=\sum_{k=1}^K w_k(x)\,\hat f_k(x).$$5 to decide whether a service character is acoustically “heard,” and modulates the update strength by the confidence of the parent service word. This allows the merger to preserve local acoustic evidence when the service transcript is likely a language-model artifact, while still using the service output as a dynamic prior over alignments.

A third family operates on features, logits, or probabilities. (Wang et al., 2024) distinguishes weight averaging from black-box operations such as logit averaging,

$$\hat f_{\text{mix}}(x)=\sum_{k=1}^K w_k(x)\,\hat f_k(x).$$6

feature averaging,

$$\hat f_{\text{mix}}(x)=\sum_{k=1}^K w_k(x)\,\hat f_k(x).$$7

and probability averaging. In purely linear models, feature-level averaging is mathematically equivalent to weight-level averaging. In nonlinear networks, the equivalence breaks because weight averaging is “average then activate,” whereas feature averaging is “activate then average.”

Search-based BMM adds a further mechanism: the models themselves may remain unchanged while an optimizer tunes how they are combined. In (Chen et al., 16 Sep 2025), Evo-Merging uses a two-stage CMA-ES pipeline with sparsity-based denoising and sign-aware scaling. In (Ackermann et al., 24 Jun 2026), the same perspective is cast in a unified spherical black-box optimization framework in which different optimizers are distinguished by fitness aggregation and consensus scope.

4. Optimization-centric BMM

The multi-objective line treats merge design as a black-box search problem over expensive evaluations. MM-MO (Li et al., 2024) uses qEHVI to optimize merging configurations defined over DARE+TIES controls. Hypervolume improvement is estimated under a surrogate posterior, and the search iterates by fitting a surrogate, selecting a batch of promising candidates, evaluating them on tasks such as C-EVAL and GSM8K, and updating the observed Pareto set. The paper reports that MM-MO reaches $$\hat f_{\text{mix}}(x)=\sum_{k=1}^K w_k(x)\,\hat f_k(x).$$8 on C-EVAL, $$\hat f_{\text{mix}}(x)=\sum_{k=1}^K w_k(x)\,\hat f_k(x).$$9 on GSM8K, and $\boldsymbol{x}$0 on HumanEval, outperforming single models and handcrafted merges.

DFS merging introduces a mixed-variable structure. (Niloy, 12 May 2026) emphasizes the conditional dependency between binary architecture variables and continuous scaling variables: if $\boldsymbol{x}$1, then $\boldsymbol{x}$2 is irrelevant. The structured approach first selects active layers via $\boldsymbol{x}$3 and then optimizes only the active scales $\boldsymbol{x}$4. In the reported experiment, Structured DFS achieved $\boldsymbol{x}$5 accuracy, Unstructured DFS achieved $\boldsymbol{x}$6, and the effective search space was reduced by $\boldsymbol{x}$7.

Evo-Merging (Chen et al., 16 Sep 2025) addresses massive repositories. Stage 1 optimizes per-model sparsity ratios $\boldsymbol{x}$8 with uniform weights, while Stage 2 optimizes signed scaling weights $\boldsymbol{x}$9. The denoising objective is

$\hat f_k(x)$0

and the scaling objective is

$\hat f_k(x)$1

A distinctive claim is that $\hat f_k(x)$2 can be positive or negative, so some models are added and others effectively subtracted. The paper also provides an error bound for asymmetric sparsification of LoRA $\hat f_k(x)$3 matrices,

$\hat f_k(x)$4

(Ackermann et al., 24 Jun 2026) generalizes several black-box optimizers through a master update,

$\hat f_k(x)$5

with consensus terms determined by fitness aggregation and interaction kernels. Within this view, ES-OVI interpolates between flatness-preferring and sharpness-preferring search, while AdaPol and SchedPol combine OVI-like global search with cCBO-like multimodal refinement. The paper reports that the CBO-OVI hybrids achieve competitive results on LLM merging under limited evaluation budgets.

5. Empirical regimes and application domains

Simulator mixing in nuclear and toy physics is one of the earliest explicit BMM settings in this set of works. In (Yannotty et al., 2023), Example 1a mixes $\hat f_k(x)$6 and $\hat f_k(x)$7: BART-BMM attains RMSE $\hat f_k(x)$8, whereas HS yields RMSE $\hat f_k(x)$9. In Example 1b, where both simulators overestimate the truth in the intermediate region, BART-BMM attains RMSE $$Y_i = f_\dagger(x_i) + \epsilon_i,\quad \epsilon_i \stackrel{iid}{\sim} N(0,\sigma^2),$$0 versus HS RMSE $$Y_i = f_\dagger(x_i) + \epsilon_i,\quad \epsilon_i \stackrel{iid}{\sim} N(0,\sigma^2),$$1, and the sum of weights drops below $$Y_i = f_\dagger(x_i) + \epsilon_i,\quad \epsilon_i \stackrel{iid}{\sim} N(0,\sigma^2),$$2 in the problematic region. In a 2D trigonometric Taylor-expansion example, BART-BMM with $$Y_i = f_\dagger(x_i) + \epsilon_i,\quad \epsilon_i \stackrel{iid}{\sim} N(0,\sigma^2),$$3 trees attains RMSE $$Y_i = f_\dagger(x_i) + \epsilon_i,\quad \epsilon_i \stackrel{iid}{\sim} N(0,\sigma^2),$$4.

Accent adaptation provides a structured-prediction instance of BMM. (Khandelwal et al., 2020) reports, for Google US as service, Indian-accent WER/CER of $$Y_i = f_\dagger(x_i) + \epsilon_i,\quad \epsilon_i \stackrel{iid}{\sim} N(0,\sigma^2),$$5 for FineMerge versus $$Y_i = f_\dagger(x_i) + \epsilon_i,\quad \epsilon_i \stackrel{iid}{\sim} N(0,\sigma^2),$$6 for the service and $$Y_i = f_\dagger(x_i) + \epsilon_i,\quad \epsilon_i \stackrel{iid}{\sim} N(0,\sigma^2),$$7 for the local model. For Australian accent, FineMerge attains $$Y_i = f_\dagger(x_i) + \epsilon_i,\quad \epsilon_i \stackrel{iid}{\sim} N(0,\sigma^2),$$8 versus $$Y_i = f_\dagger(x_i) + \epsilon_i,\quad \epsilon_i \stackrel{iid}{\sim} N(0,\sigma^2),$$9 for the service and $f_1(x),\dots,f_K(x)$0 for the local model. For British accent, FineMerge attains WER $f_1(x),\dots,f_K(x)$1 and CER $f_1(x),\dots,f_K(x)$2. The paper states that the method achieves as much as $f_1(x),\dots,f_K(x)$3 relative reduction in WER over both the local and service models.

Large-model merging is evaluated under both small and massive search spaces. MM-MO (Li et al., 2024) outperforms task arithmetic, TIES, and default DARE+TIES on C-EVAL, GSM8K, and HumanEval. Evo-Merging (Chen et al., 16 Sep 2025) reports average out-of-domain Prec $f_1(x),\dots,f_K(x)$4 and F1 $f_1(x),\dots,f_K(x)$5, versus LoRaHub Prec $f_1(x),\dots,f_K(x)$6 and F1 $f_1(x),\dots,f_K(x)$7, while operating on more than $f_1(x),\dots,f_K(x)$8 LoRA-based models. In in-domain multi-task merging across eight NER datasets, Evo-Merging reports F1 $f_1(x),\dots,f_K(x)$9 versus LoRaHub F1 $$Y_i \mid \hat f(x_i), w(x_i),\sigma^2 \overset{ind}{\sim} N\Big(\hat f(x_i)^\top w(x_i),\,\sigma^2\Big),$$0. The same paper shows that adding five noisy tasks raises Evo-Merging from F1 $$Y_i \mid \hat f(x_i), w(x_i),\sigma^2 \overset{ind}{\sim} N\Big(\hat f(x_i)^\top w(x_i),\,\sigma^2\Big),$$1 to $$Y_i \mid \hat f(x_i), w(x_i),\sigma^2 \overset{ind}{\sim} N\Big(\hat f(x_i)^\top w(x_i),\,\sigma^2\Big),$$2, whereas several baselines drop sharply.

Optimizer studies also report that the merging landscape can be multimodal. In (Ackermann et al., 24 Jun 2026), training and test accuracy over a 2D slice of merge-parameter space show multiple local maxima, and the paper reports that cCBO is beneficial under good initialization, OVI performs better under bad initialization, and SchedPol and AdaPol are competitive in both scenarios.

6. Misconceptions, boundary cases, and open problems

A recurrent misconception is to identify BMM with ordinary weight averaging. (Wang et al., 2024) makes the distinction explicit: weight-averaged soups require white-box parameter access, whereas logit averaging, feature averaging, and probability averaging are black-box operations. The paper further reports that logit ensembles almost always give the best accuracy across ResNet, DenseNet, VGG, ViT, and DeiT, while uniform soups are consistently worst and greedy soups improve but usually still trail logit ensembles. This does not imply that black-box averaging always dominates; the same study argues that weight merging can be more robust to increases in weights’ magnitudes and variances because it directly shrinks parameter norms and variances.

A second misconception is that BMM weights are always probabilities. In the BART-based simulator framework, no simplex constraint is imposed, and the weights are not posterior model probabilities (Yannotty et al., 2023). This is central to the method’s ability to let $$Y_i \mid \hat f(x_i), w(x_i),\sigma^2 \overset{ind}{\sim} N\Big(\hat f(x_i)^\top w(x_i),\,\sigma^2\Big),$$3 in regions where all simulators collectively overshoot the truth.

A third boundary concerns what counts as “black-box.” BD-Merging (Xie et al., 4 Mar 2026) is described as fitting a partially black-box paradigm: it does not use fine-tuning data from individual tasks, but it assumes access to the pretrained backbone, task-specific checkpoints, task vectors, an unlabeled auxiliary dataset, and new training of an evidential head and router. Bayesian Model Merging (Li et al., 13 May 2026) is plug-and-play and black-box in the sense of post-hoc fusion around anchor and expert checkpoints, yet it requires access to weights, shared architecture, and, in the data-assisted variant, activations; the paper explicitly notes that it is not applicable when only black-box outputs of experts are provided.

Open problems are consistent across the literature. (Niloy, 12 May 2026) highlights scaling in $$Y_i \mid \hat f(x_i), w(x_i),\sigma^2 \overset{ind}{\sim} N\Big(\hat f(x_i)^\top w(x_i),\,\sigma^2\Big),$$4, mixed-variable conditional dependencies, and the need for specialized optimizers such as CatCMA and ICatCMA. (Chen et al., 16 Sep 2025) emphasizes query efficiency, extension to multi-modal black-box models, and more sophisticated fusion functions $$Y_i \mid \hat f(x_i), w(x_i),\sigma^2 \overset{ind}{\sim} N\Big(\hat f(x_i)^\top w(x_i),\,\sigma^2\Big),$$5. (Khandelwal et al., 2020) points to multi-service fusion, dialect and domain adaptation beyond accents, and more expressive but better-regularized fusion models. (Yannotty et al., 2023) notes data demands in high dimensions, limitations when all black-box models share systematic biases, and the difficulty of extending from mean mixing to density mixing. These directions suggest that the main unresolved issue is not whether BMM is feasible, but how to control granularity, search structure, and uncertainty when the accessible signal is limited to outputs, validation metrics, or lightweight checkpoint summaries.