Bayesian Model Merging

• Bayesian model merging is a principled method that integrates multiple probabilistic models into a coherent Bayesian framework while preserving uncertainty and prior knowledge.
• It employs Gaussian, exponential-family, and EM-based approximations to merge parameters, facilitating balanced stability and plasticity across tasks.
• Its applications range from continual learning to multitask finetuning and evidence synthesis, demonstrating superior performance over naive parameter averaging.

Bayesian model merging refers to a family of principled frameworks and algorithms for combining multiple probabilistic models into a unified Bayesian structure, such that the resulting model integrates information across model sources and data partitions, while appropriately managing prior and data-induced uncertainties. This merging is critical in a range of contexts including continual learning, multitask learning, combining simulator outputs, model composition, and integrating complex evidence sources in scientific studies. Distinct from ad hoc parameter averaging, Bayesian model merging ensures theoretical coherence by explicitly modeling the combination as Bayesian inference, typically under Gaussian or exponential-family approximations, or via melding of submodel marginals.

1. Formal Bayesian Foundations of Model Merging

The Bayesian model merging paradigm is anchored in fundamental Bayesian inference, where for parameter vector θ and datasets $D_1,\ldots, D_T$, the posterior is given by

$$p(\theta | D_1:T) \propto p(\theta) \prod_{i=1}^T p(D_i | \theta)$$

In practice, merged models often correspond to optimizing an objective that balances task-specific losses with priors encoding information from earlier tasks or models (Li et al., 3 Apr 2025, Maldonado et al., 2024). The merging may target the MAP estimate by

$$\theta^* = \arg\min_\theta \Big[\sum_{t=1}^T \alpha_t \mathcal{L}_t(\theta) + R(\theta) \Big]$$

where $\mathcal{L}_t$ are task losses and $R(\theta)$ is a regularizer corresponding to the prior. Under Gaussian (Laplace) or more expressive variational approximations, model merging can be reduced to closed-form solutions or efficient EM-style procedures (Maldonado et al., 2024).

When models arise from independent fine-tunings or modular constructions, merging aims to approximate the Bayes-optimal solution by aggregating anchor models and local posteriors—typically leveraging Gaussian or exponential-family surrogates.

2. Methodological Variants and Closed-Form Solutions

Multiple methodological schemes have emerged, all leveraging the Bayesian formulation but exploiting different approximation and optimization strategies:

• Gaussian/Laplace-Based Merging. Under local quadratic approximations of the loss (e.g., via the Fisher Information matrix or Hessian), merged weights are obtained by weighted averages, with weighting determined by parameter uncertainty or task-specific curvature (Li et al., 3 Apr 2025, Maldonado et al., 2024). For example, for Gaussian single-task posteriors $q_t(\theta) = N(\theta_t, H_t^{-1})$, the merged parameter is:

$$\hat{\theta}_H(\alpha) = \bigg[\sum_t \alpha_t H_t \bigg]^{-1} \bigg[\sum_t \alpha_t H_t \theta_t\bigg]$$

• Posterior/Exponential Family Mixtures. Moving beyond unimodal Gaussians, merging may exploit mixtures of Gaussians, with an EM-like iteration to compute the optimal merged parameter (Maldonado et al., 2024).
• Module-Wise Bayesian Regression. When model parameters are structured into modules (e.g., transformer blocks), merging is framed as a Bayesian linear regression, where task-specific "task vectors" are regularized towards anchors using activation statistics. The inner-level solution is

$$U^* = (YX^\top + \lambda U^0)(XX^\top + \lambda I)^{-1}$$

with an outer-level Bayesian optimization to select regularization strengths jointly across modules (Li et al., 13 May 2026).

• Adaptive/Task-Aware Coefficients. In continual learning and multitask settings, merging coefficients (interpolation weights) are adaptively computed to balance model stability and plasticity based on curvature along the solution path, yielding formulas of the type

$$\lambda_t^* = \frac{\Delta\theta^\top F_t \Delta\theta}{\Delta\theta^\top \big[F_t + \sum_{i<t} F_i \big] \Delta\theta}$$

where $F_t$ is the Fisher Information for task $$p(\theta | D_1:T) \propto p(\theta) \prod_{i=1}^T p(D_i | \theta)$$0, and $$p(\theta | D_1:T) \propto p(\theta) \prod_{i=1}^T p(D_i | \theta)$$1 is the difference between task-specific and stability-preserving solutions (Li et al., 3 Apr 2025).

3. Model Merging in Continual and Multitask Learning

A major application of Bayesian model merging is in continual and multitask learning scenarios. In Bayesian continual learning, methods like BECAME construct a sequence in which each new task yields both a stability-preserving and a plasticity-optimized solution. The merged model is formed by Bayes-optimal interpolation, using a closed-form coefficient based on Fisher information and parameter displacement (Li et al., 3 Apr 2025). This guarantees convexity along the merge path, ensures a unique optimum, and empirically achieves superior trade-offs between stability (retaining old knowledge) and plasticity (learning new tasks).

For multitask finetuning, Bayesian merging provides fast, closed-form "previews" of multitask optima across arbitrary task weightings, circumventing costly retraining. The family of closed-form merging formulas—simple averaging, Hessian-weighted averaging, and mixture-based methods—enables rapid exploration of the accuracy landscape as a function of task weights, guiding resource allocation and experiment design (Maldonado et al., 2024).

4. Black-Box Optimization and Data-Free Merging

Modern scalable Bayesian merging often requires tuning high-dimensional module-wise or task-wise regularization and scale parameters. Bayesian optimization is systematically employed over held-out data to discover optimal hyperparameters for merging coefficients, regularization weights, or module/group scales (Li et al., 13 May 2026, Lee et al., 26 Apr 2025, Liu et al., 2024). These outer optimization loops use Gaussian-process surrogates and acquisition functions such as Expected Improvement or Upper Confidence Bound, and greatly enhance empirical accuracy relative to fixed or grid-tuned hyperparameters.

Recent frameworks also include data-free variants of merging. By leveraging empirically verified alignment between activation Gram matrices and task-vector Gram matrices under neural-collapse-like phenomena, closed-form merging formulas can be instantiated without any auxiliary data, using only pretrained and fine-tuned weight differences (Li et al., 13 May 2026). This property further pushes merging into regimes with minimal computational and data requirements.

5. Model Merging Beyond Parametric Weight Averaging

Bayesian model merging encompasses broader scenarios beyond simple averaging of parameters:

• Evidence Synthesis and Markov Melding. In modular Bayesian analysis, where submodels share parameters or latent quantities, Markov melding combines submodel posteriors via pooled priors (logarithmic or linear pools), ensuring coherent inference across heterogeneous modules (Manderson et al., 2020, Manderson et al., 2021). The melded posterior is

$$p(\theta | D_1:T) \propto p(\theta) \prod_{i=1}^T p(D_i | \theta)$$2

where $$p(\theta | D_1:T) \propto p(\theta) \prod_{i=1}^T p(D_i | \theta)$$3 is a pooled prior over shared quantities. Numerical stability in sampling is achieved by focusing on direct estimation of self-density ratios rather than marginal priors, using weighted-sample KDEs across low-density regions (Manderson et al., 2020).

• Mixtures and Additive Model Mixing. Bayesian model mixing techniques, such as BMM/BART, construct input-dependent mixtures of predictions, with flexible weight functions modeled via Bayesian additive regression trees (BART), allowing nonparametric local aggregation of simulator outputs (Yannotty et al., 2023).
• Topological Fusion of Bayesian Networks. For graphical models, merging involves fusion at the level of directed acyclic graphs (DAGs), preserving all original dependencies, and is executed via graph union, arc reversal, and conditional probability re-expression, yielding a consensus structure for prior compromise (Matzkevich et al., 2013).

6. Empirical Validation and Practical Applications

Empirical studies consistently show that Bayesian model merging frameworks outperform naïve parameter averaging or baseline merging strategies across vision, language, and simulation domains. For instance, data-assisted Bayesian merging with adaptive module-wise regularization matches or approaches the performance of the oracle average of individual task-specific fine-tuned experts, and data-free variants close much of the remaining gap (Li et al., 13 May 2026). In large-scale LLM pretraining, Bayes-optimal checkpoint merging adds up to 1% absolute accuracy at negligible computational cost compared to full retraining (Liu et al., 2024).

Bayesian model merging has proved effective in evidence synthesis for epidemiological studies (e.g., HIV prevalence, A/H1N1 influenza), in producing fast finetuning previews to guide multitask weighting, and in modular model composition for complex physical simulations (Manderson et al., 2020, Maldonado et al., 2024, Yannotty et al., 2023).

7. Limitations and Theoretical Considerations

Despite its broad applicability and strong empirical performance, Bayesian model merging exhibits several limitations:

• The convex combination or Bayesian averaging of weights may be suboptimal under pathological loss landscapes with strong parameter interactions or where the true solution involves complex non-linear integration of model components.
• The necessity to approximate posteriors with tractable surrogates (Gaussian, mixture, exponential-family) introduces approximation errors, especially in high-dimensional or highly non-linear settings (Maldonado et al., 2024, Li et al., 3 Apr 2025).
• For Markov melding, accurate prior density estimation for shared quantities is numerically sensitive. Robust estimation via weighted-sample self-density ratios is essential to avoid instability in low-density regions (Manderson et al., 2020).
• Bayesian optimization in high-dimensional hyperparameter spaces may require careful selection of acquisition functions, surrogate models, and initialization strategies to ensure convergence within practical evaluation budgets (Li et al., 13 May 2026, Lee et al., 26 Apr 2025).

Potential extensions include integrating active data selection, learning structural correspondences across modules for hierarchical or structured merging, and advancing numerical techniques for marginal density and ratio estimation in arbitrary Bayesian graphical models.

---

References:

(Li et al., 3 Apr 2025, Maldonado et al., 2024, Li et al., 13 May 2026, Lee et al., 26 Apr 2025, Liu et al., 2024, Manderson et al., 2020, Manderson et al., 2021, Yannotty et al., 2023, Matzkevich et al., 2013, Hwang et al., 2011)