---
title: 'Model Soups: Efficient Weight-Averaging'
url: https://www.emergentmind.com/topics/model-soups
type: topic
---

# Model Soups: Efficient Weight-Averaging

Model soups are weight-space ensembles obtained by averaging the parameters of multiple independently fine-tuned neural networks with the same architecture, typically initialized from the same pre-trained checkpoint. This technique provides a method to leverage the diversity generated by hyperparameter sweeps and stochastic training, resulting in improved generalization, robustness to distribution shift, and—in certain settings—extensible properties such as zero-shot continual learning or controllable trade-offs. Model soups incur no additional inference cost, as the result is a single model with a standard parameter set. The methodology has been extended from vision transformers and large language models to diffusion generative models, graph neural networks, foundation model adapters, sparse/pruned networks, federated learning agents, and multi-objective RLHF policies.

## 1. Formalism and General Principles

The canonical model soup procedure, introduced by Wortsman et al. [2203.05482], constructs a new set of weights by computing an affine (typically convex) combination of $K$ independently fine-tuned models:
\[
\theta_{\text{soup}} = \sum_{k=1}^K \lambda_k\, \theta^{(k)},
\]
with coefficients $\lambda_k \geq 0$ and $\sum_k \lambda_k = 1$. The two principal strategies are:

- **Uniform soup:** $\lambda_k = 1/K$ for all $k$.
- **Greedy soup:** Iteratively add candidate models in order of descending validation accuracy, including a model if its addition improves validation performance when averaged with the current soup.

Empirical investigation across large-scale vision and language tasks showed that these soups frequently outperform the single best model from a hyperparameter sweep in both in-distribution and out-of-distribution settings, under the prerequisite that all candidate models reside in a common, connected low-loss basin in parameter space (i.e., strong mode connectivity due to joint initialization and network architecture) [2203.05482, 2301.10092]. Parameter-space averaging, in contrast to output-level ensembling, introduces no extra computational or memory cost at inference.

## 2. Algorithms and Variants

Model soup construction encompasses several algorithmic extensions:

- **Learned soup:** Optimizes the mixing coefficients $\lambda_k$ on a validation set via gradient descent, optionally in a constrained or regularized way (e.g., block coordinate descent on the hyperplane constrained by $\sum_k \lambda_k = 1$) [2407.03641, 2503.11612].
- **Greedy and pruned soups:** Greedy soup selects a minimal subset of models whose addition to the average preserves or improves validation accuracy; pruned soup starts from the uniform average and recursively prunes away models that do not harm the validation metric when removed, using multiple passes for stability [2301.10092].
- **Layer-wise and fine-grained soups:** Mixing coefficients may be learned per-layer or per-module, rather than globally, enabling additional expressivity and marginal accuracy gains, especially in deep transformers and ViTs [2407.03641, 2407.08196].
- **Efficient construction:** Block coordinate gradient descent and partitioned subgraph optimization allow the exploitation of large model families without exceeding modest computational resource budgets [2503.11612, 2407.03641].

The following table summarizes common soup construction strategies:

| Strategy             | Description                                | Constraints           |
|----------------------|--------------------------------------------|-----------------------|
| Uniform              | Equal weights over all candidate models     | $\lambda_k=1/K$       |
| Greedy               | Incrementally add if val improves          | $\lambda_k\geq 0$     |
| Learned              | Optimize $\lambda_k$ via validation loss   | $\sum_k \lambda_k=1$  |
| Layer-wise           | $\lambda_k$ vector per layer or module     | $\sum_k \lambda_k^{(\ell)}=1$ per $\ell$ |
| Pruned               | Start with all, remove any non-harmful     | support subset may shrink |

## 3. Theoretical Foundations

The effectiveness of model soups is theoretically grounded in the geometry of the neural loss landscape and the concentration of solutions in wide, connected basins when fine-tuning from shared pre-trained initialization. Weight-averaged models land at the barycenter of these modes, benefiting from:

- **Variance reduction:** Averaging reduces the variance of predictions across models, as formalized via the classical bias-variance decomposition [2603.02181].
- **Flatness and connectivity:** Souped weights inherit the flatness properties of the containing basin; convex paths in weight space between independently fine-tuned models remain in a low-loss region [2203.05482, 2301.10092].
- **Pareto optimality under trade-offs:** In multi-objective or adversarial settings, convex or affine combinations of extremal fine-tuned models trace Pareto fronts for robustness under multiple threats or objectives [2302.10164, 2502.10762].

Negative results occur if candidate models occupy distinct, disconnected minima, as in training-from-scratch or when fine-tuning from highly divergent hyperparameters; in such cases, averaging can destroy useful structure, leading to collapse [2301.10092].

## 4. Application Domains

Model soups have been extended well beyond their origin in vision transformer fine-tuning:

- **Sparse and Pruned Networks:** Sparse Model Soups (SMS) preserve pruning masks by enforcing identical connectivity during retraining splits, yielding substantial accuracy/OOD performance gains at high sparsity rates (e.g., +1.1% on CIFAR-100 at 98% sparsity), without densifying the network after averaging [2306.16788].
- **Diffusion Models:** Soups constructed from independently trained specialist diffusion models on dataset shards enable continual training-free learning and unlearning, and provably avoid memorization by geometric-mean sampling distributions [2406.08431].
- **Adapters and Domain Adaptation:** Soup-Adapters average adapter module parameters, reparameterizing multiple adapters into a single efficient module that smooths the sensitivity to hyperparameters and improves both in-distribution and OOD performance under few-shot adaptation [2507.05807].
- **Graph Neural Networks:** Learned souping and partitioned learned souping reduce wall-time and memory overhead for GNNs, with up to 1.2% accuracy improvement and 24.5× speedup over exhaustive interpolation [2503.11612].
- **Federated Learning:** Local Superior Soups (LSS) interpolate local model checkpoints via randomized affine combinations, imposing affinity and diversity regularizers that lead to dramatic improvements in global accuracy under strict round constraints [2410.23660].
- **Multi-modal Model Fusion:** SoupLM merges contrasting LLM or VLM checkpoints (Vicuna, LLaVA, etc.) into integrated, well-generalized models via whole-model or module-wise interpolation of weights, recovering 1–1.5% additional accuracy in language and vision-language tasks [2407.08196].
- **Multi-objective RLHF:** Bone Soups construct a structured set of backbone models via MORLHF, then merge (“soup”) them via a circulant basis to produce controllably optimal multi-objective policies at inference [2502.10762].

## 5. Empirical Findings and Best Practices

Empirical studies across applications converge on several robust patterns:

- **Best performance requires pretrained initialization and shared architectures:** Success hinges on all “ingredients” (candidate models) being aligned in both initialization and final label/feature dimensions [2203.05482, 2301.10092].
- **Hyperparameter diversity among ingredients:** Gains are typically highest for candidate models fine-tuned with different learning rates, seeds, augmentations, weight decays, or data orders, provided these do not cause basin separation [2203.05482, 2306.16788].
- **Greedy/pruned selection consistently outperforms uniform in presence of outliers:** Uniform soups risk collapse if poor checkpoints are included; greedy and pruned variants provide robustness [2301.10092].
- **Layer-wise and learned soups offer small but measurable improvements over global mixing:** Layer-wise $\lambda_k^{(\ell)}$ can capture fine-grained adaptation not possible with a single global coefficient, especially in deep models [2407.03641].
- **Model soups outperform or match conventional ensembles with no additional inference cost:** Standard output- or logit-based ensembles require as many forward passes as ingredients; soups perform a single forward pass at inference [2203.05482, 2406.08431].

Model soups have been shown to deliver up to 0.6–1.5 percentage point gains in top-1 accuracy on ImageNet [2203.05482], 0.7 in macro-F1 on challenging low-resource ICH datasets [2603.02181], and superior robustness to distribution shifts, adversarially perturbed data, and compositional tasks in diffusion and RL (see individual paper metrics for architecture- and benchmark-specific deltas).

## 6. Limitations, Extensions, and Open Problems

Several limitations and frontiers for model soups are recognized:

- **Disconnected minima:** When fine-tuned models are not in a shared basin, parameter averaging is ineffective or detrimental, as observed with ResNet or EfficientNet trained from scratch [2301.10092].
- **Calibration of batch/layer normalization:** Averaged models may retain inconsistent running statistics, requiring recalibration after construction [2301.10092].
- **Mask and sparsity matching:** In sparse and pruned soups, mask mismatches can densify the model; solutions require phase-wise mask sharing or specialized pruning-aware averaging [2306.16788].
- **Efficient construction for large $K$:** Conventional learned-soup approaches are non-scalable at large $K$ due to memory and compute demand; MEHL-Soup and PLS methods are emerging solutions [2503.11612, 2407.03641].
- **Automated coefficient selection and extrapolation:** Grid search over affine weights for multi-objective or adversarial settings is viable at low dimension, but extension to high-dimensional spaces or automated coefficient learning is an open area [2302.10164, 2502.10762].
- **MonoSoup for single-checkpoint post-hoc balancing:** When only one fine-tuned checkpoint is available, SVD-based “MonoSoup” allows decomposition into high- and low-rank update directions, achieving much of the OOD benefit of full soups with trivial memory cost [2602.09689].

Future work includes extending model soups beyond homogenous families (heterogeneous architectures), combining soups with distillation or Bayesian methods, and theoretical analysis of loss landscape connectivity predictors for soup success.

## 7. Impact and Outlook

Model soups have reshaped best-practice protocols for both production and research pipelines involving large-scale model adaptation. They enable systematic exploitation of “wasted” diversity in hyperparameter sweeps, yield robust, low-variance models for deployment, and support new strategies for multi-domain or multi-objective model integration without retraining or complex ensemble infrastructure. Recent innovations such as hierarchical souping [2403.13341], real-time controllable generation [2502.10762], and continual diffusion soup updating [2406.08431] continue to expand both the theoretical and practical scope of the approach. Soups' zero–inference-cost profile and extensibility position them as a central tool in scalable, robust deep learning.

Source: https://www.emergentmind.com/topics/model-soups