---
title: Model Soup Parameter Averaging
url: https://www.emergentmind.com/topics/model-soup-parameter-averaging-be9d926a-ec67-47d6-a8d3-12abcb3b0efb
type: topic
---

# Model Soup Parameter Averaging

Model soup parameter averaging is a family of techniques in which the parameters (weights) of multiple neural network models are combined by a weighted average to form a single model. This paradigm, introduced in the context of fine-tuned foundation models, enables the benefits of ensemble-like generalization and robustness without incurring increased inference or memory cost. The approach has since diversified into several algorithmic regimes, including checkpoint averaging, greedy/learned soups, adaptive meta-ensembling, domain-specialized parameter mergers, and continual learning architectures. Its practical impact is observed across vision, NLP, diffusion models, and large language model alignment, providing an efficient, data-free means to synthesize improved models from heterogeneous training or fine-tuning runs.

## 1. Formal Definitions and Core Soup Construction Methods

Let $\{\theta_i\}_{i=1}^{N}$ denote $N$ independently trained or fine-tuned neural network checkpoints (typically obtained from a shared initialization under different hyperparameter configurations or data orders). The canonical model soup is a convex combination of these weights,
$$
\theta_{\text{soup}} = \sum_{i=1}^N \alpha_i \theta_i, \quad \sum_{i=1}^N \alpha_i = 1,\ \alpha_i \geq 0
$$
- **Uniform soup**: All $\alpha_i = 1/N$ [2203.05482].
- **Greedy soup**: Ingredients are sequentially added from a sorted list (e.g., by validation accuracy), retaining each only if the averaged validation accuracy does not decrease, ensuring a monotonic or plateau behavior [2203.05482].
- **Pruned soup**: Begin from the full uniform soup, iteratively remove ingredients whose exclusion does not decrease validation performance, yielding sparser and sometimes superior averages [2301.10092].
- **Learned soup**: Optimize $\{\alpha_i\}$ using cross-entropy or other surrogate loss on a held-out validation set; typically solved with gradient-based methods [2203.05482].

All these procedures create a single merged parameter vector that can be directly deployed, using the same inference-time budget as any single constituent model. Empirically, effective soup construction requires that all $\theta_i$ reside in a common low-loss basin of the loss landscape (i.e., exhibit mode connectivity).

## 2. Theoretical Principles and Equivalence to Ensembling

Model soups operationalize two key theoretical insights:
- **Loss landscape geometry**: In over-parameterized networks, fine-tuned models (from a strong common initialization) tend to occupy a connected low-loss region, allowing linear interpolations to approximately preserve in-distribution and sometimes out-of-distribution performance [2203.05482].
- **First-order equivalence to ensemble averages**: For sufficiently aligned checkpoints (close in weight space), the loss of the weight-averaged model approximates the loss of a prediction (logits or probability) ensemble to first-order via Taylor expansion [2401.17790]. That is,
  $$
  \mathcal{L}(f(x; \theta_{\text{soup}}), y) \approx \mathcal{L}\left(\sum \alpha_i f(x;\theta_i), y\right)
  $$
  up to quadratic corrections in $\|\theta_i - \theta_0\|$.

Model soups often perform similarly to or slightly below full probability/logit ensembles in-distribution, but can exceed ensemble performance under distribution shift due to robustness conferred by averaging multiple training trajectories [2203.05482].

## 3. Algorithmic Variants and Generalizations

The parameter-averaging paradigm supports a spectrum of methods beyond naive uniform averaging:
- **Resource-Adjusted Souping** (RADIN): Given prohibitive $2^N$ subset selection for the optimal soup, RADIN leverages approximated validation scores via cached logits to propose promising soups, followed by a small number of true evaluations, interpolating between uniform, greedy, and oracle selection costs [2401.17790].
- **PopulAtion Parameter Averaging (PAPA)**: Enforces model alignment during training by attracting each member’s weights towards the population mean at fixed intervals. This preserves diversity for ensemble-like generalization yet keeps parameters close enough to be directly averaged, closing most of the gap between ensembles and soups [2304.03094].
- **Amortized Model Ensembling (AME)**: Models the selection of an average as an explicit meta-optimization, in which model differences serve as pseudogradients, enabling adaptive or momentum-based updates and repeated meta-epochs for potentially improved neural averaging [2508.14832].
- **Specialist/Domain-Targeted SoE**: The "Soup-of-Experts" architecture parameterizes the model as a sum of a shared core and learned expert offsets, combining them at inference with MLP-generated mixing coefficients tailored to arbitrary domain mixture weights [2502.01804].
- **Pruning-Compatible Souping (SMS)**: Sparse Model Soups enforce a shared mask across all fine-tuned/pruned models, allowing valid parameter averaging without sacrificing target sparsity and achieving robustness beyond vanilla IMP [2306.16788].
- **Diffusion Soup**: Average models fine-tuned on disjoint data shards, theoretically interpolating geometric-mean distributions and supporting continual learning, unlearning, and style blending [2406.08431].

## 4. Empirical Performance and Domain Applications

Model soups have demonstrated consistent empirical gains across vision, NLP, and generative models:
- **Vision (ImageNet, ViT/CLIP/ALIGN)**: Greedy and learned soups exceed the best single checkpoint by up to $+1$ pp Top-1 on ImageNet; robustness gains ($+3$ pp on average under domain shifts) are observed over standard ensembling, all at single-model inference cost [2203.05482].
- **NLP (GLUE tasks, cross-lingual transfer)**: Fine-tuned model soups improve GLUE benchmark tasks ($\sim +0.8\%$ on MRPC, RTE) and token classification. In ZS-XLT settings, accumulative and per-run checkpoint averaging yield clear gains, closely tracking "oracle" performance based on unavailable target-dev sets [2310.10532].
- **RLHF and LLM alignment (SALSA)**: Replacing the PPO KL-reference with a soup (typically an average of two independent SFTs) yields superior exploration, reward, and alignment, with consistent $+4$ to $+14$ ppt win-rate improvements on Llama2-7B, Mistral-7B, and Gemma-2B [2411.01798].
- **Diffusion Models**: Averaged diffusion soups outperform monolithic training on domain-and-style compositionality metrics (e.g., TIFA, IR, CLIP) while preserving continual/unlearning properties [2406.08431].
- **Sparse Regimes**: SMS strictly improves out-of-distribution accuracy and often in-distribution test accuracy over both standard IMP and individual pruned models [2306.16788].

A summary of selected quantitative gains is as follows:

| Task/Domain                   | Soup Type           | Single Best | Soup      | Ensemble | Reference         |
|-------------------------------|---------------------|-------------|-----------|----------|-------------------|
| ImageNet (ViT-B/32)           | Greedy              | 80.38%      | 81.03%    | —        | [2203.05482]      |
| ImageNet (CLIP)               | Uniform/Greedy      | 80.38%      | 79.97%    | —        | [2203.05482]      |
| OOD Shift (ViT-B/32, avg)     | Greedy              | 47.83%      | 50.75%    | —        | [2203.05482]      |
| RLHF Avg Win-Rate (Llama2-7B) | SALSA (α=0.5)       | 47.50%      | 52.50%    | —        | [2411.01798]      |
| CIFAR-10 (Res18)              | PAPA AvgSoup        | 96.8%       | 97.4%     | 97.5%    | [2304.03094]      |
| CIFAR-100 (WideResNet20, imp) | SMS                 | 75.54%      | 76.59%    | —        | [2306.16788]      |
| Diffusion Domain-Sharded (IR) | Uniform Soup (n=9)  | 0.34        | 0.45      | —        | [2406.08431]      |

## 5. Practical Constraints, Limitations, and Failure Modes

While model soup parameter averaging is empirically robust, its practical deployment is conditioned on several factors:
- **Common basin requirement**: Uniform soups succeed only when all candidate checkpoints reside in the same low-loss landscape region; divergence, especially for models with batch norm or trained from scratch with different seeds, leads to collapse (random or poor performance) [2301.10092].
- **Batch norm sensitivity**: Averaging models with inconsistent batch norm statistics can lead to ill-calibrated predictions. Re-estimating BN running statistics after soup formation is critical [2301.10092].
- **Sparse soups**: Averaging models with different sparse masks destroys sparsity. SMS solves this by enforcing a shared mask at each pruning phase [2306.16788].
- **Subset selection complexity**: The greedy approach is $O(N)$ in evaluations, and full combinatorial selection is intractable at nontrivial $N$; RADIN and learned soups enable cost-accuracy trade-offs [2401.17790], [2203.05482].
- **Application specificity**: For cross-lingual transfer, checkpoint averaging must maintain aligned head architectures, and gains may depend on task taxonomy (sentence, span, or token level) [2310.10532].
- **Computational/storage cost at soup formation**: Constructing soups requires storage of $N$ full checkpoints and potentially their validation logits, but inference cost remains constant.

## 6. Extensions, Generalizations, and Domain-Specific Constructions

Model soup parameter averaging underpins a series of specialized and generalized frameworks:
- **Domain-adaptive SoE**: "Soup-of-Experts" constructs a large meta-model of $n$ expert offsets and a shared core, producing specialized models for arbitrary domain mixtures by a single linear merge, parameterized by a two-layer MLP mapping domain weights to expert coefficients [2502.01804].
- **Meta-ensembling/“neural averaging”**: AME reframes soup computation as a data-free meta-optimization problem, wherein pseudogradients derived from expert differences guide adaptive neural weight updates, yielding generalizations (multiple epochs, Adam, AdaGrad) that empirically outperform both uniform and greedy soups, especially OOD [2508.14832].
- **Populational regularization**: PAPA exposes a continuous or periodic attraction-to-mean strategy, proactively keeping training trajectories aligned and soup-compatible, increasing robustness against mode collapse and scaling to large populations [2304.03094].
- **Diffusion geometric mixing**: Diffusion Soup leverages the observation that uniform averaging in parameter space approximates geometric mean mixing of constituent data distributions under a first-order NTK approximation, yielding compositional, anti-memorization, and continual learning properties [2406.08431].
- **Alignment, RLHF, and policy anchoring**: SALSA demonstrates that anchoring PPO policy updates to a soup of SFT models relaxes the KL-divergence constraint, enlarges the viable exploration region in policy-space, and yields superior alignment and robustness [2411.01798].

## 7. Recommendations and Best Practices

Deploying model soup parameter averaging requires the following considerations:
- **Initialization and fine-tuning regime**: Use a strong common pretrained initialization and restrict hyperparameter variance sufficiently to maintain basin connectivity.
- **Selection strategy**: Prefer greedy or learned soups over uniform when checkpoint performance varies; validate with a held-out set.
- **BN re-calibration**: For architectures with batch norm, re-compute running statistics post-soup construction.
- **Evaluation budget**: Use RADIN or similar approximate screening to reduce validation cost for large $N$ [2401.17790].
- **Sparse model soups**: Ensure all averaged copies share the target mask to retain sparsity (e.g., via IMP or structured pruning [2306.16788]).
- **Combining with SWA, EMA, or bagging**: These trajectory-based averaging methods are complementary to soup merging of independently trained checkpoints and often yield additive gains [2203.05482], [2304.03094].
- **Domain/task considerations**: Tailor soup construction and selection to downstream tasks, data partitions, and desired specialization/generality ratio—e.g., SoE for rapid domain specialists or Diffusion Soup for continual model composition.

Model soup parameter averaging has evolved from a fine-tuning trick for foundation models into a principled, theoretically grounded methodology for efficient model synthesis, with broad impact across classification, generative modeling, RL, and transfer learning domains. Theoretical advances in meta-ensembling and neural averaging frameworks continue to broaden its applicability and robustness [2508.14832], [2401.17790], [2502.01804].

Source: https://www.emergentmind.com/topics/model-soup-parameter-averaging-be9d926a-ec67-47d6-a8d3-12abcb3b0efb