---
title: 'MonoSoup: Single-Checkpoint Spectral Editing'
url: https://www.emergentmind.com/topics/monosoup
type: topic
---

# MonoSoup: Single-Checkpoint Spectral Editing

Searching arXiv for MonoSoup and closely related model soup work to ground the article in current papers.
MonoSoup is a post-hoc model-editing method that seeks to recover much of the in-distribution (ID) / out-of-distribution (OOD) balance associated with multi-checkpoint model soups while using only a single fine-tuned checkpoint together with the original pre-trained model [2602.09689]. Rather than averaging weights across many independently fine-tuned models, MonoSoup analyzes the fine-tuning update of one checkpoint layer by layer, decomposes that update spectrally, and reweights its high-energy and low-energy components with automatically determined layer-wise coefficients [2602.09689]. In this sense, MonoSoup belongs to the broader family of model soup methods, but it departs from classical weight-space ensembling by replacing cross-checkpoint averaging with within-checkpoint spectral reweighting [2407.08196][2602.09689]. The method is presented as single-checkpoint, post-hoc, data-free, and hyperparameter-free in its automated form, and is evaluated on both CLIP vision models under natural distribution shifts and Qwen language models on mathematical reasoning and multiple-choice benchmarks [2602.09689].

## 1. Conceptual setting and relation to model soups

MonoSoup is motivated by the familiar tension in transfer learning between specialization and robustness. Fine-tuning often improves ID performance on the target distribution, but it can degrade OOD robustness because representations become more tailored to the fine-tuning distribution and forget broader structure acquired during pretraining [2602.09689]. The paper frames this explicitly as an ID–OOD tradeoff and positions MonoSoup as a post-hoc remedy that manipulates weights directly rather than retraining [2602.09689].

The immediate predecessor is the model soup paradigm, in which multiple fine-tuned checkpoints from a common initialization are averaged in weight space. In the canonical formulation, if one has fine-tuned weights $\{\bm{\theta}_t\}_{t=1}^T$ from a common initialization $\bm{\theta}_0$, Model Soup forms
$$
\bm{\theta} = \frac{1}{T}\sum_{t=1}^T \bm{\theta}_t.
$$
The appeal is that the merged model preserves the deployment footprint of a single checkpoint while often improving generalization [2602.09689]. Related work on LLM and multimodal merging makes the same point: model soup methods operate in a linear interpolation subspace and can integrate capabilities from different training domains without additional inference cost [2407.08196]. Other studies likewise emphasize that soups improve inference-time performance without increasing inference compute, because the combination occurs in parameter space before deployment rather than in output space during inference [2301.10092].

MonoSoup asks whether this soup-like effect can be recovered when only one fine-tuned checkpoint is available. The paper’s answer is to inspect the internal structure of a single fine-tuning update rather than average across multiple endpoints [2602.09689]. This suggests a shift from “checkpoint diversity across runs” to “spectral diversity within one update.” A plausible implication is that MonoSoup can be understood as a single-checkpoint compression of soup behavior: the beneficial alignment-and-denoising effect of classical soups is approximated by layer-wise spectral editing of one task vector [2602.09689].

## 2. Mathematical formulation

MonoSoup works on the fine-tuning update rather than the raw fine-tuned weights. For each layer $\ell$, if $W^{(\ell)}_{\mathrm{FT}}$ and $W^{(\ell)}_0$ are the fine-tuned and pre-trained weights, respectively, then the layer update is
$$
\Delta W^{(\ell)} = W^{(\ell)}_\text{FT} - W^{(\ell)}_0 \in \mathbb{R}^{m \times n}.
$$
Equivalently, the paper also writes
$$
\Delta W = W_{ft} - W_{pt}.
$$
This isolates the part of the model responsible for adaptation and forgetting, rather than conflating it with the much larger pre-trained backbone [2602.09689].

MonoSoup then applies a singular value decomposition layer by layer:
$$
\Delta W^{(\ell)} = \bm{U}^{(\ell)} \bm{\Sigma}^{(\ell)} \bm{V}^{(\ell)\top},
$$
where the singular values satisfy $\sigma_1 \geq \sigma_2 \geq \dots$ [2602.09689]. The update is split into a high-energy component and a low-energy residual. The high-energy part is
$$
\Delta W^{(\ell)}_{\text{High}} = \sum_{i \le k} \sigma_i^{(\ell)} u_i^{(\ell)} v_i^{(\ell)\top},
$$
and the low-energy part is
$$
\Delta W^{(\ell)}_{\text{Low}} = \Delta W^{(\ell)} - \Delta W^{(\ell)}_{\text{High}}.
$$
The paper’s central claim is that the top singular directions capture dominant task adaptation, whereas the low-energy tail, although weaker, can still encode residual signals useful for robustness [2602.09689].

The reweighted MonoSoup update is then defined as
$$
\Delta W_{\text{MonoSoup}}^{(\ell)} = \lambda_{\text{High}}^{(\ell)}\,\Delta W_{\text{High}}^{(\ell)} + \lambda_{\text{Low}}^{(\ell)}\,\Delta W_{\text{Low}}^{(\ell)}.
$$
The final edited layer is reconstructed as
$$
W_{\text{final}}^{(\ell)} = W_0^{(\ell)} + \Delta W_{\text{MonoSoup}}^{(\ell)}.
$$
Thus the output is a single dense checkpoint, not an ensemble and not a low-rank adapter [2602.09689].

## 3. Effective rank and layer-wise coefficient rule

A natural way to choose the split rank $k$ is an energy-retention threshold. The paper gives
$$
k = \arg\min_{j} \left\{ j \;\middle|\; \frac{\sum_{i=1}^j \sigma_i^2}{\sum_{i=1}^{\min(m, n)} \sigma_i^2} \geq R \right\}. \tag{1}
$$
This threshold-based variant appears in the paper as MonoSoup $(R=0.8)$ [2602.09689]. However, because $R$ is a hyperparameter, the automated MonoSoup variant replaces it with entropy-based effective rank.

For layer $\ell$, the normalized singular values are
$$
p_i^{(\ell)} = \frac{\sigma_i^{(\ell)}}{\sum_j \sigma_j^{(\ell)}},
$$
and the effective rank is
$$
k^{(\ell)} = \bigg\lceil \exp \bigg( -\sum_{i} p_i^{(\ell)} \ln p_i^{(\ell)} \bigg) \bigg\rceil. \tag{6}
$$
This determines the layer-specific partition point between the principal subspace and the residual tail [2602.09689].

The coefficient rule depends on two layer-wise signals. The first is the spectral decay ratio,
$$
\rho^{(\ell)} = \left(\frac{\sigma_{k+1}\left(\Delta W^{(\ell)}\right)}{\sigma_{1}\left(\Delta W^{(\ell)}\right)}\right)^2. \tag{3}
$$
The second is the low-energy fractional energy,
$$
\cos ^2 \alpha^{(\ell)}=\frac{\left\|\Delta W_{\mathrm{Low}}^{(\ell)}\right\|_F^2}{\left\|\Delta W^{(\ell)}\right\|_F^2} \in[0,1]. \tag{4}
$$
The exact coefficient rule is
$$
\lambda_{\text{Low}}^{(\ell)} = \rho^{(\ell)} + \big(1 - \rho^{(\ell)}\big)\, \cos \alpha^{(\ell)}, \qquad \lambda_{\text{High}}^{(\ell)} = 1 - \lambda_{\text{Low}}^{(\ell)}. \tag{5}
$$
These equations are the defining MonoSoup reweighting formulas [2602.09689].

The paper states that $\lambda_{\text{Low}}^{(\ell)}$ is designed to satisfy four boundary conditions: suppression when the spectrum is sharp and misaligned, retention when the spectrum is flat or highly aligned, a spectral baseline $\lambda_{\text{Low}}=\rho$ when alignment is zero, and an alignment baseline $\lambda_{\text{Low}}=\cos\alpha$ when spectral mass is negligible [2602.09689]. This suggests that the method is explicitly anisotropic: the update is not scaled by a single global coefficient, but partitioned spectrally and reweighted differently at each layer.

## 4. Algorithmic procedure and interpretation

The practical MonoSoup procedure can be reconstructed directly from the formulas. Given a pre-trained model and one fine-tuned checkpoint, the method computes the layer-wise update $\Delta W^{(\ell)}$, performs SVD on each layer, chooses a split rank either by the threshold rule or by effective rank, forms $\Delta W_{\text{High}}^{(\ell)}$ and $\Delta W_{\text{Low}}^{(\ell)}$, computes $\rho^{(\ell)}$ and $\cos\alpha^{(\ell)}$, applies the coefficient rule, reconstructs $\Delta W_{\text{MonoSoup}}^{(\ell)}$, and adds the edited update back to the pre-trained layer [2602.09689]. There is no retraining and no use of training or validation data in the automated variant [2602.09689].

This distinguishes MonoSoup from several adjacent methods. Unlike Model Soup, it does not average across many fine-tuned models. Unlike EMA or SWA, it does not need a training trajectory. Unlike LoRA, it is not a training-time low-rank parameterization. Unlike Wise-FT, it is not a scalar isotropic interpolation between the pre-trained and fine-tuned endpoints [2602.09689]. Wise-FT uses
$$
\theta_{\text{wise}} = (1-\lambda)\theta_0 + \lambda \theta_t,
$$
whereas MonoSoup applies layer-wise spectral reweighting that treats different subspaces differently [2602.09689].

The broader model soup literature helps clarify why this distinction matters. SoupLM defines a general soup over isomorphic models as
$$
f(\theta^s) = \sum_{i=1}^n \alpha^i \theta^i,\qquad \sum_{i=1}^n \alpha^i = 1,
$$
and then refines this idea by learning per-layer and per-module interpolation coefficients $\alpha_{[s,l]}$ on a development set [2407.08196]. MonoSoup preserves the post-hoc spirit of such methods but eliminates the need for multiple checkpoints or data-driven coefficient fitting [2602.09689]. A plausible implication is that MonoSoup trades the expressivity of multi-checkpoint interpolation for operational simplicity: one fine-tuned model, one pre-trained model, and one spectral editing pass.

## 5. Empirical results in vision and language

The main vision experiments use CLIP ViT-B/32 fine-tuned on ImageNet-1K and evaluated on ImageNet for ID accuracy and on five natural shifts for OOD robustness: ImageNet-V2, ImageNet-R, ImageNet-Sketch, ImageNet-A, and ObjectNet [2602.09689]. The paper uses the 70 CLIP ViT-B/32 fine-tuned checkpoints released by Wortsman et al. for reference baselines such as Uniform Model Soup and Greedy Model Soup, but MonoSoup itself requires only one checkpoint plus the initialization [2602.09689].

The general baselines reported are: Initialization at **75.4% ID** and **46.2% OOD**, Uniform Model Soup at **79.9% ID** and **51.4% OOD**, and Greedy Model Soup at **81.0% ID** and **50.7% OOD** [2602.09689]. For representative fine-tuned checkpoints, the paper reports, for example, FT model (OOD$^+$) at **78.11% ID** and **50.67% OOD**, FT model (OOD$^-$) at **76.53% ID** and **36.71% OOD**, FT model (ID$^+$) at **80.38% ID** and **47.96% OOD**, and FT model (ID$^-$) at **74.99% ID** and **38.64% OOD** [2602.09689].

MonoSoup $(R=0.8)$ improves these checkpoints in several cases. For OOD$^+$, it gives **78.29% ID** and **51.60% OOD**; for OOD$^-$, **78.55% ID** and **44.21% OOD**; for ID$^+$, **80.03% ID** and **49.95% OOD**; and for ID$^-$, **77.76% ID** and **46.54% OOD** [2602.09689]. The hyperparameter-free MonoSoup variant remains close, with OOD$^+$ at **78.21% ID** and **50.91% OOD**, OOD$^-$ at **78.03% ID** and **42.78% OOD**, ID$^+$ at **80.34% ID** and **48.94% OOD**, and ID$^-$ at **77.38% ID** and **45.22% OOD** [2602.09689].

These results support several conclusions stated in the paper. MonoSoup can exceed Greedy Soup’s **50.7% OOD** on the best OOD checkpoint, reaching **51.60% OOD** with a single checkpoint [2602.09689]. It also strongly repairs weak checkpoints, improving OOD from **36.71%** to **44.21%** for the worst OOD checkpoint and from **38.64%** to **46.54%** for the worst ID checkpoint [2602.09689].

In language, the paper evaluates Qwen3-0.6B fine-tuned on a mixture of MetaMathQA, DeepMind AquaRat, OpenBookQA, and SciQ, with evaluation on GSM8K, SciQ, GSM$_{\text{Plus}}$, GSM$_{\text{Plat}}$, and MMLU-Pro-Math [2602.09689]. For the M-2 (Cosine) setting, the reported sequence is: Standard at **56.1 / 30.8 / 58.5 / 94.5 / 35.9**, + LiNeS at **56.5 / 31.4 / 58.9 / 95.2 / 36.1**, + MonoSoup $(R=0.8)$ at **56.6 / 31.7 / 59.3 / 95.1 / 36.6**, and + MonoSoup at **56.8 / 31.9 / 59.4 / 95.3 / 36.9** [2602.09689]. The paper summarizes gains as as large as **+9.2 points** on the harder reasoning tasks relative to the base Qwen3-0.6B reference [2602.09689].

## 6. Analysis, limitations, and relation to adjacent methods

A central analytical claim is that successful multi-model merging is associated with alignment of fine-tuning updates. Using 70 CLIP ViT-B/32 ImageNet fine-tunings, the paper studies all 2,409 model pairs under Model Stock and reports that successful merges are associated with high alignment of fine-tuning updates, measured by cosine similarity [2602.09689]. MonoSoup extends this intuition from “alignment across checkpoints” to “structure within one checkpoint.” Dominant aligned directions are approximated by the high-energy spectral subspace, while robustness-relevant residual structure is retained through controlled reweighting of the low-energy tail [2602.09689].

This emphasis on geometry places MonoSoup within a broader landscape of soup-related methods. SoupLM focuses on interpolation between compatible LLM and multimodal checkpoints descended from a common base, with isomorphic structures and fixed non-shared modality components [2407.08196]. MEHL-Soup addresses the memory and graph-construction cost of learned soup by formulating the problem as hyperplane optimization and applying block coordinate gradient descent to the mixing coefficients, thereby making learned soups practical on a single GPU [2407.03641]. These methods still assume multiple checkpoints. MonoSoup removes that assumption and instead treats a single task vector as the object to be decomposed and reweighted [2602.09689].

The paper also contrasts MonoSoup with direct singular value truncation. In small task-arithmetic settings, retaining only a small number of singular values can work, but in the paper’s main setting of ImageNet plus natural shifts, removing low-energy directions hurts both ID and OOD even when preserving **95%** of spectral energy [2602.09689]. This is why MonoSoup reweights rather than discards the tail. An ablation shows that using only $\rho^\ell$ tends to preserve ID but gives little OOD gain, using only $\cos\alpha^\ell$ can improve OOD on weak checkpoints but may reduce ID, and combining both is best and most stable [2602.09689].

The method has clear limitations. The conclusion mentions a limitation tied to the variance-retention threshold $R$, although the paper also introduces the effective-rank-based hyperparameter-free version precisely to address this [2602.09689]. More generally, MonoSoup depends on the assumption that the layer-wise update admits a meaningful high/low spectral partition. This suggests that effectiveness may vary with architecture or fine-tuning regime. A plausible implication is that MonoSoup is strongest when the fine-tuning update is structured enough for SVD-based decomposition to separate concentrated adaptation from residual robustness-preserving directions.

Common misconceptions can be addressed directly. MonoSoup is not a classical model soup in the sense of averaging many independently trained models; the title itself makes the point that “model soups need only one ingredient” [2602.09689]. Nor is it a uniform interpolation like Wise-FT, since appendix analysis shows substantial separation between $\lambda_\text{High}^{(\ell)}$ and $\lambda_\text{Low}^{(\ell)}$ across layers, with a mean gap of about **0.35** [2602.09689]. It is therefore more accurate to regard MonoSoup as a layer-wise anisotropic spectral editing method within the broader post-hoc soup family.

Taken together, the available evidence positions MonoSoup as a single-checkpoint counterpart to multi-checkpoint soup methods: it preserves the one-model deployment footprint emphasized across the model soup literature, but substitutes spectral reweighting of a single update for validation-guided averaging across many checkpoints [2301.10092][2407.08196][2602.09689]. This suggests a broader taxonomy in which classical soups exploit diversity across checkpoints, learned soups optimize coefficients in a checkpoint-defined interpolation subspace, and MonoSoup exploits diversity across singular directions inside one checkpoint update.

Source: https://www.emergentmind.com/topics/monosoup