MonoSoup: Single-Checkpoint Spectral Editing

• MonoSoup is a post-hoc model-editing method that uses spectral decomposition to reweight fine-tuning updates, enhancing both ID and OOD performance.
• It decomposes layer-wise weight updates into high-energy and low-energy components, applying automated layer-wise coefficients to balance task adaptation and robustness.
• MonoSoup avoids the need for multiple checkpoints by leveraging singular value analysis within a single update, streamlining the transition from pre-trained to task-specific models.

Searching arXiv for MonoSoup and closely related model soup work to ground the article in current papers. MonoSoup is a post-hoc model-editing method that seeks to recover much of the in-distribution (ID) / out-of-distribution (OOD) balance associated with multi-checkpoint model soups while using only a single fine-tuned checkpoint together with the original pre-trained model (Abdollahpoorrostam et al., 10 Feb 2026). Rather than averaging weights across many independently fine-tuned models, MonoSoup analyzes the fine-tuning update of one checkpoint layer by layer, decomposes that update spectrally, and reweights its high-energy and low-energy components with automatically determined layer-wise coefficients (Abdollahpoorrostam et al., 10 Feb 2026). In this sense, MonoSoup belongs to the broader family of model soup methods, but it departs from classical weight-space ensembling by replacing cross-checkpoint averaging with within-checkpoint spectral reweighting (Bai et al., 2024, Abdollahpoorrostam et al., 10 Feb 2026). The method is presented as single-checkpoint, post-hoc, data-free, and hyperparameter-free in its automated form, and is evaluated on both CLIP vision models under natural distribution shifts and Qwen LLMs on mathematical reasoning and multiple-choice benchmarks (Abdollahpoorrostam et al., 10 Feb 2026).

1. Conceptual setting and relation to model soups

MonoSoup is motivated by the familiar tension in transfer learning between specialization and robustness. Fine-tuning often improves ID performance on the target distribution, but it can degrade OOD robustness because representations become more tailored to the fine-tuning distribution and forget broader structure acquired during pretraining (Abdollahpoorrostam et al., 10 Feb 2026). The paper frames this explicitly as an ID–OOD tradeoff and positions MonoSoup as a post-hoc remedy that manipulates weights directly rather than retraining (Abdollahpoorrostam et al., 10 Feb 2026).

The immediate predecessor is the model soup paradigm, in which multiple fine-tuned checkpoints from a common initialization are averaged in weight space. In the canonical formulation, if one has fine-tuned weights $\{\bm{\theta}_t\}_{t=1}^T$ from a common initialization $\bm{\theta}_0$, Model Soup forms

$$\bm{\theta} = \frac{1}{T}\sum_{t=1}^T \bm{\theta}_t.$$

The appeal is that the merged model preserves the deployment footprint of a single checkpoint while often improving generalization (Abdollahpoorrostam et al., 10 Feb 2026). Related work on LLM and multimodal merging makes the same point: model soup methods operate in a linear interpolation subspace and can integrate capabilities from different training domains without additional inference cost (Bai et al., 2024). Other studies likewise emphasize that soups improve inference-time performance without increasing inference compute, because the combination occurs in parameter space before deployment rather than in output space during inference (Dansereau et al., 2023).

MonoSoup asks whether this soup-like effect can be recovered when only one fine-tuned checkpoint is available. The paper’s answer is to inspect the internal structure of a single fine-tuning update rather than average across multiple endpoints (Abdollahpoorrostam et al., 10 Feb 2026). This suggests a shift from “checkpoint diversity across runs” to “spectral diversity within one update.” A plausible implication is that MonoSoup can be understood as a single-checkpoint compression of soup behavior: the beneficial alignment-and-denoising effect of classical soups is approximated by layer-wise spectral editing of one task vector (Abdollahpoorrostam et al., 10 Feb 2026).

2. Mathematical formulation

MonoSoup works on the fine-tuning update rather than the raw fine-tuned weights. For each layer $\ell$, if $W^{(\ell)}_{\mathrm{FT}}$ and $W^{(\ell)}_0$ are the fine-tuned and pre-trained weights, respectively, then the layer update is

$$\Delta W^{(\ell)} = W^{(\ell)}_\text{FT} - W^{(\ell)}_0 \in \mathbb{R}^{m \times n}.$$

Equivalently, the paper also writes

$\Delta W = W_{ft} - W_{pt}.$

This isolates the part of the model responsible for adaptation and forgetting, rather than conflating it with the much larger pre-trained backbone (Abdollahpoorrostam et al., 10 Feb 2026).

MonoSoup then applies a singular value decomposition layer by layer:

$$\Delta W^{(\ell)} = \bm{U}^{(\ell)} \bm{\Sigma}^{(\ell)} \bm{V}^{(\ell)\top},$$

where the singular values satisfy $\sigma_1 \geq \sigma_2 \geq \dots$ (Abdollahpoorrostam et al., 10 Feb 2026). The update is split into a high-energy component and a low-energy residual. The high-energy part is

$\bm{\theta}_0$0

and the low-energy part is

$\bm{\theta}_0$1

The paper’s central claim is that the top singular directions capture dominant task adaptation, whereas the low-energy tail, although weaker, can still encode residual signals useful for robustness (Abdollahpoorrostam et al., 10 Feb 2026).

The reweighted MonoSoup update is then defined as

$\bm{\theta}_0$2

The final edited layer is reconstructed as

$\bm{\theta}_0$3

Thus the output is a single dense checkpoint, not an ensemble and not a low-rank adapter (Abdollahpoorrostam et al., 10 Feb 2026).

3. Effective rank and layer-wise coefficient rule

A natural way to choose the split rank $\bm{\theta}_0$4 is an energy-retention threshold. The paper gives

$\bm{\theta}_0$5

This threshold-based variant appears in the paper as MonoSoup $\bm{\theta}_0$6 (Abdollahpoorrostam et al., 10 Feb 2026). However, because $\bm{\theta}_0$7 is a hyperparameter, the automated MonoSoup variant replaces it with entropy-based effective rank.

For layer $\bm{\theta}_0$8, the normalized singular values are

$\bm{\theta}_0$9

and the effective rank is

$$\bm{\theta} = \frac{1}{T}\sum_{t=1}^T \bm{\theta}_t.$$0

This determines the layer-specific partition point between the principal subspace and the residual tail (Abdollahpoorrostam et al., 10 Feb 2026).

The coefficient rule depends on two layer-wise signals. The first is the spectral decay ratio,

$$\bm{\theta} = \frac{1}{T}\sum_{t=1}^T \bm{\theta}_t.$$1

The second is the low-energy fractional energy,

$$\bm{\theta} = \frac{1}{T}\sum_{t=1}^T \bm{\theta}_t.$$2

The exact coefficient rule is

$$\bm{\theta} = \frac{1}{T}\sum_{t=1}^T \bm{\theta}_t.$$3

These equations are the defining MonoSoup reweighting formulas (Abdollahpoorrostam et al., 10 Feb 2026).

The paper states that $$\bm{\theta} = \frac{1}{T}\sum_{t=1}^T \bm{\theta}_t.$$4 is designed to satisfy four boundary conditions: suppression when the spectrum is sharp and misaligned, retention when the spectrum is flat or highly aligned, a spectral baseline $$\bm{\theta} = \frac{1}{T}\sum_{t=1}^T \bm{\theta}_t.$$5 when alignment is zero, and an alignment baseline $$\bm{\theta} = \frac{1}{T}\sum_{t=1}^T \bm{\theta}_t.$$6 when spectral mass is negligible (Abdollahpoorrostam et al., 10 Feb 2026). This suggests that the method is explicitly anisotropic: the update is not scaled by a single global coefficient, but partitioned spectrally and reweighted differently at each layer.

4. Algorithmic procedure and interpretation

The practical MonoSoup procedure can be reconstructed directly from the formulas. Given a pre-trained model and one fine-tuned checkpoint, the method computes the layer-wise update $$\bm{\theta} = \frac{1}{T}\sum_{t=1}^T \bm{\theta}_t.$$7, performs SVD on each layer, chooses a split rank either by the threshold rule or by effective rank, forms $$\bm{\theta} = \frac{1}{T}\sum_{t=1}^T \bm{\theta}_t.$$8 and $$\bm{\theta} = \frac{1}{T}\sum_{t=1}^T \bm{\theta}_t.$$9, computes $\ell$0 and $\ell$1, applies the coefficient rule, reconstructs $\ell$2, and adds the edited update back to the pre-trained layer (Abdollahpoorrostam et al., 10 Feb 2026). There is no retraining and no use of training or validation data in the automated variant (Abdollahpoorrostam et al., 10 Feb 2026).

This distinguishes MonoSoup from several adjacent methods. Unlike Model Soup, it does not average across many fine-tuned models. Unlike EMA or SWA, it does not need a training trajectory. Unlike LoRA, it is not a training-time low-rank parameterization. Unlike Wise-FT, it is not a scalar isotropic interpolation between the pre-trained and fine-tuned endpoints (Abdollahpoorrostam et al., 10 Feb 2026). Wise-FT uses

$\ell$3

whereas MonoSoup applies layer-wise spectral reweighting that treats different subspaces differently (Abdollahpoorrostam et al., 10 Feb 2026).

The broader model soup literature helps clarify why this distinction matters. SoupLM defines a general soup over isomorphic models as

$\ell$4

and then refines this idea by learning per-layer and per-module interpolation coefficients $\ell$5 on a development set (Bai et al., 2024). MonoSoup preserves the post-hoc spirit of such methods but eliminates the need for multiple checkpoints or data-driven coefficient fitting (Abdollahpoorrostam et al., 10 Feb 2026). A plausible implication is that MonoSoup trades the expressivity of multi-checkpoint interpolation for operational simplicity: one fine-tuned model, one pre-trained model, and one spectral editing pass.

5. Empirical results in vision and language

The main vision experiments use CLIP ViT-B/32 fine-tuned on ImageNet-1K and evaluated on ImageNet for ID accuracy and on five natural shifts for OOD robustness: ImageNet-V2, ImageNet-R, ImageNet-Sketch, ImageNet-A, and ObjectNet (Abdollahpoorrostam et al., 10 Feb 2026). The paper uses the 70 CLIP ViT-B/32 fine-tuned checkpoints released by Wortsman et al. for reference baselines such as Uniform Model Soup and Greedy Model Soup, but MonoSoup itself requires only one checkpoint plus the initialization (Abdollahpoorrostam et al., 10 Feb 2026).

The general baselines reported are: Initialization at 75.4% ID and 46.2% OOD, Uniform Model Soup at 79.9% ID and 51.4% OOD, and Greedy Model Soup at 81.0% ID and 50.7% OOD (Abdollahpoorrostam et al., 10 Feb 2026). For representative fine-tuned checkpoints, the paper reports, for example, FT model (OOD$\ell$6) at 78.11% ID and 50.67% OOD, FT model (OOD$\ell$7) at 76.53% ID and 36.71% OOD, FT model (ID$\ell$8) at 80.38% ID and 47.96% OOD, and FT model (ID$\ell$9) at 74.99% ID and 38.64% OOD (Abdollahpoorrostam et al., 10 Feb 2026).

MonoSoup $W^{(\ell)}_{\mathrm{FT}}$0 improves these checkpoints in several cases. For OOD$W^{(\ell)}_{\mathrm{FT}}$1, it gives 78.29% ID and 51.60% OOD; for OOD$W^{(\ell)}_{\mathrm{FT}}$2, 78.55% ID and 44.21% OOD; for ID$W^{(\ell)}_{\mathrm{FT}}$3, 80.03% ID and 49.95% OOD; and for ID$W^{(\ell)}_{\mathrm{FT}}$4, 77.76% ID and 46.54% OOD (Abdollahpoorrostam et al., 10 Feb 2026). The hyperparameter-free MonoSoup variant remains close, with OOD$W^{(\ell)}_{\mathrm{FT}}$5 at 78.21% ID and 50.91% OOD, OOD$W^{(\ell)}_{\mathrm{FT}}$6 at 78.03% ID and 42.78% OOD, ID$W^{(\ell)}_{\mathrm{FT}}$7 at 80.34% ID and 48.94% OOD, and ID$W^{(\ell)}_{\mathrm{FT}}$8 at 77.38% ID and 45.22% OOD (Abdollahpoorrostam et al., 10 Feb 2026).

These results support several conclusions stated in the paper. MonoSoup can exceed Greedy Soup’s 50.7% OOD on the best OOD checkpoint, reaching 51.60% OOD with a single checkpoint (Abdollahpoorrostam et al., 10 Feb 2026). It also strongly repairs weak checkpoints, improving OOD from 36.71% to 44.21% for the worst OOD checkpoint and from 38.64% to 46.54% for the worst ID checkpoint (Abdollahpoorrostam et al., 10 Feb 2026).

In language, the paper evaluates Qwen3-0.6B fine-tuned on a mixture of MetaMathQA, DeepMind AquaRat, OpenBookQA, and SciQ, with evaluation on GSM8K, SciQ, GSM$W^{(\ell)}_{\mathrm{FT}}$9, GSM$W^{(\ell)}_0$0, and MMLU-Pro-Math (Abdollahpoorrostam et al., 10 Feb 2026). For the M-2 (Cosine) setting, the reported sequence is: Standard at 56.1 / 30.8 / 58.5 / 94.5 / 35.9, + LiNeS at 56.5 / 31.4 / 58.9 / 95.2 / 36.1, + MonoSoup $W^{(\ell)}_0$1 at 56.6 / 31.7 / 59.3 / 95.1 / 36.6, and + MonoSoup at 56.8 / 31.9 / 59.4 / 95.3 / 36.9 (Abdollahpoorrostam et al., 10 Feb 2026). The paper summarizes gains as as large as +9.2 points on the harder reasoning tasks relative to the base Qwen3-0.6B reference (Abdollahpoorrostam et al., 10 Feb 2026).

6. Analysis, limitations, and relation to adjacent methods

A central analytical claim is that successful multi-model merging is associated with alignment of fine-tuning updates. Using 70 CLIP ViT-B/32 ImageNet fine-tunings, the paper studies all 2,409 model pairs under Model Stock and reports that successful merges are associated with high alignment of fine-tuning updates, measured by cosine similarity (Abdollahpoorrostam et al., 10 Feb 2026). MonoSoup extends this intuition from “alignment across checkpoints” to “structure within one checkpoint.” Dominant aligned directions are approximated by the high-energy spectral subspace, while robustness-relevant residual structure is retained through controlled reweighting of the low-energy tail (Abdollahpoorrostam et al., 10 Feb 2026).

This emphasis on geometry places MonoSoup within a broader landscape of soup-related methods. SoupLM focuses on interpolation between compatible LLM and multimodal checkpoints descended from a common base, with isomorphic structures and fixed non-shared modality components (Bai et al., 2024). MEHL-Soup addresses the memory and graph-construction cost of learned soup by formulating the problem as hyperplane optimization and applying block coordinate gradient descent to the mixing coefficients, thereby making learned soups practical on a single GPU (Li et al., 2024). These methods still assume multiple checkpoints. MonoSoup removes that assumption and instead treats a single task vector as the object to be decomposed and reweighted (Abdollahpoorrostam et al., 10 Feb 2026).

The paper also contrasts MonoSoup with direct singular value truncation. In small task-arithmetic settings, retaining only a small number of singular values can work, but in the paper’s main setting of ImageNet plus natural shifts, removing low-energy directions hurts both ID and OOD even when preserving 95% of spectral energy (Abdollahpoorrostam et al., 10 Feb 2026). This is why MonoSoup reweights rather than discards the tail. An ablation shows that using only $W^{(\ell)}_0$2 tends to preserve ID but gives little OOD gain, using only $W^{(\ell)}_0$3 can improve OOD on weak checkpoints but may reduce ID, and combining both is best and most stable (Abdollahpoorrostam et al., 10 Feb 2026).

The method has clear limitations. The conclusion mentions a limitation tied to the variance-retention threshold $W^{(\ell)}_0$4, although the paper also introduces the effective-rank-based hyperparameter-free version precisely to address this (Abdollahpoorrostam et al., 10 Feb 2026). More generally, MonoSoup depends on the assumption that the layer-wise update admits a meaningful high/low spectral partition. This suggests that effectiveness may vary with architecture or fine-tuning regime. A plausible implication is that MonoSoup is strongest when the fine-tuning update is structured enough for SVD-based decomposition to separate concentrated adaptation from residual robustness-preserving directions.

Common misconceptions can be addressed directly. MonoSoup is not a classical model soup in the sense of averaging many independently trained models; the title itself makes the point that “model soups need only one ingredient” (Abdollahpoorrostam et al., 10 Feb 2026). Nor is it a uniform interpolation like Wise-FT, since appendix analysis shows substantial separation between $W^{(\ell)}_0$5 and $W^{(\ell)}_0$6 across layers, with a mean gap of about 0.35 (Abdollahpoorrostam et al., 10 Feb 2026). It is therefore more accurate to regard MonoSoup as a layer-wise anisotropic spectral editing method within the broader post-hoc soup family.

Taken together, the available evidence positions MonoSoup as a single-checkpoint counterpart to multi-checkpoint soup methods: it preserves the one-model deployment footprint emphasized across the model soup literature, but substitutes spectral reweighting of a single update for validation-guided averaging across many checkpoints (Dansereau et al., 2023, Bai et al., 2024, Abdollahpoorrostam et al., 10 Feb 2026). This suggests a broader taxonomy in which classical soups exploit diversity across checkpoints, learned soups optimize coefficients in a checkpoint-defined interpolation subspace, and MonoSoup exploits diversity across singular directions inside one checkpoint update.