---
title: Rank-1 Subspace in Model Merging
url: https://www.emergentmind.com/topics/rank-1-subspace-of-model-merging
type: topic
---

# Rank-1 Subspace in Model Merging

Searching arXiv for the requested paper and closely related work on rank-1/subspace model merging.
Model merging in late-stage language-model pre-training can exhibit a distinctive geometric regularity in which averaged checkpoints, despite being formed from a raw trajectory that oscillates across many directions, collapse onto an approximately one-dimensional manifold. This phenomenon is termed the **Rank-1 Subspace** in "Extra-Merge: Tracing the Rank-1 Subspace of Model Merging in Language Model Pre-Training" [2605.26484]. In that formulation, the merged trajectory exposes a single dominant descent direction that can be extracted by PCA and used for training-free extrapolation. In the broader model-merging literature, related rank-1 or low-rank views appear in analyses of task-vector spectra, spectral over-accumulation, curvature-aware merging, multimodal subspace alignment, and subspace-level reasoning injection, but the meaning of “rank-1 subspace” differs materially across these settings [2506.16506].

## 1. Definition in late-stage pre-training

In the pre-training setting studied by Extra-Merge, one begins with a sequence of saved checkpoints $\{w_0,w_1,\dots,w_T\}$ and forms merged checkpoints by averaging the last $N$ states. The merged state at step $t$ is defined as
$$
\bar w_t = \frac{1}{N}\sum_{i=0}^{N-1} w_{t-iT}.
$$
From a window of $K$ consecutive merged checkpoints, one constructs the centered matrix
$$
W = [\bar w_{t-K+1}-\mu,\dots,\bar w_t-\mu]\in\mathbb R^{d\times K},\qquad
\mu=\frac{1}{K}\sum_{i=0}^{K-1}\bar w_{t-i}.
$$
PCA of this matrix yields eigenvalues $\lambda_1\ge\lambda_2\ge\dots\ge\lambda_K$, and the Rank-1 Subspace is defined as the span of the first principal component $u_1$, with explained-variance ratio
$$
R_1=\frac{\lambda_1}{\sum_{j=1}^K\lambda_j}\gg 0.94.
$$
Equivalently, the merged checkpoints lie approximately on the one-dimensional linear manifold
$$
M_1=\{\mu+\alpha u_1:\alpha\in\mathbb R\}
$$
[2605.26484].

This definition is specific to a sequence of *merged checkpoints* produced during a single training run. It is therefore distinct from rank-1 notions based on SVD of task vectors or weight-difference matrices in fine-tuning-based model merging. A plausible implication is that “rank-1 subspace” in current model-merging research is not a universal object but a family of closely related low-dimensional structures whose interpretation depends on whether the underlying objects are trajectories, task matrices, or layerwise updates.

## 2. Geometric observation: raw oscillations and merged rectification

The empirical motivation for the pre-training Rank-1 Subspace comes from a contrast between raw and merged trajectories. When one linearly interpolates between two consecutive raw checkpoints $w_t\to w_{t+T}$, the loss forms a U-shaped basin whose midpoint is lower than both endpoints. This is interpreted as evidence that raw SGD “bounces” across valley walls rather than descending smoothly. By contrast, interpolation between merged checkpoints $\bar w_t\to\bar w_{t+T}$ yields a strictly monotonic descent [2605.26484].

PCA on short segments with $K=5$ across GPT-2 and LLaMA models further sharpens this contrast. Raw checkpoints distribute variance across multiple principal components, with $R_1\approx 0.4$–$0.6$, whereas merged checkpoints satisfy $R_1>0.94$. Projected onto the leading direction $u_1$, raw trajectories remain non-monotonic, while merged projections evolve strictly monotonically. The paper characterizes this as spectral concentration together with trajectory rectification: averaging suppresses high-frequency oscillations and reveals a one-dimensional descent path [2605.26484].

This usage of rank-1 is trajectory-centric rather than update-centric. It does not assert that the full optimization path is intrinsically one-dimensional; rather, it states that the *merged* late-stage path becomes almost one-dimensional after averaging. This suggests that the observed low-dimensionality is a property of the averaging operator acting on a particular training regime, not merely a property of the underlying optimizer state.

## 3. River-valley analysis and PCA recovery

Extra-Merge provides a theoretical account through a “river-valley” landscape analysis. Near late-stage pre-training, the loss is modeled as
$$
L(w)\approx \ell(v^T(w-w^*))+\frac{1}{2}\|H^{1/2}P_s(w-w^*)\|^2,
$$
where $v\in\mathbb R^d$ is the flat river direction, $P_f=vv^T$, $P_s=I-P_f$ is the sharp subspace, $H\succeq 0$ has eigenvalues in $[\lambda_{\min},\lambda_{\max}]$ on $\mathrm{range}(P_s)$, and $\ell(\cdot)$ is the smooth one-dimensional loss along $v$ [2605.26484].

Under decoupled SGD dynamics with step-size $\eta$,
$$
t_{k+1}=t_k-\eta\,\ell'(t_k)+\text{small drift},
\qquad
z_{k+1}=(I-\eta H)z_k+\eta g_k,\quad g_k\sim N(0,\sigma^2P_s),
$$
the sharp directions behave as noisy high-curvature fluctuations, whereas the river direction carries the underlying descent signal. Theorem 5.3 shows that averaging acts as a geometric low-pass filter: for the merged model $\bar w=(1/N)\sum w_{k+iT}$, the expected squared deviation from the river satisfies an upper bound that decays as $O(1/N)$ when $(1-\eta\lambda_j)^T\ll 1$. In the paper’s interpretation, high-curvature mountain fluctuations are exponentially damped, so merging projects the trajectory onto the low-curvature river subspace [2605.26484].

Theorem 5.4 then addresses why PCA identifies the correct direction. The sample covariance of a sliding window of merged checkpoints takes the form
$$
\Sigma=\sigma_{\text{signal}}\,vv^T+\Sigma_{\text{noise}},
$$
with $\sigma_{\text{signal}}$ scaling like $(KT)^2$ and the top eigenvalue of $\Sigma_{\text{noise}}$ scaling like $O(1/N)$. If the signal-to-noise gap $\delta=\sigma_{\text{signal}}-\lambda_{\max}(\Sigma_{\text{noise}})>0$, then $v$ is the unique top eigenvector, and the empirical estimate $\hat u_1$ satisfies
$$
|\hat u_1^T v|^2 \ge 1 - O(\|\Sigma-\hat\Sigma\|_{\mathrm{op}}/\delta)
$$
with high probability. In this framework, PCA on merged checkpoints does not merely summarize the path; it recovers the river tangent itself [2605.26484].

A related but conceptually different geometric treatment appears in "Model Merging on Loss Landscape: A Geometry Perspective" [2605.26693], where merging is posed as a Fréchet-mean problem on a Riemannian manifold equipped with the expected-Hessian metric. In the two-model rank-1 case, the merge is restricted to $\mathrm{Span}\{u\}$ with $u=(\delta_2-\delta_1)/\|\delta_2-\delta_1\|$, and the curvature-aware solution is guaranteed to tie or improve upon flat interpolation whenever the projected curvatures differ. The shared theme is that a one-dimensional subspace can suffice when it captures the dominant geometry, but the underlying object is different: Extra-Merge tracks a pre-training trajectory, whereas EpiMer restricts merging of task vectors between fine-tuned models [2605.26693].

## 4. Extra-Merge algorithm

Extra-Merge is a training-free strategy that exploits the Rank-1 Subspace by extrapolating beyond the last merged checkpoint along the PCA direction. Its inputs are merged checkpoints $\{\bar w_{t-N+1},\dots,\bar w_t\}$, a PCA window $K$, and base fraction $a$ with default $0.1$ [2605.26484].

The first stage extracts the direction. One forms the centered matrix $W\in\mathbb R^{d\times K}$ from $\{\bar w_{t-K+1},\dots,\bar w_t\}$ and computes the top principal component $\hat u_1$ via Gram-matrix PCA. The direction is then oriented using
$$
\mathrm{sign}=\mathrm{sgn}\big((\bar w_t-\bar w_{t-1})^T\hat u_1\big),
$$
and one sets $\hat d=\mathrm{sign}\cdot \hat u_1$ [2605.26484].

The second stage performs line-search extrapolation. Let
$$
\Delta z=(\bar w_t-\bar w_{t-1})^T\hat d,
\qquad
\gamma=a\,|\Delta z|.
$$
Candidates are generated as
$$
w^{(k)}=\bar w_t+k\gamma\hat d,\qquad k=1,2,\dots
$$
The loss $\ell(w^{(k)})$ is evaluated, the procedure stops at the first $k$ where loss increases, and the algorithm returns the candidate with minimal loss. In compact form,
$$
w_{\text{extra}}=\arg\min_{k\ge 1}\ell(\bar w_t+k\gamma\hat d),
\qquad
\gamma=a\,|(\bar w_t-\bar w_{t-1})^T\hat d|.
$$
No additional gradient steps are required. The practical hyperparameters reported are merging interval $T=500$ for GPT-2 small/med or $1{,}000$ for larger models, window $N=8$–$10$, PCA window $K=4$, and base fraction $a=0.1$ [2605.26484].

The method depends on the empirical premise that merged checkpoints already lie near a one-dimensional manifold. If that premise fails, the extrapolation direction would no longer have the interpretation of a dominant descent tangent. This clarifies a common misconception: Extra-Merge is not a generic linear extrapolation heuristic applied to arbitrary checkpoints; it is specifically justified by the observed collapse of merged late-stage trajectories onto a Rank-1 Subspace.

## 5. Experimental profile

The paper reports results across GPT-2 and LLaMA families from 124M to 2B parameters. On validation loss, uniform averaging (PMA) lowers loss relative to raw checkpoints during high-learning-rate phases but converges back to the baseline as the learning rate decays, whereas Extra-Merge yields a further loss reduction of approximately $0.008$–$0.012$ across all model scales and schedulers, including WSD for GPT-2 and Cosine for LLaMA [2605.26484].

On zero-shot downstream evaluation with Pythia-12B, using 11 checkpoints from 130k to 140k steps and uniform or EMA averaging baselines, Extra-Merge achieves an average accuracy of $57.47\%$ versus $56.88\%$ for the raw model and $57.07\%$ for PMA. Reported per-task improvements are $+1.10\%$ on ARC-Challenge, $+0.84\%$ on ARC-Easy, $+0.16\%$ on HellaSwag, and $+0.26\%$ on PIQA [2605.26484].

The method also generalizes to the Muon optimizer. On GPT-2 Small trained with Muon, Extra-Merge improves upon PMA by approximately $0.009$ in validation loss, despite the optimizer’s orthogonal update rule producing a very different trajectory [2605.26484]. The claim is therefore optimizer-agnostic robustness at least across the tested setting.

These results indicate that the rank-1 phenomenon is not confined to a single architecture family or one optimizer. A plausible implication is that the low-dimensional structure emerges from late-stage optimization geometry together with checkpoint averaging rather than from a peculiar artifact of a single implementation.

## 6. Related rank-1 and subspace perspectives in model merging

The broader literature uses “rank-1 subspace” in several non-identical ways. In "Subspace-Boosted Model Merging" [2506.16506], stacking task vectors from multiple experts into a task matrix reveals **rank collapse** as more experts are merged: the leading singular value grows while the tail singular values shrink, and in ViT-B/16 vision experiments the stable rank of each layer’s $\Delta$ drops from $\sim 25$ to $<5$ when merging 14 experts by naive task arithmetic. The dominant rank-1 subspace $\mathrm{Span}\{u_1\}$ is interpreted as a shared component across tasks, but excessive concentration onto that direction causes loss of task-specific diversity. Subspace Boosting preserves the top singular direction while restoring higher-rank support by boosting the collapsed tail singular values [2506.16506].

In "When Shared Knowledge Hurts: Spectral Over-Accumulation in Model Merging" [2602.05536], each rank-1 term $\sigma^r u^r(v^r)^T$ in the SVD of an update matrix defines a spectral component, and over-accumulation occurs when multiple tasks share aligned singular vectors so that singular values add under linear merging. Singular Value Calibration rescales the inflated singular values to restore a balanced spectrum and improves Task Arithmetic by $13.0\%$ while also yielding gains on language benchmarks [2602.05536]. Here the central problem is not collapse of a trajectory onto one dimension, but repeated counting of shared rank-1 directions across tasks.

Other works operationalize rank-1 or low-rank subspaces differently. FRISM decomposes an LRM task vector into rank-1 SVD components and learns subspace-wise coefficients $\alpha_i$ for fine-grained reasoning injection into VLMs, rather than using a single layer-level scalar [2601.21187]. SSAM identifies a shared low-rank subspace for language-related parameter updates across multimodal specialists and, in a rank-1 specialization, projects two updates onto the top singular vector of their concatenation before balancing aligned and residual components with a scalar $\alpha$ [2603.21584]. SCORE studies domain generalization by extracting leading singular directions from domain-specific deltas, building a shared orthogonal basis from these directions, and pruning off-diagonal conflicts after projection into that basis [2603.05768]. The thesis "Model Merging: Foundations and Algorithms" develops yet another perspective in which the rank-1 approximation $\widehat\Delta_{t,1}^{(\ell)}=\sigma_{t,1}^{(\ell)}u_{t,1}^{(\ell)}(v_{t,1}^{(\ell)})^T$ is the optimal one-dimensional Frobenius approximation to a layerwise task vector and forms the basis of TSV-Merge [2605.01580].

Taken together, these works show that rank-1 structure can be either beneficial or pathological depending on context. In Extra-Merge, the near-1D merged trajectory is exploited as a descent signal; in Subspace-Boosting and SVC, dominance of a leading direction is precisely what must be counteracted to preserve diversity or avoid over-counting.

## 7. Conceptual significance and limitations

The principal significance of the Rank-1 Subspace in pre-training is explanatory. Standard checkpoint averaging methods such as SWA, LAWA, and PMA had been known to improve validation loss, but the geometric reason had remained unclear. Extra-Merge interprets averaging as a mechanism that filters out high-curvature oscillations and reveals the optimal descent direction in a river-valley landscape, thereby making the success of merging intelligible in terms of geometry rather than only empirical regularization [2605.26484].

At the same time, the topic admits several possible misunderstandings. First, the phrase “rank-1 subspace” does not imply that all of model merging is universally one-dimensional. In the pre-training case, the claim concerns windows of *merged late-stage checkpoints*; in fine-tuning-based merging, the relevant rank-1 objects are often top singular components of task matrices or weight deltas. Second, a dominant rank-1 direction is not always desirable. Subspace-Boosted Model Merging explicitly argues that further collapse beyond the leading shared component sacrifices task-specific diversity, and SVC argues that repeated accumulation of shared spectral directions biases the merged model toward common subspaces [2506.16506].

The current literature therefore supports a nuanced view. Rank-1 structure can expose the principal geometry of descent, provide an efficient merge parameterization, or reveal the shared component of multiple tasks. It can also indicate collapse, interference, or over-counting. The specific contribution of Extra-Merge is to isolate a case in which the rank-1 phenomenon is directly actionable: late-stage checkpoint averaging in language-model pre-training produces a stable approximately one-dimensional manifold, and extrapolating along that manifold can reduce loss without further optimization [2605.26484].

Source: https://www.emergentmind.com/topics/rank-1-subspace-of-model-merging