Rank-1 Subspace in Model Merging

• Rank-1 Subspace is characterized by merging late-stage checkpoints to reveal a one-dimensional manifold via PCA with an explained-variance ratio exceeding 0.94.
• Averaging raw checkpoints suppresses high-frequency oscillations and rectifies the trajectory, directly exposing the dominant descent direction used for training-free extrapolation.
• The framework highlights how geometric regularity in model merging varies across training and fine-tuning, informing curvature-aware and task-specific merging strategies.

Searching arXiv for the requested paper and closely related work on rank-1/subspace model merging. Model merging in late-stage language-model pre-training can exhibit a distinctive geometric regularity in which averaged checkpoints, despite being formed from a raw trajectory that oscillates across many directions, collapse onto an approximately one-dimensional manifold. This phenomenon is termed the Rank-1 Subspace in "Extra-Merge: Tracing the Rank-1 Subspace of Model Merging in LLM Pre-Training" (Zhou et al., 26 May 2026). In that formulation, the merged trajectory exposes a single dominant descent direction that can be extracted by PCA and used for training-free extrapolation. In the broader model-merging literature, related rank-1 or low-rank views appear in analyses of task-vector spectra, spectral over-accumulation, curvature-aware merging, multimodal subspace alignment, and subspace-level reasoning injection, but the meaning of “rank-1 subspace” differs materially across these settings (Skorobogat et al., 19 Jun 2025).

1. Definition in late-stage pre-training

In the pre-training setting studied by Extra-Merge, one begins with a sequence of saved checkpoints $\{w_0,w_1,\dots,w_T\}$ and forms merged checkpoints by averaging the last $N$ states. The merged state at step $t$ is defined as

$\bar w_t = \frac{1}{N}\sum_{i=0}^{N-1} w_{t-iT}.$

From a window of $K$ consecutive merged checkpoints, one constructs the centered matrix

$$W = [\bar w_{t-K+1}-\mu,\dots,\bar w_t-\mu]\in\mathbb R^{d\times K},\qquad \mu=\frac{1}{K}\sum_{i=0}^{K-1}\bar w_{t-i}.$$

PCA of this matrix yields eigenvalues $\lambda_1\ge\lambda_2\ge\dots\ge\lambda_K$, and the Rank-1 Subspace is defined as the span of the first principal component $u_1$, with explained-variance ratio

$$R_1=\frac{\lambda_1}{\sum_{j=1}^K\lambda_j}\gg 0.94.$$

Equivalently, the merged checkpoints lie approximately on the one-dimensional linear manifold

$M_1=\{\mu+\alpha u_1:\alpha\in\mathbb R\}$

(Zhou et al., 26 May 2026).

This definition is specific to a sequence of merged checkpoints produced during a single training run. It is therefore distinct from rank-1 notions based on SVD of task vectors or weight-difference matrices in fine-tuning-based model merging. A plausible implication is that “rank-1 subspace” in current model-merging research is not a universal object but a family of closely related low-dimensional structures whose interpretation depends on whether the underlying objects are trajectories, task matrices, or layerwise updates.

2. Geometric observation: raw oscillations and merged rectification

The empirical motivation for the pre-training Rank-1 Subspace comes from a contrast between raw and merged trajectories. When one linearly interpolates between two consecutive raw checkpoints $N$0, the loss forms a U-shaped basin whose midpoint is lower than both endpoints. This is interpreted as evidence that raw SGD “bounces” across valley walls rather than descending smoothly. By contrast, interpolation between merged checkpoints $N$1 yields a strictly monotonic descent (Zhou et al., 26 May 2026).

PCA on short segments with $N$2 across GPT-2 and LLaMA models further sharpens this contrast. Raw checkpoints distribute variance across multiple principal components, with $N$3–$N$4, whereas merged checkpoints satisfy $N$5. Projected onto the leading direction $N$6, raw trajectories remain non-monotonic, while merged projections evolve strictly monotonically. The paper characterizes this as spectral concentration together with trajectory rectification: averaging suppresses high-frequency oscillations and reveals a one-dimensional descent path (Zhou et al., 26 May 2026).

This usage of rank-1 is trajectory-centric rather than update-centric. It does not assert that the full optimization path is intrinsically one-dimensional; rather, it states that the merged late-stage path becomes almost one-dimensional after averaging. This suggests that the observed low-dimensionality is a property of the averaging operator acting on a particular training regime, not merely a property of the underlying optimizer state.

3. River-valley analysis and PCA recovery

Extra-Merge provides a theoretical account through a “river-valley” landscape analysis. Near late-stage pre-training, the loss is modeled as

$N$7

where $N$8 is the flat river direction, $N$9, $t$0 is the sharp subspace, $t$1 has eigenvalues in $t$2 on $t$3, and $t$4 is the smooth one-dimensional loss along $t$5 (Zhou et al., 26 May 2026).

Under decoupled SGD dynamics with step-size $t$6,

$t$7

the sharp directions behave as noisy high-curvature fluctuations, whereas the river direction carries the underlying descent signal. Theorem 5.3 shows that averaging acts as a geometric low-pass filter: for the merged model $t$8, the expected squared deviation from the river satisfies an upper bound that decays as $t$9 when $\bar w_t = \frac{1}{N}\sum_{i=0}^{N-1} w_{t-iT}.$0. In the paper’s interpretation, high-curvature mountain fluctuations are exponentially damped, so merging projects the trajectory onto the low-curvature river subspace (Zhou et al., 26 May 2026).

Theorem 5.4 then addresses why PCA identifies the correct direction. The sample covariance of a sliding window of merged checkpoints takes the form

$\bar w_t = \frac{1}{N}\sum_{i=0}^{N-1} w_{t-iT}.$1

with $\bar w_t = \frac{1}{N}\sum_{i=0}^{N-1} w_{t-iT}.$2 scaling like $\bar w_t = \frac{1}{N}\sum_{i=0}^{N-1} w_{t-iT}.$3 and the top eigenvalue of $\bar w_t = \frac{1}{N}\sum_{i=0}^{N-1} w_{t-iT}.$4 scaling like $\bar w_t = \frac{1}{N}\sum_{i=0}^{N-1} w_{t-iT}.$5. If the signal-to-noise gap $\bar w_t = \frac{1}{N}\sum_{i=0}^{N-1} w_{t-iT}.$6, then $\bar w_t = \frac{1}{N}\sum_{i=0}^{N-1} w_{t-iT}.$7 is the unique top eigenvector, and the empirical estimate $\bar w_t = \frac{1}{N}\sum_{i=0}^{N-1} w_{t-iT}.$8 satisfies

$\bar w_t = \frac{1}{N}\sum_{i=0}^{N-1} w_{t-iT}.$9

with high probability. In this framework, PCA on merged checkpoints does not merely summarize the path; it recovers the river tangent itself (Zhou et al., 26 May 2026).

A related but conceptually different geometric treatment appears in "Model Merging on Loss Landscape: A Geometry Perspective" (Lu et al., 26 May 2026), where merging is posed as a Fréchet-mean problem on a Riemannian manifold equipped with the expected-Hessian metric. In the two-model rank-1 case, the merge is restricted to $K$0 with $K$1, and the curvature-aware solution is guaranteed to tie or improve upon flat interpolation whenever the projected curvatures differ. The shared theme is that a one-dimensional subspace can suffice when it captures the dominant geometry, but the underlying object is different: Extra-Merge tracks a pre-training trajectory, whereas EpiMer restricts merging of task vectors between fine-tuned models (Lu et al., 26 May 2026).

4. Extra-Merge algorithm

Extra-Merge is a training-free strategy that exploits the Rank-1 Subspace by extrapolating beyond the last merged checkpoint along the PCA direction. Its inputs are merged checkpoints $K$2, a PCA window $K$3, and base fraction $K$4 with default $K$5 (Zhou et al., 26 May 2026).

The first stage extracts the direction. One forms the centered matrix $K$6 from $K$7 and computes the top principal component $K$8 via Gram-matrix PCA. The direction is then oriented using

$K$9

and one sets $$W = [\bar w_{t-K+1}-\mu,\dots,\bar w_t-\mu]\in\mathbb R^{d\times K},\qquad \mu=\frac{1}{K}\sum_{i=0}^{K-1}\bar w_{t-i}.$$0 (Zhou et al., 26 May 2026).

The second stage performs line-search extrapolation. Let

$$W = [\bar w_{t-K+1}-\mu,\dots,\bar w_t-\mu]\in\mathbb R^{d\times K},\qquad \mu=\frac{1}{K}\sum_{i=0}^{K-1}\bar w_{t-i}.$$1

Candidates are generated as

$$W = [\bar w_{t-K+1}-\mu,\dots,\bar w_t-\mu]\in\mathbb R^{d\times K},\qquad \mu=\frac{1}{K}\sum_{i=0}^{K-1}\bar w_{t-i}.$$2

The loss $$W = [\bar w_{t-K+1}-\mu,\dots,\bar w_t-\mu]\in\mathbb R^{d\times K},\qquad \mu=\frac{1}{K}\sum_{i=0}^{K-1}\bar w_{t-i}.$$3 is evaluated, the procedure stops at the first $$W = [\bar w_{t-K+1}-\mu,\dots,\bar w_t-\mu]\in\mathbb R^{d\times K},\qquad \mu=\frac{1}{K}\sum_{i=0}^{K-1}\bar w_{t-i}.$$4 where loss increases, and the algorithm returns the candidate with minimal loss. In compact form,

$$W = [\bar w_{t-K+1}-\mu,\dots,\bar w_t-\mu]\in\mathbb R^{d\times K},\qquad \mu=\frac{1}{K}\sum_{i=0}^{K-1}\bar w_{t-i}.$$5

No additional gradient steps are required. The practical hyperparameters reported are merging interval $$W = [\bar w_{t-K+1}-\mu,\dots,\bar w_t-\mu]\in\mathbb R^{d\times K},\qquad \mu=\frac{1}{K}\sum_{i=0}^{K-1}\bar w_{t-i}.$$6 for GPT-2 small/med or $$W = [\bar w_{t-K+1}-\mu,\dots,\bar w_t-\mu]\in\mathbb R^{d\times K},\qquad \mu=\frac{1}{K}\sum_{i=0}^{K-1}\bar w_{t-i}.$$7 for larger models, window $$W = [\bar w_{t-K+1}-\mu,\dots,\bar w_t-\mu]\in\mathbb R^{d\times K},\qquad \mu=\frac{1}{K}\sum_{i=0}^{K-1}\bar w_{t-i}.$$8–$$W = [\bar w_{t-K+1}-\mu,\dots,\bar w_t-\mu]\in\mathbb R^{d\times K},\qquad \mu=\frac{1}{K}\sum_{i=0}^{K-1}\bar w_{t-i}.$$9, PCA window $\lambda_1\ge\lambda_2\ge\dots\ge\lambda_K$0, and base fraction $\lambda_1\ge\lambda_2\ge\dots\ge\lambda_K$1 (Zhou et al., 26 May 2026).

The method depends on the empirical premise that merged checkpoints already lie near a one-dimensional manifold. If that premise fails, the extrapolation direction would no longer have the interpretation of a dominant descent tangent. This clarifies a common misconception: Extra-Merge is not a generic linear extrapolation heuristic applied to arbitrary checkpoints; it is specifically justified by the observed collapse of merged late-stage trajectories onto a Rank-1 Subspace.

5. Experimental profile

The paper reports results across GPT-2 and LLaMA families from 124M to 2B parameters. On validation loss, uniform averaging (PMA) lowers loss relative to raw checkpoints during high-learning-rate phases but converges back to the baseline as the learning rate decays, whereas Extra-Merge yields a further loss reduction of approximately $\lambda_1\ge\lambda_2\ge\dots\ge\lambda_K$2–$\lambda_1\ge\lambda_2\ge\dots\ge\lambda_K$3 across all model scales and schedulers, including WSD for GPT-2 and Cosine for LLaMA (Zhou et al., 26 May 2026).

On zero-shot downstream evaluation with Pythia-12B, using 11 checkpoints from 130k to 140k steps and uniform or EMA averaging baselines, Extra-Merge achieves an average accuracy of $\lambda_1\ge\lambda_2\ge\dots\ge\lambda_K$4 versus $\lambda_1\ge\lambda_2\ge\dots\ge\lambda_K$5 for the raw model and $\lambda_1\ge\lambda_2\ge\dots\ge\lambda_K$6 for PMA. Reported per-task improvements are $\lambda_1\ge\lambda_2\ge\dots\ge\lambda_K$7 on ARC-Challenge, $\lambda_1\ge\lambda_2\ge\dots\ge\lambda_K$8 on ARC-Easy, $\lambda_1\ge\lambda_2\ge\dots\ge\lambda_K$9 on HellaSwag, and $u_1$0 on PIQA (Zhou et al., 26 May 2026).

The method also generalizes to the Muon optimizer. On GPT-2 Small trained with Muon, Extra-Merge improves upon PMA by approximately $u_1$1 in validation loss, despite the optimizer’s orthogonal update rule producing a very different trajectory (Zhou et al., 26 May 2026). The claim is therefore optimizer-agnostic robustness at least across the tested setting.

These results indicate that the rank-1 phenomenon is not confined to a single architecture family or one optimizer. A plausible implication is that the low-dimensional structure emerges from late-stage optimization geometry together with checkpoint averaging rather than from a peculiar artifact of a single implementation.

6. Related rank-1 and subspace perspectives in model merging

The broader literature uses “rank-1 subspace” in several non-identical ways. In "Subspace-Boosted Model Merging" (Skorobogat et al., 19 Jun 2025), stacking task vectors from multiple experts into a task matrix reveals rank collapse as more experts are merged: the leading singular value grows while the tail singular values shrink, and in ViT-B/16 vision experiments the stable rank of each layer’s $u_1$2 drops from $u_1$3 to $u_1$4 when merging 14 experts by naive task arithmetic. The dominant rank-1 subspace $u_1$5 is interpreted as a shared component across tasks, but excessive concentration onto that direction causes loss of task-specific diversity. Subspace Boosting preserves the top singular direction while restoring higher-rank support by boosting the collapsed tail singular values (Skorobogat et al., 19 Jun 2025).

In "When Shared Knowledge Hurts: Spectral Over-Accumulation in Model Merging" (Li et al., 5 Feb 2026), each rank-1 term $u_1$6 in the SVD of an update matrix defines a spectral component, and over-accumulation occurs when multiple tasks share aligned singular vectors so that singular values add under linear merging. Singular Value Calibration rescales the inflated singular values to restore a balanced spectrum and improves Task Arithmetic by $u_1$7 while also yielding gains on language benchmarks (Li et al., 5 Feb 2026). Here the central problem is not collapse of a trajectory onto one dimension, but repeated counting of shared rank-1 directions across tasks.

Other works operationalize rank-1 or low-rank subspaces differently. FRISM decomposes an LRM task vector into rank-1 SVD components and learns subspace-wise coefficients $u_1$8 for fine-grained reasoning injection into VLMs, rather than using a single layer-level scalar (Huang et al., 29 Jan 2026). SSAM identifies a shared low-rank subspace for language-related parameter updates across multimodal specialists and, in a rank-1 specialization, projects two updates onto the top singular vector of their concatenation before balancing aligned and residual components with a scalar $u_1$9 (Reza et al., 23 Mar 2026). SCORE studies domain generalization by extracting leading singular directions from domain-specific deltas, building a shared orthogonal basis from these directions, and pruning off-diagonal conflicts after projection into that basis (Chaves et al., 6 Mar 2026). The thesis "Model Merging: Foundations and Algorithms" develops yet another perspective in which the rank-1 approximation $$R_1=\frac{\lambda_1}{\sum_{j=1}^K\lambda_j}\gg 0.94.$$0 is the optimal one-dimensional Frobenius approximation to a layerwise task vector and forms the basis of TSV-Merge (Crisostomi, 2 May 2026).

Taken together, these works show that rank-1 structure can be either beneficial or pathological depending on context. In Extra-Merge, the near-1D merged trajectory is exploited as a descent signal; in Subspace-Boosting and SVC, dominance of a leading direction is precisely what must be counteracted to preserve diversity or avoid over-counting.

7. Conceptual significance and limitations

The principal significance of the Rank-1 Subspace in pre-training is explanatory. Standard checkpoint averaging methods such as SWA, LAWA, and PMA had been known to improve validation loss, but the geometric reason had remained unclear. Extra-Merge interprets averaging as a mechanism that filters out high-curvature oscillations and reveals the optimal descent direction in a river-valley landscape, thereby making the success of merging intelligible in terms of geometry rather than only empirical regularization (Zhou et al., 26 May 2026).

At the same time, the topic admits several possible misunderstandings. First, the phrase “rank-1 subspace” does not imply that all of model merging is universally one-dimensional. In the pre-training case, the claim concerns windows of merged late-stage checkpoints; in fine-tuning-based merging, the relevant rank-1 objects are often top singular components of task matrices or weight deltas. Second, a dominant rank-1 direction is not always desirable. Subspace-Boosted Model Merging explicitly argues that further collapse beyond the leading shared component sacrifices task-specific diversity, and SVC argues that repeated accumulation of shared spectral directions biases the merged model toward common subspaces (Skorobogat et al., 19 Jun 2025).

The current literature therefore supports a nuanced view. Rank-1 structure can expose the principal geometry of descent, provide an efficient merge parameterization, or reveal the shared component of multiple tasks. It can also indicate collapse, interference, or over-counting. The specific contribution of Extra-Merge is to isolate a case in which the rank-1 phenomenon is directly actionable: late-stage checkpoint averaging in language-model pre-training produces a stable approximately one-dimensional manifold, and extrapolating along that manifold can reduce loss without further optimization (Zhou et al., 26 May 2026).