Extra-Merge: Optimizing LLM Parameter Space

• Extra-Merge is a set of methods that exploit low-dimensional (rank-1) manifolds in LLM parameter space to perform training-free extrapolation for improved performance.
• It applies PCA on merged checkpoints to isolate the dominant eigenvector, which explains over 94% of variance and serves as a reliable direction for extrapolation.
• Experimental findings show that Extra-Merge and its variant ExMe yield lower validation losses and higher accuracies across various models compared to traditional merging techniques.

Extra-Merge encompasses a family of methods that exploit the geometric structure of the parameter space in LLM training by tracing and extrapolating along low-dimensional manifolds revealed through model checkpoint merging. This approach seeks to improve model performance in a training-free or computationally efficient manner by operating directly in parameter space, often after pre-training or supervised fine-tuning, and does not require access to additional data or gradient updates. The Extra-Merge paradigm is grounded in recent empirical and theoretical insights about pre-training dynamics, notably the emergence of a stable rank-1 (one-dimensional) subspace in merged model trajectories, which can be leveraged for systematic extrapolation and performance gains (Lin et al., 5 Mar 2025, Zhou et al., 26 May 2026).

1. Geometric Foundations: The Rank-1 Subspace Phenomenon

Extensive empirical analysis reveals that, in the late stages of LLM pre-training, the sequence of model parameter vectors generated by vanilla stochastic gradient descent (SGD) displays pronounced oscillations when viewed in the high-dimensional parameter space. However, when these checkpoints are merged—specifically, by averaging over a set interval (as in Polyak model averaging, PMA)—their trajectory collapses onto an approximately one-dimensional linear manifold, termed the "rank-1 subspace" (Zhou et al., 26 May 2026).

Principal component analysis (PCA) performed on $K$ consecutive merged checkpoints (typically $K=5$) consistently shows that the first principal component (PC₁) explains over 94% of the variance for models such as GPT-2 Small/Medium and LLaMA-0.5B/2B, while raw checkpoints distribute variance across many components. This subspace remains globally stable even as $K$ increases (explained variance ratio $>0.8$ for $K=16$), in contrast to raw checkpoint trajectories, which do not exhibit such low-dimensional structure. This empirical regularity provides a geometric foundation for training-free, subspace-based parameter manipulation.

2. Theoretical Justification: River-Valley Landscape and Averaging as a Low-Pass Filter

The river-valley theoretical framework models the loss landscape around a well-trained LLM as a one-dimensional “river” manifold, $\mathcal{M}$, embedded in parameter space. Any parameter vector $\theta$ in a proximal neighborhood can be decomposed as

$\theta = \theta^* + t v + z$

where $v\in\mathbb{R}^d$ is the tangent ("river") direction, $t$ is a scalar river coordinate, and $K=5$0 is orthogonal to $K=5$1, lying in the "mountain" (sharp) subspace. The loss near $K=5$2 decomposes as

$K=5$3

where $K=5$4 is the positive semidefinite Hessian on the mountain subspace. SGD advances primarily along $K=5$5 while introducing high-curvature noise in $K=5$6 due to stochasticity.

Averaging $K=5$7 checkpoints acts as a geometric low-pass filter: the high-curvature noise ($K=5$8) is dampened exponentially (with rate related to $K=5$9), and in the limit the averaged point collapses onto the 1-D river. Population covariance analysis confirms that PCA on merged checkpoints recovers $K$0 as the top eigenvector, with formal guarantees (Theorems 5.3, 5.4 in (Zhou et al., 26 May 2026)) bounding the error in subspace recovery.

3. Extra-Merge Algorithm: Training-Free Extrapolation on the Manifold

Taking advantage of the stable 1-D subspace, the Extra-Merge algorithm extrapolates along this direction to discover lower-loss configurations without additional training updates:

• Given $K$1 merged checkpoints $K$2, compute their top principal component $K$3 via PCA on centered points.
• Orient $K$4 to point “forward” in parameter space:

$K$5

• Define a stride $K$6, where $K$7 and $K$8.
• Iteratively extrapolate:

$K$9

Evaluate validation loss $>0.8$0 after each step, stopping at the first $>0.8$1 where loss increases. Alternatively, a 1-D line search over $>0.8$2 may be used:

$>0.8$3

This approach discovers new configurations outside the convex hull of past checkpoints, contrasting with simple PMA (Zhou et al., 26 May 2026).

4. Extrapolation Merging (ExMe) in Instruction Fine-Tuning

A distinct yet related extrapolation-based paradigm, Extrapolation Merging (ExMe), addresses performance optimization after instruction fine-tuning (Lin et al., 5 Mar 2025). ExMe formalizes model extrapolation:

$>0.8$4

Solving for an extrapolated parameter yields

$>0.8$5

where $>0.8$6. Two such extrapolated models are then linearly merged:

$>0.8$7

The merge ratio $>0.8$8 is optimized to maximize a held-out evaluation metric over a grid $>0.8$9. The complete protocol combines supervised fine-tuning, extrapolation along the SFT–base vector for $K=16$0, and search-based merging.

5. Comparative Performance and Experimental Findings

Experimental evaluations in both pre-training and fine-tuning regimes demonstrate that Extra-Merge and ExMe outperform baseline merging strategies. In pre-training, Extra-Merge consistently improves loss across scales: for GPT-2 Small (124M parameters), validation loss decreases from 3.194 (raw endpoint) and 3.193 (PMA) to 3.184 (Extra-Merge). Similar improvements (0.008–0.018 in loss) are seen in GPT-2 Medium, GPT-2 XL, and LLaMA models.

ExMe achieves the highest average accuracy across seven evaluation tasks—GSM8K, MATH, HumanEval (pass@1), MMLU, CMMLU, C-Eval, and GaoKaoBench—for all tested LLMs, including Qwen2-7B, Qwen1.5-14B, Meta-Llama-3-8B, and Mistral-Nemo-Base-2407. Gains over best SFT checkpoints range from +0.5% to +4.7% average accuracy. On code generation (HumanEval), pass@1 for Meta-Llama-3-8B improves from 38.41 to 46.95.

On the Pythia-12B zero-shot suite, Extra-Merge delivers the following accuracy improvements relative to raw and PMA (uniformly across ARC-Challenge, ARC-Easy, HellaSwag, PIQA). For instance, Extra-Merge achieves an average of 57.47% versus 56.88% (raw) and 57.07% (PMA), with the largest absolute gain (+1.10%) on ARC-Challenge. Extra-Merge also generalizes to models optimized with the Muon optimizer, maintaining its margin over PMA.

Model/Task	Raw	PMA	Extra-Merge / ExMe	Gain Over Baseline
GPT-2 Small (ℓ)	3.194	3.193	3.184	–0.010
Pythia-12B (avg %)	56.88	57.07	57.47	+0.40
Qwen2-7B (%)	68.10	–	71.33	+4.7

Ablations indicate optimal $K=16$1 and $K=16$2 values are typically small; too large $K=16$3 leads to overshooting, while optimal $K=16$4 (0.1–0.2) places highest weight on the strongest extrapolated model.

6. Limitations and Future Directions

The current instantiations of Extra-Merge and ExMe operate with several limitations:

• There is no theoretical proof of global convergence or generalization guarantees for extrapolation and merging.
• All parameters are merged indiscriminately, without pruning redundant or low-impact weights.
• The merging direction is restricted to two (or a few) model “experts”; scaling to multiple experts or gradient-based merge ratios remains an open direction.
• ExMe’s effectiveness can be adversely impacted by the style of fine-tuning data (e.g., dialogue-format bias), especially on multiple-choice tasks.
• Future research directions include integrating parameter importance metrics (e.g., Fisher information), Bayesian hyperparameter optimization, and extensions to reinforcement learning from human feedback (RLHF) or multimodal model settings (Lin et al., 5 Mar 2025).

A plausible implication is that further refinement of subspace discovery, parameter selection, and dynamic hyperparameter adaptation may enhance the robustness and generality of Extra-Merge methods.