---
title: 'Extra-Merge: Optimizing LLM Parameter Space'
url: https://www.emergentmind.com/topics/extra-merge
type: topic
---

# Extra-Merge: Optimizing LLM Parameter Space

Extra-Merge encompasses a family of methods that exploit the geometric structure of the parameter space in Large Language Model (LLM) training by tracing and extrapolating along low-dimensional manifolds revealed through model checkpoint merging. This approach seeks to improve model performance in a training-free or computationally efficient manner by operating directly in parameter space, often after pre-training or supervised fine-tuning, and does not require access to additional data or gradient updates. The Extra-Merge paradigm is grounded in recent empirical and theoretical insights about pre-training dynamics, notably the emergence of a stable rank-1 (one-dimensional) subspace in merged model trajectories, which can be leveraged for systematic extrapolation and performance gains [2503.04834], [2605.26484].

## 1. Geometric Foundations: The Rank-1 Subspace Phenomenon

Extensive empirical analysis reveals that, in the late stages of LLM pre-training, the sequence of model parameter vectors generated by vanilla stochastic gradient descent (SGD) displays pronounced oscillations when viewed in the high-dimensional parameter space. However, when these checkpoints are merged—specifically, by averaging over a set interval (as in Polyak model averaging, PMA)—their trajectory collapses onto an approximately one-dimensional linear manifold, termed the "rank-1 subspace" [2605.26484].

Principal component analysis (PCA) performed on $K$ consecutive merged checkpoints (typically $K=5$) consistently shows that the first principal component (PC₁) explains over 94% of the variance for models such as GPT-2 Small/Medium and LLaMA-0.5B/2B, while raw checkpoints distribute variance across many components. This subspace remains globally stable even as $K$ increases (explained variance ratio $>0.8$ for $K=16$), in contrast to raw checkpoint trajectories, which do not exhibit such low-dimensional structure. This empirical regularity provides a geometric foundation for training-free, subspace-based parameter manipulation.

## 2. Theoretical Justification: River-Valley Landscape and Averaging as a Low-Pass Filter

The river-valley theoretical framework models the loss landscape around a well-trained LLM as a one-dimensional “river” manifold, $\mathcal{M}$, embedded in parameter space. Any parameter vector $\theta$ in a proximal neighborhood can be decomposed as
$$
\theta = \theta^* + t v + z
$$
where $v\in\mathbb{R}^d$ is the tangent ("river") direction, $t$ is a scalar river coordinate, and $z$ is orthogonal to $v$, lying in the "mountain" (sharp) subspace. The loss near $\mathcal{M}$ decomposes as
$$
\ell(\theta) \approx \ell_{\text{river}}(t) + \frac{1}{2} \|H^{1/2}z\|^2
$$
where $H$ is the positive semidefinite Hessian on the mountain subspace. SGD advances primarily along $v$ while introducing high-curvature noise in $z$ due to stochasticity.

Averaging $N$ checkpoints acts as a geometric low-pass filter: the high-curvature noise ($z$) is dampened exponentially (with rate related to $(I-\eta H)^T$), and in the limit the averaged point collapses onto the 1-D river. Population covariance analysis confirms that PCA on merged checkpoints recovers $v$ as the top eigenvector, with formal guarantees (Theorems 5.3, 5.4 in [2605.26484]) bounding the error in subspace recovery.

## 3. Extra-Merge Algorithm: Training-Free Extrapolation on the Manifold

Taking advantage of the stable 1-D subspace, the Extra-Merge algorithm extrapolates along this direction to discover lower-loss configurations without additional training updates:

- Given $K$ merged checkpoints $\{\bar\theta_{t-K+1},...\bar\theta_t\}$, compute their top principal component $u_1$ via PCA on centered points.
- Orient $u_1$ to point “forward” in parameter space:
  $$
  d = \text{sign}((\bar\theta_t-\bar\theta_{t-1})^Tu_1)u_1
  $$
- Define a stride $\delta = a|\Delta z|$, where $a=0.1$ and $\Delta z = u_1^T(\bar\theta_t-\bar\theta_{t-1})$.
- Iteratively extrapolate:
  $$
  \theta^{(k)} = \bar\theta_t + k\delta \cdot d
  $$
  Evaluate validation loss $\ell(\theta^{(k)})$ after each step, stopping at the first $k$ where loss increases. Alternatively, a 1-D line search over $\alpha \in [0,2]$ may be used:
  $$
  \theta_{\rm new} = \bar\theta_t + \alpha d
  $$
This approach discovers new configurations outside the convex hull of past checkpoints, contrasting with simple PMA [2605.26484].

## 4. Extrapolation Merging (ExMe) in Instruction Fine-Tuning

A distinct yet related extrapolation-based paradigm, Extrapolation Merging (ExMe), addresses performance optimization after instruction fine-tuning [2503.04834]. ExMe formalizes model extrapolation:
$$
\Theta_{\rm SFT} = (1-\lambda)\Theta_{\rm base} + \lambda\Theta_{\rm EXPO}
$$
Solving for an extrapolated parameter yields
$$
\Theta_{\rm EXPO} = \Theta_{\rm SFT} + \alpha(\Theta_{\rm SFT}-\Theta_{\rm base}) = (1+\alpha)\Theta_{\rm SFT} - \alpha\Theta_{\rm base}
$$
where $\alpha=(1/\lambda)-1$. Two such extrapolated models are then linearly merged:
$$
\Theta_{\rm merge} = \beta\Theta_{\rm EXPO-1} + (1-\beta)\Theta_{\rm EXPO-2}
$$
The merge ratio $\beta$ is optimized to maximize a held-out evaluation metric over a grid $\beta\in\{0.1,...,0.9\}$. The complete protocol combines supervised fine-tuning, extrapolation along the SFT–base vector for $\alpha\in\{0.1,...,0.5\}$, and search-based merging.

## 5. Comparative Performance and Experimental Findings

Experimental evaluations in both pre-training and fine-tuning regimes demonstrate that Extra-Merge and ExMe outperform baseline merging strategies. In pre-training, Extra-Merge consistently improves loss across scales: for GPT-2 Small (124M parameters), validation loss decreases from 3.194 (raw endpoint) and 3.193 (PMA) to 3.184 (Extra-Merge). Similar improvements (0.008–0.018 in loss) are seen in GPT-2 Medium, GPT-2 XL, and LLaMA models.

ExMe achieves the highest average accuracy across seven evaluation tasks—GSM8K, MATH, HumanEval (pass@1), MMLU, CMMLU, C-Eval, and GaoKaoBench—for all tested LLMs, including Qwen2-7B, Qwen1.5-14B, Meta-Llama-3-8B, and Mistral-Nemo-Base-2407. Gains over best SFT checkpoints range from +0.5% to +4.7% average accuracy. On code generation (HumanEval), pass@1 for Meta-Llama-3-8B improves from 38.41 to 46.95.

On the Pythia-12B zero-shot suite, Extra-Merge delivers the following accuracy improvements relative to raw and PMA (uniformly across ARC-Challenge, ARC-Easy, HellaSwag, PIQA). For instance, Extra-Merge achieves an average of 57.47% versus 56.88% (raw) and 57.07% (PMA), with the largest absolute gain (+1.10%) on ARC-Challenge. Extra-Merge also generalizes to models optimized with the Muon optimizer, maintaining its margin over PMA.

| Model/Task         | Raw    | PMA    | Extra-Merge / ExMe | Gain Over Baseline |
|--------------------|--------|--------|--------------------|--------------------|
| GPT-2 Small (ℓ)    | 3.194  | 3.193  | 3.184              | –0.010             |
| Pythia-12B (avg %) | 56.88  | 57.07  | 57.47              | +0.40              |
| Qwen2-7B (%)       | 68.10  | –      | 71.33              | +4.7               |

Ablations indicate optimal $\alpha$ and $\beta$ values are typically small; too large $\alpha$ leads to overshooting, while optimal $\beta$ (0.1–0.2) places highest weight on the strongest extrapolated model.

## 6. Limitations and Future Directions

The current instantiations of Extra-Merge and ExMe operate with several limitations:

- There is no theoretical proof of global convergence or generalization guarantees for extrapolation and merging.
- All parameters are merged indiscriminately, without pruning redundant or low-impact weights.
- The merging direction is restricted to two (or a few) model “experts”; scaling to multiple experts or gradient-based merge ratios remains an open direction.
- ExMe’s effectiveness can be adversely impacted by the style of fine-tuning data (e.g., dialogue-format bias), especially on multiple-choice tasks.
- Future research directions include integrating parameter importance metrics (e.g., Fisher information), Bayesian hyperparameter optimization, and extensions to reinforcement learning from human feedback (RLHF) or multimodal model settings [2503.04834].

A plausible implication is that further refinement of subspace discovery, parameter selection, and dynamic hyperparameter adaptation may enhance the robustness and generality of Extra-Merge methods.

Source: https://www.emergentmind.com/topics/extra-merge