Plateau-Guided Model Merging

• Plateau-guided model merging is a principled method for combining neural models by identifying performance plateaus that signal diminishing returns.
• It relies on theoretical foundations such as variance reduction, Gaussian width analysis, and kinematic phase transitions to set precise merging limits.
• Empirical results demonstrate that merging benefits peak with 4-6 experts, informing algorithmic strategies for efficient distributed training and continual learning.

Plateau-guided model merging encompasses a family of principled techniques for optimally combining multiple neural models—typically LLMs or multimodal LLMs (MLLMs)—by leveraging empirical or theoretical performance plateaus to guide the merging process. This approach explicitly identifies points of diminishing marginal return when aggregating models or expert modules, selecting merging parameters and schedules that maximize utility while avoiding redundancy and degraded performance. Plateau-guided model merging has catalyzed new methodologies for distributed training, continual learning, multi-task adaptation, and efficient parameter reuse across domains.

1. Theoretical Foundations of Plateau-Guided Model Merging

Plateau-guided merging is underpinned by three complementary branches of theory: variance reduction upper bounds, Gaussian width analysis of the effective parameter space, and kinematic phase transitions. Together, these results provide non-heuristic criteria for identifying when further merging is no longer beneficial or becomes detrimental (Wang et al., 27 May 2025).

Let $n$ denote the number of expert models being merged and $\rho$ the pairwise cosine similarity among expert weight vectors. For uniformly weighted merges under Gaussian prior assumptions, the post-merge variance satisfies

$$\sigma^2_{\text{merge}}(n) = \sigma^2 \left[ \rho + \frac{1-\rho}{n} \right]$$

where $\sigma^2$ is the variance of the individual experts. As $n \to \infty$, $\sigma^2_{\text{merge}}(n)$ attains a lower bound of $\sigma^2 \rho$, showing that variance reduction is fundamentally capped by inter-expert correlation. Setting a minimum required absolute variance drop $\Delta$ per newly merged expert, the usable maximum number of experts is

$n_{\max} \leq \frac{\sigma^2 (1-\rho)}{\Delta}$

This sharp upper bound implies that, for any target improvement, one cannot benefit from unlimited merging—the process hits a "plateau."

Gaussian width analysis further quantifies diminishing marginal returns in terms of the parameter-space geometry. The effective parameter space ("$\varepsilon$-sublevel set") after $\rho$0 experts are merged is characterized by width

$\rho$1

where $\rho$2 are top eigenvalues of the Hessian at the solution. The gain from adding each successive expert decreases strictly, reflecting saturation of the accessible parameter volume (Wang et al., 27 May 2025).

Approximate kinematics theory provides a unique, model-dependent critical point $\rho$3: beyond this number of merged experts, the likelihood of parameter redundancy or destructive interference rises sharply, often leading to stalled or even degraded task performance.

2. Empirical Scaling Laws and Power-Law Plateau Behaviors

Empirical studies confirm that model-merging performance consistently follows a "floor plus tail" power-law: $\rho$4 where $\rho$5 is the expected merged-model cross-entropy loss, $\rho$6 is the base model parameter count, $\rho$7 is number of merged experts, $\rho$8 is typically in $\rho$9, and $$\sigma^2_{\text{merge}}(n) = \sigma^2 \left[ \rho + \frac{1-\rho}{n} \right]$$0 is the irreducible loss in the large-$$\sigma^2_{\text{merge}}(n) = \sigma^2 \left[ \rho + \frac{1-\rho}{n} \right]$$1, large-$$\sigma^2_{\text{merge}}(n) = \sigma^2 \left[ \rho + \frac{1-\rho}{n} \right]$$2 limit (Wang et al., 29 Sep 2025). $$\sigma^2_{\text{merge}}(n) = \sigma^2 \left[ \rho + \frac{1-\rho}{n} \right]$$3 constitutes a model-size-dependent lower bound (the merging “floor”), while the $$\sigma^2_{\text{merge}}(n) = \sigma^2 \left[ \rho + \frac{1-\rho}{n} \right]$$4 tail encapsulates the diminishing returns from merging additional experts.

Key empirical regularities include:

• Most gains accrue early: $$\sigma^2_{\text{merge}}(n) = \sigma^2 \left[ \rho + \frac{1-\rho}{n} \right]$$5 of the possible reduction in loss occurs by $$\sigma^2_{\text{merge}}(n) = \sigma^2 \left[ \rho + \frac{1-\rho}{n} \right]$$6 experts.
• Plateau location insensitive to merge method or domain: For multiple domains (algebra, biology, compositional tasks) and merging frameworks (Average, Task Arithmetic, TIES, DARE), the elbow/plateau occurs at comparable $$\sigma^2_{\text{merge}}(n) = \sigma^2 \left[ \rho + \frac{1-\rho}{n} \right]$$7.
• Variance reduction scales as $$\sigma^2_{\text{merge}}(n) = \sigma^2 \left[ \rho + \frac{1-\rho}{n} \right]$$8: The variance across expert-merge permutation choices shrinks with $$\sigma^2_{\text{merge}}(n) = \sigma^2 \left[ \rho + \frac{1-\rho}{n} \right]$$9, with order sensitivity becoming negligible at moderate $\sigma^2$0.

This law enables predictive planning for merging runs: by fitting loss at $\sigma^2$1, one can extrapolate the plateau location, estimate the marginal utility of further experts, and balance model scaling versus merging (Wang et al., 29 Sep 2025).

3. Algorithmic Instantiations: Plateau-Guided Scheduling and Merging

Algorithmic implementations of plateau-guided merging operationalize these insights in diverse contexts, including:

• Selective late-layer parameter interpolation: As in PlaM, merged parameters in MLLMs are defined as

$\sigma^2$2

where $\sigma^2$3 is the plateau onset layer and $\sigma^2$4 are optimized per task. Only self-attention projections ($\sigma^2$5) are merged; the rest of the model stays frozen at the vision-language solution. This preserves early cross-modal connections and late textual reasoning (Wang et al., 12 Jan 2026).

• Adaptive merge scheduling via learning/forgetting signals: In continual learning, AimMerging monitors rates of parameter change ($\sigma^2$6) and forgetting events ($\sigma^2$7), dynamically stretching or compressing the intervals between merges in response to detected plateaus. Merging is triggered either when new learning abates ("plateau"), or when historical-task loss spikes ("forgetting event"), with weights assigned to new/past knowledge according to recent trends (Feng et al., 22 Sep 2025).
• Heavy-tailed reparameterization beyond the plateau: If strictly guided merging plateaus, a reparameterized heavy-tailed (RHT) transformation extends functional coverage, producing further—albeit less pronounced—gains by nonlinear amplification of weight directions (Wang et al., 27 May 2025).

4. Practical Guidance: Plateau Detection, Stopping Criteria, and Resource Trade-offs

Practical application of plateau-guided merging requires

1. Quantifying the location of the plateau (elbow point):
   • Via masking or performance curves (PlaM), the elbow $\sigma^2$8 is the inflection where improvements slow sharply.
   • For expert merging, fit observed losses to the power-law and solve for $\sigma^2$9 achieving desired loss bandwidth $n \to \infty$0.
2. Stopping criteria:
   • Stop merging additional experts when variance or performance gain $n \to \infty$1, or when the upper bound on effective parameter space is reached (Wang et al., 27 May 2025, Wang et al., 29 Sep 2025).
   • For iterative/continual scenarios, pause merges during learning plateaus to allow acquisition of significant new knowledge, as reflected in low parameter update rates (Feng et al., 22 Sep 2025).
3. Compute–accuracy budgeting:
   • Compare expected marginal benefit $n \to \infty$2 for an added expert versus increasing base model size. Marginal gains from new experts fall as $n \to \infty$3, while scaling the base model improves only the floor term—a trade-off addressed quantitatively in plateau-law parameters (Wang et al., 29 Sep 2025).

5. Mechanistic Analysis and Empirical Results

In PlaM, plateau-guided merging in MLLMs corrects late-stage semantic degradation and enhances visual grounding. This is evidenced by a shift in attention mass from diffuse to focused, task-relevant image regions. Layer-wise inspection indicates:

• Attention mass from instruction tokens to vision tokens in post-merge layers rises sharply at the plateau onset ($n \to \infty$4), e.g., from $n \to \infty$5 to $n \to \infty$6 during prompt encoding (Wang et al., 12 Jan 2026).
• Qualitative heatmaps illustrate consolidation of attention on semantically critical features (e.g., clock hands, object boundaries).

Quantitative improvements are robust:

• Across five open-source models and nine multimodal benchmarks, PlaM provides consistent gains over standard VLMs, with the largest increments ($n \to \infty$7 to $n \to \infty$8 points) on benchmarks requiring deep cross-modal reasoning. Gains are smaller but systematic ($n \to \infty$9 to $\sigma^2_{\text{merge}}(n)$0 points) for hallucination- or composition-resistance tasks (Wang et al., 12 Jan 2026).
• In continual learning, plateau-guided AimMerging improves forward transfer (FWT) by $\sigma^2_{\text{merge}}(n)$1 and backward transfer (BWT) by $\sigma^2_{\text{merge}}(n)$2 compared to fixed-interval and static-weight baselines (Feng et al., 22 Sep 2025).
• Merging more experts always helps initially, but performance peaks and may drop beyond $\sigma^2_{\text{merge}}(n)$3–$\sigma^2_{\text{merge}}(n)$4 experts, matching theoretical predictions (Wang et al., 27 May 2025).

Model/Method	Plateau Onset ($\sigma^2_{\text{merge}}(n)$5 or $\sigma^2_{\text{merge}}(n)$6)	Gain at Plateau	Empirical Signature
PlaM (MM LLM)	$\sigma^2_{\text{merge}}(n)$7 via masking curve	$\sigma^2_{\text{merge}}(n)$8–$\sigma^2_{\text{merge}}(n)$9 points	Late-layer attention focus
LLM Merging (power-law)	$\sigma^2 \rho$0	$\sigma^2 \rho$1 total gain	Loss/variance plateau
AimMerging	Adaptive interval	$\sigma^2 \rho$2, $\sigma^2 \rho$3 (FWT/BWT gain)	Dynamic interval oscillation

6. Extensions and Generalizations

Plateau-guided model merging has been generalized to a range of settings:

• Multimodal networks: Identifying representation alignment/degradation points (e.g., via token-masking curves) enables architecture-specific merging, as in selective merging of self-attention projections in PlaM.
• Cross-domain/multi-task fusion: Synergistic merging across domains exploits increased diversity, modestly lowering the performance floor, but still adheres to the same scaling laws and plateau constraints (Wang et al., 29 Sep 2025).
• Beyond-plateau function expansion: Heavy-tailed reparameterization via nonlinear transforms permits further coverage expansion, with observed $\sigma^2 \rho$4–$\sigma^2 \rho$5 additional relative gain post-plateau at the potential cost of increased interference (Wang et al., 27 May 2025).
• Continual learning controllers: Plateau detection via learning/forgetting trajectory signals enables dynamic, non-uniform merge timing and fusion weights, decreasing catastrophic forgetting and supporting knowledge integration (Feng et al., 22 Sep 2025).

A plausible implication is that plateau-guided principles offer a general planning framework for resource-constrained composition of expert or specialized subnetworks—turning merging from an empirical art into an analytically tractable subproblem across LLM, VLM, and continual learning domains.

7. Common Misconceptions and Limitations

Certain misconceptions are addressed by the theoretical and empirical literature:

• Unlimited merging always yields improvement: Both formal analysis and empirical results show that after a small number ($\sigma^2 \rho$6) of experts, additional merging yields vanishing or negative returns due to parameter space saturation and ridge interference (Wang et al., 27 May 2025, Wang et al., 29 Sep 2025).
• Method choice dominates performance at high $\sigma^2 \rho$7: Multiple model merging algorithms (Average, Task Arithmetic, TIES, DARE) yield negligible differences near or past the plateau, converging to the same floor-plus-tail scaling (Wang et al., 29 Sep 2025).
• Plateau corresponds to hardware or optimization limits: The plateau is a statistical-geometry phenomenon intrinsic to stochastic weight-space coverage and loss geometry, not an artifact of compute or dataset bottlenecks.
• Post-plateau improvement is impossible: Controlled reparameterizations (RHT) can partially circumvent the plateau, but standard merging must respect the upper bound imposed by the effective parameter space (Wang et al., 27 May 2025).

Limitations of plateau-guided model merging include reliance on observability of performance/loss curves (which may be noisy for rare or very large-scale domains) and the need for accurate estimation of merge-interval gains in dynamic/online settings. Nonetheless, the plateau-guided paradigm constitutes a critical advance in scalable, budget-aware, and principled neural model composition.