Manifold-Based Model Merging

• Manifold-based model merging is a framework that treats neural network parameters as points on low-loss manifolds, enabling effective multi-task integration.
• The approach employs techniques like local linear approximations, tangent-space views, and quotient geometry to align and merge fine-tuned models.
• These methods preserve key functionalities and improve mergeability by respecting the intrinsic geometric structure of the model adaptation process.

Searching arXiv for papers on manifold-based model merging and related model merging geometry. Manifold-based model merging denotes a family of approaches that combine neural network parameters by treating trained models not merely as vectors in Euclidean weight space, but as points, directions, or subspaces associated with structured low-loss geometry. In the contemporary model-merging literature, this perspective is tied to mode connectivity, linear mode connectivity, low-loss manifolds, quotient geometries induced by architectural symmetries, low-rank adapter manifolds, and feature-space task manifolds (Ruan et al., 12 Mar 2025, Song et al., 10 Mar 2026, Silva et al., 29 Apr 2026). The common objective is to produce a single merged model—or, in dynamic settings, a merged system—that preserves capabilities from multiple fine-tuned experts without retraining from scratch and often without access to the original training data (Ruan et al., 12 Mar 2025). In practice, manifold-based formulations range from local linearizations such as task arithmetic, through SVD-based changes of coordinates and output-space projections, to explicit Riemannian constructions on the orthogonal group, quotient manifolds for LoRA, and Fisher–Rao Karcher means (Chaichana et al., 29 May 2025, Yang et al., 5 Feb 2026, Silva et al., 29 Apr 2026, Evans et al., 27 May 2026).

1. Conceptual definition and geometric basis

In the broad model-merging setting, if a backbone has parameters $\theta_0$ and task-specific fine-tuning yields $\theta_t$, a merged model is written as

$$\theta_{\text{merge}} = f(\theta_{t_1}, \theta_{t_2}, \dots, \theta_{t_k}; \theta_0),$$

with standard examples including weight averaging and task-vector arithmetic (Ruan et al., 12 Mar 2025). The manifold interpretation begins from the observation that good solutions are often not isolated minima. Classical mode-connectivity work shows that two trained models $\theta_A,\theta_B$ can be connected by a continuous low-loss path $\gamma(t)$ satisfying $\gamma(0)=\theta_A$, $\gamma(1)=\theta_B$, with small $\mathcal{L}(\gamma(t))$ throughout (Ruan et al., 12 Mar 2025). In the survey literature this is recast as a connected manifold $\mathcal{M}$ of low-loss points, or as wide low-loss basins $\mathcal{B}_\epsilon(\theta^*)$, especially in overparameterized regimes (Song et al., 10 Mar 2026).

A central formal object is the task manifold

$\theta_t$0

which is described as a connected, high-dimensional manifold or a union of manifolds in parameter space (Ruan et al., 12 Mar 2025). When fine-tuned branches share a common pretrained initialization and Cross-Task Linearity holds, linear interpolation

$\theta_t$1

can approximate a low-loss geodesic on this manifold (Ruan et al., 12 Mar 2025). The same geometric picture appears in the FUSE survey, which formalizes linear mode connectivity as the condition that

$\theta_t$2

remains low-loss, and gives a loss-barrier metric

$\theta_t$3

as a practical measure of mergeability (Song et al., 10 Mar 2026).

This geometric framing separates manifold-based merging from naïve averaging. Simple Euclidean averaging presumes that the line segment between checkpoints stays inside a flat enough low-loss region. Manifold-based methods instead treat nonlinearity, curvature, and symmetry as first-class concerns: they seek aligned coordinates, tangent-space directions, geodesic-like interpolants, or constrained submanifolds on which the merged model remains functionally stable (Ruan et al., 12 Mar 2025, Silva et al., 29 Apr 2026). A plausible implication is that manifold-based merging is best understood not as a single algorithmic family, but as a unifying viewpoint under which several existing methods become special cases distinguished by the manifold, metric, and averaging rule they employ (Silva et al., 29 Apr 2026).

2. Linear approximations, tangent-space views, and local submanifolds

The simplest manifold approximation is local linearization around the pretrained base. In task arithmetic one defines

$\theta_t$4

which interprets $\theta_t$5 as a tangent vector at the base point $\theta_t$6 (Ruan et al., 12 Mar 2025). In this reading, $\theta_t$7 is a point on the manifold and the task vectors are first-order directions of specialization. The procedure assumes that the manifold is sufficiently flat around $\theta_t$8 that additions in tangent space remain meaningful (Ruan et al., 12 Mar 2025). The FUSE survey makes the same point when it characterizes task arithmetic as movement in an affine subspace spanned by task directions, explicitly linking it to tangent-space formulations and local mode connectivity (Song et al., 10 Mar 2026).

Model soups provide an even simpler linear manifold approximation. A soup computes

$\theta_t$9

that is, a convex combination of task-specific points (Ruan et al., 12 Mar 2025). In geometric language, this takes a barycenter in a locally flat region of the low-loss manifold. The FUSE survey interprets soups exactly in this manner, as convex combinations of points in a shared low-loss basin induced by common initialization (Song et al., 10 Mar 2026).

Sparsification methods can also be reinterpreted as submanifold selection. TIES, DARE, DELLA, Model Breadcrumbs, and related methods prune small or conflicting coordinates before merging, thereby restricting movement to important, compatible directions (Ruan et al., 12 Mar 2025). The 2026 survey describes TIES as a three-stage procedure—trim, elect sign, merge—that keeps only sparse directions on which multiple tasks agree, thereby avoiding high-curvature or contradictory directions (Song et al., 10 Mar 2026). This suggests that the practical role of sparsification is to carve out a lower-dimensional manifold, or at least a lower-dimensional subspace, on which multi-task compatibility is higher.

Activation-based pruning methods such as SurgeryV2, ZipIt!, Localize-and-Stitch, and AdaMerging push this perspective closer to function-space geometry. Instead of selecting parameters solely by magnitude, they use activations or representation importance, effectively restricting merging to functionally salient subspaces (Ruan et al., 12 Mar 2025). In the model-merging review this is described as merging inside a functionally important subspace rather than raw coordinate space (Ruan et al., 12 Mar 2025). A plausible implication is that activation-aware pruning can be seen as an empirical approximation to manifold learning in representation space.

3. Coordinate changes, symmetry, and quotient geometries

A persistent difficulty in model merging is that coordinate-wise comparison is often ill-posed. Fine-tuned models may differ by neuron permutations, rotations within subspaces, or other symmetries, so entries at the same tensor index need not represent the same function (Chaichana et al., 29 May 2025). This issue is central to permutation-based methods, which aim to resolve permutation invariance so that functionally similar models become nearby in parameter space (Ruan et al., 12 Mar 2025). In the geometric language of the review, these methods place model parameters on a common chart of the manifold, after which a linear path in coordinates better corresponds to a low-loss path in function space (Ruan et al., 12 Mar 2025).

"Decom-Renorm-Merge: Model Merging on the Right Space Improves Multitasking" (Chaichana et al., 29 May 2025) gives a particularly explicit coordinate-change interpretation. DRM starts from shared-base weight deltas

$$\theta_{\text{merge}} = f(\theta_{t_1}, \theta_{t_2}, \dots, \theta_{t_k}; \theta_0),$$0

and constructs a joint SVD space by stacking task deltas and decomposing

$$\theta_{\text{merge}} = f(\theta_{t_1}, \theta_{t_2}, \dots, \theta_{t_k}; \theta_0),$$1

For the horizontal variant, the stacked matrix is

$$\theta_{\text{merge}} = f(\theta_{t_1}, \theta_{t_2}, \dots, \theta_{t_k}; \theta_0),$$2

Because $$\theta_{\text{merge}} = f(\theta_{t_1}, \theta_{t_2}, \dots, \theta_{t_k}; \theta_0),$$3 is shared across tasks, all deltas are expressed in a common output-feature basis (Chaichana et al., 29 May 2025). DRM then renormalizes the task-specific right singular blocks $$\theta_{\text{merge}} = f(\theta_{t_1}, \theta_{t_2}, \dots, \theta_{t_k}; \theta_0),$$4, absorbing row norms into $$\theta_{\text{merge}} = f(\theta_{t_1}, \theta_{t_2}, \dots, \theta_{t_k}; \theta_0),$$5, so that

$$\theta_{\text{merge}} = f(\theta_{t_1}, \theta_{t_2}, \dots, \theta_{t_k}; \theta_0),$$6

with $$\theta_{\text{merge}} = f(\theta_{t_1}, \theta_{t_2}, \dots, \theta_{t_k}; \theta_0),$$7 having unit-norm rows (Chaichana et al., 29 May 2025). Pruning, sign election, and disjoint averaging are then performed in this SVD-renormalized space, and the result is mapped back to weight space (Chaichana et al., 29 May 2025).

The significance of this construction is explicitly geometric. DRM argues that raw parameter space is the wrong coordinate system because of non-unique parameterizations and polysemanticity; SVD yields a shared linear subspace in which directions have more stable semantics across tasks (Chaichana et al., 29 May 2025). Renormalization is described as making that joint basis more isotropic, producing a “robust and even joint space” for interference-aware arithmetic (Chaichana et al., 29 May 2025). Quantitatively, the paper reports that renormalization alone yields large gains: ViT-B/32 +4.0 points (77.5 vs 73.5), T5-Base +5.0 (73.6 vs 68.6), DeBERTa-Base +8.8 (66.9 vs 58.1), and Llama3.1-8B LoRA +6.8 (69.5 vs 62.7) (Chaichana et al., 29 May 2025).

A more formal symmetry treatment appears in "Generalizing the Geometry of Model Merging Through Frechet Averages" (Silva et al., 29 Apr 2026). There the central claim is that both the geometry and the averaging procedure must be symmetry-invariant. If a total parameter manifold $$\theta_{\text{merge}} = f(\theta_{t_1}, \theta_{t_2}, \dots, \theta_{t_k}; \theta_0),$$8 carries a group action by $$\theta_{\text{merge}} = f(\theta_{t_1}, \theta_{t_2}, \dots, \theta_{t_k}; \theta_0),$$9, one passes to the quotient manifold

$\theta_A,\theta_B$0

whose points are equivalence classes of functionally identical parameterizations (Silva et al., 29 Apr 2026). The corresponding quotient distance is

$\theta_A,\theta_B$1

so merging requires orbit alignment before averaging (Silva et al., 29 Apr 2026). In the LoRA case, a rank-$\theta_A,\theta_B$2 update $\theta_A,\theta_B$3 with $\theta_A,\theta_B$4, $\theta_A,\theta_B$5, and $\theta_A,\theta_B$6 carries an $\theta_A,\theta_B$7 symmetry

$\theta_A,\theta_B$8

leading to a quotient manifold

$\theta_A,\theta_B$9

as the natural domain for symmetry-aware LoRA merging (Silva et al., 29 Apr 2026).

This quotient viewpoint clarifies why coordinate-space averages can fail even when checkpoints are functionally equivalent. It also suggests that “alignment” in many practical methods is not merely a preprocessing trick but an attempt to approximate intrinsic averaging on quotient manifolds (Silva et al., 29 Apr 2026). A plausible implication is that future scalable methods may increasingly replace ad hoc alignment with explicit quotient-geometric constructions.

4. Riemannian manifolds and metric-aware merging

Some recent work makes the manifold structure explicit by choosing a non-Euclidean model manifold and performing averaging or composition there. "Orthogonal Model Merging" (Yang et al., 5 Feb 2026) exemplifies this strategy with the orthogonal group

$\gamma(t)$0

viewed as a compact Lie group and Riemannian manifold (Yang et al., 5 Feb 2026). The motivation is that pretrained weight matrices possess intrinsic geometric structure, including hyperspherical energy, and arbitrary Euclidean updates destroy inner products, norms, and spectral properties (Yang et al., 5 Feb 2026). If task-specific adaptation is represented by orthogonal transformations $\gamma(t)$1, OrthoMerge maps them to the Lie algebra $\gamma(t)$2 via the inverse Cayley transform

$\gamma(t)$3

averages in the algebra, applies a magnitude correction

$\gamma(t)$4

and maps back using

$\gamma(t)$5

The merged structural component $\gamma(t)$6 thereby stays on the orthogonal manifold (Yang et al., 5 Feb 2026).

The paper extends this beyond OFT by orthogonal-residual decoupling: for a target weight $\gamma(t)$7, solve the orthogonal Procrustes problem

$\gamma(t)$8

with closed-form solution $\gamma(t)$9 from the SVD of $\gamma(0)=\theta_A$0, merge the orthogonal parts on the manifold, and merge residuals additively in Euclidean space (Yang et al., 5 Feb 2026). Empirically, on Llama-3.1-8B with OFT, average accuracy over five tasks improves from 44.97 for the best Euclidean baseline to 46.25 for OrthoMerge, and out-of-domain average reaches 41.80 (Yang et al., 5 Feb 2026).

A distinct metric-aware line of work uses information geometry. "Functionality-Oriented LLM Merging on the Fisher--Rao Manifold" (Wang et al., 5 Mar 2026) formulates merging as a weighted Karcher mean on the Fisher–Rao manifold, with objective

$\gamma(0)=\theta_A$1

where the local Fisher–Rao distance approximates KL divergence between predictive distributions (Wang et al., 5 Mar 2026). The first-order condition is

$\gamma(0)=\theta_A$2

so the target is a Riemannian center of mass in function space rather than an arithmetic average in weight space (Wang et al., 5 Mar 2026). Because exact FR log/exp maps are intractable for LLMs, the paper introduces a blockwise spherical proxy that normalizes blocks to the sphere, computes a spherical Karcher mean, and rescales by representative norms (Wang et al., 5 Mar 2026). The method is reported to be stable under increasing heterogeneity and number of experts, outperforming baselines for both $\gamma(0)=\theta_A$3 and $\gamma(0)=\theta_A$4 expert merges on 14B-scale Qwen models (Wang et al., 5 Mar 2026).

The Fréchet-averaging framework in (Silva et al., 29 Apr 2026) generalizes such ideas. On a metric space or Riemannian manifold $\gamma(0)=\theta_A$5, the Fréchet mean is defined as

$\gamma(0)=\theta_A$6

Euclidean averaging is recovered as the flat case, whereas Fisher merging emerges as a particular approximation under Fisher–Rao geometry, Gaussian/Laplace assumptions, and a quadratic surrogate (Silva et al., 29 Apr 2026). The same paper argues, however, that Fisher merging remains symmetry-unaware because it refines the metric but not the averaging domain; quotient geometry is required for full invariance (Silva et al., 29 Apr 2026). This distinction is important: not every metric-weighted average is manifold-aware in the strong sense.

5. Feature-space manifolds, output-space subspaces, and dynamic routing

Not all manifold-based approaches operate in parameter space. A second major strand treats tasks or model residuals as living on subspaces or manifolds in feature or output space. "Training-free Task Classification for Multi-Task Model Merging" (Son et al., 21 Jun 2026) models each task as a low-rank affine subspace in the pretrained backbone’s feature space. For task $\gamma(0)=\theta_A$7, one extracts support features $\gamma(0)=\theta_A$8, computes the mean

$\gamma(0)=\theta_A$9

forms the centered matrix $\gamma(1)=\theta_B$0, and computes

$\gamma(1)=\theta_B$1

Using the top-$\gamma(1)=\theta_B$2 right singular vectors $\gamma(1)=\theta_B$3, the task manifold is

$\gamma(1)=\theta_B$4

Routing is performed by the projection residual

$\gamma(1)=\theta_B$5

with hard assignment

$\gamma(1)=\theta_B$6

This is explicitly a training-free nearest-subspace classifier in feature space (Son et al., 21 Jun 2026).

The importance of this construction is that manifold geometry becomes the control mechanism for dynamic merging. Instead of training a router or assuming task IDs at inference, SiM precomputes low-rank manifolds from a small support set—often 32 examples per task—and uses them to select which compressed task vector $\gamma(1)=\theta_B$7 to activate, yielding

$\gamma(1)=\theta_B$8

for each input group (Son et al., 21 Jun 2026). On ViT-B/32 with 20 vision tasks, the paper reports experts at 91.3, best static merge at 77.1, best dynamic baseline at 79.3/74.4, and SiM+TSV-C at 88.6 (Son et al., 21 Jun 2026). Task classification accuracy on eight vision tasks is reported as approximately 99% with $\gamma(1)=\theta_B$9 support examples (Son et al., 21 Jun 2026).

A related but distinct subspace formulation appears in "Model Merging by Output-Space Projection" (Evans et al., 27 May 2026). There the merged residual at one layer is optimized by a convex quadratic program over calibration outputs, and merging is interpreted as projection of base-model output residuals onto a chosen output subspace $\mathcal{L}(\gamma(t))$0 (Evans et al., 27 May 2026). If $\mathcal{L}(\gamma(t))$1 denotes the base-model residual on calibration point $\mathcal{L}(\gamma(t))$2, then the optimal residual after projection is $\mathcal{L}(\gamma(t))$3, and the minimal calibration loss is

$\mathcal{L}(\gamma(t))$4

Writing

$\mathcal{L}(\gamma(t))$5

the captured residual energy is

$\mathcal{L}(\gamma(t))$6

which serves as a closed-form diagnostic of how well the chosen basis captures the residual manifold (Evans et al., 27 May 2026). The optimal $\mathcal{L}(\gamma(t))$7-dimensional output subspace is given by the top eigenvectors of $\mathcal{L}(\gamma(t))$8, yielding a spectral criterion for basis design (Evans et al., 27 May 2026). This is a clear instance in which the “manifold” is explicitly approximated by a low-dimensional output subspace induced by calibration data.

Feature-space and output-space formulations differ from parameter-manifold methods in their ontology. They define closeness by explainability of representations or residual outputs rather than by parameter geometry. Yet they share the same structural logic: good merged behavior lies in a lower-dimensional geometric object, and the algorithm succeeds insofar as it remains near that object (Son et al., 21 Jun 2026, Evans et al., 27 May 2026).

6. Modular, continual, and pre-training variants

The manifold view extends beyond one-shot static merging. In continual settings, "Towards Adaptive Continual Model Merging via Manifold-Aware Expert Evolution" (Qiu et al., 24 Apr 2026) models each task-specific update $\mathcal{L}(\gamma(t))$9 for module $\mathcal{M}$0 by a truncated SVD

$\mathcal{M}$1

yielding input and output subspaces on Grassmann manifolds (Qiu et al., 24 Apr 2026). The similarity between experts is measured by a projection-based subspace affinity

$\mathcal{M}$2

which equals the mean squared cosine of principal angles and is a normalized inverse chordal distance (Qiu et al., 24 Apr 2026). A distribution-aware threshold

$\mathcal{M}$3

decides whether to merge a new expert into an existing subspace or create a new expert (Qiu et al., 24 Apr 2026). Routing is then performed by feature–subspace alignment rather than learned gating. MADE-IT reports higher accuracy and lower forgetting than both backbone-centric and MoE baselines across long task sequences, with expert reduction rates up to approximately 95% in generic modules (Qiu et al., 24 Apr 2026). This is an explicitly Grassmannian formulation of manifold-based model merging.

A modular interpretation also governs "Fine-Grained Model Merging via Modular Expert Recombination" (Qiu et al., 6 Feb 2026). MERGE decomposes models into components $\mathcal{M}$4, groups homologous components across tasks via a discrete matrix $\mathcal{M}$5, merges each group with a static merger $\mathcal{M}$6, optionally quantizes it via $\mathcal{M}$7, and reconstructs task-specific or routed models from these modular experts (Qiu et al., 6 Feb 2026). The paper frames the search over configurations as a bi-objective optimization over cross-task performance and storage, producing a Pareto front of efficient modular expert libraries (Qiu et al., 6 Feb 2026). Although the paper does not formalize manifolds, its own exposition notes that this can be interpreted as a product space $\mathcal{M}$8 over component manifolds, with routing selecting points in a discrete approximation to that product manifold (Qiu et al., 6 Feb 2026).

At the scale of pre-training trajectories, "Extra-Merge: Tracing the Rank-1 Subspace of Model Merging in LLM Pre-Training" (Zhou et al., 26 May 2026) identifies a distinct manifold phenomenon. For late-stage pre-training checkpoints, principal model averaging produces merged checkpoints whose centered parameter matrix has first explained variance ratio $\mathcal{M}$9 across GPT-2 and LLaMA models up to 2B parameters (Zhou et al., 26 May 2026). The paper interprets this as a stable approximate one-dimensional linear manifold, or Rank-1 Subspace, arising because averaging acts as a geometric low-pass filter in a river–valley landscape (Zhou et al., 26 May 2026). Extra-Merge estimates the tangent direction of this 1D manifold by PCA and extrapolates

$\mathcal{B}_\epsilon(\theta^*)$0

with $\mathcal{B}_\epsilon(\theta^*)$1 chosen by validation loss, thereby moving beyond the convex hull of observed checkpoints without gradient updates (Zhou et al., 26 May 2026). This is a local manifold-extrapolation method rather than a multi-task merge, but it reinforces the broader claim that mergeable structure can collapse to unexpectedly low-dimensional manifolds in the right regime.

Finally, "Model Assembly Learning with Heterogeneous Layer Weight Merging" (Zhang et al., 27 Mar 2025) extends manifold-like reasoning to heterogeneous architectures. MAL performs layer-wise generalized permutation alignment with zero-padding, then convexly combines aligned layers across different architectures and widths (Zhang et al., 27 Mar 2025). Its central empirical finding is that low-loss connectivity can often be preserved locally at the layer level, even when global architecture differs, provided merging is depth-consistent and the base architecture is competent (Zhang et al., 27 Mar 2025). This suggests that layer-wise solution manifolds may overlap across architectures more than whole-model manifolds do.

7. Empirical regularities, mergeability, and open problems

Several empirical regularities recur across the literature. First, shared initialization matters. Both surveys emphasize that linear interpolation works surprisingly well when models share the same pretrained initialization, tasks are not strongly conflicting, and fine-tuning is relatively mild (Ruan et al., 12 Mar 2025, Song et al., 10 Mar 2026). This is one reason larger instruction-tuned LLMs tend to merge better: in MergeBench, the best methods exceed 90% normalized multi-task performance on 8B–9B and instruction-tuned bases, whereas 2B–3B pretrained bases are notably worse (2505.10833). The benchmark also reports that model merging tends to perform better on stronger base models, and that coefficient tuning and sparsification improve knowledge retention (2505.10833).

Second, mergeability is not uniform across tasks. "Will it Merge? On The Causes of Model Mergeability" (Rahamim et al., 10 Jan 2026) defines mergeability as the probability that a task update’s knowledge survives random merges and finds that base model knowledge is a dominant predictor: tasks or examples the base already “knows” better are more mergeable (Rahamim et al., 10 Jan 2026). This suggests that manifold proximity to the base model may be more informative than raw update norm, which shows only weak correlations except in extremely non-mergeable cases (Rahamim et al., 10 Jan 2026). The same paper proposes weighted averaging based on inverse base-task accuracy,

$\mathcal{B}_\epsilon(\theta^*)$2

to better preserve weak base knowledge (Rahamim et al., 10 Jan 2026). A plausible implication is that mergeability metrics may eventually become geometry-aware priors for deciding whether to merge, route, or keep tasks separate.

Third, where one merges matters. DRM’s results show that moving to a better-aligned SVD-renormalized space can outperform several state-of-the-art merging baselines across ViT, DeBERTa, T5, and Llama3.1-8B (Chaichana et al., 29 May 2025). Output-space projection shows that the quality of a chosen basis can be predicted from calibration residual energy alone (Evans et al., 27 May 2026). SiM shows that dynamic routing by feature-space manifolds can narrow the gap to experts without training a router (Son et al., 21 Jun 2026). These results collectively weaken any simplistic view that “model merging” is primarily about inventing better arithmetic on raw weights.

The field’s open problems are correspondingly geometric. The surveys highlight the lack of theory for large transformers and LLMs, the difficulty of scalable permutation matching and Fisher/Hessian approximations, the interaction between compression and merging, the need for dual constraints in weight and activation space, and the absence of predictive mergeability metrics (Ruan et al., 12 Mar 2025, Song et al., 10 Mar 2026). Fréchet-averaging work points to open questions about geodesic convexity, efficient quotient-manifold alignment, and richer metrics beyond Euclidean or diagonal Fisher (Silva et al., 29 Apr 2026). Fisher–Rao Karcher methods still rely on approximations such as spherical proxies (Wang et al., 5 Mar 2026). Continual and modular methods raise unresolved questions about automatic hyperparameter selection, scaling to larger models, and how to learn better decompositions than fixed transformer blocks (Qiu et al., 24 Apr 2026, Qiu et al., 6 Feb 2026).

A common misconception is that manifold-based merging necessarily requires explicit differential geometry or expensive geodesic solvers. The evidence across these papers does not support that claim. Many practically successful methods are only approximately manifold-aware: they use local linearizations, low-rank subspaces, alignment transforms, or sparse submanifolds as tractable surrogates (Ruan et al., 12 Mar 2025, Chaichana et al., 29 May 2025, Evans et al., 27 May 2026). Conversely, another misconception is that successful merging is purely empirical and unrelated to geometry. The consistent role of mode connectivity, symmetry alignment, output residual subspaces, low-rank feature manifolds, and metric-aware averaging argues against that view (Song et al., 10 Mar 2026, Silva et al., 29 Apr 2026).

Taken together, the literature supports an informal but precise synthesis. Manifold-based model merging is the practice of exploiting the fact that useful model adaptations are geometrically structured: as low-loss manifolds in parameter space, quotient manifolds under symmetry, low-rank manifolds of adapters, subspaces of feature distributions, or low-dimensional residual subspaces in output space. Methods differ in what manifold they posit and how they move on it—linear interpolation, tangent-vector addition, Fréchet averaging, Karcher means, quotient alignment, subspace projection, modular recombination, or nearest-subspace routing—but they are unified by a single constraint: the merged model should remain on or near the geometry that makes the source experts good in the first place (Ruan et al., 12 Mar 2025, Silva et al., 29 Apr 2026, Wang et al., 5 Mar 2026).