---
title: Linear Mode Connectivity in Neural Networks
url: https://www.emergentmind.com/topics/linear-mode-connectivity
type: topic
---

# Linear Mode Connectivity in Neural Networks

Linear mode connectivity (LMC) is the empirical and theoretical phenomenon that two independently trained solutions (modes) of a neural network—often obtained via stochastic gradient descent (SGD) from different random seeds or data permutations—can, after proper symmetry alignment, be connected by a straight line segment in weight space along which the training or test loss remains essentially constant, with no intervening barrier. This structure, which appears surprising in highly non-convex loss landscapes, has significant implications for model merging, ensembling, optimization stability, architecture analysis, and the understanding of the deep learning loss geometry.

## 1. Formal Definition and Fundamental Characterization

Let $\theta_A, \theta_B \in \mathbb{R}^d$ represent two trained parameter vectors of a neural network and let $\ell(\theta)$ denote the empirical risk or loss. The straight-line interpolation is
\[
\theta(t) = (1-t)\theta_A + t\theta_B,\qquad t \in [0,1].
\]
The pointwise interpolation loss is $L_{\mathrm{interp}}(t) = \ell(\theta(t))$, and the barrier is
\[
B(\theta_A, \theta_B) = \max_{t \in [0,1]} \ell(\theta(t)).
\]
$\theta_A$ and $\theta_B$ are linearly mode connected if $B(\theta_A, \theta_B) \approx \max \{\ell(\theta_A), \ell(\theta_B)\}$ (typically, a near-zero increase above endpoint losses), i.e., the loss remains flat or has a negligible hump between the modes. In practice, one also measures the difference versus a convex combination, i.e., $B' = \max_{t}\{\ell(\theta(t)) - [(1-t)\ell(\theta_A) + t\ell(\theta_B)]\}$, but these are functionally equivalent in the overparameterized setting [2506.22712], [2312.09832], [2110.06296].

## 2. Symmetry Structure and Alignment Mechanisms

Neural networks possess extensive parameter-space symmetries that render many minima functionally equivalent yet parametrically distinct. These symmetries arise from:

- **Permutations:** Hard one-to-one neuron/channel reorderings: $P \theta$ for permutation matrix $P$, common in MLPs (hidden units) and CNNs (channels).
- **Semi-permutations:** Sub-stochastic, sparse matrices mapping one-to-many or many-to-one, particularly in residual or attention head blocks in deep transformers.
- **Orthogonal Transformations:** $O^\top O = I$, relevant to normalization or residual subspaces (e.g., RMSNorm).
- **Invertible Linear Maps (GL):** Full-rank weight matrices in key modules, such as QK/OV projections in attention circuits.

Let $G$ be a symmetry group acting on parameter space. We seek
\[
\phi^\star = \arg\min_{\phi \in G} \max_{t \in [0,1]} \ell\big((1-t)\theta_A + t\,\phi(\theta_B)\big),
\]
that is, the optimal symmetry $\phi$ (composition of permutations, semi-permutations, orthogonal maps, invertibles) to align $\theta_B$ so that linear interpolation with $\theta_A$ avoids loss barriers [2506.22712], [2503.06001].

**Algorithmic approaches:**
- **Weight matching:** Bilinear assignment (Hungarian algorithm) layerwise, possibly extended with Procrustes for orthogonal matching. Attention heads employ cost matrices based on Frobenius distances. [2506.22712], [2402.04051]
- **Activation matching:** Data-driven, matches neuron activations empirically to align functional subspaces.
- **Learned matching (end-to-end):** Unconstrained alignments parametrized and optimized jointly with projections to the relevant symmetry classes, enabling gradient-based refinement and fully exploiting $G$ [2506.22712].

## 3. Theoretical Foundations and Empirical Findings

### 3.1. Emergence of LMC in Practice and Theory

Early empirical studies identified large barriers when interpolating between independently trained networks, but barrier-free interpolation emerges after symmetry alignment (permutation, etc.) [2110.06296], [2210.06671]. The phenomenon is theoretically underpinned in wide two-layer networks via optimal transport and in multi-layer settings via recursively controlled neuron alignments. Localization of the minimal barrier is rate-limited by width and intrinsic layerwise dimension [2310.19103]:
\[
B_{\rm perm}(\theta_A,\theta_B) = O(m^{-1/2})
\]
for two-layer ReLU with width $m$ [2503.06001]; general bounds scale as $O(m^{-2/d})$ for $d$-dimensional supports [2310.19103].

### 3.2. Architectural and Optimization Dependence

LMC strongly depends on:
- **Width:** Sufficiently overparameterized (wide) networks are a prerequisite; double descent in the LMC barrier is observed as width increases [2503.06001].
- **Symmetry complexity:** Transformers require more than permutations: semi-permutations, orthogonals, invertibles [2506.22712].
- **Optimization regime:** SGD at appropriate learning rates, batch sizes, and scheduling can promote or destroy LMC; adaptive optimizers (Adam) may break LMC by pushing iterates out of shared basins unless careful warmup is provided [2312.09832].
- **Dataset complexity:** Harder classification tasks and deep architectures fragment the loss landscape, breaking LMC in the absence of massive overparameterization or symmetry exploitation [2312.09832].
- **Initialization and training phase:** Models become LMC-stable only after early training; initializations rarely yield stability, and the timing of the SGD trajectory fork is crucial [1912.05671], [2406.16300].

### 3.3. Quantitative Results

Representative experiments on state-of-the-art vision transformers (ViT) and GPT-2 demonstrate that, for independently trained pairs:

|              Method                  | LMC Barrier (CIFAR-10) |
|:------------------------------------- |:----------------------:|
| Vanilla averaging                    |          1.69          |
| Activation matching (AM)             |          1.27          |
| Weight matching (WM)                 |          0.36          |
| Learned symmetries (perm only)       |          0.45          |
| Learned all symmetries (full $G$)    |     **0.00**           |

Zero-barrier interpolation is achieved only by accounting for the full hierarchy of symmetries in transformers. Weight or activation matching alone reduces, but does not entirely eliminate, the barrier. Similar findings apply to GPT-2: learned full-symmetry matching yields a barrier of 0.41 compared to 4.3 for vanilla and 1.58 for permutation/WM alone [2506.22712].

## 4. Generalizations and Special Cases

### 4.1. Layer-wise Connectivity

Even when joint LMC fails, interpolating only one layer at a time rarely induces a barrier. In deep linear networks, layerwise interpolation implies convexity of the loss along that direction [2307.06966]; in non-linear nets, empirical heatmaps show no layer-wise barrier—especially in early and late layers, with possible exceptions in mid-blocks.

### 4.2. Mixture-of-Experts, Tree Ensembles, and Other Models

LMC extends beyond standard MLPs/CNNs:
- **Mixture-of-Experts (MoE):** Mode connectivity is preserved up to permutation of experts and gating functions; matching algorithms efficiently align functional components [2509.11348].
- **Differentiable tree ensembles:** Require accounting for subtree flip and split order invariance in addition to tree-permutation, or, for decision-list variants, only tree permutation [2405.14596].
- **Sparse Networks:** Synthetic-distilled subnetworks exhibit stable, flat, linearly connected basins after pruning, unlike standard sparse or dense networks [2310.18769].

### 4.3. Star-Shaped and Two-Piece Linear Connectivity

For multiple minima, there exists (in overparameterized teacher-student and linear regimes) a center mode such that all minima are two-piece linearly connected to it, rendering the landscape nearly convex (normalized geodesic distance close to 1) [2404.06391].

## 5. Implications for Optimization, Model Fusion, and Ensembling

LMC reveals that—even in ostensibly rugged landscapes—practical minima from stochastic optimization are not separated by insurmountable energy ridges but are connected (after symmetry alignment) by flat directions in parameter space. This permits:
- **Model merging:** Linear interpolation after symmetry matching yields intermediary models with loss indistinguishable from the original endpoints [2210.06671], [2402.04051].
- **Fine-tuning and continual learning:** Sequentially found minima can be constrained to lie on the same (possibly linear) path as multitask solutions, reducing catastrophic forgetting [2010.04495].
- **Ensembling and federated learning:** Layer- or block-wise averaging enables efficient fusion without loss barriers [2307.06966].
- **Analysis of risk and generalization:** Flat, connected basins correlate with improved generalization; LMC serves as an indicator of robustness to training noise [1912.05671], [2511.04514].

## 6. Theoretical Analyses and Open Problems

- **Barriers and Hessians:** Second-order approximations predict the loss barrier: $B_{\max} \approx \frac18 \Delta^\top H \Delta$, with $\Delta = \theta_B - \theta_A$ and $H$ the Hessian [2406.16300].
- **Symmetry group topology:** The structure and action of $G$ determine the number and connectivity of minima; skip connections reduce disconnectedness [2505.23681].
- **Failure modes:** Unaligned scales or incomplete symmetry removal can induce arbitrarily large barriers; continuous scaling symmetries in homogeneous nets must be factored out [2505.23681].
- **Extensions to further architectures:** Ongoing work is extending LMC and symmetry analysis to transformers with richer symmetries (beyond permutations), to MoEs, and to differentiable trees, as well as to applications in multitask and federated settings [2506.22712], [2509.11348], [2405.14596].

## 7. Future Research Directions

Promising avenues include:
- Scaling symmetry-aware alignment to larger and more diverse models (LLama, multi-task transformers).
- Automated identification and exploitation of soft and continuous symmetries (e.g., via Sinkhorn relaxations).
- Analysis of triangle inequalities and transitivity of symmetries for multi-model fusion.
- Detailed study of optimizer dynamics, initialization choices, and their effect on basin selection and LMC [2506.22712], [2312.09832].
- Generalization to sequence and tree architectures as well as practical ensembling in the presence of data and domain shifts.

In summary, linear mode connectivity—once elusive and relegated to small or idealized settings—has been rigorously characterized, both empirically and theoretically, across diverse architectures and learning paradigms. Proper consideration of latent parameter symmetries is essential; with this, the loss geometry of modern networks is revealed to be unexpectedly benign, with broad, connected valleys supporting practical innovations in merging, ensembling, and continual learning [2506.22712].

Source: https://www.emergentmind.com/topics/linear-mode-connectivity