Mode Connectivity in Learning Landscapes

• Mode connectivity is the existence of continuous, low-loss paths in parameter or input space that connect distinct minima, revealing the geometry of nonconvex landscapes.
• Mechanisms like dropout, symmetry alignment, and curvature analysis are key to constructing paths with minimal loss barriers between different model solutions.
• Its applications span deep learning, quantum circuits, and federated settings, offering practical benefits in generalization, robustness, and model repair.

Mode connectivity refers to the existence of continuous, low-loss paths in parameter space (or, more recently, in input space) connecting distinct minima of a loss function, typically in neural network or variational circuit settings. Contrary to classical expectations from nonconvex optimization, empirically observed minima in modern high-dimensional machine learning landscapes are often not isolated but reside on a connected manifold. Mode connectivity thus characterizes a central aspect of the geometry and trainability of these landscapes, with ramifications spanning deep learning theory, quantum variational algorithms, federated learning, pre-trained LLMs, sparse network design, and beyond.

1. Formal Definitions and Mathematical Frameworks

The standard setting fixes a parameterized model (e.g., DNN, QCBM) with a real parameter vector $\theta\in\mathbb{R}^N$ and scalar loss $L(\theta)$. Two local minima $\theta_A$, $\theta_B$ are said to exhibit mode connectivity if there exists a continuous path $\gamma:[0,1]\to\mathbb{R}^N$ with $\gamma(0)=\theta_A$, $\gamma(1)=\theta_B$ such that

$$\max_{t\in[0,1]} L(\gamma(t)) \leq \max\{L(\theta_A),L(\theta_B)\} + \epsilon$$

for small $\epsilon$ (often $\epsilon=0$ in empirical studies). The simplest instance is linear mode connectivity (LMC), where $L(\theta)$0 for $L(\theta)$1.

The barrier height $L(\theta)$2 is defined as

$L(\theta)$3

with $L(\theta)$4 measured along the interpolating path. Low or vanishing barriers ($L(\theta)$5) across paths indicate strong mode connectivity. Recent theoretical developments extend mode connectivity to Riemannian geometry contexts, considering geodesics with respect to the Fisher–Rao metric or other information-geometric measures, as in (Tan et al., 2023).

2. Mechanisms and Theoretical Explanations

Mode connectivity is not a generic feature of arbitrary nonconvex landscapes. Two principal mechanisms, empirically supported and theoretically analyzed, underpin its ubiquity in practical models:

• Dropout and Noise Stability: The property that solutions are robust to stochastic dropout or Gaussian noise, so that even after ablation of a large proportion of hidden units, the loss raises only marginally. This enables construction of explicit piecewise-linear (polygonal chain) paths of $L(\theta)$6 (typically 2–13) segments with bounded loss barrier between minima, based on sparsifying and permuting active units (Kuditipudi et al., 2019).
• Permutation and Scaling Symmetries: Many weight-space symmetries (e.g., permutation of hidden units, rescaling in linear or homogeneous nets) induce a large connected symmetry group in parameter space. Locally optimal solutions within the same symmetry orbit can be joined by symmetry-induced curves with exactly constant loss. Linear mode connectivity is frequently observed after aligning these symmetries (e.g., via neuron alignment or layer permutations) (Zhao et al., 29 May 2025, Tatro et al., 2020).
• Curvature and Distance Constraints: Quantitative analysis on the loss surface shows the barrier along a linear path between minima is determined by the quadratic form $L(\theta)$7 (where $L(\theta)$8 is the Hessian, and $L(\theta)$9 is the endpoint parameter difference), with higher curvature or longer endpoints increasing the barrier (Singh et al., 2024).

3. Algorithms for Path Construction and Barrier Measurement

Multiple methodologies are employed across settings:

Algorithm	Description	Typical Barrier Outcome
Linear Interpolation	$\theta_A$0	Suffices in many overparametrized or aligned settings; may see barriers otherwise (Lubana et al., 2022, McDermott et al., 2023)
Polygonal Chain (Piecewise)	Path with one or more “bends” (auxiliary points) optimized to minimize average loss along the path	Consistently produces low-loss curves bridging minima in vision, NLP, federated learning (Gotmare et al., 2018, Zhou et al., 2023)
Quadratic Bézier / Geodesic	$\theta_A$1, with learned $\theta_A$2; or Fisher–Rao geodesic discretizations	Eliminates barriers where linear fails (e.g., narrow nets, quantum circuits) (Tan et al., 2023, Hamilton et al., 2021)
Neuron Alignment + Path Opt	Optimize discrete neuron permutations for maximum activation correlation before path search	Removes artificially induced barriers, reveals intrinsic connectivity (Tatro et al., 2020)
Input Space Mode Connectivity	Paths in input space (e.g., $\theta_A$3 between images) under fixed model $\theta_A$4, leveraging high-dimensional percolation	Explains linear or near-linear input-space connectivity; barrier height predicted by percolation theory (Vrabel et al., 2024)

4. Empirical Findings Across Domains

• Deep Learning (Vision, NLP): In overparametrized supervised DNNs, independently trained minima are consistently joined by low-loss paths, both linearly (after symmetry alignment) or by simple nonlinear curves. This holds across variations in optimizer, batch size, initialization, training schedule, and even occasionally across domains or tasks after pretraining (Gotmare et al., 2018, Qin et al., 2022, Singh et al., 2024).
• Quantum Circuits: In Quantum Circuit Born Machines (QCBMs) and related VQAs, mode connectivity is sensitive to the choice of rotation gate parameterization and entangling layer pattern. Richer parameterizations (e.g., AGP) admit geodesic low-loss paths even when linear interpolation barriers are large, whereas sparse ensembles may connect trivially but remain uninformative (Hamilton et al., 2021).
• Graph Neural Networks: Mode connectivity is pronouncedly non-linear; linear interpolation between minima often yields high barriers, but quadratic Bézier paths can nearly always connect. The presence and severity of the barrier is controlled not by GNN architecture, but by properties of the graph structure (homophily, density, spectral gap) (Li et al., 18 Feb 2025).
• Sparse and Distilled Nets: Iterative magnitude-pruned subnetworks found via synthetic (distilled) data exhibit much stronger linear mode connectivity and flatter minima than those discovered with full real data IMP, up to extreme sparsities (90%), with the distillation procedure filtering away directions of sharpness and instability (McDermott et al., 2023).
• Federated and Distributed Settings: Heterogeneity (in data distributions across clients) induces barriers between modes trained in different contexts. Non-linear paths (piecewise-linear/polygonal) eliminate such barriers, and both theory (mean-field limit, dropout stability) and experiment show that increased model width restores connectivity (Zhou et al., 2023).
• Optimization Dependence: At large width, solutions found by distinct optimizers (AdamW, Muon, Signum) within each optimizer's implicit regularization set are connected, but different optimizers can select disjoint minimizer islands separated by large barriers at finite width or under sufficiently strong regularization. Spectral characteristics of interpolated models can transition smoothly or sharply, revealing optimizer-induced geometric structure (Zhang et al., 11 May 2026).

5. Extensions and Interpretations: Beyond Standard Parameter Space

• Input Space Mode Connectivity: For a fixed trained model, there exist low-loss curves in input space connecting diverse images of the same class or even between synthetic modes. In high dimensions, such connectivity can be explained via percolation theory—mode clusters percolate the input space in the large $\theta_A$5 limit. This extension enables new adversarial detection schemes and interpretability approaches (Vrabel et al., 2024).
• Mechanistic Connectivity: Absence of linear mode connectivity correlates tightly with mechanistic dissimilarity (e.g., reliance on different input features, spurious attributes). Thus, mode connectivity serves as a diagnostic for the invariance or attribute reliance of minima, and can be directly enforced or disrupted by targeted regularization or fine-tuning (e.g., CBFT) (Lubana et al., 2022).
• Applications in Machine Unlearning and Repair: Mode connectivity enables efficient machine unlearning by facilitating nonlinear unlearning pathways and the construction of a spectrum of unlearning models along a connectivity curve. In the context of adversarial or backdoored models, mode connectivity-based repair methods interpolate models along low-loss paths using only a small clean dataset (2505.10859, Cheng et al., 8 Apr 2025, Zhao et al., 2020).
• Differentiable Mechanism Design: Neural architectures in auction design (e.g., RochetNet, AMAs) exhibit explicit, theoretically justified mode connectivity for any two solutions under large enough "menu" size or under suitable reducibility. Paths are realized by simple multi-segment linear interpolations (Hertrich et al., 2023).
• Curvature and Path Length: The loss barrier along the linear chord between two minima is bounded by the product of maximum curvature (from local Hessians) and the squared chord length, furnishing explicit quantitative criteria for approximate linear mode connectivity (Singh et al., 2024, Zhao et al., 29 May 2025).

6. Practical Implications, Limitations, and Open Problems

• Generalization and Ensembling: Connectivity is strongly tied to generalization; minima with low connecting barriers tend to generalize better, forming the basis for robust ensembling strategies (curve ensembling, fast weight averaging) (Qin et al., 2022).
• Landscape Geometry and Diagnoses: Path barrier height and smoothness serve as geometric diagnostics for overfitting, domain mismatch, optimizer-induced diversity, and spurious correlation reliance. This enables model selection and hyperparameter tuning sensitive to landscape topology (Li et al., 18 Feb 2025).
• Robustness and Security: The location and size of robustness barriers (as opposed to standard loss barriers) quantify the "cost" of adversarial defense, with input Hessian curvature tightly correlated with adversarial vulnerability along connectivity paths (Zhao et al., 2020).
• Limitations: Mode connectivity may not hold universally, especially at small width, extreme regularization, or for dissimilar data/task regimes without suitable symmetry alignment. Curvature analysis reveals quantitative thresholds for when barriers emerge (Zhang et al., 11 May 2026, Singh et al., 2024).
• Open Directions: Research continues on establishing sufficiency and necessity of conditions for connectivity at scale, optimizing higher-dimensional or nonlinear connecting surfaces, exploring unsupervised or reinforcement contexts, extending results to discrete domains (NLP, GNNs), and leveraging connectivity for domain adaptation, continual learning, and mechanism editing.

7. Domain-Specific Examples and Applications

• Quantum Circuit Optimization: QCBMs with expressive (AGP-type) rotations admit NEB-found geodesic paths between minima, even when simple linear interpolation fails, demonstrating the necessity of detour-based pathfinding for variational quantum algorithms (Hamilton et al., 2021).
• Pre-trained LLMs: Mode connectivity is preserved across different data-orders, hyperparameters, and even between different domain/task fine-tunings after sufficient pretraining. Ensembling along the path often outperforms the original minima. Weightwise or module-wise interpolation further improves transfer and multi-tasking (Qin et al., 2022).
• Wave Physics: The concept of mode connectivity generalizes to physical wave systems, quantifying, via the cross density of states, the extent to which eigenmodes of a disordered medium connect spatial locations at a given frequency; critically, the connectivity discriminates between diffusive and Anderson localized regimes, with practical measurement protocols via second-order coherence (Canaguier-Durand et al., 2018).

In conclusion, mode connectivity constitutes a unifying geometric property of modern high-dimensional learning landscapes, underpinned by generic stability and symmetry mechanisms, and enables practical algorithmic advances in training, generalization, robustness, and model lifecycle management across domains (Gotmare et al., 2018, Lubana et al., 2022, Zhou et al., 2023, Zhang et al., 11 May 2026).