Saddle-to-Saddle Dynamics in Optimization

• Saddle-to-Saddle Dynamics is a regime in optimization where trajectories traverse unstable saddle points via heteroclinic orbits, explaining stage-wise complexity growth.
• It employs high-index saddle formulations using Hessian eigen decomposition to invert unstable directions, guiding efficient mapping of solution landscapes.
• This framework underpins observed phenomena like loss plateaus and bursts in neural training, and aids in transition state analysis in physical and chemical systems.

Saddle-to-saddle dynamics describes a regime in nonlinear optimization and dynamical systems where trajectories connect a sequence of saddles—unstable stationary points—via heteroclinic orbits, typically before ultimately reaching a minimum. This structure underlies a broad class of phenomena in machine learning (deep networks), computational chemistry, mathematical physics, and dynamical systems, and has prompted the development of specialized algorithms to systematically trace solution landscapes through their saddle points. Saddle-to-saddle regimes are responsible for pronounced stages, plateaus, and bursts in training and optimization, as well as for the emergence of simplicity and low-rank biases in overparameterized models.

1. Foundational Framework and Mathematical Formulation

Saddle-to-saddle dynamics formally arise in settings where the loss, energy, or Hamiltonian landscape contains a hierarchy of critical points of increasing or varying Morse index (number of unstable directions). For a parameter space $x \in \mathbb{R}^d$ and a smooth energy $E(x)$ or loss $L(x)$, critical points $x^*$ satisfy $\nabla E(x^*) = 0$, with the Hessian $H(x^*) = \nabla^2 E(x^*)$ having both positive and negative eigenvalues at saddles.

In high-index saddle dynamics, the trajectory evolves as

$$\dot{x} = \beta \left( I - 2\sum_{i=1}^k v_i v_i^\top \right)(-\nabla E(x)),$$

where $\{v_i\}_{i=1}^k$ are orthonormal bases of the subspace spanned by the $k$ negative Hessian directions at the saddle, ensuring the flow inverts descent in unstable subspaces while descending in stable ones (Liu et al., 2024, Liu et al., 3 Jan 2026).

In systems with symmetry (e.g., deep networks), the landscape contains continuously embedded families of saddles—fixed points in lower-complexity submanifolds—which serve as waypoints for the dynamics (Zhang et al., 23 Dec 2025). Escape from one saddle generically leads, via a heteroclinic orbit, to a saddle of higher complexity.

2. Saddle-to-Saddle Regimes in Deep Networks

In deep linear and ReLU networks with small initialization, the origin is a degenerate saddle where all weights are zero and the gradient vanishes. Analyzing the local expansion around this saddle reveals escape directions with strong low-rank bias: in depth-$L$ ReLU networks, the leading singular value $\sigma_1(W^{(\ell)})$ of the $\ell$-th layer outpaces others by at least a factor of $\ell^{1/4}$ during the first escape, resulting in weight matrices with pronounced bottlenecks (Bantzis et al., 27 May 2025). Subsequent evolution is characterized by a sequence of escapes from saddles of increasing rank, each associated with incrementally more complex solutions as learning progresses stage-wise (Jacot et al., 2021, Abbe et al., 2023, Zhang et al., 23 Dec 2025).

Table: Saddle-to-Saddle Staging in Overparameterized Networks

Stage	Critical Point Structure	Measured Complexity
Initial Plateau	Saddle at origin (low bottleneck rank)	Minimal rank/kinks
Escape 1	Saddle of rank 1 (low complexity)	First singular/kink
Escape 2	Saddle of rank 2 (moderate complexity)	Two singular/kinks
...	...	...
Final Minimum	(Approximate) global minimizer, full rank	Maximal complexity

Saddle-to-saddle dynamics thus explain the empirically observed stage-wise recruitment of features: plateaus in loss, followed by bursts of complexity and gradient spikes, are universal signatures that match the predicted heteroclinic transitions (Abbe et al., 2023, Zhang et al., 23 Dec 2025).

3. General Saddle Dynamics and Solution Landscape Algorithms

Outside machine learning, high-index saddle dynamics (HiSD) and its variants—improved HiSD (iHiSD), shrinking-dimer methods, Gaussian-process- and neural-network-based surrogates—enable the systematic construction of solution landscapes by tracing index-$k$ saddles and their connecting orbits (Liu et al., 2024, Su et al., 6 Feb 2025, Liu et al., 3 Jan 2026, Zhang et al., 2022).

A typical workflow is:

1. Locate an initial high-index saddle by upward or downward search.
2. From the current saddle $(x^*, V^*)$, perturb along each unstable eigenvector $v_i^*$ to initialize downward search.
3. Iteratively trace orbits to the next lower-index saddle, recording directed connections.
4. Repeat recursively to obtain minima and the full directed graph of saddle connections (solution landscape).

Completeness results (under Morse–Smale conditions) guarantee that the entire landscape of stationary points is accessible by chaining iHiSD trajectories (Su et al., 6 Feb 2025). This is critical in physical chemistry, where transition states (saddles) govern reaction networks, and in nonlinear PDEs where states of differing energy or stability are connected by saddle-to-saddle orbits.

4. Saddle-to-Saddle Phenomena in Dynamical Systems

In dynamical systems such as the double pendulum and three-body problem, saddle-to-saddle transport is mediated by families of codimension-1 invariant manifolds and hyperbolic periodic orbits surrounding index-1 saddles. Robust heteroclinic and homoclinic connections can be constructed between these orbits, organizing the global structure of phase space and enabling engineered itineraries over arbitrarily long durations with precisely controlled transitions (Kaheman et al., 2022).

Theoretical results guarantee the existence of true trajectories shadowing any prescribed sequence of heteroclinic jumps among saddles. This directly underpins chaos, global mixing, and control in chaotic Hamiltonian systems, and has practical analogs in energy-efficient space mission trajectory planning.

5. Complexity Growth and Simplicity Bias from Saddle-to-Saddle Learning

In overparameterized neural architectures, saddle-to-saddle dynamics generically induce a “simplicity bias”—networks progressively increase solution complexity over time, learning functions expressible with more units, higher rank, or additional nonlinearities one-by-one (Zhang et al., 23 Dec 2025).

• Linear and convolutional networks learn intermediate solutions of increasing matrix rank or kernel count.
• ReLU networks increase the number of "kinks" (distinct linear regions).
• Attention models incrementally activate more heads.
• Each invariant manifold corresponding to a solution with $h$ effective units is an embedded saddle, and gradient flow alternates between plateaus on these manifolds and transitions ("bursts") to higher complexity.

The durations and locations of plateaus are governed by singular-value/feature gaps of the data and initialization scale.

This universal regime is supported by explicit constructions in deep linear/diagonal networks via mirror-flow and arc-length reparameterizations, and by staged SGD dynamics matching the “leap complexity” of target functions (Pesme et al., 2023, Abbe et al., 2023).

6. Algorithmic and Practical Implementations

Recent software frameworks, such as SaddleScape V1.0, systematically implement HiSD, iHiSD, and GHiSD methods to identify all critical points (including high-index saddles) and their saddle-to-saddle connections, generating directed graphs of solution landscapes (Liu et al., 3 Jan 2026). These frameworks automate Hessian-vector products (analytic, numeric, autodiff), eigenpair solvers, and offer data-driven surrogate modeling (via NNs or GPs) to drastically reduce computation for expensive force or energy function evaluations (Liu et al., 2024, Zhang et al., 2022). Acceleration variants (heavy-ball, Nesterov) further improve efficiency.

A key theoretical advance is that iHiSD enables nonlocal, stable convergence even from initial points outside the basin of attraction of a saddle, guaranteeing that a finite chain of saddle connections suffices to reach any other critical point (Su et al., 6 Feb 2025).

7. Implications and Broader Significance

Saddle-to-saddle dynamics unify phenomena across domains: they provide an explanatory and predictive framework for staged feature learning, simplicity bias, and incremental complexity in neural networks; enable the systematic construction and visualization of solution landscapes in chemistry, physics, and optimization; and organize transport and mixing in chaotic dynamical systems by invariant manifold structure.

The transition between saddles—quantified by analysis of escape directions, invariant manifolds, and explicit algorithms—sets fundamental timescales and complexity-theoretic lower bounds for learning and optimization systems (Abbe et al., 2023). These regimes illuminate the interplay of initialization, data geometry, symmetry, and overparameterization in governing the efficiency and ultimate structure of learned or reacted solutions.