Saddle-to-Saddle Jump Process & Learning Dynamics

• Saddle-to-Saddle Jump Process is a sequential dynamical mechanism where trajectories linger near saddle points before rapidly transitioning to states with higher effective complexity.
• It models incremental learning in gradient-based systems by progressively activating rank components, hidden units, or attention heads as effective degrees of freedom.
• The process links theoretical analyses from deep linear and ReLU networks with practical insights in SGD modulation, low-rank optimization, and nonlinear dynamics.

A saddle-to-saddle jump process is a stagewise dynamical pattern in which a trajectory remains near a saddle, metastable plateau, or lower-complexity invariant set for an extended interval and then undergoes a relatively fast transition to another saddle-related state. In optimization and learning theory, this terminology is used for gradient-based dynamics that progressively activate additional effective degrees of freedom—such as rank components, active coordinates, hidden units, convolutional kernels, or attention heads—so that complexity increases one stage at a time. In nonlinear dynamics, closely related language also appears for rare transitions across stable manifolds and, in some numerical settings, for algorithmic reinitializations that continue tracing a single chaotic saddle rather than motion between distinct saddles (Jacot et al., 2021, Zhang et al., 23 Dec 2025, Wagemakers et al., 2019).

1. Conceptual scope and basic mechanism

In the optimization literature, the core mechanism is a recurrence of three ingredients: embedded saddles, invariant manifolds, and timescale separation. Smaller or simpler solutions are realized inside a larger parameter space; these embedded solutions become saddles or near-saddles for the wider model; and the dynamics spend long intervals near such structures before escaping along a distinguished unstable direction toward a successor state. In the framework developed for a general class of neural networks, a trajectory can start near a low-effective-width manifold, move along it or near it, approach a saddle corresponding to an embedded low-width fixed point, then leave that saddle by breaking exactly one constraint, and finally land near the next invariant manifold with one more effective unit. The resulting learning curve alternates between plateaus and fast transitions, and the transitions are described as heteroclinic-like connections between saddles along invariant manifolds (Zhang et al., 23 Dec 2025).

A distinct but related formalization appears in diagonal linear networks. After an arc-length time reparametrization, the limiting trajectory alternates between a stick phase, in which the predictor remains at a saddle, and a jump phase, in which the path follows a normalized heteroclinic orbit to a new saddle. Here the saddle is not an arbitrary critical point: it is the minimizer of the loss constrained to the currently active coordinates and sign pattern. This makes the process an incremental-learning dynamics in which each visited saddle is the best predictor available under the current support constraints (Pesme et al., 2023).

The same expression does not always denote literal transitions between different invariant saddles. In the saddle-straddle method for Wada basins, repeated expansion-and-refinement cycles generate a chain of short segment approximations that track the same chaotic saddle. The paper explicitly states that these are not jumps between different saddles in the dynamical system; they are algorithmic reinitializations that keep the computed segment close to the invariant saddle (Wagemakers et al., 2019).

2. Deep linear networks and exact saddle sequences

Deep linear networks provide the cleanest analytical setting for saddle-to-saddle dynamics. For a depth-$L$ model with output matrix

$A_\theta = W_L \cdots W_1,$

the initialization scale $\sigma^2=w^{-\gamma}$ produces a phase transition at $\gamma=1$. When $\gamma<1$, initialization is very close to a global minimum and far from saddles; when $\gamma>1$, initialization is very close to a saddle and far from global minima; and $\gamma=1$ is the transition point. In the extreme small-initialization limit, the proposed dynamics are

$$\vartheta^0=0 \;\to\; \vartheta^1 \;\to\; \cdots \;\to\; \vartheta^K,$$

where each intermediate $\vartheta^k$ is a saddle corresponding to a rank-$k$ linear map. The paper further proves the first step of this picture: under a simple top-singular-value condition at the origin, the escape path converges to a deterministic width-1 trajectory embedded in the wider network, up to hidden-layer rotation. The escape time diverges as $A_\theta = W_L \cdots W_1,$0 for $A_\theta = W_L \cdots W_1,$1 and $A_\theta = W_L \cdots W_1,$2 for $A_\theta = W_L \cdots W_1,$3, which explains the long plateaus characteristic of the regime (Jacot et al., 2021).

An exact trajectory-level description is available for two-layer diagonal linear networks. With regression loss

$A_\theta = W_L \cdots W_1,$4

diagonal parametrization $A_\theta = W_L \cdots W_1,$5, vanishing initialization, and time rescaling by $A_\theta = W_L \cdots W_1,$6, the limiting predictor satisfies

$A_\theta = W_L \cdots W_1,$7

This forces the limiting motion to be piecewise constant in physical time: the trajectory sits at one saddle, jumps when a coordinate of the subgradient state hits $A_\theta = W_L \cdots W_1,$8, then sits at a new saddle. The visited saddles and jump times are generated by a recursive algorithm reminiscent of LARS, and the process terminates in finitely many steps at the unique minimum $A_\theta = W_L \cdots W_1,$9-norm solution. In sparse regimes satisfying an RIP-type assumption, the support can be learned one coordinate at a time; more generally, the active set can be non-monotone, with activations and deactivations (Pesme et al., 2023).

These two deep-linear analyses give complementary views of the same phenomenon. One emphasizes rank growth in wide deep parametrizations; the other gives an exact active-set path to the minimum $\sigma^2=w^{-\gamma}$0-norm interpolator. Both identify the saddle-to-saddle pattern with an incremental reconstruction of the final solution rather than a smooth direct descent.

3. Simplicity bias and effective-width growth in modern architectures

A general theoretical framework extends the saddle-to-saddle picture beyond linear models. In that framework, “simple” means expressible with fewer effective units. The specialization depends on architecture: linear networks learn solutions of increasing rank, ReLU networks learn solutions with an increasing number of kinks, convolutional networks learn solutions with an increasing number of convolutional kernels, and self-attention models learn solutions with an increasing number of attention heads. The structural basis is twofold. First, fixed points of narrower networks are embedded inside wider networks. Second, invariant manifolds preserve relations such as equal weights, proportional weights, zeroed-out units, or linear dependence among units. A trajectory can therefore move through a hierarchy of lower-effective-width manifolds and saddles, adding one effective unit at a time (Zhang et al., 23 Dec 2025).

The same work distinguishes two microscopic mechanisms behind the jumps. In linear networks, timescale separation occurs across directions, because components aligned with the leading singular vectors of the data correlation matrix grow as $\sigma^2=w^{-\gamma}$1. In quadratic models such as linear self-attention, the dominant effect is a rich-get-richer competition across units: one head grows fastest because of initialization asymmetry, then another, and so on. This yields the same plateau-and-jump phenomenology, but the source of the stage structure differs. The paper distills saddle-to-saddle learning into two requirements: the escape path from a saddle must closely follow an invariant manifold with one additional effective unit, and initialization must be close to an invariant manifold with fewer effective units than needed to reach zero loss (Zhang et al., 23 Dec 2025).

For deep ReLU networks, the most developed rigorous result concerns the first saddle escape from the origin. Near tiny initialization, the origin is a critical point of the localized homogeneous loss, and escape directions are defined as critical points of that localized loss on the sphere. The main theorem shows a low-rank bias in the optimal escape direction: the first singular value of the $\sigma^2=w^{-\gamma}$2-th layer weight matrix is at least $\sigma^2=w^{-\gamma}$3 larger than any other singular value. The deeper layers are therefore more strongly rank-1 and more nearly linear. The authors present this as a first step toward proving a full saddle-to-saddle dynamics in deep ReLU networks, where gradient descent would visit a sequence of saddles with increasing bottleneck rank rather than ordinary matrix rank (Bantzis et al., 27 May 2025).

This body of work identifies saddle-to-saddle dynamics with a dynamical simplicity bias. Complexity is not acquired all at once; it is recruited through successive departures from lower-complexity invariant structures.

4. Stochastic gradient descent, noise, and learning-time scales

Stochasticity modifies the timing of saddle-to-saddle transitions without necessarily changing their order. In deep linear networks trained with SGD, the dynamics can be approximated by an anisotropic Langevin SDE,

$\sigma^2=w^{-\gamma}$4

where the gradient-noise covariance is state-dependent and anisotropic. Under aligned and balanced weights, the full stochastic dynamics reduce exactly to decoupled one-dimensional SDEs for the singular modes. The diffusion coefficient along mode $\sigma^2=w^{-\gamma}$5 is

$\sigma^2=w^{-\gamma}$6

The paper shows that the maximal diffusion along a mode occurs before the corresponding feature is fully learned, so peak SGD noise forecasts feature completion. At the same time, SGD does not fundamentally alter the saddle-to-saddle regime: the network still learns singular modes in descending order, and noise mainly modulates the time spent near each saddle. Without label noise, the stationary marginal law is a Dirac mass at the teacher singular value; with label noise, it becomes approximately Boltzmann-like and approximately Gaussian near the stable point (Corlouer et al., 7 Apr 2026).

A more combinatorial SGD theory appears in the study of leap complexity for fully connected networks on isotropic data. There the saddle-to-saddle dynamics describe sequential support learning: SGD drifts near a plateau while it infers part of the target’s support, escapes once enough coordinates have aligned, then lands in another saddle or plateau associated with a more complex monomial or feature block. The complexity of each transition is controlled by the leap,

$\sigma^2=w^{-\gamma}$7

The main conjecture states that the total runtime scales as

$\sigma^2=w^{-\gamma}$8

and the proven special-case results establish sequential alignment times consistent with that stagewise picture for Gaussian data and two-layer networks under additional technical assumptions (Abbe et al., 2023).

A useful contrast is perturbed gradient descent for general nonconvex optimization. That method rigorously analyzes escape from a saddle neighborhood by adding a random perturbation when the gradient is small and negative curvature is present. Its central geometric result is that the stuck region is a thin band in the most negative curvature direction, so a random perturbation almost surely moves the iterate into an escaping region. This is a saddle-escape mechanism rather than an explicit saddle-to-saddle sequence, and it is best regarded as adjacent terminology rather than the same process (Jin et al., 2017).

5. Low-rank matrix optimization and incremental eigenpair learning

In low-rank matrix optimization, the saddle-to-saddle picture appears as incremental eigenpair learning. Starting from the Burer–Monteiro formulation

$\sigma^2=w^{-\gamma}$9

the direction-magnitude decomposition writes $\gamma=1$0 with $\gamma=1$1 and relaxes the magnitude variable to obtain

$\gamma=1$2

The resulting DMD dynamics decouple direction learning from magnitude learning, and the paper identifies a sequence of approximate low-rank saddles

$\gamma=1$3

where $\gamma=1$4 is the best rank-$\gamma=1$5 approximation of $\gamma=1$6. A point $\gamma=1$7 is a saddle whenever $\gamma=1$8 and $\gamma=1$9 for $\gamma<1$0 (Wei et al., 30 Jun 2026).

The continuous-time flow limit,

$\gamma<1$1

learns eigenspaces sequentially through alignment variables $\gamma<1$2. Under the spectral gap condition

$\gamma<1$3

and sufficient overparameterization $\gamma<1$4, there exist times

$\gamma<1$5

such that

$\gamma<1$6

for $\gamma<1$7. Thus the trajectory successively passes near the best rank-$\gamma<1$8 approximations, escaping each saddle and settling near the next. The same analysis motivates recursive DMD, which explicitly turns the stagewise mechanism into a deflation-based rank-1 scheme (Wei et al., 30 Jun 2026).

The computational significance is also explicit. Overparameterized gradient descent on the Burer–Monteiro formulation cannot converge faster than $\gamma<1$9, whereas overparameterized DMD regains linear convergence. The paper therefore interprets DMD as exponentially faster in iteration count than the overparameterized BM baseline, with the saddle-to-saddle mechanism providing both the geometric explanation and the algorithmic motivation for the recursive variant (Wei et al., 30 Jun 2026).

6. Broader dynamical-systems uses, limits, and common misconceptions

Outside optimization, closely related jump processes are studied in deterministic and stochastic dynamical systems. In the theory of Wada basins, the saddle-straddle method computes the chaotic saddle embedded in the boundary between one basin $\gamma>1$0 and the merged complement

$\gamma>1$1

For connected Wada basins, the equality

$\gamma>1$2

implies a unique connected boundary and hence a unique chaotic saddle. The numerical procedure repeatedly refines a segment that straddles the boundary, iterates it forward so that it stretches along the unstable manifold, and bisects again to recover the desired scale. The resulting chain of segment approximations can look like a saddle-to-saddle motion, but the paper is explicit that these are not physical jumps between different saddles; they are numerical steps used to trace one chaotic saddle accurately (Wagemakers et al., 2019).

In stochastic rare-event theory, a more literal jump interpretation is available. For small-noise systems

$\gamma>1$3

rare escape from a sink across the stable manifold of a saddle is governed by a large deviation principle with action

$\gamma>1$4

The Method of Division splits an infinite-time transition into linearized sink-side escape, a finite-time nonlinear middle segment, and linearized saddle-side approach. The paper states that this structure is naturally reusable for sequential saddle transitions: a saddle-to-saddle process can be viewed as a concatenation of rare-event segments, each governed by a similar reduced action (Shao et al., 29 Aug 2025).

A further non-bifurcative use appears in complex networks. Near a saddle equilibrium with stable eigenvalue $\gamma>1$5 and unstable eigenvalue $\gamma>1$6, a trajectory may appear stable for a long interval, then escape rapidly along the unstable direction. The proposed early warning statistic is

$\gamma>1$7

with negative slope during saddle approach and positive slope during escape. This work emphasizes that saddle-escape transitions differ from bifurcation-driven tipping: the system need not lose stability through parameter change; it may already be near a hyperbolic saddle and escape after perturbation (Kuehn et al., 2014).

These examples delimit the term’s scope. In some settings it refers to a genuine sequence of saddles linked by heteroclinic-like motion; in others it denotes concatenated rare escapes; and in still others the apparent jumps are numerical or observational artifacts of how one tracks an invariant set. The recurring motif is prolonged residence near a saddle-structured state followed by a faster transition, but the ontological status of the “jump” depends on the model class and on whether the saddles are dynamical, variational, stochastic, or algorithmic.