Selection of visited rank‑r fixed points during saddle‑to‑saddle dynamics

Determine, for two-layer linear networks trained by gradient flow and exhibiting saddle-to-saddle dynamics, which specific rank‑r fixed point among the combinatorial family determined by selecting r eigen-directions of the symmetric matrix Eyz Ezz Xyz is approached at each stage of training.

Background

The linear case yields many embedded fixed points at each effective width r, corresponding to choices of r eigenmodes. Although stage-like dynamics increases r over time, the particular fixed point selected within each level is not determined by current theory.

Characterizing this selection mechanism would refine predictions about the trajectory and function learned at each plateau and transition.

References

When a linear network undergoes saddle- to-saddle dynamics and approaches fixed points of effective width r = 0,1, ... , D sequentially, determining which of the (") fixed points it approaches is a non-trivial open problem.

— Saddle-to-Saddle Dynamics Explains A Simplicity Bias Across Neural Network Architectures (2512.20607 - Zhang et al., 23 Dec 2025) in Appendix G.3 — Fixed points of linear networks (Remark 2)