Distance-to-manifold threshold for invariant‑manifold‑guided dynamics

Quantify how close a parameter vector must be to an invariant manifold in networks defined by Equation (1) to guarantee that gradient flow trajectories approach a fixed point on that manifold before leaving it, thereby enabling rigorous proofs of saddle-to-saddle dynamics beyond diagonal linear networks.

Background

Saddle-to-saddle dynamics relies on trajectories evolving near invariant manifolds associated with low effective width. A quantitative basin-of-attraction criterion would formalize when such guidance occurs and extend existing rigorous results beyond special cases like diagonal linear networks.

This would bridge heuristic arguments and empirical observations with provable guarantees across more architectures.

References

Several interesting technical questions remain open. First, how close must a point in weight space be to an invariant manifold in order to approach a fixed point on that manifold before leaving the manifold?

— Saddle-to-Saddle Dynamics Explains A Simplicity Bias Across Neural Network Architectures (2512.20607 - Zhang et al., 23 Dec 2025) in Appendix C — Technical future directions