Locally linearized ODE regime below the critical initialization scale

Establish whether correlated ResNets with scaling exponent \(H<\gamma<1\), equivalently \(\lambda_L=L^{-\gamma}\), exhibit a locally linearized ordinary-differential-equation training regime with parameter changes of order \(L^{\gamma-1}\).

Background

For H<γ<1H<\gamma<1, the paper proves that the network converges to the identity at initialization because the scaling is sub-critical. It does not prove the behavior during training.

By extrapolating an analysis developed for independent initialization, the authors conjecture that parameter changes vanish while the training dynamics remain locally linearized and converge to an ODE-type regime.

References

We therefore conjecture a locally linearized ODE regime, analogous to the ``Lazy ODE'' regime of \citet[Fig.~4, Thm.~2]{chizatHiddenWidthDeep2026}.

— Correlated initialization of deep residual networks  (2609.03589 - Benning et al., 3 Sep 2026) in Section 3.1.2, “A conjectural phase diagram,” bullet “Locally linearized ODE”