Persistence of the maximal-local-update regime at critical ODE scaling

Establish whether correlated ResNets with scaling \(\lambda_L=L^{-1}\) exhibit a maximal-local-update training regime in which individual residual-block features change by order one while the overall network dynamics remains stable.

Background

The scaling L−1L^{-1} corresponds to the maximal-local-update boundary in the independent-initialization phase diagram discussed by the paper. At initialization, the correlated model is generally close to the identity under this scaling.

The unresolved issue is whether the same training regime persists when the initialization is correlated across layers rather than independent.

References

We conjecture that an analogous regime persists under our correlated initialization.

— Correlated initialization of deep residual networks  (2609.03589 - Benning et al., 3 Sep 2026) in Section 3.1.2, “A conjectural phase diagram,” bullet “Critical maximal local updates”

Our experiments therefore cannot establish whether the critical initialization scaling or the maximal-local-update scaling \lambda_\Layer=\Layer{-1} is the more favorable choice for training.

— Correlated initialization of deep residual networks  (2609.03589 - Benning et al., 3 Sep 2026) in Section 3.2, “Experiments”