Necessity of the critical scaling for nontrivial initialization limits

Establish whether the scaling \(\lambda_L\asymp L^{-H}\ell(L)^{q/2}\), where \(H=1-\alpha q/2\), the correlation index is \(\alpha\), the Hermite rank is \(q\), and \(\ell\) is the slowly varying correlation factor, is necessary for a nontrivial large-depth limit of a ResNet with correlated initialization.

Background

The paper proves convergence to a nontrivial Young differential equation at the proposed critical scaling and proves that smaller, sub-critical scalings yield the identity limit. It does not establish that every larger scaling must produce divergence or that the proposed scaling is the unique scaling yielding a nontrivial limit.

The conjecture concerns the full phase transition for ResNets initialized with long-range correlated weights generated from a Gaussian sequence through a feature function of Hermite rank qq.

References

Before moving on we highlight the natural conjecture that the scaling

\lambda_\Layer \asymp \frac{\Layer{-\hurst}{\slowVar(\Layer){\frac{\hermRank}2}

is necessary for a non-trivial limit of the ResNet at initialization.

— Correlated initialization of deep residual networks  (2609.03589 - Benning et al., 3 Sep 2026) in Section 2, immediately before Corollary 2.6 (Sub-critical scaling)

Based on Chizat's analysis of the training we conjecture that the parameter changes should be of order \Layer{\hurst-1}, and hence vanish, while their accumulated first-order effect remains of order one. We therefore expect the training dynamics to be locally linearized.

— Correlated initialization of deep residual networks  (2609.03589 - Benning et al., 3 Sep 2026) in Section 3.1.2, “A conjectural phase diagram,” bullet “Critical SDE (Non-trivial initialization)”