Blow-up under super-critical correlated initialization

Prove that a ResNet with correlated initialization blows up at initialization under super-critical scaling \(\lambda_L\) larger than \(L^{-H}\ell(L)^{q/2}\), under suitable non-degeneracy or lower-bound assumptions on the activation function.

Background

The sub-critical identity limit is established rigorously, but the corresponding super-critical behavior is unresolved. The proposed conclusion is that amplification of the limiting driver by a diverging factor should cause the hidden states to blow up.

The paper explains that driver amplification alone is insufficient because the activation function may suppress growth; examples include the zero activation and a bounded radial activation. Consequently, a proof requires appropriate lower bounds or non-degeneracy conditions on the activation function.