Critical-gradient initialization via vanishing gradient Lyapunov exponent (λg = 0)

Determine whether setting the initialization hyperparameters of the deep transformer architecture defined in Section 2 so that the gradient Lyapunov exponent λg equals zero—i.e., exactly at the boundary between exponentially exploding and vanishing end-to-end input–output Jacobian norms—constitutes a good initialization for training deep randomly initialized transformers.

Background

The authors paper the backpropagation of gradients through deep transformers by analyzing the expected squared Frobenius norm of the end-to-end input–output Jacobian, which typically scales exponentially with depth. This scaling is characterized by a gradient Lyapunov exponent λg: λg < 0 corresponds to vanishing gradients and λg > 0 to exploding gradients.

Motivated by the need to avoid both extremes for trainability, the authors propose a conjecture that initializing at the boundary (λg = 0) provides a good initialization for training.