Alternative mechanisms for inverse depth scaling and hidden-state patterns

Determine whether mechanisms other than ensemble averaging can produce both inverse-depth scaling of loss with depth and the observed hidden-state signatures (uniform small per-layer rotations and weak inter-layer update correlations) in Transformer-based large language models trained for next-token prediction, and characterize any such mechanisms if they exist.

Background

The paper argues that most layers in LLMs behave like an ensemble of similar transformations, leading to an approximate inverse-depth loss scaling, and presents toy-model and hidden-state evidence supporting this explanation.

However, because the analysis is not derived from first principles, the authors acknowledge that other mechanisms could, in principle, yield the same inverse-depth scaling and similar hidden-state behaviors, and they cannot currently rule out such alternatives.