Alternative mechanisms for inverse depth scaling and hidden-state patterns

Determine whether mechanisms other than ensemble averaging can produce both inverse-depth scaling of loss with depth and the observed hidden-state signatures (uniform small per-layer rotations and weak inter-layer update correlations) in Transformer-based large language models trained for next-token prediction, and characterize any such mechanisms if they exist.

Background

The paper argues that most layers in LLMs behave like an ensemble of similar transformations, leading to an approximate inverse-depth loss scaling, and presents toy-model and hidden-state evidence supporting this explanation.

However, because the analysis is not derived from first principles, the authors acknowledge that other mechanisms could, in principle, yield the same inverse-depth scaling and similar hidden-state behaviors, and they cannot currently rule out such alternatives.

References

Due to the lack of first-principle derivations, we cannot rigorously exclude the possibility that there are other mechanisms that can produce the same inverse depth scaling and similar hidden state behaviors.

— Inverse Depth Scaling From Most Layers Being Similar  (2602.05970 - Liu et al., 5 Feb 2026) in Section 6 (Discussion)

The next step is a controlled grid varying residual update scale, initialisation, normalisation, and architectures that weaken the unit identity path. Such experiments would turn the family-conditioned patterns observed here into causal architectural statements, and would clarify whether designs such as mixture-of-depths, residual scaling, or parallel depth can deliberately move models away from the standard residual-accumulation regime.

— The Residual Stream's Effective Depth  (2609.31098 - Gahtan et al., 25 Sep 2026) in Section 6, “Limitations and future work”