Determine whether mitigating the curse of depth changes scaling exponents

Determine whether mitigating the curse of depth in pre-norm transformer architectures changes the compute-optimal scaling exponent of pretraining loss.

Background

The paper discusses the curse of depth in pre-norm transformers, where the residual stream grows with depth and successive blocks increasingly contribute smaller updates, causing deeper layers to drift toward having little effect. Prior interventions—including scaling the pre-normalization output, normalizing the residual stream, and injecting earlier representations into later layers—have been reported to improve loss at fixed model size.

The unresolved issue is whether these interventions merely improve the scaling constant or can alter the scaling exponent itself. The paper’s boundary operator combines residual-stream normalization with reinjection of an earlier representation and subsequently reports empirical exponent improvements, but the cited background passage explicitly identifies the general relationship between mitigating the curse of depth and changing the scaling law as an open question.