Explain cross-layer and higher-order mechanisms underlying training-spike discrepancies

Explain how the discrepancies between layer-wise quadratic simulations and observed Transformer language-model validation-loss spikes arise, and determine how correlations across layers and Taylor-expansion terms beyond second order contribute to the spike mechanism.

Background

The paper models individual Transformer layers using second-order Taylor expansions and layer-wise quadratic random-feature models. These simulations reproduce some bump-like behavior associated with training spikes, but they do not fully match the empirical dynamics of the total validation loss or the observed first- and second-order responses of individual layers.

The authors specifically identify two unresolved deficiencies: empirical second-order terms can exhibit jumps rather than transient spikes, and first-order terms can display bumps, jumps, or dives at a scale and rate not captured by independent layer-wise quadratic models. They suggest that interactions between layers and Taylor terms of order higher than two may be responsible, but do not resolve the issue.

References

How we can explain these discrepancies, and how the correlations across layers and Taylor expansions beyond 2nd order may play a role in the mechanism, are intriguing questions for future work.

— Simulating Training Spikes of Deep Neural Networks by Layer-wise Random Feature Models  (2610.13524 - Tian et al., 9 Oct 2026) in Section 3, subsection “Simulation of Transformer LM Training,” immediately following Figure 3 and before Figure 4