The Stability of Singular Distribution: A Spectral Perspective on the Two-Phase Dynamics of Language Model Pre-training

Abstract: LLM pre-training typically exhibits a two-phase trajectory: a fast initial loss drop followed by a prolonged slow improvement. We identify an underlying spectral phenomenon, Stability of Singular Distribution (SoSD), where the trace-normalized singular value spectrum stabilizes early, even as parameter matrices continue to evolve. We demonstrate that synchronization between SoSD and the slow-descent regime is widely observed across diverse architectures (GPT-2, LLaMA) and settings, including various schedules (Step-wise, WSD, Cosine Decay), weight decays, and optimizers (AdamW, Muon). By analyzing a simplified Transformer, we prove that growing weight norms inevitably precipitate an early SoSD threshold, after which the rate of loss decrease becomes theoretically bounded by the variation in the singular distribution. We further interpret strategies like WSD and Muon through their ability to modulate the SoSD scale, offering a spectral lens for understanding efficient pre-training dynamics.

• The paper shows that early stabilization of singular distributions predicts the shift from rapid to slow loss descent during LM pre-training.
• Empirical analysis across GPT-2 and LLaMA models confirms that singular value spectra converge rapidly, revealing a robust spectral phenomenon.
• Theoretical modeling links norm growth to spectral stability, suggesting new optimization strategies to overcome training bottlenecks.

The Stability of Singular Distribution: A Spectral Perspective on Two-Phase Dynamics in LM Pre-training

Introduction

This paper provides a comprehensive spectral analysis of the training dynamics in large-scale LLM (LM) pre-training, focusing on the ubiquitous two-phase convergence: an initial rapid decrease in loss, followed by a slow asymptotic descent. It identifies and rigorously characterizes the phenomenon of Stability of Singular Distribution (SoSD), where the trace-normalized singular value spectrum of critical model weight matrices stabilizes well before the parameter matrices themselves converge. The work establishes an explicit connection between this spectral stabilization and the onset of slow loss descent, demonstrating both empirical universality (across architectures, datasets, and optimizers) and theoretical inevitability. By doing so, the analysis links microscopic parameter trajectory dynamics with macroscopic convergence regimes and offers new insight into the mechanistic bottlenecks of Transformer optimization.

Empirical Characterization of SoSD and Its Synchronization with Training Loss

The core empirical finding is the early stabilization of the singular value distribution of parameter matrices in Transformer LLMs, specifically in both attention and MLP projections. Analysis with cosine similarity reveals that singular value spectra align with their terminal configuration much earlier than the parameters themselves, indicating a rapid convergence of the spectrum even as the weights continue to drift (Figure 1).

Figure 1: (a) Early stabilization of the trace-normalized singular spectrum with prolonged parameter drift. (b) Onset of SoSD temporally aligns with transition to slow loss descent.

This effect is generic across architectures and scales, as substantiated by a suite of experiments on GPT-2 (Small/Medium) and LLaMA (0.5B/2B), and robust to optimizer (AdamW, Muon) and learning rate schedule variants. The synchronization of SoSD onset with the inflection point between rapid and slow validation loss descent is striking and persistent (Figure 2).

Figure 2: SoSD–loss synchronization across architectures and projection layers; vertical lines denote spectral stabilization coinciding with slow-descent onset.

Quantitative analysis of the singular distribution variation (SDV) further corroborates this effect: SDV is large during the initial sharp descent, then collapses to a narrow band ($\approx 10^{-4}$) as the loss curve enters its slow asymptotic regime. Notably, both attention and MLP projections synchronize their SDV transition with loss saturation, ruling out a layer-specific or artifact explanation.

Theoretical Foundations: Emergence and Consequence of SoSD

The paper develops a detailed theoretical model for pre-training dynamics in single-layer/single-head Transformer settings, incorporating sequence-level cross-entropy loss and standard optimization practices. The analysis is grounded on four key technical conditions: strictly increasing weight norms (in the absence of WD), gradient boundedness, condition number constraints, and smoothness plus margin conditions in the late training regime.

Under these assumptions, it is proven that, due to norm growth, the trace-normalized singular spectra of all projection matrices (query, key, value) must stabilize early in training, at a timescale $T^* = O(\max\{\|W\|\})$, where the stability bound for SDV is $O(\eta/\|W\|)$. This establishes SoSD as a kinetic, not solely geometric, effect; the rate of change of the singular distribution is monotonically suppressed by increasing parameter norms.

Critically, the analysis proves that after the SoSD threshold $T^*$, further reduction in training loss is strictly upper-bounded by the magnitude of SDV. The loss decrement per iteration transitions from a regime with a constant lower bound (permitting rapid optimization) to an asymptotically vanishing rate governed by $\Delta \mathcal{L}(t) \le O(\varepsilon^p)$, where $\varepsilon$ is the upper bound on SDV and $p>2$ (see Section 4). Thus, the two-phase loss dynamics (sharp initial improvement followed by slow descent) are shown to be a direct consequence of spectral stabilization: the learning process is bottlenecked kinetically once the singular spectra become stationary.

Spectral Interpretation of Pre-training Strategies

The spectral SoSD framework yields sharp qualitative predictions about the effect of common optimization strategies. Since SoSD stability bound $\varepsilon \propto \eta/ \|W\|$, interventions can be interpreted spectrally:

• Learning Rate Scheduling: Both step-wise and continuous decay (WSD, Cosine) effect a proportional tightening in the permissible SDV per iteration as $\eta$ is decreased. Empirically, sharp drops in $\eta$ produce synchronized drops in SDV and unlock further rapid loss reduction until the new, tighter SoSD regime is reached. Continuous annealing prolongs the window for meaningful loss descent (Figure 3).
• Weight Decay: By constraining weight norm growth, weight decay relaxes the SoSD bound (larger $T^* = O(\max\{\|W\|\})$0 at fixed $T^* = O(\max\{\|W\|\})$1), empirically resulting in higher SDV and lower terminal loss (Figure 5a). This uncouples the typically assumed relationship between low SDV and improved loss, confirming SoSD as a kinetic constraint rather than a direct performance metric.
• Alternative Optimizers (Muon vs. Adam): The SoSD phenomenon is robust to choice of optimizer, but different optimizers exhibit distinct SDV scales in the stationary phase, with Muon enabling lower SDV and superior convergence compared to Adam (Figure 5b). This differential efficiency is explained by the optimizer’s role in controlling the spectral update scale.

  Figure 3: SDV and validation loss trajectories under various LR schedules, showing SoSD as a function of spectral step size modulation.

  

  Figure 4: (a) Weight decay induces higher SDV but enables lower loss. (b) Muon produces faster convergence and lower SDV than Adam, with both displaying clear SoSD regimes.

Implications and Future Directions

This work presents strong evidence that the transition to slow loss descent in pre-training is generically controlled by early-onset spectral stabilization, positioning SoSD as a mechanistic bottleneck. The findings yield multiple implications:

• Optimization and Practical Training: Pre-training efficiency is fundamentally limited by SoSD-induced kinetic constraints. Strategies that dynamically relax or modulate this spectral bottleneck (via schedule, regularization, or spectral-aware optimizers) afford further gains.
• Theory: The spectral perspective connects with literature on implicit regularization, max-margin dynamics, and monotonic norm growth. The direct coupling between spectral distance and loss descent offers a new avenue for global, rather than strictly circuit/feature-level, analysis of Transformer learning.
• Generalization and Extensibility: Though the core theory is developed for single-layer settings, the universality of SoSD across depth, width, and scale suggests broad extension. Future directions include layer-wise hierarchy of SoSD thresholds, interaction with explicit spectral penalties, and systematic development of spectral-phase-aware adaptivity in learning rates and optimizers.

Conclusion

Through spectral analysis of LM pre-training, this paper identifies and rigorously characterizes the Stability of Singular Distribution phenomenon, establishing its universal role as the controlling mechanism for two-phase convergence dynamics. By anchoring empirical signatures in theoretical inevitability, it enables more principled design of optimization protocols, connects fine-scale parameter evolution with global training efficiency, and offers a pathway for future research into spectral control in deep learning.

---

Reference: "The Stability of Singular Distribution: A Spectral Perspective on the Two-Phase Dynamics of LLM Pre-training" (2605.26489)