Effective Learning Rate Governs Loss Dynamics in Language Model Pretraining

Abstract: We uncover ELR collapse in LLM pretraining: learning rate (LR) and parameter norm govern loss dynamics primarily through their ratio, the effective learning rate (ELR). When ELR is matched across runs, their loss trajectories collapse throughout training despite substantially different LRs and parameter norms. Across optimizers, architectures, datasets, and model scales, mean collapse errors are typically a few x 10^-3, below the seed-to-seed variation measured in a representative configuration. Systematic ablations identify normalization design and the timescale of LR-norm variation as key determinants of collapse precision. Controlled interventions further show that weight decay and Hyperball shape loss dynamics primarily through the ELR schedules they induce. Replacing LR with ELR enables a fitted functional scaling law (FSL) to transfer across norm-control methods. The resulting ELR-based FSL also explains delayed acceleration, a recurring effect of norm control. Together, these results establish ELR as a common coordinate linking LR scheduling, norm control, and loss dynamics.

• The paper empirically demonstrates that during language model pretraining, the effective learning rate (ELR)—defined as the nominal learning rate divided by the parameter norm—is the primary driver of training loss.
• This study shows effective learning rate (ELR) collapse, where identical ELR trajectory produces consistent loss curves regardless of distinctly different nominal learning rate and parameter norm trajectories
• Experiments across various models and datasets confirm that norm-control mechanisms like weight decay and Hyperball primarily influence loss dynamics through the ELR they induce, and higher ELR collapse precision depends on adaptive normalization gains

The paper advances a precise empirical claim about pretraining dynamics: for the configurations examined, learning rate and parameter norm influence training loss primarily through their ratio, termed the effective learning rate (ELR). This claim is tested by constructing training runs with substantially different learning-rate and norm trajectories while enforcing the same ELR trajectory. The resulting loss curves remain closely aligned across the full training horizon. The central contribution is therefore not a new optimizer or norm-control algorithm, but an empirical reduction of several optimization degrees of freedom to a common dynamical coordinate (2608.24814).

Central hypothesis and experimental construction

For matrix-valued parameters updated directly by an optimizer, the paper defines

$$\eta_k^{\mathrm{eff}}=\frac{\eta_k}{\|\mathbf W_k\|_F},$$

where $\eta_k$ is the nominal learning rate and $\|\mathbf W_k\|_F$ is the parameter Frobenius norm. The motivation comes from analyses of scale-insensitive optimization, where parameter norm determines the angular magnitude of an update. The paper extends this perspective to Transformer pretraining, despite the fact that Transformers do not possess exact global scale invariance.

The experimental design separates ELR from its two constituents. A reference run induces an ELR schedule $\gamma_k$. A comparison run is assigned a different learning-rate schedule $\eta_k^B$, and its target parameter norm is set to $\eta_k^B/\gamma_k$. Projection after each update enforces the prescribed norm trajectory. Consequently, the compared runs have distinct nominal learning rates and parameter norms but identical ELRs. Because initialization, data order, optimizer state, and other randomness are held fixed, the measured loss discrepancy is attributable primarily to the LR--norm intervention.

The paper evaluates alignment using the mean absolute trajectory discrepancy

$$\Delta_{\mathrm{coll}} = \frac{1}{|\mathcal T|} \sum_{k\in\mathcal T} \left|L_k-L_k^{\mathrm{ref}}\right|,$$

computed over post-warmup steps after independently applying exponential loss smoothing. The resulting phenomenon is called ELR collapse: matched ELR schedules produce nearly coincident loss trajectories even when their LR and norm schedules differ substantially.

Quantitative evidence for ELR collapse

The initial experiments use a 124M-parameter Llama model and a 586M-parameter Qwen3 mixture-of-experts model trained with AdamW. A common warmup--stable--decay ELR schedule is realized through a conventional WSD learning-rate schedule and three alternative schedules: linear-up, linear-down, and sinusoidal. The norm trajectory is adjusted separately for every alternative so that the ELR remains identical to the reference.

For Llama-124M, the collapse errors for the three alternative schedules are $1.8\times10^{-3}$, $2.5\times10^{-3}$, and $2.6\times10^{-3}$. For Qwen3-MoE, they are $\eta_k$0, $\eta_k$1, and $\eta_k$2. These discrepancies are small relative to the differences in the underlying LR and norm schedules.

Figure 1: Distinct learning-rate and parameter-norm schedules produce nearly identical losses when their ELR schedules are matched.

The residual trajectories provide additional evidence that the agreement is not merely a similar final loss. The signed residuals generally remain centered near zero and show no systematic temporal drift, although transient deviations occur. Thus, the result is a trajectory-level alignment rather than an endpoint equivalence or exact pointwise identity.

Figure 2: ELR-matched loss residuals fluctuate around zero without sustained directional drift.

The authors calibrate these errors against ordinary stochastic variation. Repeated Llama-124M runs with different initialization seeds have a mean pairwise discrepancy of $\eta_k$3, while changing the data-order seed produces $\eta_k$4. The ELR-collapse errors of $\eta_k$5--$\eta_k$6 are therefore approximately 4.3 to 9 times smaller than the corresponding stochastic baselines. This comparison supports the interpretation that ELR matching removes a substantial deterministic source of loss variation, although the calibration is configuration-specific rather than a universal perceptual threshold.

The validation extends to Llama-1B, dense Qwen3, Kimi Delta Attention, C4, OpenWebText, Signum, and Muon. Across 26 prescribed-ELR comparisons, the median collapse error is $\eta_k$7 and every reported comparison is below $\eta_k$8. The Llama-1B stress test yields $\eta_k$9 for both tested schedules. Additional experiments span model sizes from approximately 100M to 1B parameters and include dense, MoE, and linear-attention architectures.

Figure 3: ELR collapse persists in the 1B-parameter Llama configuration.

Figure 4: Cross-architecture, cross-dataset, and cross-optimizer validation of prescribed-ELR collapse.

The result is also observed in a 9.8M-parameter ViT trained on ImageNet-1K. After branching from a shared state, three nonconstant LR--norm realizations with matched ELR remain aligned with a constant-LR reference, with collapse errors between $\|\mathbf W_k\|_F$0 and $\|\mathbf W_k\|_F$1. This experiment broadens the empirical scope beyond autoregressive language modeling, but it remains a small-scale vision experiment and does not establish universality across all architectures or datasets.

Figure 5: Prescribed-ELR branches remain aligned in a pre-norm ViT trained on ImageNet-1K.

Collapse is conditional rather than a consequence of scale symmetry

A central strength of the paper is that it explicitly rejects a simplistic explanation based on exact scale invariance. Transformer loss is not invariant to a uniform rescaling of all matrix parameters. Pre-norm residual blocks combine an unnormalized identity branch with a normalized residual branch; attention value and output projections interact multiplicatively; SwiGLU introduces nonhomogeneous and multiplicative dependence on several matrices; and embeddings, output logits, biases, and normalization gains introduce additional scale-sensitive components.

The empirical ablations reinforce this distinction. The default Llama configuration uses QK-Norm and RMSNorm with learnable gains. Removing QK-Norm increases the mean collapse error from $\|\mathbf W_k\|_F$2 to $\|\mathbf W_k\|_F$3. Removing QK-Norm and fixing the remaining RMSNorm gains increases it further to $\|\mathbf W_k\|_F$4.

Figure 6: Removing QK-Norm and fixing RMSNorm gains progressively degrade ELR-collapse precision.

The gain result is particularly important and contradicts an explanation based solely on increased scale invariance. Fixing RMSNorm gains should make the parameterization more scale-invariant in a static sense, yet it worsens collapse by approximately a factor of 3.5 relative to the learnable-gain configuration. The implication is that adaptive normalization gains contribute dynamically to the compensation underlying ELR collapse. The paper does not identify the microscopic mechanism; it establishes the dependence empirically.

The same ordering appears in the ViT experiment. Mean collapse error rises from $\|\mathbf W_k\|_F$5 with QK-Norm and learnable gains, to $\|\mathbf W_k\|_F$6 without QK-Norm, and to $\|\mathbf W_k\|_F$7 after additionally fixing RMSNorm gains. This cross-domain replication indicates that the normalization dependence is not peculiar to one language-model implementation.

Figure 7: The normalization-dependent ordering of ELR-collapse precision is reproduced in ViTs.

The second identified condition concerns the timescale of LR--norm variation. The authors modulate both LR and target norm by the same sinusoidal factor, thereby preserving ELR while increasing the modulation frequency. Mean collapse error grows monotonically from $\|\mathbf W_k\|_F$8 at two cycles to $\|\mathbf W_k\|_F$9 at 32 cycles. The residuals oscillate at the imposed frequency rather than exhibiting unbounded drift.

Figure 8: Faster LR--norm modulation produces larger, frequency-matched residual oscillations.

This result constrains the interpretation of ELR as a sufficient state variable. ELR matching is highly accurate for slowly varying realizations, but the parameterization has a finite dynamical response time. Two schedules with identical instantaneous ELR can therefore differ when LR and norm vary too rapidly. The paper consequently presents ELR as an accurate macroscopic coordinate, not an exact Markovian description of the full optimization process.

Norm control acts through the ELR it induces

The prescribed-norm experiments establish collapse under direct control of the parameter norm. The paper then tests whether practical norm-control mechanisms can be understood through the same coordinate. This is done using LR-only interventions: the target run retains its norm-control method, while the comparison run retains a different norm-control method and adapts only its LR to match the target ELR. The comparison norm is not prescribed.

For AdamW, a run with weight decay $\gamma_k$0 serves as the target, while a no-decay run is given an adapted LR. Removing weight decay under the original LR changes the norm evolution and substantially alters the loss trajectory. Matching the target ELR with the no-decay run recovers the target loss with a collapse error of $\gamma_k$1.

For Muon, a Hyperball run serves as the target and a weight-decayed Muon run is adapted through its LR. The resulting collapse error after the initial matching transient is $\gamma_k$2. Reverse-direction interventions also preserve close alignment, indicating that the result is not an artifact of choosing one particular target configuration.

Figure 9: LR adaptation that matches the target ELR recovers loss dynamics across weight decay and Hyperball.

Figure 10: Bidirectional LR-only ELR matching aligns AdamW runs with and without weight decay.

Figure 11: Bidirectional ELR matching aligns Muon with weight decay and Hyperball-controlled Muon.

These interventions support the paper’s stronger mediation claim: weight decay and Hyperball affect loss dynamics primarily through the ELR schedules they induce, rather than by contributing independent loss-dynamical coordinates at the tested scale. This statement is explicitly limited to training loss. It does not imply equivalence of parameter trajectories, learned representations, stability margins, numerical behavior, or downstream performance.

The normalization ablation remains relevant here. When RMSNorm gains are fixed, LR-only ELR matching produces sustained residuals at the $\gamma_k$3 scale in both matching directions. Hence, practical norm-control transfer depends on the same architectural conditions as prescribed-ELR collapse.

Figure 12: Fixing RMSNorm gains substantially weakens LR-only ELR matching.

ELR improves functional scaling-law transfer

The paper tests whether ELR supports not only pairwise trajectory alignment but also predictive modeling across norm-control regimes. It adopts the functional scaling law (FSL), which decomposes loss into a signal-learning term and an optimization-noise term. In the original formulation, both the accumulated intrinsic time and the noise contribution are parameterized by the nominal LR. The paper defines elr-FSL by replacing LR with ELR throughout the same functional form.

The transfer experiment uses ten Llama-124M trajectories trained on FineWeb: configurations with no weight decay, weight decay $\gamma_k$4, and Hyperball. Both FSL variants are fitted on four non-Hyperball trajectories. Four additional non-Hyperball trajectories test held-out in-distribution transfer, and two Hyperball trajectories test out-of-distribution transfer to a norm-control method absent from fitting.

The results show a marked distinction:

Evaluation split	lr-FSL RMSE	elr-FSL RMSE	Relative improvement
Fit	0.0183	0.0131	$\gamma_k$5
Held-out ID	0.0239	0.0133	$\gamma_k$6
OOD Hyperball	0.2508	0.0212	$\gamma_k$7

The OOD result is the strongest numerical evidence for the proposed coordinate. A model fitted without Hyperball data obtains an RMSE of $\gamma_k$8 under elr-FSL but $\gamma_k$9 under lr-FSL. In the per-trajectory results, the improvement reaches $\eta_k^B$0 for Hyperball with WSD-10% and $\eta_k^B$1 for Hyperball with cosine decay.

Figure 13: ELR parameterization enables FSL transfer to unseen Hyperball trajectories, whereas LR parameterization develops a large systematic bias.

Figure 14: elr-FSL predictions remain accurate across fitted, held-out, and OOD trajectories.

The implication is stronger than simple curve matching: ELR provides a more transferable input coordinate for a functional loss model across norm-control mechanisms. Nevertheless, the evidence is based on a small model, a single primary dataset, a limited number of trajectories, and a fixed FSL functional form. The results demonstrate transfer within this experimental design, not that ELR universally resolves scaling-law extrapolation.

ELR explains delayed acceleration

The paper applies the ELR framework to delayed acceleration, in which a norm-controlled run initially has higher loss than an uncontrolled baseline but later overtakes it. Under a shared nominal LR schedule, the unregularized run’s parameter norm grows more rapidly, causing its ELR to decay faster. Weight decay restrains norm growth and sustains a larger ELR during the earlier phase.

This produces a nontrivial signal--noise tradeoff. A larger early ELR increases effective training time and accelerates signal learning, but also increases optimization-noise accumulation. The signal advantage can therefore remain hidden while the noise term dominates. During terminal LR decay, new noise injection decreases and previously accumulated noise is forgotten, revealing the signal-learning advantage acquired earlier. Delayed acceleration is consequently interpreted as a temporal separation between acquisition and observation of the gain.

Figure 15: Norm-controlled training sustains ELR earlier and exhibits delayed overtaking as late-stage ELR decay reduces noise accumulation.

The paper tests this explanation by directly reshaping the norm trajectory. An explicit norm-control schedule follows the weight-decay trajectory early but increases the norm more rapidly late in training. This preserves the early ELR advantage while inducing faster late-stage ELR decay, producing a lower final loss than the weight-decay baseline. The result is contradictory to the idea that uniformly smaller parameter norms are intrinsically beneficial: the advantageous intervention deliberately increases norm growth in the terminal phase because the relevant object is the ELR schedule, not norm magnitude alone.

A controlled WSD experiment varies the duration of the terminal decay phase. With a decay ratio of 0.1, the low-ELR phase is too short for the weight-decayed run to overtake the unregularized baseline. With a ratio of 0.3, the longer decay phase reveals the earlier signal-learning advantage and produces a crossover. Hyperball exhibits the same qualitative delayed-acceleration pattern under Muon.

Figure 16: Extending the WSD decay phase reveals a signal-learning advantage accumulated earlier.

Figure 17: Hyperball produces delayed acceleration analogous to that observed under weight decay.

The FSL interpretation thus connects three observations: early ELR preservation, elevated intermediate noise, and late-stage overtaking. It also turns the explanation into a control intervention, although the proposed norm schedule is heuristic rather than optimized.

Limitations and open questions

The paper’s principal limitation is that ELR collapse is established empirically and conditionally rather than derived from a microscopic theory. Exact scale invariance is absent, and the dependence on QK-Norm, learnable RMSNorm gains, and modulation timescale indicates that ELR alone is not a complete state description. The underlying compensation mechanism remains unresolved.

The ELR definition is also optimizer-dependent. For direct-update optimizers, the denominator is the parameter norm. For Hyperball, because the optimizer normalizes the update before projection, the relevant quantity is $\eta_k^B$2. This makes ELR a structured but not entirely universal scalar; extending the definition to other update geometries requires care.

The controlled comparisons share initialization and data order, which is necessary for measuring intervention effects but does not establish robustness under independent stochastic realizations. The stochastic baseline is reported for one Llama-124M/FineWeb/AdamW configuration. Likewise, most experiments use relatively small models and short pretraining runs compared with production-scale LLM training.

Finally, the demonstrated equivalence concerns smoothed training loss. Matched loss trajectories need not imply matched parameter norms at the layer level, internal representations, gradient statistics, optimization stability, or downstream task behavior. Specific questions remain open: what dynamical mechanism makes learnable normalization gains improve collapse; how should ELR be defined for optimizers with more complex preconditioning or non-Euclidean updates; and whether an ELR-first design can predict the appropriate schedule across model size, batch size, data budget, and training horizon.

Conclusion

The paper presents ELR collapse as a quantitative empirical law for a broad set of pretraining experiments. When LR and parameter-norm schedules are varied while ELR is held fixed, loss trajectories typically align at the few-$\eta_k^B$3 level, below measured seed-induced variation. The result survives changes in architecture, optimizer, dataset, model scale, and, in one experiment, modality.

The evidence further indicates that weight decay and Hyperball influence loss primarily by reshaping ELR, that ELR substantially improves FSL transfer across norm-control regimes, and that delayed acceleration follows from the temporal interaction between effective training time and optimization noise. The claims are explicitly conditional: normalization design and variation timescale govern precision, and ELR does not characterize every aspect of training. Within those limits, the paper motivates treating ELR as the principal design coordinate for loss dynamics, with nominal LR schedules and norm-control mechanisms regarded as alternative realizations of that coordinate (2608.24814).