Small Initialization Matters for Large Language Models

Abstract: LLMs provide a tractable system for asking how intelligence itself emerges, rather than only how LLMs can be engineered. Although progress is usually attributed to scale, data and architecture, we show that parameter initialization is a gene-like determinant of training and, in particular, of model capacity. Reducing the initialization scale consistently improves pretraining, with the largest gains on reasoning-demanding tasks. We identify two widely used empirical settings that restrain the advantage of small initialization, and show how relaxing them restores favorable scaling. We further uncover a critical initialization that balances the reasoning and training. Mechanistically, small initialization drives a distinct developmental trajectory: parameters first condense into low-complexity structures and later expand into richer representations, giving concrete form to the idea that compression is intelligence. Token-level analyses show that the gains concentrate on non-trivial, context-constrained predictions rather than all tokens uniformly. These results motivate a simple $γ$-initialization rule: expose initialization rage as an explicit knob and use small initialization by default, an almost cost-free intervention that improves pretraining and strengthens reasoning across model scales.

• The paper demonstrates that small initialization (γ=1) consistently lowers validation loss and increases accuracy by over 4% on benchmarks.
• It employs empirical and mechanistic analyses to reveal how tuning normalization and attention parameters mitigates issues like RMSNorm saturation and attention sink.
• The study uncovers a condensation-to-expansion trajectory in learned representations, advocating for γ as a key hyperparameter in LLM pretraining.

Revisiting Initialization: Empirical and Mechanistic Insights for LLM Training

Introduction

This work, "Small Initialization Matters for LLMs" (2606.17945), investigates the under-explored but critical role of parameter initialization scale in LLM pretraining. Classical approaches typically employ "standard" strategies such as Xavier or Kaiming initialization. However, this study presents detailed empirical evidence and mechanistic analyses demonstrating that smaller parameter initialization—parameterized as $\gamma$ in the variance $d_{\mathrm{in}}^{-2\gamma}$—can enable substantially better pretraining and downstream performance, especially on reasoning-intensive tasks. Importantly, the paper identifies architectural constraints that can suppress or expose the benefits of small initialization, and elucidates how initialization scale drives distinct training dynamics, particularly a compression-to-expansion trajectory in learned representations.

Empirical Characterization of Initialization Scale

The authors systematically probe Transformer models of varying scales, contrasting the standard $\gamma=0.5$ initialization (Xavier-like) with smaller (i.e., $\gamma=1$) initialization. Initial findings are robust: small initialization always reduces validation loss, but the benefit diminishes with increasing model size in the default architecture (Figure 1a,b).

Figure 1: Effects of initialization scale ($\gamma=0.5$ vs. $\gamma=1$) and architectural modifications on loss, attention sink, and normalization sensitivity.

Two major architectural phenomena are identified that blunt small initialization's benefits:

1. RMSNorm Epsilon Saturation: For sufficiently small initializations, the hidden-state variance drops below the stabilization constant $\epsilon$ used in normalization layers, causing the RMSNorm operation to be insensitive to further changes in parameter scale (Figure 1c,d,e).
2. Attention Sink Amplification: Small initialization exacerbates the tendency for attention to disproportionately focus on the first token, a phenomenon known as the attention sink (Figure 1f).

By decreasing the normalization $\epsilon$ from $10^{-5}$ to $10^{-12}$ and introducing gated attention (thus directly counteracting the attention sink), the performance benefit of small initialization is not only recovered but grows with model size (Figure 2a). Task-level evaluation across knowledge, commonsense, and math benchmarks confirms that $d_{\mathrm{in}}^{-2\gamma}$0 consistently outperforms $d_{\mathrm{in}}^{-2\gamma}$1, with absolute accuracy gains exceeding 4% on TriviaQA, HellaSwag, GSM8K, and MATH500.

Figure 2: Loss scaling trends with and without architectural adjustments for both dense and MoE LLMs.

The findings extend to Mixture-of-Experts (MoE) architectures (Figure 2b,c), confirming generality beyond dense Transformers.

Theoretical and Mechanistic Analysis: Residual Flow and Complexity Trajectories

To determine optimal initialization magnitude, the authors analyze the interplay between residual updates and the identity pathway inherent to pre-norm Transformers. For $d_{\mathrm{in}}^{-2\gamma}$2, the cumulative residual flow dominates; for $d_{\mathrm{in}}^{-2\gamma}$3, the model approaches an identity map and loses expressivity (Figure 3a). Empirically, validation loss improves from $d_{\mathrm{in}}^{-2\gamma}$4 to $d_{\mathrm{in}}^{-2\gamma}$5 but deteriorates for $d_{\mathrm{in}}^{-2\gamma}$6, pinpointing $d_{\mathrm{in}}^{-2\gamma}$7 as the practical and theoretical optimum (Figure 3b).

Figure 3: (a) Relative scale of cumulative residual flow vs. input embedding as a function of $d_{\mathrm{in}}^{-2\gamma}$8; (b) Validation loss as a function of $d_{\mathrm{in}}^{-2\gamma}$9.

Token-Level Performance Analysis

Detailed per-token analysis reveals that the improvement from small initialization is non-uniform. The symmetric probability gap $\gamma=0.5$0 (Figure 4a) is positive for a focused subset of moderately difficult, context-constrained tokens (Figure 4b,c), rather than uniformly across the vocabulary. This indicates that small initialization selectively enhances predictions requiring non-trivial contextual integration and reasoning.

Figure 4: (a) Distribution of $\gamma=0.5$1 favoring small initialization; (b) Aggregated gap as a function of token difficulty.

Condensation: From Compression to Expansion in Representation Dynamics

Mechanistically, small initialization induces a condensation process: weight matrices rapidly compress into low-rank, highly aligned structures early in training, then expand to higher complexity as demanded by the task (Figure 5; see also Figures 6–8 for all layers and module types). This low-to-high complexity trajectory is characterized by a sharp initial drop then recovery in stable rank, observed for all major parameter matrices. In contrast, standard initialization lacks a pronounced condensation phase.

Figure 5: Stable-rank evolution in layers 1–6 under different initialization scales demonstrates pronounced condensation with $\gamma=0.5$2.

Implications and Future Directions

The authors establish initialization scale as a primary design axis for LLMs, on par with architectural, data, and optimization choices. They advocate for treating $\gamma=0.5$3 as an explicit hyperparameter, exposing it in model interfaces and setting $\gamma=0.5$4 as a principled default. This choice yields more sample-efficient pretraining and stronger reasoning capacity with minimal computational or engineering overhead.

These findings suggest immediate changes to LLM pretraining pipelines: involve explicit small initialization, adapt normalization layer stabilizers, and introduce architectural elements (such as gated attention) to manage emergent phenomena like the attention sink. Furthermore, the observed condensation-expansion trajectory may have connections to compression-based principles in intelligence and could inspire novel inductive biases, curricula, or regularization protocols.

Conclusion

This study provides a rigorous empirical and mechanistic exposition of the consequences of initialization scale in LLM pretraining. By delineating the interactions between initialization, normalization, and attention mechanisms, the work demonstrates that small initialization is an efficient, widely-applicable lever for improving learning dynamics and final task performance in both dense and sparse LLM architectures. The theoretical identification of a balance point at $\gamma=0.5$5 consolidates practical recommendations. Future research should further explore initialization-induced inductive biases, their relationship to representation compression, and their influence on scaling laws and robust generalization.