Stable Pretraining: Techniques for Reliable Convergence

• Stable-pretraining is the study of methods that ensure reliable convergence of large language models under challenges like scale, depth, and domain shifts.
• Techniques such as output embedding centering, geometric control, precision-aware quantization, and progressive residual warmup mitigate training instability.
• Approaches in continual pretraining and domain adaptation extend stable-pretraining by preserving downstream performance despite data shifts and constrained numerical precision.

Searching arXiv for the cited paper and closely related stable-pretraining work to ground the article. Across recent arXiv work, stable-pretraining denotes a cluster of problems and methods concerned with making pretraining and continual pretraining of foundation models converge reliably under scale, depth, domain shift, and constrained numerical precision, while preserving downstream utility. In LLMs, the topic includes output-logit divergence at large learning rates, gradient explosion and dissipation, instability under stage transitions, temporary performance drops during continual pretraining, and fragility under FP4 arithmetic. Corresponding remedies include geometric control of output embeddings, depth- and time-aware residual schedules, linear-layer reparameterizations, precision-aware quantization recipes, data-mixture and curriculum design, and regularized continued pretraining (Stollenwerk et al., 5 Jan 2026, Chen et al., 5 Mar 2026, Wang et al., 21 Feb 2025, NVIDIA et al., 29 Sep 2025, Guo et al., 2024).

1. Scope and historical development

The literature uses the term in more than one sense. One line of work studies the numerical stability of the pretraining process itself, especially for decoder-only Transformers and LLMs. Another studies stability during continual pre-training, where a pretrained model is adapted to a new domain and temporarily loses downstream competence before recovering. A separate systems usage appears in the software stack “stable-pretraining-v1,” which is a research framework rather than a new optimization algorithm (Balestriero et al., 23 Nov 2025). This suggests that “stable-pretraining” is best understood as a research area rather than a single method.

Within current LLM work, the most concrete optimization target is often the prevention of divergence-like events late in training or under aggressive hyperparameters. “Output Embedding Centering for Stable LLM Pretraining” isolates output-logit divergence as a practically important failure mode when learning rates are large, especially toward the end of training (Stollenwerk et al., 5 Jan 2026). “Progressive Residual Warmup for LLM Pretraining” treats early-training coordination across depth as the key instability source in deep Transformers (Chen et al., 5 Mar 2026). “Scale-Distribution Decoupling” locates instability in the fully connected layers themselves, arguing that standard linear maps entangle output scale and distributional structure in a way that worsens conditioning (Wang et al., 21 Feb 2025). “Pretraining LLMs with NVFP4” extends the theme to narrow-precision training, where stability depends on outlier control, quantization geometry, stochastic rounding, and selective precision allocation (NVIDIA et al., 29 Sep 2025).

There are also important antecedents outside contemporary Transformers. “Stabilizing RNN Gradients through Pre-training” treats pre-training as a preparatory phase that places a network in a locally stable dynamical regime before the main task training begins, using the Local Stability Condition for deep recurrent, neuromorphic, and state-space models (Herranz-Celotti et al., 2023). “Contracting Implicit Recurrent Neural Networks” develops a convexly characterized stable recurrent family together with a near-critical initialization-plus-projection procedure intended to make early training stable, expressive, and fast (Revay et al., 2019). These works broaden stable-pretraining beyond the Transformer setting.

A frequent source of confusion is the distinction between pretraining stability and adaptation stability. “A Stability Analysis of Fine-Tuning a Pre-Trained Model” is explicitly about downstream fine-tuning after pretraining, not about the pretraining process itself; it analyzes leave-one-out stability for full fine-tuning and head tuning (Fu et al., 2023). It is adjacent to stable-pretraining, but not identical with it.

2. Output geometry, logit divergence, and embedding-space diagnostics

A particularly influential mechanistic account comes from the geometry of the language-model head. In a standard decoder-only Transformer, the final hidden state $h \in \mathbb{R}^H$ is mapped to vocabulary logits by $l_i = e_i \cdot h$, where $e_i$ is the output embedding for token $i$. “Output Embedding Centering for Stable LLM Pretraining” argues that output-logit divergence is driven by a common shift of the output embedding cloud away from the origin. If $\mu = \frac{1}{V}\sum_i e_i$ is the mean output embedding and $\bar l = \frac{1}{V}\sum_i l_i$ is the mean logit, then $\bar l = \mu \cdot h$; the mean output embedding therefore directly controls the mean logit (Stollenwerk et al., 5 Jan 2026).

This geometric diagnosis is used to explain the limitations of the standard mitigation, z-loss. The paper shows that z-loss suppresses positive divergence of a single logit and negative divergence only when all logits move down together, but does not prevent a single logit from diverging negatively. On that basis it proposes Output Embedding Centering (OEC), with two implementations. The deterministic version, $\mu$-centering, explicitly recenters the output embedding matrix after each optimizer step via $e_i^\star = e_i - \mu$. The regularized version, $\mu$-loss, adds $l_i = e_i \cdot h$0, with default $l_i = e_i \cdot h$1. The paper proves that $l_i = e_i \cdot h$2-centering makes the mean logit exactly zero, preserves the logits’ standard deviation, and leaves softmax probabilities and cross-entropy loss unchanged, while empirically reducing the worst-case absolute logit under a condition that held for all 35 baseline models examined (Stollenwerk et al., 5 Jan 2026).

The empirical results position OEC as a direct stable-pretraining method. Across dense decoder-only Transformers trained on FineWeb with GPT-2 tokenization, model sizes from 16M to 221M, and learning rates from $l_i = e_i \cdot h$3 to $l_i = e_i \cdot h$4, both $l_i = e_i \cdot h$5-loss and $l_i = e_i \cdot h$6-centering outperform z-loss in learning-rate robustness. At 57M parameters, learning-rate sensitivity is $l_i = e_i \cdot h$7 for the baseline, $l_i = e_i \cdot h$8 for z-loss, $l_i = e_i \cdot h$9 for $e_i$0-loss, and $e_i$1 for $e_i$2-centering; at 221M, the corresponding values are $e_i$3, $e_i$4, $e_i$5, and $e_i$6. The paper also reports that neither OEC variant shows significant divergence in its experiments, whereas the baseline diverges first and z-loss still exhibits occasional divergence at sufficiently large learning rates (Stollenwerk et al., 5 Jan 2026).

A complementary diagnostic perspective appears in “Stable LLM Pre-training by Reducing Embedding Variability.” That work proposes Token Embedding Variability (TEV) as a cheap proxy for pre-training stability in pre-layer-normalized Transformers. TEV is the rowwise standard deviation of the token embedding matrix, summarized by its vocabulary-wide mean and standard deviation. The paper argues that stable pre-training tends to produce lower $e_i$7 and lower $e_i$8, and reports that TEV closely tracks gradient-variance trends in GPT-2-style models (Chung et al., 2024).

The same paper proposes Multi-head Low-Rank Attention (MLRA) as an architectural stabilization method. By factorizing the attention projection matrices, MLRA reduces the initialization-time output variance of a projection from $e_i$9 to $i$0, and derives an attention-output variance upper bound below $i$1 under its assumptions. Empirically, MLRA lowers gradient variance, yields lower TEV statistics than GPT-2 and $i$2Reparam, and improves zero-shot perplexity most strongly as model depth increases; notably, a 192-layer GPT-2 baseline failed five times during pre-training, whereas MLRA remained trainable (Chung et al., 2024).

3. Architectural and parameterization methods for stable optimization

A second major strand of stable-pretraining work changes the optimization trajectory across depth rather than the output head. “Progressive Residual Warmup” introduces a training-phase-aware residual multiplier $i$3 into each Transformer block, replacing $i$4 with $i$5. Its default schedule is $i$6, so deeper layers warm up more slowly and “early layer learns first” (Chen et al., 5 Mar 2026).

The method is motivated by the sequential dependency of stacked layers: if shallow representations are still unstable, deep residual branches learn on top of noisy inputs and feed back noisy gradients. In the paper’s depth-scaling experiment, a 71M base Transformer is scaled from 12 layers to 120 layers while keeping hidden dimensions fixed, and Pre-LN with ProRes maintains near-zero loss spikes as depth increases. Stability is measured using the OLMo 2 “spike score,” defined as the percentage of points at least seven standard deviations away from a rolling average over the previous 1000 steps, computed during 10%–90% of training progress. ProRes also reduces gradient spike scores and improves C4-en test perplexity across Pre-LN, Post-LN, DeepNorm, and LNS configurations; for standard Pre-LN, the reported perplexities improve from $i$7 to $i$8 at 130M/350M/1.3B (Chen et al., 5 Mar 2026).

“Scale-Distribution Decoupling” addresses a different failure mechanism: the standard linear layer $i$9 requires a single matrix $\mu = \frac{1}{V}\sum_i e_i$0 to carry both magnitude and directional structure. SDD replaces it with

$\mu = \frac{1}{V}\sum_i e_i$1

where $\mu = \frac{1}{V}\sum_i e_i$2 is a learnable matrix, $\mu = \frac{1}{V}\sum_i e_i$3 is a learnable scaling vector, and $\mu = \frac{1}{V}\sum_i e_i$4 uses an RMS-style norm. The paper’s argument is that normalization bounds activations and yields well-conditioned gradients, while $\mu = \frac{1}{V}\sum_i e_i$5 restores controllable scale without forcing $\mu = \frac{1}{V}\sum_i e_i$6 to learn it implicitly (Wang et al., 21 Feb 2025).

The empirical case for SDD is strongest in difficult settings for Post-Norm Transformers. On a dense 1B model trained on 200B tokens, SDD reaches training loss $\mu = \frac{1}{V}\sum_i e_i$7, compared with $\mu = \frac{1}{V}\sum_i e_i$8 for OLMo2-1B, $\mu = \frac{1}{V}\sum_i e_i$9 for a Post-Norm baseline, and $\bar l = \frac{1}{V}\sum_i l_i$0 for DeepNorm. In hyperparameter perturbation tests on 581M models, PostNorm-581M becomes non-convergent under $\bar l = \frac{1}{V}\sum_i l_i$1 learning rate, $\bar l = \frac{1}{V}\sum_i l_i$2 initialization standard deviation, and removal of warmup, whereas SDD-581M remains stable with losses $\bar l = \frac{1}{V}\sum_i l_i$3, $\bar l = \frac{1}{V}\sum_i l_i$4, and $\bar l = \frac{1}{V}\sum_i l_i$5, respectively. The paper also reports flatter layerwise gradient norms and superior depth scaling from 8 to 32 layers (Wang et al., 21 Feb 2025).

A related but conceptually distinct line studies stagewise growth of effective model capacity. “Efficient Stagewise Pretraining via Progressive Subnetworks” keeps the full residual Transformer present throughout training but optimizes only random subnetworks in early stages, progressively increasing active subnetwork size until the full model is trained. Its concrete algorithm, RaPTr, uses depth-wise random subnetworks and relies on residual connections and layer norms to keep loss smooth across stage transitions. The paper reports smooth train and eval losses across transitions, up to 33% speedup on standard benchmarks, and a 1.5% improvement on QA tasks and SuperGLUE in UL2, while providing a formal account of why loss discontinuities remain small when dropped layers are bypassed through residual paths (Panigrahi et al., 2024).

4. Precision, curriculum, and systems-level stabilization

Stable-pretraining also has a pronounced systems dimension. “Pretraining LLMs with NVFP4” studies whether LLMs can be pretrained for long horizons at true 4-bit floating-point precision without divergence or severe quality loss. The paper’s headline result is a 12B-parameter hybrid Mamba-Transformer trained for 10 trillion tokens in NVFP4, described as the longest publicly documented training run in 4-bit precision to date, with training loss and downstream task accuracies comparable to an FP8 baseline (NVIDIA et al., 29 Sep 2025).

The NVFP4 recipe is explicitly multi-component. It uses Random Hadamard Transforms to bound block-level outliers, a two-dimensional quantization scheme for weights so that forward and backward use the same quantized representation, stochastic rounding for gradients, and selective high-precision layers. At 12B scale, the authors report that removing any of these components worsens convergence relative to the full method. The 12B run keeps roughly 15%–16% of linear layers in BF16, mainly near the beginning and end of the network, and the paper notes that purely FP4 linear layers diverge. This suggests that stable narrow-precision pretraining is a coupled design problem spanning quantization geometry, numerical bias, and layer sensitivity, rather than a simple matter of lowering operand precision (NVIDIA et al., 29 Sep 2025).

Curriculum and data processing are treated as stabilization mechanisms in “daVinci-LLM: Towards the Science of Pretraining.” That work trains a 3.09B-parameter Qwen2-style decoder-only model from random initialization across 8T tokens using a two-stage adaptive curriculum. In Stage 1, the authors report “smooth and consistent convergence,” gradient norm that “remained stable throughout,” and that “No significant gradient spikes or loss divergences were encountered during the run,” which proceeded “without manual interventions or restarts.” The paper attributes this not only to optimization choices but also to Data Darwinism, a processing taxonomy from L0 to L9, and to staged shifts in domain balance and format, culminating in a Stage 2 that introduces structured QA while explicitly trying to prevent overfitting or domain collapse (Qin et al., 28 Mar 2026).

The same paper treats data composition as a stability control. It reports that general knowledge plateaus early, code and science improve longer, and high QA concentration can cause code collapse or sharp degradation in general knowledge if staged incorrectly. A balanced Stage 2-1 mixture is presented as necessary to prevent over-specialization when reasoning-intensive data are introduced. This suggests that stable-pretraining is partly an interaction between optimization and nonstationary data design rather than an optimizer-only problem (Qin et al., 28 Mar 2026).

At the infrastructure level, “stable-pretraining-v1: Foundation Model Research Made Simple” addresses the engineering burden of SSL and foundation-model experimentation. Its contribution is a modular library built on PyTorch, Lightning, Hugging Face, and TorchMetrics, with a Manager abstraction, dictionary-first batches, online probes, $\bar l = \frac{1}{V}\sum_i l_i$6-NN probes, collapse detection metrics such as RankMe and LiDAR, shared-memory callback scheduling, and a design principle of “logging everything.” The paper’s relevance to stable-pretraining is not a new optimization method, but the claim that monitoring, debugging, and reproducibility should be first-class components of pretraining research (Balestriero et al., 23 Nov 2025).

5. Continual pretraining, domain adaptation, and reusable structure

Stable-pretraining is equally prominent in work on continual pre-training, where a pretrained model is adapted to a new corpus and can become temporarily or permanently unstable under distribution shift. “Efficient Continual Pre-training by Mitigating the Stability Gap” formalizes the stability gap as a temporary performance drop at the beginning of continual pre-training, followed by recovery. In medical adaptation of OpenLlama-3B, the paper reports that its strategies improve average medical task performance from 36.2% to 40.7% with only 40% of the original training budget. The proposed mitigations are training on a properly sized subset for multiple epochs, filtering to a high-quality sub-corpus, and mixing in data similar to the original pre-training distribution (Guo et al., 2024).

The key point is that perplexity can improve monotonically while downstream task accuracy follows a V-shaped curve. The paper interprets this as an imbalance between plasticity and stability under a shifted corpus. Smaller repeated subsets accelerate recovery because repeated exposure reduces novelty pressure; high-quality filtering makes each token more useful; and replay-like mixtures reduce the effective distance between source and target distributions. This is a different notion of stable-pretraining from output-logit control, but it is central for domain-adaptive LLMs (Guo et al., 2024).

In speech, “Stable Distillation” addresses a related problem in continued self-supervised pre-training for low-resource ASR. Vanilla continued pre-training on a small target-domain corpus can overfit and forget useful upstream knowledge, so the paper proposes a teacher-student setup in which a student is regularized toward the final-layer representations of a teacher that has already undergone target-domain continued pre-training. The combined loss adds an MSE-style representation-matching term to the original SSL pretext loss. Across multiple languages and domains, the method improves WER by 0.8–7 relative to baselines, and on SwitchBoard the paper shows that vanilla CP worsens test WER as the number of continued pre-training steps increases, while Stable Distillation avoids that trend (Seth et al., 2023).

A broader corpus-level perspective appears in “Downstream Datasets Make Surprisingly Good Pretraining Corpora.” That paper studies self-pretraining, where the same downstream training text is used both for pretraining and for later fine-tuning, starting from random initialization. It is not an optimization-stability paper in the narrow sense, but it shows that self-pretraining can rival standard BookWiki pretraining despite using roughly $\bar l = \frac{1}{V}\sum_i l_i$7–$\bar l = \frac{1}{V}\sum_i l_i$8 less data, outperforming BookWiki on 7 of 10 datasets for ELECTRA and 5 of 10 for RoBERTa. This suggests that, in some regimes, more controlled and corpus-local pretraining can preserve much of the gain usually attributed to large external corpora (Krishna et al., 2022).

Finally, “Pretraining Induces a Reusable Spectral Basis for Downstream Task Adaptation” argues that stability is not only about optimization trajectories but also about what pretraining leaves invariant for later use. Across vision and LLMs, the leading singular vectors of pretrained weight matrices remain highly stable under finetuning and are shared across unrelated downstream tasks. The paper proposes a parameter-efficient adaptation method that freezes pretrained singular vectors and optimizes only leading spectral coefficients, achieving GLUE average 86.73 with 0.36M trainable parameters, about 0.2% of a 184M DeBERTa-v3-base model. This suggests that stable directions in pretrained models often encode reusable task-relevant structure rather than irrelevant noise (Yu et al., 8 May 2026).

6. Measurement, forecasting, and open problems

Because stability is multi-faceted, the literature uses several distinct diagnostics. In OEC, the main robustness metric is learning-rate sensitivity

$\bar l = \frac{1}{V}\sum_i l_i$9

supplemented by best test loss, mean embedding norm $\bar l = \mu \cdot h$0, sample-estimated mean logit, logit standard deviation, and maximum absolute logit (Stollenwerk et al., 5 Jan 2026). In TEV-based work, embedding-table statistics stand in for costly gradient-variance tracking (Chung et al., 2024). In ProRes, stability is measured with loss and gradient spike scores adapted from OLMo 2 (Chen et al., 5 Mar 2026). In continual pretraining, stability is often operationalized by transient accuracy drops, relative layerwise update magnitudes, and forgetting-free retention of general competence (Guo et al., 2024).

A newer development is the study of stable pretraining laws for generative evaluations. “Pretraining Scaling Laws for Generative Evaluations of LLMs” compares three ways of predicting pass-at-$\bar l = \mu \cdot h$1: a compute law, a parameters-and-tokens law, and a law based on the model’s likelihood assigned to gold reference solutions. On GSM8K, the compute and parameters+tokens laws stabilize only over the last $\bar l = \mu \cdot h$2 orders of magnitude before the target model, whereas the gold-reference-likelihood law is reported to converge across $\bar l = \mu \cdot h$3 orders. The same paper also shows that $\bar l = \mu \cdot h$4 itself is a control lever for scaling behavior: it changes irreducible error terms, fitted exponents, and predictive reliability (Schaeffer et al., 28 Sep 2025). This suggests that stable-pretraining research increasingly includes not only how to train stably, but how to forecast generative capability stably from smaller runs.

Several open questions recur across the literature. OEC is validated only up to 221M parameters and leaves interactions with tied embeddings, encoder-decoder models, mixture-of-experts, and stabilizers such as logit soft-capping or NormSoftMax unresolved (Stollenwerk et al., 5 Jan 2026). ProRes is demonstrated up to 7B but does not provide a formal convergence proof or a settled automatic schedule-selection rule (Chen et al., 5 Mar 2026). SDD shows strong mid-scale results, yet direct comparisons to other reparameterization baselines and frontier-scale deployments remain open (Wang et al., 21 Feb 2025). NVFP4 still depends on selective high-precision islands and hardware support on Blackwell Tensor Cores (NVIDIA et al., 29 Sep 2025). Continual-pretraining work repeatedly highlights incomplete knowledge of the original pretraining mixture as a bottleneck for replay-style stabilization (Guo et al., 2024). And the scaling-law work leaves open why gold-reference likelihood is such a stable proxy for pass-at-$\bar l = \mu \cdot h$5, particularly when many valid generative solutions may exist (Schaeffer et al., 28 Sep 2025).

Taken together, the literature presents stable-pretraining as a layered problem. At the lowest level it involves conditioning linear maps, logits, and residual branches; at the systems level it involves quantization formats, callbacks, and stagewise compute allocation; at the data level it involves mixture rates, subset sizes, and curriculum timing; and at the transfer level it involves preserving reusable structure while adapting to new domains. This suggests that no single intervention exhausts the topic: stable-pretraining is the joint study of how foundation models can be pretrained, continually pretrained, and monitored so that larger scale, broader data, and more aggressive efficiency targets do not turn training into an unstable regime.