Progressive Residual Warmup (ProRes)

• ProRes is a training-phase-aware modification that gradually scales Transformer residual connections to stabilize early optimization.
• It uses a depth-progressive scalar, defined as α(l, t) = min(t/(T×l), 1), to control activation growth and coordinate learning across layers.
• Experimental results show that ProRes reduces perplexity and improves zero-shot accuracy, particularly in deep and Post-LN models.

Progressive Residual Warmup (ProRes) is a training-phase-aware modification of Transformer residual connections for LLM pretraining. It multiplies each layer’s residual branch by a scalar $\alpha(l,t)$ that gradually warms from $0$ to $1$, with deeper layers taking longer to reach full strength. In the default form,

$$\alpha(l,t)=\min\!\left(\frac{t}{T\times l},1\right),$$

so layer $l$ reaches full residual contribution after $T\cdot l$ steps. The method is motivated by the sequential dependency of stacked Transformer layers and by an “early layer learns first” philosophy: shallow layers are allowed to stabilize earlier, while deeper layers are delayed until upstream representations are more reliable (Chen et al., 5 Mar 2026).

1. Conceptual basis and training objective

ProRes is defined for Transformer pretraining rather than for inference-time adaptation or post hoc fine-tuning. Its core claim is that standard Transformer training allows all residual branches to contribute immediately from initialization, even though deeper layers consume representations produced by earlier layers and shallow layers receive gradients influenced by deeper layers. The method therefore coordinates learning across depth by attenuating residual contributions early in training and restoring them progressively over time (Chen et al., 5 Mar 2026).

The optimization problem ProRes addresses is described in terms of several related phenomena: chaotic early optimization, uncoordinated residual updates across depth, excessive or poorly controlled model updates at initialization and early training, activation growth across depth in Pre-LN, and training instability with loss or gradient spikes in deeper models. ProRes addresses these issues without changing the architecture itself; instead, it changes how much each residual branch contributes during the early phase of pretraining (Chen et al., 5 Mar 2026).

A central implication is that ProRes makes the model effectively shallower early in training and gradually restores full depth. This suggests an intermediate position between static residual scaling and explicit depth growing: unlike static methods, the attenuation is temporary; unlike progressive stacking, the architecture remains fixed throughout training.

2. Formal definition across Transformer variants

The canonical presentation begins with a Pre-LN Transformer block. Without ProRes, the block is

$x_{l+1} = x_l + \mathcal{F}(\mathrm{Norm}(x_l)).$

With ProRes, the residual branch is scaled as

$$x_{l+1} = x_l + \alpha(l,t)\cdot\mathcal{F}(\mathrm{Norm}(x_l)).$$

Here $x_l$ is the input representation to layer $l$, $0$0 is the attention or FFN module, and $0$1 is a predefined scalar determined by layer index and training step (Chen et al., 5 Mar 2026).

The same construction is instantiated for other normalization schemes. For Post-LN,

$0$2

For Sandwich-LN,

$0$3

For DeepNorm,

$0$4

For LayerNorm Scaling,

$0$5

In every case, the scaling is applied to the residual branch output rather than to the skip path (Chen et al., 5 Mar 2026).

At $0$6, for schedules that start at zero, $0$7, so the residual stack is an exact identity map: $0$8 This identity-start property is used by the paper as a unifying optimization intuition: it controls activation growth, gives well-behaved gradients, and prevents deep randomly initialized residual branches from perturbing representations immediately.

3. Schedule design and implementation

The default ProRes schedule is linear in both training time and layer depth: $0$9 Equivalently, the layer-specific warmup length is

$1$0

so layer $1$1 warms up over $1$2 steps, layer $1$3 over $1$4 steps, and layer $1$5 over $1$6 steps. All residuals are fully active after $1$7 steps (Chen et al., 5 Mar 2026).

The paper studies several alternative schedules. “Linear-sqrt” uses

$1$8

which activates residuals more aggressively early than linear. “Linear-square” uses

$1$9

which is gentler near initialization. “Equal” removes depth progression by setting

$$\alpha(l,t)=\min\!\left(\frac{t}{T\times l},1\right),$$0

“Reverse” prioritizes deep layers: $$\alpha(l,t)=\min\!\left(\frac{t}{T\times l},1\right),$$1 The paper also evaluates stagewise schedules and fixed downscaling baselines such as fix-$$\alpha(l,t)=\min\!\left(\frac{t}{T\times l},1\right),$$2, fix-$$\alpha(l,t)=\min\!\left(\frac{t}{T\times l},1\right),$$3, and fix-$$\alpha(l,t)=\min\!\left(\frac{t}{T\times l},1\right),$$4 (Chen et al., 5 Mar 2026).

Implementation is deliberately minimal. One computes the current layer index $$\alpha(l,t)=\min\!\left(\frac{t}{T\times l},1\right),$$5, computes $$\alpha(l,t)=\min\!\left(\frac{t}{T\times l},1\right),$$6 from the chosen schedule and global step $$\alpha(l,t)=\min\!\left(\frac{t}{T\times l},1\right),$$7, multiplies the residual branch output by $$\alpha(l,t)=\min\!\left(\frac{t}{T\times l},1\right),$$8, and then performs the usual residual addition. The scalar is predefined rather than learned, so the method adds essentially no parameter overhead and only a scalar multiply per residual branch. The main experiments use

$$\alpha(l,t)=\min\!\left(\frac{t}{T\times l},1\right),$$9

that is, $l$0, with no tuning for the main result table (Chen et al., 5 Mar 2026).

4. Optimization interpretation and relation to adjacent stabilization methods

The paper frames ProRes through three optimization principles. First, it enforces identity behavior at initialization. Second, it extends the idea of bounded model update from initialization to the early training trajectory: prior methods such as Fixup, T-Fixup, and DeepNorm focus on bounded updates at initialization, whereas ProRes imposes dynamic, layerwise, temporary residual attenuation during the unstable early phase. Third, it respects sequential learning order by activating residuals from shallow to deep (Chen et al., 5 Mar 2026).

This yields a distinctive optimization trajectory. Early training is effectively shallower, because deep layers contribute little or nothing while shallow layers learn a representational basis. Later training restores full capacity once residual warmup completes. The paper treats this as complementary to learning-rate warmup: learning-rate warmup operates at the optimizer level and globally, while ProRes operates at the architectural level and layerwise. The experiments retain standard LR warmup while adding ProRes (Chen et al., 5 Mar 2026).

The mechanistic distinction from related methods is explicit. Compared with residual scaling, LayerScale, or ReZero-like ideas, ProRes uses a predefined time-varying depth-progressive schedule rather than static or learned gating. Compared with progressive stacking or depth growing, it leaves the architecture fixed and modulates residual contribution continuously. Compared with stochastic depth, it is deterministic and monotone. Compared with static depth-dependent attenuation such as LayerNorm Scaling, it constrains updates when instability is highest but relaxes the constraint later, rather than permanently shrinking deep residual contributions (Chen et al., 5 Mar 2026).

A common misconception is that any warmup over residual connections is equivalent to mere global attenuation. The ablations indicate otherwise: warmup alone is not sufficient; the shallow-to-deep order matters. In particular, “equal” schedules often underperform and long “equal” schedules can become worse than baseline, while “reverse” schedules are harmful, especially as warmup grows (Chen et al., 5 Mar 2026).

5. Experimental findings in LLM pretraining

The main experiments use decoder-only Transformers with Llama-style components, including RMSNorm, SwiGLU, Rotary Position Embedding, and Llama-based Attention and MLP. The primary corpus is a 50B-token subset of C4-en, tokenized with the Llama tokenizer, with sequence length $l$1 and evaluation on 10M held-out C4-en test tokens. Training uses AdamW with $l$2, weight decay $l$3, $l$4, gradient clipping $l$5, global batch size $l$6, 100k training steps, and a Warmup-Stable-Decay schedule with 2000 warmup steps and linear decay to zero in the final 10% of steps; Post-LN and DeepNorm instead use a warmup of 10% of training steps (Chen et al., 5 Mar 2026).

Across all tested architectures and scales, ProRes improves C4-en perplexity. For plain Pre-LN, C4-en perplexity improves from $l$7 to $l$8 at 130M, from $l$9 to $T\cdot l$0 at 350M, and from $T\cdot l$1 to $T\cdot l$2 at 1.3B. For Post-LN, the gains are larger: $T\cdot l$3 at 130M, $T\cdot l$4 at 350M, and $T\cdot l$5 at 1.3B. The paper identifies Post-LN as the architecture with the largest benefit, especially at 1.3B, where the improvement is $T\cdot l$6 PPL (Chen et al., 5 Mar 2026).

The benefits are not limited to one normalization or initialization scheme. ProRes also improves Pre-LN combined with DS-Init or Scaled Init, as well as Sandwich-LN, DeepNorm, and LayerNorm Scaling. The paper interprets this as evidence that ProRes is orthogonal to initialization choices rather than redundant with them. It also reports that gains are generally larger at larger model scales (Chen et al., 5 Mar 2026).

Downstream evaluation uses perplexity on WikiText and LAMBADA and zero-shot accuracy on PIQA, SIQA, HellaSwag, WinoGrande, ARC-Easy, ARC-Challenge, OpenBookQA, RACE, LAMBADA, and MMLU. For 1.3B models, ProRes improves average zero-shot accuracy by $T\cdot l$7 points on average across architectures. For plain Pre-LN, the average rises from $T\cdot l$8 to $T\cdot l$9; HellaSwag improves from $x_{l+1} = x_l + \mathcal{F}(\mathrm{Norm}(x_l)).$0 to $x_{l+1} = x_l + \mathcal{F}(\mathrm{Norm}(x_l)).$1, WinoGrande from $x_{l+1} = x_l + \mathcal{F}(\mathrm{Norm}(x_l)).$2 to $x_{l+1} = x_l + \mathcal{F}(\mathrm{Norm}(x_l)).$3, ARC-Easy from $x_{l+1} = x_l + \mathcal{F}(\mathrm{Norm}(x_l)).$4 to $x_{l+1} = x_l + \mathcal{F}(\mathrm{Norm}(x_l)).$5, ARC-Challenge from $x_{l+1} = x_l + \mathcal{F}(\mathrm{Norm}(x_l)).$6 to $x_{l+1} = x_l + \mathcal{F}(\mathrm{Norm}(x_l)).$7, and LAMBADA accuracy from $x_{l+1} = x_l + \mathcal{F}(\mathrm{Norm}(x_l)).$8 to $x_{l+1} = x_l + \mathcal{F}(\mathrm{Norm}(x_l)).$9 (Chen et al., 5 Mar 2026).

The paper also emphasizes generalization. For 1.3B models, the average perplexity reduction across methods is $$x_{l+1} = x_l + \alpha(l,t)\cdot\mathcal{F}(\mathrm{Norm}(x_l)).$$0 on WikiText and $$x_{l+1} = x_l + \alpha(l,t)\cdot\mathcal{F}(\mathrm{Norm}(x_l)).$$1 on LAMBADA, which is larger than the in-domain C4-en reduction. For example, plain Pre-LN improves from $$x_{l+1} = x_l + \alpha(l,t)\cdot\mathcal{F}(\mathrm{Norm}(x_l)).$$2 to $$x_{l+1} = x_l + \alpha(l,t)\cdot\mathcal{F}(\mathrm{Norm}(x_l)).$$3 on WikiText and from $$x_{l+1} = x_l + \alpha(l,t)\cdot\mathcal{F}(\mathrm{Norm}(x_l)).$$4 to $$x_{l+1} = x_l + \alpha(l,t)\cdot\mathcal{F}(\mathrm{Norm}(x_l)).$$5 on LAMBADA. On ClimbMix, ProRes likewise improves held-out perplexity for Pre-LN: $$x_{l+1} = x_l + \alpha(l,t)\cdot\mathcal{F}(\mathrm{Norm}(x_l)).$$6 at 130M, $$x_{l+1} = x_l + \alpha(l,t)\cdot\mathcal{F}(\mathrm{Norm}(x_l)).$$7 at 350M, and $$x_{l+1} = x_l + \alpha(l,t)\cdot\mathcal{F}(\mathrm{Norm}(x_l)).$$8 at 1.3B (Chen et al., 5 Mar 2026).

Depth-scaling results reinforce the stabilization claim. In a 71M-width sweep from 12 to 120 layers, Pre-LN with ProRes is reported as the best-performing method overall, with the lead widening as depth increases. Stability is quantified using the spike score from OLMo2, defined as the fraction of points at least 7 standard deviations away from a rolling average over the previous 1000 steps within the 10%–90% training window. ProRes maintains near-zero loss spikes as depth increases (Chen et al., 5 Mar 2026).

6. Schedule ablations, scope, and relation to other residual warmup ideas

The most detailed ablation uses 350M, 24 layers, 6B tokens, and 60k steps. For the linear schedule, total warmup that is neither too short nor too long is preferred. In Pre-LN, perplexity goes from $$x_{l+1} = x_l + \alpha(l,t)\cdot\mathcal{F}(\mathrm{Norm}(x_l)).$$9 at 1k total warmup to $x_l$0 at 12k and then weakens to $x_l$1 at 48k; in Post-LN, it improves from $x_l$2 at 1k to $x_l$3 at 24k; in Sandwich-LN, it improves from $x_l$4 at 1k to $x_l$5 at 48k. The paper therefore recommends “linear” as the most robust default, while noting that for Post-LN, “linear-square” and stagewise-$x_l$6 can be even better (Chen et al., 5 Mar 2026).

The layer-ordering ablations are central to the method’s interpretation. “Equal” schedules, in which all layers warm together, are usually worse and more sensitive; for Pre-LN, equal 24k reaches $x_l$7, worse than baseline, and for Sandwich-LN, equal 24k reaches $x_l$8. “Reverse” schedules, in which deeper layers warm earlier than shallow ones, are explicitly harmful: for Pre-LN, reverse 24k yields $x_l$9, and for Sandwich-LN, reverse 24k yields $l$0. These results support the paper’s “early layer learns first” premise and distinguish ProRes from undifferentiated residual attenuation (Chen et al., 5 Mar 2026).

The scope of the evidence is nevertheless bounded. The paper states that it does not establish whether ProRes works equally well for encoder-only models, encoder-decoder models, vision transformers, or multimodal transformers. It also does not explicitly disentangle attention residual versus FFN residual scaling, embedding warmup behavior, or whether separate sublayer schedules would help more. Schedule optimality is architecture-dependent, and some figures, including the 7B and depth-scaling experiments, report trends visually without full numeric tables (Chen et al., 5 Mar 2026).

The name “Progressive Residual Warmup” can invite comparison with other residual-warmup mechanisms, but those connections are analogical rather than direct. In recommender systems, RESUS warms cold users by learning a shared global predictor and then a user-specific residual correction, yet it is described as decoupled and sequential rather than explicitly progressive (Shen et al., 2022). In residual reinforcement learning, DAWN uses Data-Anchored Warmup and Normalization to address critic cold start pathology and structural scale mismatch, but its warmup is replay-based and critic-centered rather than a depth-progressive residual schedule (Ma et al., 11 Feb 2026). In optimizer design, adaptive warm-up for norm-constrained methods derives warm-up followed by decay from a gap-dependent curvature model, which is state-dependent but not a residual-branch schedule (Riabinin et al., 5 Feb 2026). These adjacent uses of “warmup” and “residual” clarify what ProRes specifically is: a depth-progressive residual scaling schedule for Transformer pretraining, not a general name for any staged residual adaptation.