Pre-Norm Residual Connections in Transformers

Updated 11 December 2025

Pre-Norm Residual Connections are a Transformer design that applies LayerNorm before each sub-layer to ensure identity skip paths and improved gradient flow.
This configuration enhances training stability and convergence in low-resource machine translation, yielding consistent BLEU gains and reduced training failures.
However, the method introduces challenges like representation collapse and overfitting, prompting the exploration of hybrid designs and alternative normalization techniques.

Pre-Norm residual connections ("PreNorm") are a foundational design in Transformer architectures, wherein layer normalization (LayerNorm) is applied before each attention or feedforward sub-layer, rather than after the residual addition. This architectural choice yields identity skip paths, changes the propagation of gradients, and fundamentally influences training stability, convergence, overfitting, and computational efficiency in deep neural models. The PreNorm design has been adopted as standard in large-scale LLMs and machine translation, but also presents unique limitations and trade-offs explored in recent work.

1. Formal Definition and Mathematical Properties

In the Pre-Norm Transformer, each residual block comprises the following sequence:

For an input $x_\ell$ to the $\ell$ th block:

$u_\ell = \mathrm{LayerNorm}(x_\ell), \quad r_\ell = \mathcal{F}_\ell(u_\ell)$

$x_{\ell+1} = x_\ell + r_\ell$

where $\mathcal{F}_\ell$ represents either a multi-head attention or MLP sublayer. This construction ensures that the residual, or skip connection, is a pure identity: $\partial x_{\ell+1} / \partial x_\ell = I + \partial \mathcal{F}_\ell(\mathrm{LN}(x_\ell))/\partial x_\ell$ .

By contrast, the Post-Norm scheme computes

$a_\ell = x_{\ell-1} + F(x_{\ell-1}), \quad x_\ell = \mathrm{LayerNorm}(a_\ell)$

in which the skip path traverses a further normalization transformation. In PreNorm, typically a final LayerNorm is appended after all residual blocks before the output head (Jiang et al., 2023, Shleifer et al., 2021, Nguyen et al., 2019).

2. Gradient Flow and Representation Collapse

The placement of LayerNorm before each sublayer in PreNorm yields unimpeded backward gradient flow directly from deep layers to shallow ones via identity skips. This contrasts with Post-Norm, where gradients must traverse a product of LayerNorm affine transforms, potentially inducing exponential vanishing or explosion as depth increases. PreNorm's identity skip path results in more uniform and lower-variance gradient magnitudes, facilitating stable training without the need for gradual learning rate warmup or highly tuned initialization (Nguyen et al., 2019, Xie et al., 2023).

However, PreNorm introduces its own notable pathology—representation collapse. As depth $N$ increases, the incremental contribution $\Delta x_\ell = x_\ell - x_{\ell-1}$ of each block shrinks as $\mathcal{N}(0, O(1/\ell))$ , with $\mathbb{E}[|\Delta x_\ell|] = O(1/\sqrt{\ell}) \rightarrow 0$ . This leads to situations in which the output representation $x_N$ varies negligibly when additional layers are added, implying a collapse of model expressivity at depth (Xie et al., 2023).

3. Empirical Performance, Stability, and Initialization

Empirical studies consistently show that PreNorm enables more robust and faster convergence on low-resource machine translation (MT) tasks and in small-batch regimes. It is less sensitive to initialization scale and does not require warmup scheduling to avoid training divergence—a significant practical advantage over PostNorm (Nguyen et al., 2019, Shleifer et al., 2021). BLEU gains of +0.27 or greater have been observed in PreNorm variants; training failure rates are sharply reduced, and global gradient norms are more consistent and less spiky across training runs.

However, in high-resource MT (e.g., WMT En→De) and multilingual or zero-shot settings, PreNorm's expressivity limitations become more pronounced. PostNorm still slightly outperforms PreNorm in ultimate BLEU scores and supports stronger language-agnostic representations in zero-shot translation tasks, with differences of up to 12.3 BLEU observed in favor of PostNorm (Mao et al., 2023).

Empirical results as reported:

Setting	PreNorm BLEU	PostNorm BLEU	Gap
IWSLT’14 En→De (12+12)	35.18	Fail	n/a
WMT De→En (18+18)	26.57	Fail	n/a
OPUS Zero-shot (Ex→X)	27.9	28.7	+0.8
OPUS Zero-shot S→T tag	10.1	16.8	+6.7

In very deep or high-resource models, PreNorm's representation collapse can degrade generalization or fine-tuned performance (Xie et al., 2023, Mao et al., 2023).

4. Extensions: RMSNorm, CRMSNorm, and Computational Efficiency

The PreNorm architecture admits efficient variants through alternative normalization schemes. RMSNorm replaces LayerNorm's mean subtraction and variance scaling with a root-mean-square scaling operation, removing the need to compute the mean:

$\mathrm{RMSNorm}(x) = \gamma \odot \frac{x}{\sqrt{\frac{1}{d}\sum_{i=1}^d x_i^2 + \epsilon}}$

CRMSNorm further compresses zero-mean vectors—by storing only the first $d-1$ coordinates—and operates on this compressed space for further computational saving (Jiang et al., 2023).

For PreNorm Transformers, LayerNorm and RMSNorm at each block are provably arithmetically equivalent (assuming zero-mean vectors via mean-centering at initialization), and CRMSNorm yields equivalent function with lower dimensionality per residual block. This equivalence enables direct replacement of Pre-LN with Pre-RMSNorm/Pre-CRMSNorm to obtain 1–10% speedups in both training and inference, without changing model behavior, loss, or output (Jiang et al., 2023).

Normalization	Mean subtraction	Scaling	FLOPs	Empirical speedup
LayerNorm	Yes	Variance	~5d	Baseline
RMSNorm	No	RMS	~3d	1–9%
CRMSNorm	No (compressed)	RMS	~3(d-1)	up to 10%

5. Variants, Limitations, and Hybrid Designs

While PreNorm is the dominant default for deep, supervised, or LLMs due to stability, it has drawbacks in overfitting and poor zero-shot generalization. The risk of overfitting is exacerbated by LayerNorm’s trainable gain and bias, which can enable memorization along shallow bypasses in deep architectures (Mao et al., 2023). In zero-shot multilingual MT, PreNorm's hidden states retain excessively strong source-language signals, increasing off-target decoding rates (e.g., PreNorm off-target ≈42% vs. PostNorm ≈8.6% on OPUS) (Mao et al., 2023).

NormFormer introduces three micro-modifications to standard PreNorm—head-wise attention scaling and two extra LayerNorms per layer—equalizing gradient magnitudes across layers to optimize convergence and allow higher learning rates. The overhead is negligible (+0.4% params), but total required compute to reach target perplexity drops by ≈40%, illustrating the benefit of tuning normalization placement and granularity (Shleifer et al., 2021).

ResiDual proposes a hybrid Pre-Post-LN design by fusing both PreNorm and PostNorm residual paths, ensuring both non-vanishing gradients (via PreNorm identity skips) and robust layerwise representation diversity (via PostNorm normalization after each residual addition). This architecture achieves stable, faster, and more expressive training in both shallow and very deep models, outperforming both baselines in BLEU on diverse MT tasks. It also avoids the collapse and overfitting phenomena endemic to pure PreNorm (Xie et al., 2023).

6. Practical Recommendations and Implementation

For purely supervised, deep Transformer models or LLM pretraining, PreNorm (with potential enhancements such as RMSNorm or NormFormer additions) is recommended for its stability, ease of training, and reduced sensitivity to initialization. Final LayerNorm after $L$ residual blocks should be appended prior to the task head. For multilingual and zero-shot scenarios, PostNorm or Pre-Post-LN hybridization is favored for generalization and language-agnosticity.

Minimal PreNorm Transformer pseudocode:

def pre_norm_layer(x):
    z1 = LayerNorm(x)
    y1 = SelfAttention(z1)
    x1 = x + y1
    z2 = LayerNorm(x1)
    y2 = FeedForward(z2)
    return x1 + y2

Conversion to Pre-RMSNorm or Pre-CRMSNorm requires mean-centering of residual states and corresponding parameter reparametrization, with no change in training or inference semantics (Jiang et al., 2023).

7. Open Issues and Research Directions

Despite the practical success of PreNorm, its limitations—including representation collapse, overfitting to supervised pairs, and poor cross-lingual abstraction—remain unresolved in their general form. The hybrid dual-residual paradigm (Xie et al., 2023) and the decoupling of normalization from residual structure (e.g., NormFormer (Shleifer et al., 2021), CRMSNorm (Jiang et al., 2023)) suggest a broader design space for retaining both stable training and expressive, generalizable representations. The application of these insights to architectures beyond Transformers, such as vision models and adapter-based systems, remains an open avenue. Furthermore, the computational advantages of RMSNorm/CRMSNorm incentivize continued work in optimizing normalization for both algorithmic and hardware efficiency, especially as model sizes scale.

Key References:

"ResiDual: Transformer with Dual Residual Connections" (Xie et al., 2023)
"NormFormer: Improved Transformer Pretraining with Extra Normalization" (Shleifer et al., 2021)
"Transformers without Tears: Improving the Normalization of Self-Attention" (Nguyen et al., 2019)
"Exploring the Impact of Layer Normalization for Zero-shot Neural Machine Translation" (Mao et al., 2023)
"Pre-RMSNorm and Pre-CRMSNorm Transformers: Equivalent and Efficient Pre-LN Transformers" (Jiang et al., 2023)