---
title: Pre-LayerNorm Residual Blocks in Transformers
url: https://www.emergentmind.com/topics/pre-layernorm-residual-blocks
type: topic
---

# Pre-LayerNorm Residual Blocks in Transformers

Pre-LayerNorm (Pre-LN) residual blocks are a foundational architectural choice in transformer models, designed to stabilize training and enhance optimization by positioning normalization upstream of sublayer computations. This design contrasts with Post-LayerNorm placement and has become the prevailing standard in contemporary large language models and vision transformers due to its effects on gradient flow, optimization stability, and empirical efficiency. Several rigorous lines of research have analyzed the mathematical structure, functional properties, equivalence with normalization variants, and practical implications of the Pre-LN scheme [2305.14858][2511.10566].

## 1. Mathematical Structure of Pre-LayerNorm Residual Blocks

In a transformer employing the Pre-LayerNorm paradigm, each block processes an input $x_\ell\in\mathbb{R}^d$ as follows:
\[
\tilde x_\ell = \mathrm{LayerNorm}(x_\ell)
\]
\[
y_\ell = \mathcal{S}_\ell(\tilde x_\ell), \quad (\mathcal{S}_\ell \text{ often MLP}_\ell \text{ or Attention}_\ell)
\]
\[
x_{\ell+1} = x_\ell + y_\ell
\]

Here, the LayerNorm operation is defined as:
\[
\mathrm{LayerNorm}(x) = \frac{x - \mu(x)\mathbf{1}}{\sqrt{\frac{1}{d}\|x\|_2^2 - \mu(x)^2 + \epsilon}}, \quad \mu(x) = \frac{1}{d}\mathbf{1}^T x, \quad \epsilon>0
\]
After passage through $L$ such blocks, a final LayerNorm is typically applied:
\[
\hat x = \mathrm{LayerNorm}(x_L)
\]
In standard implementations, Pre-LN transformers decompose each block further, applying LayerNorm before both the Multi-Head Self-Attention (MHSA) and Feed-Forward Network (FFN) sublayers:
\[
\begin{aligned}
u_\ell^{(1)} &= \mathrm{LN}_1(x_\ell)\\
m_\ell &= \mathrm{MHSA}(u_\ell^{(1)})\\
y_\ell &= x_\ell + m_\ell\\
u_\ell^{(2)} &= \mathrm{LN}_2(y_\ell)\\
f_\ell &= \mathrm{FFN}(u_\ell^{(2)})\\
x_{\ell+1} &= y_\ell + f_\ell
\end{aligned}
\]
This dual placement enhances optimization stability and modulates both learning and memorization [2511.10566].

## 2. Functional Properties and Gradient Flow

Pre-LN residual blocks address the "vanishing/exploding gradient" issues associated with Post-LN alternatives. For a Pre-LN transformer with $N$ layers, one can upper-bound the $L_2$-norm of the gradient $\frac{\partial \mathcal{L}}{\partial x_i}$ (with respect to input $x_i$ to the $i$-th LN layer) as:
\[
\left\|\frac{\partial \mathcal{L}}{\partial x_i}\right\|_2 \leq s_{\max}(P_2)\times \prod_{j=i}^{N}\left(1+s_{\max}(J_{\mathrm{FFN}^{\mathrm{LN}_2(x_j')}})\right) \prod_{j=i}^{N}\left(1+s_{\max}(J_{\mathrm{MHSA}^{\mathrm{LN}_1(x_j)}})\right)
\]
where $s_{\max}(\cdot)$ denotes spectral norm, and $P_2$ gathers downstream head Jacobians. Each factor is ensured to be at least $1$, meaning gradients do not degrade or explode geometrically as in Post-LN. Notably, the upper bound is largest for early layers and decays monotonically:
\[
\mathrm{UB}\bigl(\|g_{x_1}\|\bigr)\ge\mathrm{UB}\bigl(\|g_{x_2}\|\bigr)\ge\cdots\ge\mathrm{UB}\bigl(\|g_{x_N}\|\bigr)
\]
Furthermore, the norm of the gradient driving genuine learning ($g_x^{\mathrm{learn}}$) dominates the gradient driving memorization of noise ($g_x^{\mathrm{mem}}$):
\[
\|g_x^{\mathrm{learn}}\|_2 \ge \|g_x^{\mathrm{mem}}\|_2, \quad \forall\text{ layers}
\]
This supports the empirical robustness of Pre-LN transformers across deep and wide architectures [2511.10566].

## 3. Pre-LayerNorm versus RMSNorm and CRMSNorm: Computational Unification

While LayerNorm recenters and rescales vectors, RMSNorm performs only RMS-based rescaling:
\[
\mathrm{RMSNorm}(x) = \frac{x}{\sqrt{\frac{1}{d}\|x\|_2^2+\epsilon}}
\]
If $x$ is zero mean, $\mathrm{LayerNorm}(x) = \mathrm{RMSNorm}(x)$. Pre-LN transformers allow all main-branch activations to be zero mean by re-centering on the fly:
\[
x_\ell \mapsto x_\ell - \mu(x_\ell)\mathbf{1},\quad y_\ell \mapsto y_\ell - \mu(y_\ell)\mathbf{1}
\]
This enables LayerNorm to be algebraically replaced by RMSNorm, with all redundancy in the mean eliminated. Further, any zero-mean vector $v\in\mathbb{R}^d$ can be losslessly compressed to its first $d-1$ components; this leads to the "Compressed RMSNorm" (CRMSNorm) variant:
\[
c = (v_1,\ldots,v_{d-1})\in\mathbb{R}^{d-1},\quad v_d = -\sum_{i=1}^{d-1}v_i
\]
\[
\mathrm{CRMSNorm}(c) = \frac{c}{\sqrt{\frac{1}{d}\left(\sum_{i=1}^{d-1}c_i^2+\left(\sum_{i=1}^{d-1}c_i\right)^2\right)+\epsilon}}
\]
Replacing Pre-LN by Pre-RMSNorm or Pre-CRMSNorm produces variants with no change in function and strictly reduced floating-point operations [2305.14858].

## 4. Equivalence Theorems and Reparameterization

Pre-LN, Pre-RMSNorm, and Pre-CRMSNorm transformers are proven to be arithmetically equivalent at both training and inference:
\[
\forall\,x,\;\; f_{\mathrm{Pre\text{-}LN}(x;\theta)} = f_{\mathrm{Pre\text{-}RMSNorm}(x;\phi(\theta))} = f_{\mathrm{Pre\text{-}CRMSNorm}(x;\psi(\theta))}
\]
This equivalence is established through three key properties:
- LayerNorm on zero-mean inputs is identical to RMSNorm.
- Mean-centering can be algebraically absorbed into linear weights/biases (Lemma 1).
- The $d$-to-$(d-1)$ vector compression on zero-mean activations is lossless; surrounding linear layers can be rewritten correspondingly.
Training equivalence is maintained by conceptual "master copy" weights from Pre-LN, with forward and backward passes executed via transformed parameters without affecting gradient trajectories. This unification demonstrates that any Pre-LN transformer can be exchanged for more efficient variants without fine-tuning or loss of function [2305.14858].

## 5. Empirical Findings: Efficiency and Learning Dynamics

LayerNorm accounts for approximately 10–15% of runtime in a Pre-LN block. Replacing Pre-LN with Pre-RMSNorm yields consistent efficiency gains: 1–10% speedup in inference and 1–3% in end-to-end training is observed on Vision Transformer and GPT-3-like benchmarks using A100 GPUs, CPUs, and JAX. Efficiency improvements arise from RMSNorm being 20–60% cheaper than LayerNorm. Pre-CRMSNorm offers up to a further 10% inference speedup when hardware efficiently accommodates the $(d-1)$ compression, though on current GPUs, dimensions are often restored to $d$, making Pre-CRMSNorm and Pre-RMSNorm nearly identical in speed [2305.14858].

Empirically, the role of LayerNorm parameters is pivotal. In Pre-LN models, removing LN parameters (i.e., setting $\gamma=1$, $\beta=0$) results in catastrophic failure to learn: test accuracy collapses irrecoverably, and memorization persists ($\approx 100 \%$ of noisy samples are memorized), with a sharp increase in overfitting gap. This underscores the necessity of normalization for both gradient stability and genuine learning in Pre-LN blocks [2511.10566].

## 6. Influence of Early, Middle, and Late Layer Normalization

LayerNorm's impact in Pre-LN blocks is stratified by depth. Removing normalization in early layers leads to the most severe destabilization of learning and highest memorization rates. This is quantitatively supported by the decay in gradient-norm upper bounds from early to late layers. Conversely, in Post-LN models, removing early LN parameters suppresses memorization and restores genuine label recovery, demonstrating an architectural dichotomy in the function of layer normalization [2511.10566]. Practical recommendations include preserving LN parameters in early Pre-LN layers to ensure optimization stability and generalization, and preferring Pre-LN design over Post-LN in new architectures where stable training is critical.

## 7. Summary Table: Pre-LN, Pre-RMSNorm, and Pre-CRMSNorm

| Variant         | Normalization Operation                           | Efficiency (%)       |
|-----------------|---------------------------------------------------|----------------------|
| Pre-LayerNorm   | $\mathrm{LayerNorm}(x)$                           | Baseline             |
| Pre-RMSNorm     | $\mathrm{RMSNorm}(x - \mu(x)\mathbf{1})$          | 1–10% speedup        |
| Pre-CRMSNorm    | $\mathrm{CRMSNorm}(c)$, $c$ is compressed vector  | Up to 10% further*   |

*When hardware efficiently utilizes $(d-1)$ dimension; in practice often similar to Pre-RMSNorm.

The equivalence of these variants enables transformer designers to directly substitute more efficient Pre-RMSNorm or Pre-CRMSNorm blocks in existing Pre-LN architectures, preserving all functional, optimization, and learning-theoretic properties [2305.14858].

Source: https://www.emergentmind.com/topics/pre-layernorm-residual-blocks