---
title: Pre-Norm Residual Connections in Transformers
url: https://www.emergentmind.com/topics/pre-norm-residual-connections-prenorm
type: topic
---

# Pre-Norm Residual Connections in Transformers

Pre-Norm residual connections ("PreNorm") are a foundational design in Transformer architectures, wherein layer normalization (LayerNorm) is applied before each attention or feedforward sub-layer, rather than after the residual addition. This architectural choice yields identity skip paths, changes the propagation of gradients, and fundamentally influences training stability, convergence, overfitting, and computational efficiency in deep neural models. The PreNorm design has been adopted as standard in large-scale language models and machine translation, but also presents unique limitations and trade-offs explored in recent work.

## 1. Formal Definition and Mathematical Properties

In the Pre-Norm Transformer, each residual block comprises the following sequence:

- For an input $x_\ell$ to the $\ell$th block:
  $$
  u_\ell = \mathrm{LayerNorm}(x_\ell), \quad r_\ell = \mathcal{F}_\ell(u_\ell)
  $$
  $$
  x_{\ell+1} = x_\ell + r_\ell
  $$
where $\mathcal{F}_\ell$ represents either a multi-head attention or MLP sublayer. This construction ensures that the residual, or skip connection, is a pure identity: $\partial x_{\ell+1} / \partial x_\ell = I + \partial \mathcal{F}_\ell(\mathrm{LN}(x_\ell))/\partial x_\ell$.

By contrast, the Post-Norm scheme computes
$$
a_\ell = x_{\ell-1} + F(x_{\ell-1}), \quad x_\ell = \mathrm{LayerNorm}(a_\ell)
$$
in which the skip path traverses a further normalization transformation. In PreNorm, typically a final LayerNorm is appended after all residual blocks before the output head [2305.14858, 2110.09456, 1910.05895].

## 2. Gradient Flow and Representation Collapse

The placement of LayerNorm before each sublayer in PreNorm yields unimpeded backward gradient flow directly from deep layers to shallow ones via identity skips. This contrasts with Post-Norm, where gradients must traverse a product of LayerNorm affine transforms, potentially inducing exponential vanishing or explosion as depth increases. PreNorm's identity skip path results in more uniform and lower-variance gradient magnitudes, facilitating stable training without the need for gradual learning rate warmup or highly tuned initialization [1910.05895, 2304.14802].

However, PreNorm introduces its own notable pathology—representation collapse. As depth $N$ increases, the incremental contribution $\Delta x_\ell = x_\ell - x_{\ell-1}$ of each block shrinks as $\mathcal{N}(0, O(1/\ell))$, with $\mathbb{E}[|\Delta x_\ell|] = O(1/\sqrt{\ell}) \rightarrow 0$. This leads to situations in which the output representation $x_N$ varies negligibly when additional layers are added, implying a collapse of model expressivity at depth [2304.14802].

## 3. Empirical Performance, Stability, and Initialization

Empirical studies consistently show that PreNorm enables more robust and faster convergence on low-resource machine translation (MT) tasks and in small-batch regimes. It is less sensitive to initialization scale and does not require warmup scheduling to avoid training divergence—a significant practical advantage over PostNorm [1910.05895, 2110.09456]. BLEU gains of +0.27 or greater have been observed in PreNorm variants; training failure rates are sharply reduced, and global gradient norms are more consistent and less spiky across training runs.

However, in high-resource MT (e.g., WMT En→De) and multilingual or zero-shot settings, PreNorm's expressivity limitations become more pronounced. PostNorm still slightly outperforms PreNorm in ultimate BLEU scores and supports stronger language-agnostic representations in zero-shot translation tasks, with differences of up to 12.3 BLEU observed in favor of PostNorm [2305.09312].

### Empirical results as reported:
| Setting                  | PreNorm BLEU | PostNorm BLEU | Gap   |
|--------------------------|--------------|---------------|-------|
| IWSLT’14 En→De (12+12)   | 35.18        | Fail          | n/a   |
| WMT De→En (18+18)        | 26.57        | Fail          | n/a   |
| OPUS Zero-shot (Ex→X)    | 27.9         | 28.7          | +0.8  |
| OPUS Zero-shot S→T tag   | 10.1         | 16.8          | +6.7  |

In very deep or high-resource models, PreNorm's representation collapse can degrade generalization or fine-tuned performance [2304.14802, 2305.09312].

## 4. Extensions: RMSNorm, CRMSNorm, and Computational Efficiency

The PreNorm architecture admits efficient variants through alternative normalization schemes. RMSNorm replaces LayerNorm's mean subtraction and variance scaling with a root-mean-square scaling operation, removing the need to compute the mean:
$$
\mathrm{RMSNorm}(x) = \gamma \odot \frac{x}{\sqrt{\frac{1}{d}\sum_{i=1}^d x_i^2 + \epsilon}}
$$
CRMSNorm further compresses zero-mean vectors—by storing only the first $d-1$ coordinates—and operates on this compressed space for further computational saving [2305.14858].

For PreNorm Transformers, LayerNorm and RMSNorm at each block are provably arithmetically equivalent (assuming zero-mean vectors via mean-centering at initialization), and CRMSNorm yields equivalent function with lower dimensionality per residual block. This equivalence enables direct replacement of Pre-LN with Pre-RMSNorm/Pre-CRMSNorm to obtain 1–10% speedups in both training and inference, without changing model behavior, loss, or output [2305.14858].

| Normalization | Mean subtraction | Scaling | FLOPs | Empirical speedup |
|---------------|------------------|---------|-------|-------------------|
| LayerNorm     | Yes              | Variance| ~5d   | Baseline          |
| RMSNorm       | No               | RMS     | ~3d   | 1–9%              |
| CRMSNorm      | No (compressed)  | RMS     | ~3(d-1)| up to 10%         |

## 5. Variants, Limitations, and Hybrid Designs

While PreNorm is the dominant default for deep, supervised, or large language models due to stability, it has drawbacks in overfitting and poor zero-shot generalization. The risk of overfitting is exacerbated by LayerNorm’s trainable gain and bias, which can enable memorization along shallow bypasses in deep architectures [2305.09312]. In zero-shot multilingual MT, PreNorm's hidden states retain excessively strong source-language signals, increasing off-target decoding rates (e.g., PreNorm off-target ≈42% vs. PostNorm ≈8.6% on OPUS) [2305.09312].

NormFormer introduces three micro-modifications to standard PreNorm—head-wise attention scaling and two extra LayerNorms per layer—equalizing gradient magnitudes across layers to optimize convergence and allow higher learning rates. The overhead is negligible (+0.4% params), but total required compute to reach target perplexity drops by ≈40%, illustrating the benefit of tuning normalization placement and granularity [2110.09456].

ResiDual proposes a hybrid Pre-Post-LN design by fusing both PreNorm and PostNorm residual paths, ensuring both non-vanishing gradients (via PreNorm identity skips) and robust layerwise representation diversity (via PostNorm normalization after each residual addition). This architecture achieves stable, faster, and more expressive training in both shallow and very deep models, outperforming both baselines in BLEU on diverse MT tasks. It also avoids the collapse and overfitting phenomena endemic to pure PreNorm [2304.14802].

## 6. Practical Recommendations and Implementation

For purely supervised, deep Transformer models or large language model pretraining, PreNorm (with potential enhancements such as RMSNorm or NormFormer additions) is recommended for its stability, ease of training, and reduced sensitivity to initialization. Final LayerNorm after $L$ residual blocks should be appended prior to the task head. For multilingual and zero-shot scenarios, PostNorm or Pre-Post-LN hybridization is favored for generalization and language-agnosticity.

Minimal PreNorm Transformer pseudocode:
```python
def pre_norm_layer(x):
    z1 = LayerNorm(x)
    y1 = SelfAttention(z1)
    x1 = x + y1
    z2 = LayerNorm(x1)
    y2 = FeedForward(z2)
    return x1 + y2
```
Conversion to Pre-RMSNorm or Pre-CRMSNorm requires mean-centering of residual states and corresponding parameter reparametrization, with no change in training or inference semantics [2305.14858].

## 7. Open Issues and Research Directions

Despite the practical success of PreNorm, its limitations—including representation collapse, overfitting to supervised pairs, and poor cross-lingual abstraction—remain unresolved in their general form. The hybrid dual-residual paradigm [2304.14802] and the decoupling of normalization from residual structure (e.g., NormFormer [2110.09456], CRMSNorm [2305.14858]) suggest a broader design space for retaining both stable training and expressive, generalizable representations. The application of these insights to architectures beyond Transformers, such as vision models and adapter-based systems, remains an open avenue. Furthermore, the computational advantages of RMSNorm/CRMSNorm incentivize continued work in optimizing normalization for both algorithmic and hardware efficiency, especially as model sizes scale.

---
**Key References:**  
- "ResiDual: Transformer with Dual Residual Connections" [2304.14802]  
- "NormFormer: Improved Transformer Pretraining with Extra Normalization" [2110.09456]  
- "Transformers without Tears: Improving the Normalization of Self-Attention" [1910.05895]  
- "Exploring the Impact of Layer Normalization for Zero-shot Neural Machine Translation" [2305.09312]  
- "Pre-RMSNorm and Pre-CRMSNorm Transformers: Equivalent and Efficient Pre-LN Transformers" [2305.14858]

Source: https://www.emergentmind.com/topics/pre-norm-residual-connections-prenorm