SiameseNorm: Dual-Stream Norm for Transformers

• SiameseNorm is a dual-stream architecture that decouples gradient flow and representational boundedness, ensuring robust optimization via shared transformation weights.
• It unifies the benefits of Pre-Norm and Post-Norm by preserving identity gradients and enforcing bounded hidden states, which improves perplexity and downstream task scores.
• Empirical results on 1.3B-parameter models show enhanced arithmetic accuracy and stable performance under aggressive learning rates with minimal additional parameters.

SiameseNorm is a two-stream residual architecture for Transformer models that addresses the long-standing incompatibility between Pre-Norm and Post-Norm normalization schemes. By explicitly decoupling gradient flow and representational boundedness via parallel streams with shared transformation parameters, SiameseNorm achieves both optimization stability and high expressive capacity. Empirical results on 1.3B-parameter LLMs demonstrate that SiameseNorm remains robust at aggressive learning rates and outperforms established baselines in both perplexity and downstream task scores, with nearly negligible parameter overhead (Li et al., 8 Feb 2026).

1. Structural Incompatibility of Pre-Norm and Post-Norm

Transformer architectures traditionally use either Pre-Norm or Post-Norm residual blocks. In the Pre-Norm scheme, LayerNorm (LN) is applied before the residual transformation: $X_{i+1} = X_i + F_i(\mathrm{LN}(X_i))$ This structure guarantees that the “skip” path from $X_i$ to $X_{i+1}$ yields an exact identity Jacobian ($\frac{\partial X_{i+1}}{\partial X_i}=I$) during backpropagation, enabling robust optimization by preventing vanishing or exploding gradients. However, as $X_i$ is never renormalized, its norm typically grows with depth, requiring deeper blocks to contribute inflated residuals—resulting in the dilution problem, where later layers’ influence on representations is diminished.

Conversely, in the Post-Norm setup, LN follows the residual addition: $X_{i+1} = \mathrm{LN}(X_i + F_i(X_i))$ This placement stabilizes the hidden state scale, keeping $\|X_i\|$ approximately constant across depth and preserving deep sequence dynamics (signal expressivity). However, gradients then traverse the LN Jacobian repeatedly, which skews the spectral norm away from 1 and amplifies vanishing (or exploding) dynamics with depth—the distortion problem.

Single-stream hybrid approaches attempting to combine these paradigms are forced to compromise: applying LN after the main path eliminates the identity gradient, while omitting normalization allows uncontrolled growth. SiameseNorm directly addresses this incompatibility by deploying two explicitly coupled streams.

2. Canonical Pre-Norm and Post-Norm Residual Block Formulations

The difference between Pre-Norm and Post-Norm can be succinctly captured by their forward equations:

• Pre-Norm residual block:

$Y = x + \mathrm{Module}(\mathrm{LN}(x))$

• Post-Norm residual block:

$Y = \mathrm{LN}(x + \mathrm{Module}(x))$

The choice directly determines the propagation and stability of both activations and gradients. Pre-Norm preserves the gradient highway at the expense of unbounded activity scale. Post-Norm enforces boundedness while distorting gradients through repeated LN applications.

3. SiameseNorm Two-Stream Architecture

SiameseNorm introduces two interdependent streams within each Transformer block:

• $X_i$: the “bounded” (Post-Norm–like) stream
• $X_i$0: the “unbounded” (Pre-Norm–like) stream

Both streams share residual transformation weights $X_i$1 (for attention/MLP). The forward computations are as follows: $X_i$2 where $X_i$3 is depth-wise scaling to maintain X-stream stability.

After $X_i$4 layers, the final model output fuses both: $X_i$5 The two additional per-layer LNs (LN$X_i$6, LN$X_i$7), plus a single fuse LN (LN$X_i$8), introduce negligible parameter increase.

During gradient backpropagation, the parameter update at layer $X_i$9 is influenced by both Pre-Norm and Post-Norm pathways, as captured by the block Jacobian: $X_{i+1}$0 This construction ensures that every $X_{i+1}$1 receives both stable and expressive gradient signals.

4. Resolution of the Stability–Performance Trade-Off

SiameseNorm’s two-stream design stabilizes gradient flow and deepens effective network capacity without requiring compromise:

• The $X_{i+1}$2-stream delivers an unbroken identity gradient (as in Pre-Norm), thereby governing training stability and preventing gradient vanishing/explosion.
• The $X_{i+1}$3-stream enforces bounded, normalized representations at each layer (as in Post-Norm), preserving expressivity and favorable signal propagation dynamics.
• The shared residual transformation $X_{i+1}$4 is updated by gradients influenced by both streams, avoiding performance trade-offs imposed by single-stream hybrids.

The block Jacobian’s structure guarantees one eigen-block with identity elements (for stable gradients) and another bounded in spectral norm by LN. Empirical findings support this—gradient norm plots indicate that both Pre-Norm and SiameseNorm maintain $X_{i+1}$5–$X_{i+1}$6 during training, even as Post-Norm-based variants often exceed $X_{i+1}$7 and diverge. The $X_{i+1}$8-stream’s hidden state norms are stable, while growth in the $X_{i+1}$9-stream asserts deep sequential reasoning capacity lost in standard Pre-Norm.

Notably, arithmetic evaluation accuracy increases from $\frac{\partial X_{i+1}}{\partial X_i}=I$0 (Pre-Norm) to $\frac{\partial X_{i+1}}{\partial X_i}=I$1 (SiameseNorm), demonstrating the restoration of deep reasoning capabilities.

5. Experimental Configuration and Benchmark Results

SiameseNorm was benchmarked in large-scale pre-training experiments with an OLMo-style Transformer featuring:

• 16 layers, hidden size $\frac{\partial X_{i+1}}{\partial X_i}=I$2, 16 attention heads, MLP size $\frac{\partial X_{i+1}}{\partial X_i}=I$3, RoPE positional encoding, SwiGLU, and RMSNorm ($\frac{\partial X_{i+1}}{\partial X_i}=I$4).
• Training conducted on $\frac{\partial X_{i+1}}{\partial X_i}=I$5B tokens (with $\frac{\partial X_{i+1}}{\partial X_i}=I$6B tokens for extended evaluation) of FineWeb-Edu data.
• Learning rate settings:
  • A: $\frac{\partial X_{i+1}}{\partial X_i}=I$7 (conservative)
  • B: $\frac{\partial X_{i+1}}{\partial X_i}=I$8 (moderate)
  • C: $\frac{\partial X_{i+1}}{\partial X_i}=I$9 (aggressive)
  • D: $X_i$0, $X_i$1B tokens

Baselines included Pre-Norm, Post-Norm, DeepNorm, ResiDual, HybridNorm, and Hyper-Connections.

Key performance metrics are provided below:

Setting	Learning Rate	SiameseNorm (PPL)	Pre-Norm (PPL)	Arithmetic Accuracy (%)	Avg. Downstream Score	Notes
A	$X_i$2e$X_i$3	10.57	11.21	–	–	HybridNorm: 10.91
B	$X_i$4e$X_i$5	10.43	10.84	–	53.5 vs. 51.9	Post/HybridNorm diverge
C	$X_i$6e$X_i$7	10.48	10.89	39.6 (vs. 28.1)	–	All Post-Norms highly unstable
D	$X_i$8e$X_i$9	9.42	9.67	–	58.7 vs. 57.2	350B tokens

SiameseNorm is the only scheme to maintain stability at aggressive learning rates, consistently reducing perplexity by $X_{i+1} = \mathrm{LN}(X_i + F_i(X_i))$0–$X_{i+1} = \mathrm{LN}(X_i + F_i(X_i))$1 and improving downstream scores by $X_{i+1} = \mathrm{LN}(X_i + F_i(X_i))$2–$X_{i+1} = \mathrm{LN}(X_i + F_i(X_i))$3 points compared to Pre-Norm.

6. Implementation Considerations and Pseudocode

SiameseNorm does not require significant changes to the computational backbone of self-attention or MLP blocks. Its primary architectural modification is the inclusion of two additional per-layer LNs and explicit management of two parallel streams, with an extra normalization on fused inputs before the residual transformation—found to be critical for stability.

Example PyTorch-style pseudocode (layer level):

$X_{i+1} = \mathrm{LN}(X_i + F_i(X_i))$7

Two “small” LNs ($X_{i+1} = \mathrm{LN}(X_i + F_i(X_i))$4) and a critical additional LN ($X_{i+1} = \mathrm{LN}(X_i + F_i(X_i))$5) are introduced per layer. The layer-wise scaling $X_{i+1} = \mathrm{LN}(X_i + F_i(X_i))$6 in the X-stream is required for stability at depth. No duplication of self-attention or MLP weights is required, ensuring parameter efficiency.

In summary, SiameseNorm runs Pre-Norm and Post-Norm paradigms in parallel on two coupled streams with shared residual weights. This duality preserves both the identity-gradient highway of Pre-Norm and the bounded dynamics of Post-Norm, enabling robust and performant deep Transformer training with marginal computational overhead (Li et al., 8 Feb 2026).