PI-Compatible Alternatives to LN

• PI-compatible normalization methods are techniques that preserve output consistency regardless of feature or token order, ensuring order invariance.
• They encompass statistical, elementwise, and proxy approaches—such as PONO, IterL2Norm, and DyISRU—that improve convergence and hardware efficiency.
• These alternatives enable robust performance in transformer and convolutional networks by reducing memory bandwidth needs and computational overhead.

Permutation-invariant (PI) alternatives to Layer Normalization (LN) provide normalization or regularization mechanisms that preserve invariance to channel or token ordering and introduce either no or only computationally minimal dependency on the ordering or grouping of feature axes. The motivation for PI compatibility stems from both the mathematical structure of self-attention—which requires that normalization layers commute with permutations of the token or channel dimension—and from hardware and modeling considerations, where reduction of memory bandwidth, support for quantized formats, or removal of on-the-fly statistics is desirable.

1. Structural Properties of PI-Compatible Alternatives

Layer Normalization, although itself permutation-invariant with respect to the order of features within a single sample, relies on full-vector reductions (mean and variance calculations) across each position's feature dimension. This motivates research into PI-compatible alternatives that operate with the following guarantees:

• Permutation invariance (PI): For any permutation $\pi$ of the feature (channel, token) axis, applying the permutation to the inputs before normalization yields the equivalently permuted outputs; this property ensures no bias is introduced by channel ordering.
• Tokenwise operation: In transformer and attention models, normalization must treat each position/timestep equivalently (no cross-token reductions).
• Drop-in replacement: Ideal alternatives match or improve LN’s effect on convergence and generalization, minimize hardware cost, and maintain PI.

Alternatives fall into three main categories:

• Moment-based (statistical) normalization: operate over reduced spacings (channel-only, spatial-only, or both).
• Elementwise dynamic activation functions: replace normalization with suitably parameterized pointwise nonlinearity.
• Meta or proxy normalization: introduce auxiliary predictors or proxy distributions to emulate batch- or layer-wise statistical effects.

2. Statistical Normalization: PONO, IterL2Norm, UN

PONO (Positional Normalization)

PONO (Li et al., 2019) normalizes exclusively across the channel dimension at each spatial location $(i,j)$:

$$\hat X_{b,c,i,j} = \frac{X_{b,c,i,j} - \mu_{b,i,j}}{\sqrt{\sigma^2_{b,i,j} + \varepsilon}}$$

with $\mu_{b,i,j}$ and $\sigma^2_{b,i,j}$ computed as averages over $c$, not $i,j$. This ensures that for any channel permutation, the output distribution at each $(i,j)$ remains unchanged, constituting strict permutation invariance in channel space. Moment-Shortcut (MS) and Dynamic Moment Shortcut (DMS) variants re-inject these moments into later layers to preserve structural information, an inductive bias that empirically improves generative translation and style transfer FID by 10–15%.

IterL2Norm

IterL2Norm (Ye et al., 2024) performs L2 normalization over features without division or square roots, using a fixed-point iterative scheme suitable for hardware:

$a_{i+1} = a_i + \lambda m a_i (1 - m a_i^2)$

where $m = \|y\|_2^2$ and $y = x - \text{mean}(x)$, converging in five steps. This method is PI, as all operations are symmetric with respect to feature indices and no ordering-dependent branching occurs. Empirical precision is within $10^{-3}$ of exact norm in FP32/BFloat16, supporting deployment in PIM and low-power hardware at up to $0.44$ million norms/sec, with area and power budgets far below division-based LN.

Unified Normalization (UN)

Unified Normalization (Yang et al., 2022) eliminates on-the-fly mean/variance computation at inference by precomputing per-feature $\mu, \sigma^2$ during training, then fusing normalization into subsequent linear layers:

$$Y = \gamma \cdot \frac{X - \mu}{\sigma^2 + \varepsilon} + \beta$$

with $\mu,\sigma^2$ “frozen” and geometric mean smoothing over recent training iterations to stabilize activation statistics, together with rigorous outlier filtration. UN is functionally PI, as all featurewise statistics are computed per feature and not dependent on input order. Empirically, UN preserves or slightly improves upon LN accuracies in transformers, with 31% inference speedup and 17% memory reduction.

3. Elementwise Dynamic Activations: DyT, DyISRU, and NoMorelization

Dynamic Tanh (DyT) and Dynamic ISRU (DyISRU)

DyT (Zhu et al., 13 Mar 2025, Stollenwerk, 27 Mar 2025) exploits the observation that LN layers, when plotted (pre- vs. post-activations), realize a tanh-like $S$-shaped mapping:

$$\mathrm{DyT}(x) = \gamma \odot \tanh(\alpha x) + \beta$$

with $\alpha$ learned and $\gamma, \beta$ per channel. The operation is fully elementwise and thus strictly PI. DyT matches or exceeds LN performance on ImageNet, Diffusion, and LLM tasks, with up to 52% layerwise inference speedup. A first-order ODE approximation of LN substantiates the mapping.

DyISRU (Stollenwerk, 27 Mar 2025) derives from the exact LN-induced ODE, yielding:

$$y_i = \sqrt{D-1}\frac{x_i - \mu}{\sqrt{\beta + (x_i - \mu)^2}}$$

or, for outliers/zero mean, $y_i = \sqrt{D-1} x_i / \sqrt{\beta + x_i^2}$, with a global learned $\beta$. DyISRU tracks LN’s per-sample mapping with mean absolute residual $<0.01$, is PI, and requires only constant-time, coordinate-wise operations.

NoMorelization

NoMorelization (Liu et al., 2022) replaces normalization with two trainable per-layer scalars $\alpha,\beta$ (affine) and, during training, a coordinate-wise injected noise $\gamma\delta$:

$$y = \alpha f(x) + \beta + \gamma\delta\,,\qquad \delta \sim \mathcal N(0, I)$$

PI is preserved as these operations are coordinate-wise and inject only zero-mean i.i.d. noise. NoMorelization achieves accuracy exceeding BN/LN in both convolutional and self-attention architectures with speed improvements, and its theoretical justification (sample noise correction) ensures no spurious batch-induced bias (Liu et al., 2022).

4. Proxy, Meta, and Orthonormalizing PI Normalizations

Proxy Normalization (PN)

Proxy Normalization (Labatie et al., 2021) addresses LN-induced failure modes—collapsing expressivity by enforcing too-strong channel constancy—by normalizing via a proxy distribution:

$$l'_{\alpha,c} = \frac{\phi(\gamma_c l_{\alpha,c} + \beta_c) - \mathbb{E}_{Y_c}[\phi(\gamma_c Y_c + \beta_c)]}{\sqrt{\mathrm{Var}_{Y_c}[\phi(\gamma_c Y_c + \beta_c)] + \epsilon}}$$

with $$Y_c \sim \mathcal N(\tilde\beta_c,(1+\tilde\beta_c)^2)$$. This yields PI compatibility, preserves channel-wise normalization iteratively through layers, and improves top-1 accuracy over both LN and GN. PN is strictly batch-independent and restores BN-like behavior in batch-insensitive settings, robustly closing the performance gap.

Instance-Level Meta Normalization (ILM Norm)

ILM Norm (Jia et al., 2019) meta-learns affine normalization parameters using a lightweight hyper-autoencoder, conditioned on per-instance, PI group statistics:

$$\hat{X}_{b,c,i,j} = \frac{X_{b,c,i,j}-\mu_{b,n(c)}}{\sqrt{\sigma^2_{b,n(c)}+\epsilon}}$$

with affine parameters predicted via both gradient-based and meta-encoders. This yields strong empirical gains over LN/IN/GN, especially for small batches, with negligible computational or parameter overhead.

ZCA Whitening

ZCA whitening (Blanchette et al., 2018) is a classic PI normalization, decorrelating and scaling channels via the eigendecomposition of the covariance matrix:

$$A = U \Lambda^{-1/2} U^T,\quad \text{where}\ \Sigma = U\Lambda U^T$$

Implementation with care for degeneracies ensures PI. Across MNIST, SVHN, and similar data, ZCA outperforms BN and works as a robust PI alternative, although with $O(N^3)$ computational cost.

5. Empirical Performance and Hardware Considerations

Table: Summary of Empirical and Implementation Properties

Method	PI Guarantee	Extra Parameters	Hardware Efficiency	Key Empirical Outcomes
PONO	Full (channel)	None or minimal (DMS)	Negligible overhead	-15% FID, improved structure transfer
IterL2Norm	Full	None	Division/sqrt free	< $10^{-3}$ norm error, 0.44M norm/s
UN	Full	None	Fused inference	+31% speed, ~acc parity w/ LN
DyT, DyISRU	Full	1–3 scalars	Elementwise only	Matches/exceeds LN, strong LLM results
NoMorelization	Full	2 scalars (+noise)	Minimal	Exceeds BN/LN, best speed-accuracy tradeoff
Proxy Normalization	Full	Minor (per-channel, per act.)	Minor	Improves GN/LN, matches BN
ILM Norm	Full (group)	<0.1% per layer	Minimal	Beats GN/LN/IN, robust to batch size
ZCA Whitening	Full	None	High cost	Best error (entropy variant)

PONO and DyISRU provide architectural and hardware efficiency by relying only on per-position or per-feature statistics/affine transforms, reducing memory and computation relative to traditional LN. Unified Normalization and NoMorelization further improve training and inference speed by removing all online statistics and, where needed, fusing parameters offline.

6. Expressivity, Failure Modes, and Limitations

LN and IN suffer from expressivity collapse (constant channel behavior or loss of per-instance variation) (Labatie et al., 2021). PN, NoMorelization, and DyISRU address this via careful construction—either proxying normalization, injecting batch-independent noise, or parameterizing elementwise behavior. However, ZCA whitening, while highly effective, incurs nontrivial computational and memory cost at scale. PONO’s preservation of structure via moment re-injection introduces a low-dimensional structural prior but is most beneficial for tasks with explicit spatial structure (e.g., translation, style transfer) and only marginal in multi-modal architectures (e.g., MUNIT).

Parameter-free or nearly so methods (DyISRU, IterL2Norm, NoMorelization) are preferred in accelerator-constrained or quantized settings; meta-parameter methods (ILM Norm) offer the best results in resource-rich environments.

7. Open Problems and Future Directions

Current open questions for PI-compatible alternatives center on:

• Theoretical characterization of inductive biases imparted by moment re-injection and elementwise squashing.
• Extending channel-PI normalization to non-image or temporal domains (sequences, graphs, video, 3D).
• Reducing whitening costs (approximate SVD, block-diagonalization), especially for very high-dimensional hidden representations in large models.
• Integration of dynamic or proxy-based normalization with normalization-free or NF-regularization frameworks in large pretraining pipelines.
• Further empirical assessment of interplay with regularization methods (mixup, CutMix, Dropout).

Remarkably, advances in elementwise PI-normalization (DyT, DyISRU, NoMorelization) challenge the assumption that high-performing deep nets require normalization layers with expensive reductions and per-sample statistics, motivating broader adoption of PI-compatible, hardware-efficient alternatives across both generative and discriminative deep learning architectures (Zhu et al., 13 Mar 2025, Liu et al., 2022, Stollenwerk, 27 Mar 2025, Li et al., 2019, Ye et al., 2024, Yang et al., 2022, Blanchette et al., 2018, Jia et al., 2019, Labatie et al., 2021).