ERMS LN: RMS-Based Equivariance in CNNs

• ERMS LN is a normalization method that replaces variance with the root mean square computed over spatial dimensions to ensure translation equivariance.
• It eliminates mean subtraction while using spatially uniform affine parameters, ensuring consistent scaling across CNN feature maps.
• The design leverages theoretical analyses of aliasing and equivariance, with empirical validations comparing performance with conventional LayerNorm variants.

Searching arXiv for the cited papers and closely related normalization work. Running a lightweight lookup against arXiv identifiers to verify bibliographic metadata. $y=\frac{x}{\sqrt{\mathbb{E}[x^2]}+\varepsilon},$9 $$r_b=\sqrt{\frac{1}{CHW}\sum_{c=0}^{C-1}\sum_{h=0}^{H-1}\sum_{w=0}^{W-1}x_{bchw}^2},$$0 $$r_b=\sqrt{\frac{1}{CHW}\sum_{c=0}^{C-1}\sum_{h=0}^{H-1}\sum_{w=0}^{W-1}x_{bchw}^2},$$1 Equivariant Root-Mean-Square Layer Normalization (ERMS LN) is an RMS-based normalization design for convolutional feature maps in which the normalization statistic is computed over axes that include the spatial dimensions and any learnable affine parameters are spatially uniform over the image plane. In the translation-equivariance framework for normalization layers, this construction yields architectural equivariance to discrete shifts, and, under the paper’s aliasing analysis, to continuous translations as well, provided the rest of the architecture is designed accordingly (Scanvic et al., 26 May 2025). Its normalization core follows RMSNorm in replacing variance-based standardization by division through a root mean square, thereby omitting mean subtraction and using the second raw moment as the scale statistic (Zhang et al., 2019).

1. Formal definition

The relevant setting is a feature map tensor

$x \in \mathbb{R}^{B\times C\times H\times W},$

with entries $x_{bchw}$. The normalization framework decomposes a layer into three conceptual steps: centering,

$x \mapsto x - \mathbb{E}[x],$

scaling,

$x \mapsto \frac{x}{\sqrt{\mathrm{Var}(x)}},$

and an affine transform,

$x \mapsto \gamma \odot x + \beta.$

For ERMS LN, the variance term is replaced by an RMS statistic,

$\mathrm{RMS}(x)=\sqrt{\mathbb{E}[x^2]},$

so that the core normalizer is of the form

$y=\frac{x}{\sqrt{\mathbb{E}[x^2]}+\varepsilon},$

possibly followed by a spatially uniform affine step (Scanvic et al., 26 May 2025).

A canonical per-sample form, closely aligned with the alias-free LayerNorm configuration but using RMS rather than variance, is

$$r_b=\sqrt{\frac{1}{CHW}\sum_{c=0}^{C-1}\sum_{h=0}^{H-1}\sum_{w=0}^{W-1}x_{bchw}^2},$$

$$y_{bchw}=\gamma_c\frac{x_{bchw}}{r_b+\varepsilon}+\beta_c.$$

Equivalently,

$$y_{b,c,h,w}=\gamma_c\frac{x_{b,c,h,w}}{\sqrt{\frac{1}{CHW}\sum_{c',h',w'}x_{b,c',h',w'}^2}+\varepsilon}+\beta_c.$$

In this form the denominator is constant over $x_{bchw}$0, and $x_{bchw}$1 are broadcast over $x_{bchw}$2 (Scanvic et al., 26 May 2025).

A central consequence of the theory is that centering is optional from the standpoint of equivariance. The decisive design choices are the axes over which the RMS is computed and the axes on which $x_{bchw}$3 are allowed to vary. This makes ERMS LN a family of constructions rather than a single fixed layer definition.

2. Translation-equivariance framework

The analysis of ERMS LN depends on a precise definition of the symmetry action on feature maps. For a discrete displacement $x_{bchw}$4, the circular shift operator is

$x_{bchw}$5

with modulo-$x_{bchw}$6 indexing. For a real-valued displacement $x_{bchw}$7, continuous translation is defined through Fourier phase shifts,

$x_{bchw}$8

which coincides with the discrete shift for integer displacements by the shift theorem (Scanvic et al., 26 May 2025).

A layer $x_{bchw}$9 is architecturally equivariant if

$x \mapsto x - \mathbb{E}[x],$0

for all parameters $x \mapsto x - \mathbb{E}[x],$1, all transformations $x \mapsto x - \mathbb{E}[x],$2, and all inputs $x \mapsto x - \mathbb{E}[x],$3. The requirement is architectural rather than empirical: it must hold for every admissible parameter value, not merely after training (Scanvic et al., 26 May 2025).

Within the normalization decomposition, centering always commutes with translations because it is a linear average. The only stages that can break equivariance are the scaling step and the affine step. The affine step fails when $x \mapsto x - \mathbb{E}[x],$4 depend on spatial indices, because a shift permutes content but leaves spatially indexed parameters fixed. The scaling step fails for continuous translations when the denominator varies over space, because entrywise multiplication by a spatially varying factor becomes convolution in the Fourier domain and can broaden bandwidth beyond Nyquist, inducing aliasing (Scanvic et al., 26 May 2025).

This framework places ERMS LN at the intersection of two constraints: parameter sharing over space and scale computation over space. The layer is not defined merely by using RMS rather than variance; its equivariance hinges on how the RMS statistic is indexed.

3. Necessary and sufficient conditions

The shift-equivariance theorem states that a normalization layer is equivariant to discrete shifts if and only if its affine step does not operate on the spatial dimensions, or if it has no affine step altogether. For ERMS LN, this means

$x \mapsto x - \mathbb{E}[x],$5

or, in the common CNN setting, $x \mapsto x - \mathbb{E}[x],$6 broadcast over $x \mapsto x - \mathbb{E}[x],$7 (Scanvic et al., 26 May 2025).

The translation-equivariance theorem adds a second condition: the scale statistic must be computed at least on the spatial dimensions. In RMS form, the denominator must average over $x \mapsto x - \mathbb{E}[x],$8, possibly together with $x \mapsto x - \mathbb{E}[x],$9 and/or $x \mapsto \frac{x}{\sqrt{\mathrm{Var}(x)}},$0. When this holds, the scaling factor is spatially constant and therefore does not create spatial modulation. In the paper’s formulation, a normalization layer is equivariant to continuous translations if and only if it is shift-equivariant and the standard deviation is computed at least on the spatial dimensions; the same axis-wise conclusion carries over when variance is replaced by the second moment because centering is irrelevant to equivariance and only the denominator’s spatial dependence matters (Scanvic et al., 26 May 2025).

Several common normalization patterns illustrate the distinction. BatchNorm and InstanceNorm are translation-equivariant because their scaling factors are spatially uniform. A full LayerNorm over $x \mapsto \frac{x}{\sqrt{\mathrm{Var}(x)}},$1 with affine parameters varying over $x \mapsto \frac{x}{\sqrt{\mathrm{Var}(x)}},$2 is equivariant to neither shifts nor translations. A channel-only LayerNorm with per-pixel statistics is shift-equivariant but not translation-equivariant, because the per-pixel denominator varies spatially and introduces aliasing. The alias-free LayerNorm variant, with centering over $x \mapsto \frac{x}{\sqrt{\mathrm{Var}(x)}},$3, scaling over $x \mapsto \frac{x}{\sqrt{\mathrm{Var}(x)}},$4, and affine over $x \mapsto \frac{x}{\sqrt{\mathrm{Var}(x)}},$5, is both shift- and translation-equivariant; ERMS LN is the RMS analogue of this construction (Scanvic et al., 26 May 2025).

A common misconception is that spatially uniform affine parameters alone suffice. They do not. Spatially uniform affine parameters guarantee shift-equivariance, but continuous translation-equivariance additionally requires a spatially uniform scale factor.

4. Relation to RMSNorm and normalization geometry

RMSNorm replaces LayerNorm’s mean-and-variance standardization with a pure RMS normalization. For a pre-activation vector $x \mapsto \frac{x}{\sqrt{\mathrm{Var}(x)}},$6,

$x \mapsto \frac{x}{\sqrt{\mathrm{Var}(x)}},$7

In vector form,

$x \mapsto \frac{x}{\sqrt{\mathrm{Var}(x)}},$8

with $x \mapsto \frac{x}{\sqrt{\mathrm{Var}(x)}},$9 and $x \mapsto \gamma \odot x + \beta.$0 denoting elementwise multiplication (Zhang et al., 2019). RMSNorm removes explicit re-centering, preserves re-scaling invariance for the whole weight matrix, the entire dataset, and individual inputs, and is computationally simpler than LayerNorm; the reported runtime reductions relative to LayerNorm range from $x \mapsto \gamma \odot x + \beta.$1 to $x \mapsto \gamma \odot x + \beta.$2 across the evaluated models and tasks (Zhang et al., 2019).

At the feature-vector level, the scalar RMS normalizer depends only on the Euclidean norm. Consequently, for orthogonal transformations $x \mapsto \gamma \odot x + \beta.$3,

$x \mapsto \gamma \odot x + \beta.$4

The divide-by-RMS step therefore commutes with orthogonal transformations, including permutations, and is naturally compatible with equivariance provided the gain and bias structure also respects the relevant group action (Zhang et al., 2019).

This contrasts with standard LayerNorm geometry. LayerNorm can be decomposed into projection onto the hyperplane orthogonal to the uniform vector, nonlinear scaling, and an affine map; its image lies within the intersection of an $x \mapsto \gamma \odot x + \beta.$5-dimensional hyperplane and an $x \mapsto \gamma \odot x + \beta.$6-dimensional hyperellipsoid, and the distinguished role of the uniform vector and diagonal gain makes the operation basis-dependent (Riechers, 2024). A related geometric analysis expresses LayerNorm standardization as

$x \mapsto \gamma \odot x + \beta.$7

with $x \mapsto \gamma \odot x + \beta.$8, and reports that several LayerNorm-based and RMSNorm-based LLMs operate, on average, nearly orthogonal to the uniform vector at inference time. That result was used to argue that mean subtraction can be redundant in practice (Gupta et al., 2024).

For ERMS LN, the significance of these feature-space results is indirect but important. They reinforce the CNN equivariance theorem’s claim that centering is not the critical issue. The symmetry-sensitive part of the design is the scale statistic and the parameter sharing pattern, not the presence or absence of mean subtraction.

5. Canonical constructions and empirical support

Three ERMS LN configurations satisfy the translation-equivariance conditions derived for CNNs. The first is the global per-sample form already given,

$x \mapsto \gamma \odot x + \beta.$9

The second is a per-channel, per-sample variant,

$\mathrm{RMS}(x)=\sqrt{\mathbb{E}[x^2]},$0

The third is a batch-plus-space, per-channel form,

$\mathrm{RMS}(x)=\sqrt{\mathbb{E}[x^2]},$1

All three compute the RMS over axes that include $\mathrm{RMS}(x)=\sqrt{\mathbb{E}[x^2]},$2, so their scaling factors are spatially uniform and satisfy the paper’s translation-equivariance conditions (Scanvic et al., 26 May 2025).

By contrast, a non-equivariant comparison point is the “LayerNorm-C-style” RMS construction

$\mathrm{RMS}(x)=\sqrt{\mathbb{E}[x^2]},$3

Here the denominator depends on spatial position, so the layer remains shift-equivariant but ceases to be translation-equivariant because the spatially varying gain induces aliasing (Scanvic et al., 26 May 2025).

The empirical evidence comes from feature maps extracted from 20 BatchNorm layers of a pretrained ResNet-18 on ImageNet validation images. The equivariance error is defined as the expected cosine distance between $\mathrm{RMS}(x)=\sqrt{\mathbb{E}[x^2]},$4 and $\mathrm{RMS}(x)=\sqrt{\mathbb{E}[x^2]},$5, separately for discrete shifts $\mathrm{RMS}(x)=\sqrt{\mathbb{E}[x^2]},$6 and continuous translations $\mathrm{RMS}(x)=\sqrt{\mathbb{E}[x^2]},$7. The reported averages are approximately $\mathrm{RMS}(x)=\sqrt{\mathbb{E}[x^2]},$8, $\mathrm{RMS}(x)=\sqrt{\mathbb{E}[x^2]},$9 for BatchNorm; $y=\frac{x}{\sqrt{\mathbb{E}[x^2]}+\varepsilon},$0, $y=\frac{x}{\sqrt{\mathbb{E}[x^2]}+\varepsilon},$1 for InstanceNorm; $y=\frac{x}{\sqrt{\mathbb{E}[x^2]}+\varepsilon},$2, $y=\frac{x}{\sqrt{\mathbb{E}[x^2]}+\varepsilon},$3 for LayerNorm-CHW; $y=\frac{x}{\sqrt{\mathbb{E}[x^2]}+\varepsilon},$4, $y=\frac{x}{\sqrt{\mathbb{E}[x^2]}+\varepsilon},$5 for LayerNorm-C; and $y=\frac{x}{\sqrt{\mathbb{E}[x^2]}+\varepsilon},$6, $y=\frac{x}{\sqrt{\mathbb{E}[x^2]}+\varepsilon},$7 for LayerNorm-AF (Scanvic et al., 26 May 2025).

These measurements align with the theory. For ERMS LN, they support two concrete conclusions. First, per-pixel RMS over channels is insufficient for continuous translation-equivariance even when affine parameters are spatially uniform. Second, global or spatially averaged RMS yields the desired translation behavior and is the appropriate design choice in alias-aware CNN pipelines.

6. Limitations, misconceptions, and open directions

ERMS LN does not by itself guarantee global equivariance of a network. The translation-equivariance theorem assumes that the surrounding architecture is also designed to be shift/translation-equivariant and alias-limited, including convolutions, anti-aliased activations, and downsampling or upsampling components. If other layers inject aliasing or use spatially indexed parameters, an ERMS LN block cannot repair that failure (Scanvic et al., 26 May 2025).

The theory also does not settle performance–equivariance trade-offs. The normalization-layer paper explicitly states that it does not study performance–equivariance trade-offs for normalization, and it notes that some equivariant designs may underperform non-equivariant ones in practice; it recommends empirical validation when adopting ERMS LN. The same source identifies extension of the framework from CNNs to Vision Transformers as an open question (Scanvic et al., 26 May 2025).

A further caveat concerns approximate RMS estimation. Partial RMSNorm estimates the RMS from only the first $y=\frac{x}{\sqrt{\mathbb{E}[x^2]}+\varepsilon},$8 components and preserves the same rescaling invariances as full RMSNorm, but a fixed subset breaks full permutation symmetry of coordinates unless the subset choice is itself group-compatible. For strict equivariant constructions, the safest option is full RMS, or group-wise RMS over subsets closed under the relevant symmetry action (Zhang et al., 2019).

The main conceptual misconception is to treat ERMS LN as merely “RMSNorm applied to CNNs.” That description is incomplete. ERMS LN is RMS-based, but its defining feature is the conjunction of two axis constraints: the scale statistic must include the spatial dimensions, and the affine parameters must be independent of spatial coordinates. Under those conditions, RMS normalization becomes compatible with the translation-equivariance framework developed for convolutional architectures (Scanvic et al., 26 May 2025).