PatchNorm: Patchwise Normalization in Convolutions

• PatchNorm is a patch-based normalization method that intrinsically modifies convolutional layers by rescaling local receptive-field patches based on affine divergence.
• It offers two variants—norm-like and affine-like—with distinct scale invariance and numerical stability characteristics, ensuring more precise activation updates.
• Empirical evaluations on CIFAR-10 demonstrate that PatchNorm performs comparably to traditional normalization techniques, highlighting its potential in diagnosing convolutional update approximations.

Searching arXiv for papers and terminology around “PatchNorm” to ground the article in current literature. PatchNorm denotes a family of patch-based normalization constructions whose meaning depends on context. In the convolutional setting, the term is introduced for a new convolution-specific normalization/correction family derived from the theory of affine divergence, the mismatch between the ideal activation update given by backpropagation and the effective activation update induced by parameter gradient descent (Bird, 24 Dec 2025). In the Vision Transformer literature, the related but distinct term Dual PatchNorm denotes two LayerNorm layers placed around the patch embedding layer, one before and one after patch embedding (Kumar et al., 2023). The term should therefore be interpreted with care: in one usage it names a convolution-intrinsic, patchwise modification of the convolutional map itself; in another it names a stem-only LayerNorm placement strategy for ViTs.

1. Affine divergence and the origin of convolutional PatchNorm

The 2025 paper "The Affine Divergence: Aligning Activation Updates Beyond Normalisation" defines the motivating problem in terms of the gap between the mathematically ideal steepest-descent direction for activations and the update actually induced when parameters are updated by gradient descent (Bird, 24 Dec 2025). For an affine layer,

$\vec{z}=\mathbf{W}\vec{x}+\vec{b},$

with activation gradient

$\frac{\partial \mathcal{L}}{\partial z_i}=g_i,$

a gradient step on parameters yields

$$W_{ij}'=W_{ij}-\eta g_i x_j,\qquad b_i'=b_i-\eta g_i.$$

Propagating this parameter change into the same layer output gives

$z_i' = z_i - \eta g_i\left(\|\vec{x}\|^2+1\right),$

so that

$$\frac{\Delta \mathcal{L}}{\Delta z_i} = g_i\left(\|\vec{x}\|^2+1\right),$$

rather than the ideal gradient $g_i$. The paper names this multiplicative mismatch the affine divergence.

Within that framework, the desideratum is convergence of effective and ideal activation updates,

$$\frac{\Delta \mathcal{L}}{\Delta \vec{z}}=\frac{\partial \mathcal{L}}{\partial \vec{z}}.$$

Solving this condition leads to two broad affine-layer modifications: a norm-like form,

$z_i = W_{ij}\left(\frac{x_j}{s}\right)+b_i,$

and an affine-like form,

$z_i = \frac{W_{ij}x_j+b_i}{s}.$

With suitable $s$, both cancel the mismatch under the paper’s first-order, single-layer approximation. This is the conceptual route by which the paper states that normalization can be “derived from first principles” rather than justified only post hoc.

PatchNorm arises when the same reasoning is generalized from affine layers to convolution. A standard convolution,

$\frac{\partial \mathcal{L}}{\partial z_i}=g_i,$0

is rewritten by patch unfolding as

$\frac{\partial \mathcal{L}}{\partial z_i}=g_i,$1

Because this patch-indexed form has the same algebraic divergence structure as batched affine layers, the proposed correction is performed patchwise. The paper emphasizes that PatchNorm is not merely the application of an existing normalization near a convolutional layer; it is a modification of convolution itself and is described as “inseparable from convolution.”

2. Mathematical forms and tensor structure

In the convolutional formulation, $\frac{\partial \mathcal{L}}{\partial z_i}=g_i,$2 denotes the input feature map at spatial location $\frac{\partial \mathcal{L}}{\partial z_i}=g_i,$3 and input channel $\frac{\partial \mathcal{L}}{\partial z_i}=g_i,$4, $\frac{\partial \mathcal{L}}{\partial z_i}=g_i,$5 the convolution kernel, $\frac{\partial \mathcal{L}}{\partial z_i}=g_i,$6 the bias, and $\frac{\partial \mathcal{L}}{\partial z_i}=g_i,$7 the output at spatial position $\frac{\partial \mathcal{L}}{\partial z_i}=g_i,$8 and output channel $\frac{\partial \mathcal{L}}{\partial z_i}=g_i,$9. After patch unfolding, $$W_{ij}'=W_{ij}-\eta g_i x_j,\qquad b_i'=b_i-\eta g_i.$$0 is the flattened receptive-field patch indexed by $$W_{ij}'=W_{ij}-\eta g_i x_j,\qquad b_i'=b_i-\eta g_i.$$1, $$W_{ij}'=W_{ij}-\eta g_i x_j,\qquad b_i'=b_i-\eta g_i.$$2 indexes elements within the flattened receptive field, $$W_{ij}'=W_{ij}-\eta g_i x_j,\qquad b_i'=b_i-\eta g_i.$$3 is the flattened kernel, $$W_{ij}'=W_{ij}-\eta g_i x_j,\qquad b_i'=b_i-\eta g_i.$$4 is the output for patch $$W_{ij}'=W_{ij}-\eta g_i x_j,\qquad b_i'=b_i-\eta g_i.$$5, and $$W_{ij}'=W_{ij}-\eta g_i x_j,\qquad b_i'=b_i-\eta g_i.$$6 is one scalar normalization factor per patch (Bird, 24 Dec 2025).

The paper defines two PatchNorm variants exactly. The affine-like PatchNorm form is

$$W_{ij}'=W_{ij}-\eta g_i x_j,\qquad b_i'=b_i-\eta g_i.$$7

and is the convolutional analogue of

$$W_{ij}'=W_{ij}-\eta g_i x_j,\qquad b_i'=b_i-\eta g_i.$$8

The norm-like PatchNorm form is

$$W_{ij}'=W_{ij}-\eta g_i x_j,\qquad b_i'=b_i-\eta g_i.$$9

and is the convolutional analogue of

$z_i' = z_i - \eta g_i\left(\|\vec{x}\|^2+1\right),$0

The defining structural feature is that PatchNorm normalizes local receptive-field patches. It is therefore not channelwise normalization, not global layerwise normalization, not tokenwise normalization in the transformer sense, and not a batch-statistics method. It is patchwise over each convolutional receptive field. The paper further distinguishes the two variants by scale behavior: the norm-like form uses

$z_i' = z_i - \eta g_i\left(\|\vec{x}\|^2+1\right),$1

and is scale-invariant with respect to patch magnitude, whereas the affine-like form uses

$z_i' = z_i - \eta g_i\left(\|\vec{x}\|^2+1\right),$2

and is not scale-invariant.

A plausible implication is that the paper uses PatchNorm not only as a new “normalizer,” but also as a family of alternative convolutional functional forms derived from an optimization-geometric criterion rather than from batch-statistics or variance-control considerations.

3. Mechanistic interpretation and relation to other normalization schemes

The central mechanistic claim is that normalization works, at least in part, by reducing or canceling the mismatch between the ideal activation update $z_i' = z_i - \eta g_i\left(\|\vec{x}\|^2+1\right),$3 and the effective update induced by parameter descent $z_i' = z_i - \eta g_i\left(\|\vec{x}\|^2+1\right),$4 (Bird, 24 Dec 2025). In this interpretation, normalizing transformations are not primarily justified by internal covariate shift, variance control, or scale invariance, but by geometric alignment of activation updates.

This framing explains why the paper describes PatchNorm as “compositionally inseparable.” Standard normalization layers such as BatchNorm, LayerNorm, and RMSNorm are usually composed before or after a layer. PatchNorm instead alters the internal formula of convolution: $z_i' = z_i - \eta g_i\left(\|\vec{x}\|^2+1\right),$5 Because the scalar $z_i' = z_i - \eta g_i\left(\|\vec{x}\|^2+1\right),$6 depends on the patch being convolved, the normalization is intrinsic to the receptive-field computation rather than a generic post-hoc tensor transformation.

The paper’s mathematical comparison to existing normalizers is correspondingly structural. BatchNorm is described as using batch statistics and, in this framing, only mitigating divergence by damping variance in $z_i' = z_i - \eta g_i\left(\|\vec{x}\|^2+1\right),$7; LayerNorm is samplewise and global over a layer or token representation rather than patch-local; RMSNorm resembles $z_i' = z_i - \eta g_i\left(\|\vec{x}\|^2+1\right),$8-style samplewise scaling but typically over a whole feature vector rather than a flattened receptive field. GroupNorm, InstanceNorm, and EvoNorm are not analyzed explicitly. The most precise standalone description given in the paper is that PatchNorm is a patchwise, convolution-intrinsic normalization/correction where each receptive field is rescaled by a scalar computed from that patch’s own norm, either before kernel application or on the convolution output including bias.

This suggests that PatchNorm occupies a different conceptual category from common “normalization layers.” It is presented as an attempt to modify the geometry of the local convolutional map itself so that effective representation updates better approximate steepest descent.

4. Architectural use, implementation properties, and numerical considerations

For fair comparison in the paper’s experiments, all normalizers and corrections are applied as precomposed modifications at the layer; for PatchNorm, this means replacing a standard convolution with the modified patchwise rule (Bird, 24 Dec 2025). A standard patchwise convolution computation,

$z_i' = z_i - \eta g_i\left(\|\vec{x}\|^2+1\right),$9

is replaced either by

$$\frac{\Delta \mathcal{L}}{\Delta z_i} = g_i\left(\|\vec{x}\|^2+1\right),$$0

or by

$$\frac{\Delta \mathcal{L}}{\Delta z_i} = g_i\left(\|\vec{x}\|^2+1\right),$$1

The forms presented are parameterless; no extra learnable $$\frac{\Delta \mathcal{L}}{\Delta z_i} = g_i\left(\|\vec{x}\|^2+1\right),$$2 are included in the ablations. The paper also does not introduce separate train/inference rules. Since there are no running statistics, PatchNorm is the same functional map at train and test time.

The computational overhead is not quantified explicitly, but the paper states that PatchNorm requires evaluating each receptive-field norm, computing one scalar $$\frac{\Delta \mathcal{L}}{\Delta z_i} = g_i\left(\|\vec{x}\|^2+1\right),$$3 per patch, and then scaling the patch or patch output. It is explicitly contrasted with a full correction that would require a patchwise pseudo-root-inverse, described as computationally prohibitive. Thus PatchNorm is presented as a local approximation rather than the full algebraic solution.

The paper is explicit about numerical stability. The norm-like correction has a singularity as $$\frac{\Delta \mathcal{L}}{\Delta z_i} = g_i\left(\|\vec{x}\|^2+1\right),$$4, and the same reasoning applies patchwise: $$\frac{\Delta \mathcal{L}}{\Delta z_i} = g_i\left(\|\vec{x}\|^2+1\right),$$5 Adding $$\frac{\Delta \mathcal{L}}{\Delta z_i} = g_i\left(\|\vec{x}\|^2+1\right),$$6 is said to be possible but theoretically ad hoc and to reintroduce distortion. By contrast, affine-like PatchNorm,

$$\frac{\Delta \mathcal{L}}{\Delta z_i} = g_i\left(\|\vec{x}\|^2+1\right),$$7

is non-singular and is therefore theoretically preferred by the paper’s logic. The method is explicitly designed for convnets, but the paper does not develop residual-specific PatchNorm theory and notes that residual architectures complicate the single-layer approximation underlying the analysis.

5. Empirical evaluation in convolutional networks

PatchNorm is evaluated on CIFAR-10 classification using two CNN families, each trained for 100 epochs, 5 repeats, using ADAM, with parameterless comparison methods. The paper states overall that 160 networks tested and 32 unique convolutional networks are included in this appendix evaluation (Bird, 24 Dec 2025).

In the first experiment, a CNN with a global average pooling (GAP) layer is compared against no normalization, BatchNorm, LayerNorm, RMSNorm, layerwise $$\frac{\Delta \mathcal{L}}{\Delta z_i} = g_i\left(\|\vec{x}\|^2+1\right),$$8-Norm, and PatchNorm variants. For Tanh, the paper reports that “all PatchNorm variants, RMSNorm and LayerNorm perform comparably well,” followed by no normalization, then BatchNorm, and substantially worse layerwise $$\frac{\Delta \mathcal{L}}{\Delta z_i} = g_i\left(\|\vec{x}\|^2+1\right),$$9-norm. For Leaky-ReLU, the results are described as much closer, with LayerNorm highest, then RMSNorm, then No-Norm, then BatchNorm and affine-like PatchNorm, then $g_i$0-like PatchNorms, and layerwise $g_i$1-norm last.

In the second experiment, a CNN without GAP gradually reduces spatial dimensions into channels instead of using global average pooling. For Tanh, LayerNorm is best, followed by the three PatchNorm forms, then RMSNorm with high variability, then BatchNorm, then layerwise $g_i$2-norm, and No-Norm worst by a substantial gap. For Leaky-ReLU, BatchNorm is best, followed by LayerNorm, then RMSNorm, then the two full-$g_i$3 PatchNorms, with the other PatchNorm, NoNorm, and layerwise $g_i$4-norm similar.

The author’s summary is deliberately restrained. PatchNorm remains successful, but is more comparable to existing normalizers than the affine structural corrections were in fully connected networks, and other norms can slightly outperform it. The interpretation offered is that the single-patch approximation breaks down in convolution because patches are not independent the way samples approximately are in the batched affine analysis. The paper attributes this to nonlinear downstream mixing between patches, repeated values caused by patch unfolding, and gradient interactions from overlapping patches. In that sense, PatchNorm is presented less as a dominant replacement for standard convolutional normalization than as a principled convolutional extension whose mixed results help diagnose the limits of the approximation.

6. Terminological variants and related but distinct usages

The term Dual PatchNorm refers to a different method proposed for Vision Transformers in "Dual PatchNorm" (Kumar et al., 2023). There, the defining equation is

$g_i$5

Operationally, one LayerNorm is applied after patch extraction and flattening but before the linear patch embedding projection, and a second LayerNorm is applied after the projection. The rest of the Transformer remains unchanged. This method is a stem-only modification of ViT input processing rather than a convolution-intrinsic receptive-field normalization. Although the name contains “PatchNorm,” its mechanism, architecture, and intended scope are distinct from convolutional PatchNorm.

The ViT paper compares this stem modification against an exhaustive search over LayerNorm placement inside Transformer blocks, including nine configurations formed by placing LayerNorm before, after, or both before and after self-attention and MLP sublayers. It concludes that the standard pre-LN Transformer block is already close to optimal for ImageNet-1k classification, while adding the two LayerNorms around patch embedding produces more consistent gains. The paper also reports that Dual PatchNorm reduces the disproportionately large gradient norm at the patch embedding layer. This establishes a second major usage of “PatchNorm” in the literature: not as a new convolutional functional form, but as a pair of LayerNorm operations around ViT patch embedding.

A third nearby term is PatchNR, introduced in "PatchNR: Learning from Very Few Images by Patch Normalizing Flow Regularization" (Altekrüger et al., 2022). That work does not use the term PatchNorm explicitly. It defines a patch-based probabilistic regularizer for inverse problems, where a normalizing flow is learned on small patches and the average patch negative log-likelihood becomes a variational regularizer: $g_i$6 This is not a normalization layer in the deep-learning sense. Its relevance is terminological and conceptual only: it is patch-based and statistics-driven, but it belongs to learned priors for imaging inverse problems rather than to network normalization.

These distinctions matter because “PatchNorm” is not a single universally standardized method name across the literature. In precise usage, the convolutional method in (Bird, 24 Dec 2025), Dual PatchNorm in ViTs (Kumar et al., 2023), and PatchNR (Altekrüger et al., 2022) should be treated as separate constructs with different mathematical objects, optimization roles, and empirical domains.

7. Limitations, misconceptions, and broader significance

A frequent misconception is to treat convolutional PatchNorm as simply “apply a normalization layer near a conv layer.” The 2025 paper explicitly rejects that interpretation: PatchNorm is a family of normalisation applied patch-wise and is said to be inseparable from convolution (Bird, 24 Dec 2025). Another misconception is to assume that its motivation is standard scale-invariance. The paper instead argues that empirical success need not come from scale invariance, since the affine-like family often performs well despite lacking it.

The main caveat is that the affine-layer approximation transfers imperfectly to convolution. Although convolution unrolled into patches is algebraically identical to a batched affine map, the paper stresses that the analogy is not semantically valid because patches are not independent samples, unfolding duplicates values, and patches are repeatedly and nonlinearly recombined within the same example. A full correction would require a patchwise Gram-like inverse or root-inverse across patches, but that would be computationally prohibitive, would nonlocally mix patches, and would break desirable convolutional inductive biases. PatchNorm is therefore presented as a local approximation rather than the exact correction.

From an encyclopedic perspective, the broader significance of PatchNorm lies in the reframing it proposes. Instead of treating normalization primarily as a statistical device, the affine-divergence account interprets it as local alignment of effective activation updates with steepest descent in representation space. This suggests an alternative taxonomy in which some “normalizers” are better understood as activation-function-like maps with parameterized scaling or as alternatives to the affine map itself. Whether that framework will generalize beyond the settings studied in the paper remains open, but the article-length contribution of (Bird, 24 Dec 2025) is precisely to make that theoretical proposal concrete in both affine layers and convolutional PatchNorm.