Batchless Normalization (BlN) Techniques

• Batchless Normalization (BlN) is a family of techniques that remove minibatch dependencies during neural network training by using per-example statistics or learned parameters.
• BlN methods replace traditional batch statistics with sample-based normalization or gradient-driven parameter updates, enabling stable training in small-batch, distributed, and recurrent settings.
• Empirical results show that BlN approaches achieve competitive performance to Batch Normalization on benchmarks like CIFAR and ImageNet while reducing memory overhead and improving convergence.

Searching arXiv for the cited normalization papers and closely related batchless/normalizer-free work. Batchless Normalization (BlN) denotes a set of techniques that remove or replace minibatch-dependent activation statistics during neural-network training. Across the literature, the term has been used for several distinct mechanisms: per-example normalization within a layer, sample-by-sample running-statistic normalization, gradient-based learning of activation-distribution parameters, and normalizer-free architectural or initialization schemes that aim to recover the stabilizing effects usually associated with Batch Normalization (BN). This suggests that BlN is best understood as a family of batch-independent training strategies rather than a single algorithmic recipe (Ba et al., 2016, Chiley et al., 2019, Berger et al., 2022).

1. Scope and defining properties

A common motivation across BlN methods is the removal of BN’s dependence on minibatch statistics. In the formulations summarized here, BN is described as requiring means and variances over multiple examples, requiring special treatment for running statistics, introducing train-versus-test discrepancies, and complicating very small-batch, distributed, recurrent, or memory-constrained training (Berger et al., 2022, Civitelli et al., 2021).

Variant	Core mechanism	Stated consequence
Layer Normalization	Compute mean and variance from all summed inputs to neurons in a layer on a single training case	No batch statistics; training and test computations are identical
Online Normalization	Maintain exponentially-weighted running estimates sample by sample	Works without batches and introduces an unbiased technique for computing the gradient of normalized activations
Batchless Normalization (2022)	Add per-activation negative log-likelihood loss for a learned Gaussian $N(\mu,\sigma^2)$	No cross-sample dependencies; process each example independently
Normalization-Free ResNet initialization	Remove BN and modify residual summation to $x_l = c \cdot [h(x_{l-1}) + f_l(x_{l-1})]$ with careful He-style init	Stable forward and backward variance at initialization
BN-free BNN training	Replace BN with Adaptive Gradient Clipping, Scaled Weight Standardization, and a Specialized Bottleneck Block	BNs can be completely removed from BNN training and inference regimes
NoMorelization	Use two trainable scalars and a zero-centered noise injector	Normalizer-free substitute for BN, LN, IN, or other normalizers

The category is therefore heterogeneous. Some BlN methods still normalize activations explicitly, but do so without batch coupling; others eliminate explicit normalization layers and instead target variance preservation, deterministic scaling, or regularization by additive noise. A common source of ambiguity is that “batchless” does not imply a single shared statistical model.

2. Per-example normalization: Layer Normalization

Ba et al. introduced Layer Normalization (LN) by transposing BN from minibatch statistics to within-example statistics over the hidden units of a layer (Ba et al., 2016). For a single example with pre-activations $a=(a_1,\dots,a_H)\in\mathbb{R}^H$, LN computes

$$\mu = \frac{1}{H}\sum_{i=1}^H a_i,\qquad \sigma^2 = \frac{1}{H}\sum_{i=1}^H (a_i-\mu)^2 + \epsilon,$$

then normalizes

$\hat a_i = \frac{a_i-\mu}{\sqrt{\sigma^2}},$

and applies per-unit gain and bias,

$y_i = \gamma_i \hat a_i + \beta_i.$

Because all sums, means, and variances are computed per single example over the $H$ units, no batch statistics are used.

The learnable parameters are $\gamma=(\gamma_1,\dots,\gamma_H)$ and $\beta=(\beta_1,\dots,\beta_H)$, both of dimension $H$. They are initialized to $x_l = c \cdot [h(x_{l-1}) + f_l(x_{l-1})]$0 and $x_l = c \cdot [h(x_{l-1}) + f_l(x_{l-1})]$1 so that initially LN is the identity transform, and are updated by back-propagation. In a feed-forward layer, normalization is applied to the summed inputs before the nonlinearity. In recurrent settings, LN computes normalization statistics separately at each time step and can be inserted separately into each RNN gate or each sub-block of a GRU or LSTM.

LN’s defining properties in this formulation are that no running averages are needed, train and test time are identical, it works with batch size $x_l = c \cdot [h(x_{l-1}) + f_l(x_{l-1})]$2, and it is straightforward to apply to recurrent networks. The paper states that LN is very effective at stabilizing the hidden state dynamics in recurrent networks and can substantially reduce training time compared with previously published techniques. Reported examples include Order-Embeddings, where LN converged $x_l = c \cdot [h(x_{l-1}) + f_l(x_{l-1})]$3 faster than a baseline GRU and improved Recall@1 from $x_l = c \cdot [h(x_{l-1}) + f_l(x_{l-1})]$4 on MS-COCO; DRAW on binarized MNIST, where LN-DRAW converges in $x_l = c \cdot [h(x_{l-1}) + f_l(x_{l-1})]$5 epochs versus $x_l = c \cdot [h(x_{l-1}) + f_l(x_{l-1})]$6 for the baseline, with final test NLL $x_l = c \cdot [h(x_{l-1}) + f_l(x_{l-1})]$7 nats versus $x_l = c \cdot [h(x_{l-1}) + f_l(x_{l-1})]$8 nats; and permutation-invariant MNIST, where BN degrades at batch size $x_l = c \cdot [h(x_{l-1}) + f_l(x_{l-1})]$9 but LN remains robust.

Within the broader BlN landscape, LN is the canonical per-example normalizer: it preserves explicit normalization, but relocates the statistics from the sample dimension to the feature dimension.

3. Streaming statistics and unbiased gradients: Online Normalization

Online Normalization also removes batches, but unlike LN it still normalizes along the sample dimension by replacing batch estimates with sample-by-sample running estimates (Chiley et al., 2019). Let $a=(a_1,\dots,a_H)\in\mathbb{R}^H$0 denote the incoming activation sample at time $a=(a_1,\dots,a_H)\in\mathbb{R}^H$1. The forward pass maintains exponentially-weighted estimates:

$a=(a_1,\dots,a_H)\in\mathbb{R}^H$2

$a=(a_1,\dots,a_H)\in\mathbb{R}^H$3

$a=(a_1,\dots,a_H)\in\mathbb{R}^H$4

An optional layer-scale step computes

$a=(a_1,\dots,a_H)\in\mathbb{R}^H$5

The distinctive technical claim of Online Normalization is not only that it avoids batches, but that it resolves a theoretical limitation of BN by introducing an unbiased technique for computing the gradient of normalized activations. The paper derives the exact Jacobian of normalization as a double orthogonal projection,

$a=(a_1,\dots,a_H)\in\mathbb{R}^H$6

and approximates this geometry online through two error accumulators, $a=(a_1,\dots,a_H)\in\mathbb{R}^H$7 and $a=(a_1,\dots,a_H)\in\mathbb{R}^H$8, in the backward pass. As $a=(a_1,\dots,a_H)\in\mathbb{R}^H$9, these corrections recover the exact projections and make the gradient unbiased in the limit.

The stated integration model is a new primitive operator in C++/CUDA or a custom op, with persistent buffers for the backward accumulators. This keeps the interface close to BatchNorm while removing batch requirements. The theoretical guarantees given are asymptotic correctness of statistics under boundedness and variance assumptions, and asymptotic unbiasedness of backward gradients when the incoming gradient is bounded and $$\mu = \frac{1}{H}\sum_{i=1}^H a_i,\qquad \sigma^2 = \frac{1}{H}\sum_{i=1}^H (a_i-\mu)^2 + \epsilon,$$0 is close to $$\mu = \frac{1}{H}\sum_{i=1}^H a_i,\qquad \sigma^2 = \frac{1}{H}\sum_{i=1}^H (a_i-\mu)^2 + \epsilon,$$1.

Empirically, the paper reports ResNet-20 results of $$\mu = \frac{1}{H}\sum_{i=1}^H a_i,\qquad \sigma^2 = \frac{1}{H}\sum_{i=1}^H (a_i-\mu)^2 + \epsilon,$$2 versus BN’s $$\mu = \frac{1}{H}\sum_{i=1}^H a_i,\qquad \sigma^2 = \frac{1}{H}\sum_{i=1}^H (a_i-\mu)^2 + \epsilon,$$3 on CIFAR-10 and $$\mu = \frac{1}{H}\sum_{i=1}^H a_i,\qquad \sigma^2 = \frac{1}{H}\sum_{i=1}^H (a_i-\mu)^2 + \epsilon,$$4 versus BN’s $$\mu = \frac{1}{H}\sum_{i=1}^H a_i,\qquad \sigma^2 = \frac{1}{H}\sum_{i=1}^H (a_i-\mu)^2 + \epsilon,$$5 on CIFAR-100, both as median of $$\mu = \frac{1}{H}\sum_{i=1}^H a_i,\qquad \sigma^2 = \frac{1}{H}\sum_{i=1}^H (a_i-\mu)^2 + \epsilon,$$6 runs. For ImageNet with ResNet-50, Online Normalization achieved $$\mu = \frac{1}{H}\sum_{i=1}^H a_i,\qquad \sigma^2 = \frac{1}{H}\sum_{i=1}^H (a_i-\mu)^2 + \epsilon,$$7 versus BN’s $$\mu = \frac{1}{H}\sum_{i=1}^H a_i,\qquad \sigma^2 = \frac{1}{H}\sum_{i=1}^H (a_i-\mu)^2 + \epsilon,$$8. For volumetric segmentation with a U-Net, the reported memory usage is $$\mu = \frac{1}{H}\sum_{i=1}^H a_i,\qquad \sigma^2 = \frac{1}{H}\sum_{i=1}^H (a_i-\mu)^2 + \epsilon,$$9 for Online Normalization versus $\hat a_i = \frac{a_i-\mu}{\sqrt{\sigma^2}},$0 for BN at batch size $\hat a_i = \frac{a_i-\mu}{\sqrt{\sigma^2}},$1, and the method is reported to outperform BN and other norms on the synthetic-shapes Jaccard task. These results place Online Normalization among the most literal forms of batchless normalization: it preserves normalization over instances while removing the need to process instances simultaneously.

4. Learning population statistics by gradient descent

The paper “Batchless Normalization: How to Normalize Activations Across Instances with Minimal Memory Requirements” formulates BlN as a learned probabilistic surrogate for BN’s sample moments (Berger et al., 2022). Standard BN is summarized there as

$\hat a_i = \frac{a_i-\mu}{\sqrt{\sigma^2}},$2

followed by

$\hat a_i = \frac{a_i-\mu}{\sqrt{\sigma^2}},$3

The proposed alternative introduces learnable $\hat a_i = \frac{a_i-\mu}{\sqrt{\sigma^2}},$4 and $\hat a_i = \frac{a_i-\mu}{\sqrt{\sigma^2}},$5 per activation or per channel, normalizes by $\hat a_i = \frac{a_i-\mu}{\sqrt{\sigma^2}},$6, and adds a regularizer derived from the negative log likelihood of a Gaussian $\hat a_i = \frac{a_i-\mu}{\sqrt{\sigma^2}},$7.

For a pre-normalized activation $\hat a_i = \frac{a_i-\mu}{\sqrt{\sigma^2}},$8 with

$\hat a_i = \frac{a_i-\mu}{\sqrt{\sigma^2}},$9

the negative log-likelihood is

$y_i = \gamma_i \hat a_i + \beta_i.$0

With multiplier $y_i = \gamma_i \hat a_i + \beta_i.$1 and a gauge subtraction, the gauged per-activation BlN loss is

$y_i = \gamma_i \hat a_i + \beta_i.$2

The overall objective is

$y_i = \gamma_i \hat a_i + \beta_i.$3

A central implementation detail is the use of stop-gradient operators to separate the roles of normalization and parameter learning. The forward normalization uses

$y_i = \gamma_i \hat a_i + \beta_i.$4

while the BlN loss uses $y_i = \gamma_i \hat a_i + \beta_i.$5 and $y_i = \gamma_i \hat a_i + \beta_i.$6 for gradient flow. The stated consequence is that $y_i = \gamma_i \hat a_i + \beta_i.$7, so the normalization layer does not push preceding layers toward $y_i = \gamma_i \hat a_i + \beta_i.$8. The paper emphasizes that $y_i = \gamma_i \hat a_i + \beta_i.$9 and $H$0 are ordinary optimizer parameters, that no running-mean or running-variance bookkeeping is needed, and that examples can be streamed one at a time or across sharded devices without cross-device synchronization in forward or backward passes.

Initialization is described as collecting a small representative sample of $H$1–$H$2 examples, running the untrained network forward, and initializing $H$3 and $H$4 from sample mean and standard deviation. The hyperparameter $H$5 is stated as $H$6 in a typical setting, with optional annealing $H$7 in final epochs. Reported empirical behavior includes substantial advantages at very small batch sizes: on toy 3-spirals, BlN greatly outperforms BN or no normalization at batch sizes $H$8–$H$9 and roughly matches BN/BRN for batch $\gamma=(\gamma_1,\dots,\gamma_H)$0; on CIFAR-10 with a simple conv net, BN fails or diverges at tiny batches $\gamma=(\gamma_1,\dots,\gamma_H)$1–$\gamma=(\gamma_1,\dots,\gamma_H)$2, whereas BlN still achieves $\gamma=(\gamma_1,\dots,\gamma_H)$3 accuracy, and for batch $\gamma=(\gamma_1,\dots,\gamma_H)$4 it reaches within $\gamma=(\gamma_1,\dots,\gamma_H)$5–$\gamma=(\gamma_1,\dots,\gamma_H)$6 of BN’s best accuracy.

This formulation is notable because it treats normalization statistics as gradient-trained model parameters rather than as sample estimates or auxiliary running buffers.

5. Normalizer-free residual and binary architectures

A different BlN line removes explicit normalization layers entirely and instead designs the network so that activation and gradient scales are controlled structurally. In “A Robust Initialization of Residual Blocks for Effective ResNet Training without Batch Normalization,” Civitelli et al. modify the residual update from

$\gamma=(\gamma_1,\dots,\gamma_H)$7

to

$\gamma=(\gamma_1,\dots,\gamma_H)$8

where $\gamma=(\gamma_1,\dots,\gamma_H)$9 is the shortcut and $\beta=(\beta_1,\dots,\beta_H)$0 is a constant chosen to preserve forward and backward variance (Civitelli et al., 2021). Under the paper’s assumptions, when the shortcut is identity and both shortcut and residual branch have unit variance, one obtains $\beta=(\beta_1,\dots,\beta_H)$1. Three concrete setups are stated: IdShort with $\beta=(\beta_1,\dots,\beta_H)$2 and $\beta=(\beta_1,\dots,\beta_H)$3; LearnScalar with $\beta=(\beta_1,\dots,\beta_H)$4, $\beta=(\beta_1,\dots,\beta_H)$5 learned and initialized at $\beta=(\beta_1,\dots,\beta_H)$6, and $\beta=(\beta_1,\dots,\beta_H)$7; and ConvShort with $\beta=(\beta_1,\dots,\beta_H)$8, a $\beta=(\beta_1,\dots,\beta_H)$9 convolution initialized by He’s backward rule, again with $H$0.

The initialization prescription uses He initialization inside each residual branch, with

$H$1

for forward and

$H$2

for backward, implemented by sampling $H$3 and then empirically standardizing each weight-channel to zero mean and variance $H$4. The paper reports that pre-activation ResNets without BN and with classical He init drift, while IdShort, LearnScalar, or ConvShort with the proposed initialization keep both forward and backward variances approximately $H$5 at every depth. Reported results include ResNet-50@BlN-ConvShort on ImageNet with Top-1 $H$6 versus a BN baseline of $H$7, and ResNet-50@BlN-ConvShort on CIFAR-100 with test $H$8 versus BN $H$9.

For binary neural networks, “BNN - BN = ?” extends BN-free training with a three-part recipe: Adaptive Gradient Clipping (AGC), Scaled Weight Standardization (SWS), and a Specialized Bottleneck Block (Chen et al., 2021). AGC clips each parameter’s gradient relative to its own norm; SWS standardizes each filter as

$x_l = c \cdot [h(x_{l-1}) + f_l(x_{l-1})]$00

with $x_l = c \cdot [h(x_{l-1}) + f_l(x_{l-1})]$01 for ReLU-like activations, while binary networks with RPReLU and RSign may keep $x_l = c \cdot [h(x_{l-1}) + f_l(x_{l-1})]$02; and the block structure uses deterministic scaling by $x_l = c \cdot [h(x_{l-1}) + f_l(x_{l-1})]$03, $x_l = c \cdot [h(x_{l-1}) + f_l(x_{l-1})]$04, $x_l = c \cdot [h(x_{l-1}) + f_l(x_{l-1})]$05, and $x_l = c \cdot [h(x_{l-1}) + f_l(x_{l-1})]$06 around WS-Conv operations. The reported outcome is that BNs can be completely removed from BNN training and inference regimes. On ReActNet-A trained on ImageNet, the BN-based top-1 is $x_l = c \cdot [h(x_{l-1}) + f_l(x_{l-1})]$07, naïve removal of BN gives $x_l = c \cdot [h(x_{l-1}) + f_l(x_{l-1})]$08, and the BlN pipeline achieves $x_l = c \cdot [h(x_{l-1}) + f_l(x_{l-1})]$09. On ReActNet-18, the corresponding CIFAR-10 and CIFAR-100 results are $x_l = c \cdot [h(x_{l-1}) + f_l(x_{l-1})]$10 and $x_l = c \cdot [h(x_{l-1}) + f_l(x_{l-1})]$11, with marginal drops of $x_l = c \cdot [h(x_{l-1}) + f_l(x_{l-1})]$12 and $x_l = c \cdot [h(x_{l-1}) + f_l(x_{l-1})]$13 from the BN baselines.

These two lines share a batch-free objective but differ fundamentally from explicit activation normalization. Their central claim is that stable signal propagation can be engineered through initialization, residual scaling, weight-only standardization, and gradient control.

6. Sample-centric noise surrogates and comparative interpretation

NoMorelization provides another normalizer-free interpretation of BlN by arguing, from a sample’s perspective, that normalized training can be decomposed into deterministic residual scaling and a regularizing stochastic perturbation (Liu et al., 2022). Its forward definition is

$x_l = c \cdot [h(x_{l-1}) + f_l(x_{l-1})]$14

with training-mode form

$x_l = c \cdot [h(x_{l-1}) + f_l(x_{l-1})]$15

and evaluation-mode form

$x_l = c \cdot [h(x_{l-1}) + f_l(x_{l-1})]$16

The parameters $x_l = c \cdot [h(x_{l-1}) + f_l(x_{l-1})]$17 are trainable scalars initialized to zero so that the block begins as an identity, and $x_l = c \cdot [h(x_{l-1}) + f_l(x_{l-1})]$18 is sampled independently for each output activation. The paper presents this as a substitute for BN, LN, IN, or other normalizers, retaining only a fixed, zero-mean Gaussian perturbation while avoiding batch-dependent bias.

Its backward behavior is described as that of a learned affine transformation with no gradient through the noise:

$x_l = c \cdot [h(x_{l-1}) + f_l(x_{l-1})]$19

The computational claim is that NoMorelization adds only two scalars per layer plus a random-tensor draw and an element-wise addition, whereas BN, LN, and IN require reduction operations and temporary statistics. The paper states that NoMorelization shows the best speed-accuracy trade-off compared with existing mainstream normalizers and state-of-the-art normalizer-free methods.

Reported examples include CIFAR-10 with ResNet-56, where the BN baseline is $x_l = c \cdot [h(x_{l-1}) + f_l(x_{l-1})]$20 top-1 and NoMorelization gives $x_l = c \cdot [h(x_{l-1}) + f_l(x_{l-1})]$21 at $x_l = c \cdot [h(x_{l-1}) + f_l(x_{l-1})]$22 faster, while NoMorelization plus the NF regularizer gives $x_l = c \cdot [h(x_{l-1}) + f_l(x_{l-1})]$23 at $x_l = c \cdot [h(x_{l-1}) + f_l(x_{l-1})]$24 faster. On ImageNet with ResNet-50, standard BN achieves $x_l = c \cdot [h(x_{l-1}) + f_l(x_{l-1})]$25, NFNet reaches $x_l = c \cdot [h(x_{l-1}) + f_l(x_{l-1})]$26, NoMorelization alone matches $x_l = c \cdot [h(x_{l-1}) + f_l(x_{l-1})]$27 at $x_l = c \cdot [h(x_{l-1}) + f_l(x_{l-1})]$28 cost, and NoMorelization plus the NF regularizer gives $x_l = c \cdot [h(x_{l-1}) + f_l(x_{l-1})]$29. In CycleGAN, replacing Instance Normalization with NoMorelization improves both FID and IS on Summer$x_l = c \cdot [h(x_{l-1}) + f_l(x_{l-1})]$30Winter and Horse$x_l = c \cdot [h(x_{l-1}) + f_l(x_{l-1})]$31Zebra. The ablation on noise amplitude $x_l = c \cdot [h(x_{l-1}) + f_l(x_{l-1})]$32 further states that CNNs peak at $x_l = c \cdot [h(x_{l-1}) + f_l(x_{l-1})]$33 in the tested ResNet setting, while Swin requires smaller noise, with $x_l = c \cdot [h(x_{l-1}) + f_l(x_{l-1})]$34 optimal.

Taken together, these results clarify a recurring misconception: batchless normalization is not always a normalization operator in the strict statistical sense. In some papers it remains an explicit normalization rule with non-batch statistics; in others it becomes a normalization-free surrogate whose goals are variance control, memory reduction, batch independence, or regularization. The literature therefore uses the label “BlN” for a broader design space unified by the removal of minibatch coupling rather than by a single mathematical construction.