Online Normalization in Neural Networks

• Online normalization is a method that replaces traditional batch-dependent statistics with streaming or per-sample estimators, ensuring stable training across various architectures.
• It introduces unbiased gradient computation and dynamic moment estimation, which improve performance in online, recurrent, and continual learning settings.
• Variants such as L1 normalization, online whitening, and gradient preconditioning address the limitations of BatchNorm in small-batch and non-stationary environments.

Searching arXiv for the specified papers and closely related online normalization work. {"query":"Online normalization Batch Normalization streaming normalization continual normalization MixNorm arXiv", "max_results": 10} {"query":"(Chiley et al., 2019)", "max_results": 5} {"query":"(Liao et al., 2016)", "max_results": 5} {"query":"(Hu et al., 2021)", "max_results": 5} Online normalization denotes a class of normalization procedures that replace or relax the dependence of Batch Normalization on large, stationary mini-batches by using streaming, per-sample, or otherwise online estimators of normalization statistics. In the narrow sense established by "Online Normalization for Training Neural Networks" (Chiley et al., 2019), it is a normalization method that, like Batch Normalization, normalizes the sample dimension but does not use batches, and that introduces an unbiased technique for computing the gradient of normalized activations. In the broader literature, the term also covers closely related streaming and adaptive schemes that maintain running moments across samples or timesteps, normalize under domain shift at test time, or reformulate normalization as whitening or gradient preconditioning rather than a conventional BatchNorm layer (Liao et al., 2016, Hu et al., 2021, Zhang et al., 2021, Pham et al., 2022, Lange et al., 2021).

1. Conceptual scope and motivation

The central motivation for online normalization is the breakdown of conventional Batch Normalization in regimes where its assumptions do not hold. BatchNorm computes per-channel mean and variance over a mini-batch and assumes access to a sufficiently large, stationary batch of samples. In pure online learning, where batch size $=1$, or in very small mini-batches, estimates of mean and variance become too noisy, leading to training instabilities or outright failure. In recurrent networks, one simple remedy—time-specific BatchNorm—stores separate normalization statistics for each timestep, but this requires a large number of stored statistics, does not generalize to unseen sequence lengths, and is biologically implausible (Liao et al., 2016).

"Online Normalization for Training Neural Networks" states the problem more directly for deep learning: like BatchNorm, it normalizes the sample dimension, but it does not use batches, and it is intended for recurrent networks, fully connected networks, and networks with activation memory requirements prohibitive for batching (Chiley et al., 2019). "Batch Normalization Preconditioning for Neural Network Training" further emphasizes that BatchNorm is not suitable for use with very small mini-batch sizes or online learning, and proposes a method that is not constrained on the mini-batch size and works in the online learning setting (Lange et al., 2021).

In this broader sense, online normalization is not a single algorithm. It is a family of mechanisms whose common feature is that normalization statistics are updated in an online fashion, or that the effect of normalization is achieved without requiring batch-wide forward statistics at training or inference time. This suggests that the main axis of variation is not whether normalization is present, but where the estimator is anchored: in a stream of past activations, within each sample, in a running covariance, or directly in the parameter-gradient geometry.

2. Streaming activation normalization and unbiased backpropagation

The canonical formulation appears in "Online Normalization for Training Neural Networks" (Chiley et al., 2019). For activations $x_t \in \mathbb{R}^f$ at time $t$, it maintains per-feature running mean and variance with exponential decay factor $\alpha_f$: $\mu_t = \alpha_f \mu_{t-1} + (1-\alpha_f)x_t,$

$$\sigma_t^2 = \alpha_f \sigma_{t-1}^2 + \alpha_f(1-\alpha_f)(x_t-\mu_{t-1})^2.$$

The normalized output is then

$$\hat y_t = \frac{x_t-\mu_{t-1}}{\sqrt{\sigma_{t-1}^2+\epsilon}},$$

optionally followed by learned per-feature scale $\gamma$ and bias $\beta$ (Chiley et al., 2019).

A defining feature of this formulation is its treatment of the backward pass. For the ideal normalization operator $y=(x-\mu[x])/\sigma[x]$, the paper gives the exact Jacobian

$x_t \in \mathbb{R}^f$0

with the resulting exact backpropagation rule

$x_t \in \mathbb{R}^f$1

The paper characterizes this as the unbiased gradient of the normalization operator and introduces two accumulators, $x_t \in \mathbb{R}^f$2 and $x_t \in \mathbb{R}^f$3, to enforce the orthogonality constraints associated with the normalization geometry during streaming backpropagation (Chiley et al., 2019).

A related but more generalized framework appears in "Streaming Normalization: Towards Simpler and More Biologically-plausible Normalizations for Online and Recurrent Learning" (Liao et al., 2016). It defines, for each neuron $x_t \in \mathbb{R}^f$4, a normalization reference $x_t \in \mathbb{R}^f$5, normalization statistics $x_t \in \mathbb{R}^f$6, and a normalization operation $x_t \in \mathbb{R}^f$7. Within that framework, Sample Normalization, General Batch Normalization, and Streaming Normalization become special cases distinguished by the choice of $x_t \in \mathbb{R}^f$8 and the manner in which $x_t \in \mathbb{R}^f$9 is estimated. For Streaming Normalization, $t$0 is a running estimate over all past samples and timesteps. Forward normalization and backward gradient stabilization are both implemented through long-term and short-term running-stat storage, so the online property extends to both statistics and their gradients (Liao et al., 2016).

Taken together, these formulations separate online normalization from merely using small batches. The issue is not only the forward estimator of mean and variance, but also whether the gradient of the normalization operator remains well-defined, stable, and unbiased when the batch dimension is absent or non-stationary.

3. Variants: moments, whitening, and preconditioning

A major variant class replaces conventional second-moment scaling with other moment estimators. "Streaming Normalization" proposes $t$1 Normalization: $t$2 The special case $t$3 recovers standard deviation, while $t$4 uses mean absolute deviation. For streaming $t$5 normalization, if $t$6, the running statistic is

$t$7

and $t$8 is normalized by $t$9. The paper reports that $\alpha_f$0 normalization is well-performing, simple to implement, fast to compute, more biologically-plausible, and ideal for GPU or hardware implementations (Liao et al., 2016).

A second variant replaces diagonal normalization with online whitening. "Stochastic Whitening Batch Normalization" maintains a running full covariance $\alpha_f$1 and whitening matrix $\alpha_f$2 across training steps: $\alpha_f$3 A Newton–Schulz refinement updates the whitening transform via

$\alpha_f$4

The method thus moves from per-channel normalization to an online estimate of $\alpha_f$5, while retaining BatchNorm-style affine parameters $\alpha_f$6 (Zhang et al., 2021).

A third variant moves the effect of BatchNorm from activations to gradients. "Batch Normalization Preconditioning for Neural Network Training" constructs a preconditioner $\alpha_f$7, where

$\alpha_f$8

and applies normalization by conditioning the parameter gradients directly during training. A key update is

$\alpha_f$9

The paper states that this improves the Hessian matrix of the loss function and hence convergence during training, while avoiding a dependence on batch size in the forward network (Lange et al., 2021).

An additional, less conventional line appears in "Inherent Weight Normalization in Stochastic Neural Networks" (Detorakis et al., 2019). There, always-on multiplicative stochasticity in Neural Sampling Machines yields an activation probability

$\mu_t = \alpha_f \mu_{t-1} + (1-\alpha_f)x_t,$0

which mirrors Weight Normalization. The paper explicitly describes this as a self-normalizing property that fulfills many of the features of Batch Normalization in an online fashion (Detorakis et al., 2019).

These variants show that online normalization is not restricted to running estimates of mean and variance. It also includes alternative moment choices, online full-covariance whitening, direct gradient preconditioning, and stochastic mechanisms whose normalization effect is inherent to the forward dynamics.

4. Online normalization beyond stationary training

A distinct branch of the literature uses online normalization for non-stationary evaluation or continual data streams rather than only for ordinary training.

"MixNorm: Test-Time Adaptation Through Online Normalization Estimation" initializes global moments $\mu_t = \alpha_f \mu_{t-1} + (1-\alpha_f)x_t,$1 from the original training BatchNorm statistics, updates them online using momentum $\mu_t = \alpha_f \mu_{t-1} + (1-\alpha_f)x_t,$2,

$\mu_t = \alpha_f \mu_{t-1} + (1-\alpha_f)x_t,$3

computes local moments from spatial augmentations, and forms mixed moments

$\mu_t = \alpha_f \mu_{t-1} + (1-\alpha_f)x_t,$4

The method then normalizes features with these mixed moments and applies the usual affine parameters. In batch mode it uses the heuristic

$\mu_t = \alpha_f \mu_{t-1} + (1-\alpha_f)x_t,$5

so that the update interpolates between large-batch BatchNorm behavior and single-sample adaptation. The paper emphasizes that these updates happen on the fly and no backward-gradient updates are required (Hu et al., 2021).

"Continual Normalization: Rethinking Batch Normalization for Online Continual Learning" addresses a different failure mode: in online continual learning, BatchNorm’s running moments become biased toward the most recent task, and at inference time this cross-task normalization distorts older-task features. Continual Normalization first applies Group Normalization without affine parameters,

$\mu_t = \alpha_f \mu_{t-1} + (1-\alpha_f)x_t,$6

then applies standard BatchNorm with affine parameters,

$\mu_t = \alpha_f \mu_{t-1} + (1-\alpha_f)x_t,$7

so that BN’s moments are computed on spatially normalized features. The paper presents this as a direct replacement for BatchNorm that retains cross-mini-batch smoothing while injecting per-sample adaptivity (Pham et al., 2022).

A broader data-engineering perspective appears in "Adaptive Normalization in Streaming Data" (Gupta et al., 2019). That work treats an unbounded real-valued data stream $\mu_t = \alpha_f \mu_{t-1} + (1-\alpha_f)x_t,$8 in disjoint windows of fixed size $\mu_t = \alpha_f \mu_{t-1} + (1-\alpha_f)x_t,$9, computes window mean, variance, minimum, and maximum, and performs Min–Max normalization with reference values $$\sigma_t^2 = \alpha_f \sigma_{t-1}^2 + \alpha_f(1-\alpha_f)(x_t-\mu_{t-1})^2.$$0: $$\sigma_t^2 = \alpha_f \sigma_{t-1}^2 + \alpha_f(1-\alpha_f)(x_t-\mu_{t-1})^2.$$1 A user-specified threshold $$\sigma_t^2 = \alpha_f \sigma_{t-1}^2 + \alpha_f(1-\alpha_f)(x_t-\mu_{t-1})^2.$$2 is applied to the percentage change in window means,

$$\sigma_t^2 = \alpha_f \sigma_{t-1}^2 + \alpha_f(1-\alpha_f)(x_t-\mu_{t-1})^2.$$3

to decide whether the normalization range should be reset to the current window or conservatively expanded. Implemented on Apache Storm through a Spout, WindowBolt, and NormalizeBolt, the method is explicitly positioned as adaptive normalization for Big data stream rather than for a specific downstream classifier (Gupta et al., 2019).

This broader literature makes clear that online normalization is also a mechanism for distribution tracking. In some settings the goal is stable optimization under batch-size $$\sigma_t^2 = \alpha_f \sigma_{t-1}^2 + \alpha_f(1-\alpha_f)(x_t-\mu_{t-1})^2.$$4; in others it is robust inference under domain shift, mitigation of catastrophic forgetting, or scalable preprocessing of unbounded streams.

5. Empirical results across architectures and tasks

The empirical record in the cited works is heterogeneous because the compared methods solve different normalization problems, but several patterns recur.

Setting	Reported result	Interpretation
ResNet-20 / CIFAR-10, batch128	Online Norm: loss $$\sigma_t^2 = \alpha_f \sigma_{t-1}^2 + \alpha_f(1-\alpha_f)(x_t-\mu_{t-1})^2.$$5, acc $$\sigma_t^2 = \alpha_f \sigma_{t-1}^2 + \alpha_f(1-\alpha_f)(x_t-\mu_{t-1})^2.$$6; BatchNorm: $$\sigma_t^2 = \alpha_f \sigma_{t-1}^2 + \alpha_f(1-\alpha_f)(x_t-\mu_{t-1})^2.$$7, $$\sigma_t^2 = \alpha_f \sigma_{t-1}^2 + \alpha_f(1-\alpha_f)(x_t-\mu_{t-1})^2.$$8; GroupNorm: $$\sigma_t^2 = \alpha_f \sigma_{t-1}^2 + \alpha_f(1-\alpha_f)(x_t-\mu_{t-1})^2.$$9, $$\hat y_t = \frac{x_t-\mu_{t-1}}{\sqrt{\sigma_{t-1}^2+\epsilon}},$$0; LayerNorm: $$\hat y_t = \frac{x_t-\mu_{t-1}}{\sqrt{\sigma_{t-1}^2+\epsilon}},$$1, $$\hat y_t = \frac{x_t-\mu_{t-1}}{\sqrt{\sigma_{t-1}^2+\epsilon}},$$2	Online Normalization matches BatchNorm and exceeds non-batch alternatives (Chiley et al., 2019)
CIFAR-10 feedforward and convolutional models	Streaming Normalization converges as fast as BN in large batches, but remains stable with batch-size 1 or 2, where BN fails and LayerNorm degrades	Streaming statistics preserve stability in tiny-batch regimes (Liao et al., 2016)
CIFAR-10C, $$\hat y_t = \frac{x_t-\mu_{t-1}}{\sqrt{\sigma_{t-1}^2+\epsilon}},$$3	TENT $$\hat y_t = \frac{x_t-\mu_{t-1}}{\sqrt{\sigma_{t-1}^2+\epsilon}},$$4 $$\hat y_t = \frac{x_t-\mu_{t-1}}{\sqrt{\sigma_{t-1}^2+\epsilon}},$$5 MixNorm $$\hat y_t = \frac{x_t-\mu_{t-1}}{\sqrt{\sigma_{t-1}^2+\epsilon}},$$6	Test-time online moment estimation is effective in the single-sample case (Hu et al., 2021)
ImageNet-C, $$\hat y_t = \frac{x_t-\mu_{t-1}}{\sqrt{\sigma_{t-1}^2+\epsilon}},$$7	TENT $$\hat y_t = \frac{x_t-\mu_{t-1}}{\sqrt{\sigma_{t-1}^2+\epsilon}},$$8 $$\hat y_t = \frac{x_t-\mu_{t-1}}{\sqrt{\sigma_{t-1}^2+\epsilon}},$$9 MixNorm $\gamma$0	The same pattern holds on a larger benchmark (Hu et al., 2021)
Split CIFAR-100 / Split Mini-ImageNet, ER, batch size $\gamma$1, one epoch	BN: ACC $\gamma$2, FM $\gamma$3, LA $\gamma$4; CN: ACC $\gamma$5, FM $\gamma$6, LA $\gamma$7	Continual Normalization reduces forgetting while improving average accuracy (Pham et al., 2022)
CIFAR-10 many-shot, VGG-9 / ResNet-32	BN $\gamma$8 / $\gamma$9; IterNorm $\beta$0 / $\beta$1; SWBN $\beta$2 / $\beta$3	Online whitening improves accuracy and epochs-to-90% train acc. (Zhang et al., 2021)
Synthetic stream, 160 000 points, window size 5	Proposed adaptive method RMSE $\beta$4 versus method 2 RMSE $\beta$5	Adaptive streaming normalization improves over the first-window static baseline by $\beta$6 (Gupta et al., 2019)

For the narrow neural-network sense of online normalization, the most direct comparisons come from (Chiley et al., 2019). On ResNet-20 / CIFAR-100, Online Normalization reports loss $\beta$7 and accuracy $\beta$8, BatchNorm reports $\beta$9 and $y=(x-\mu[x])/\sigma[x]$0, and GroupNorm reports $y=(x-\mu[x])/\sigma[x]$1 and $y=(x-\mu[x])/\sigma[x]$2. On ResNet-50 / ImageNet with batch32, Online Normalization reports validation loss $y=(x-\mu[x])/\sigma[x]$3 and $y=(x-\mu[x])/\sigma[x]$4 top-1, while BatchNorm reports $y=(x-\mu[x])/\sigma[x]$5 and $y=(x-\mu[x])/\sigma[x]$6 top-1 (Chiley et al., 2019). The same paper also reports applications to image segmentation and language modeling, including a 3D U-Net setting where BatchNorm is infeasible because Table 1 shows $y=(x-\mu[x])/\sigma[x]$7 GB for batch32 versus $y=(x-\mu[x])/\sigma[x]$8 GB for Online Normalization (Chiley et al., 2019).

For streaming activation statistics in recurrent settings, (Liao et al., 2016) reports that a dense conv-RNN unrolled for $y=(x-\mu[x])/\sigma[x]$9 timesteps requires $x_t \in \mathbb{R}^f$00 separate statistics under time-specific BatchNorm, whereas Streaming Normalization uses one shared set of statistics. On character-level language modeling with Shakespeare text, using a vanilla RNN and GRU with one hidden layer of $x_t \in \mathbb{R}^f$01 units, truncated BPTT of $x_t \in \mathbb{R}^f$02 steps, and decoupled accumulation and update with $x_t \in \mathbb{R}^f$03 batches per weight update, Streaming Normalization yields faster training convergence and slightly lower validation loss than LayerNorm or time-specific BatchNorm, and GRU+SN achieves best validation perplexity among the three methods (Liao et al., 2016).

For test-time adaptation, (Hu et al., 2021) reports that MixNorm is essentially flat in $x_t \in \mathbb{R}^f$04 from $x_t \in \mathbb{R}^f$05, whereas TENT’s error drops only when $x_t \in \mathbb{R}^f$06 and the test distribution is homogeneous. Under mixed corruptions, TENT degrades severely even at $x_t \in \mathbb{R}^f$07, while MixNormBN stays $x_t \in \mathbb{R}^f$08–$x_t \in \mathbb{R}^f$09 on CIFAR-10C. In ablations on CIFAR-10C mixed corruptions, the reported errors are $x_t \in \mathbb{R}^f$10 for instance-only, $x_t \in \mathbb{R}^f$11 for local augmentation only, $x_t \in \mathbb{R}^f$12 for fixed global only, $x_t \in \mathbb{R}^f$13 for moving global only, and $x_t \in \mathbb{R}^f$14 for the mixed setting with $x_t \in \mathbb{R}^f$15 (Hu et al., 2021).

For online continual learning, (Pham et al., 2022) gives a concrete diagnostic of BatchNorm’s bias. In a toy 5-task permuted MNIST experiment, without replay, the $x_t \in \mathbb{R}^f$16-difference in first-layer mean is $x_t \in \mathbb{R}^f$17 and in variance $x_t \in \mathbb{R}^f$18, with forgetting $x_t \in \mathbb{R}^f$19. Even with a tiny episodic memory, $x_t \in \mathbb{R}^f$20, $x_t \in \mathbb{R}^f$21, and $x_t \in \mathbb{R}^f$22. The paper states that this gap in moments directly maps to worse accuracy on older tasks, approximately $x_t \in \mathbb{R}^f$23–$x_t \in \mathbb{R}^f$24 lower (Pham et al., 2022).

The whitening and preconditioning lines report gains of a somewhat different kind. SWBN reports that BN often requires $x_t \in \mathbb{R}^f$25–$x_t \in \mathbb{R}^f$26 more epochs to reach a given training loss, and that SWBN converges as fast as IterNorm in early epochs while outperforming BN by $x_t \in \mathbb{R}^f$27–$x_t \in \mathbb{R}^f$28 fewer epochs in practice. It also reports that SWBN is approximately $x_t \in \mathbb{R}^f$29–$x_t \in \mathbb{R}^f$30 slower than BN per layer, but approximately $x_t \in \mathbb{R}^f$31 faster than IterNorm for the same whitening quality (Zhang et al., 2021). BNP, in turn, reports that on a 5-layer CNN for CIFAR-10 with $x_t \in \mathbb{R}^f$32, BN diverges, BatchRenorm helps slightly, and BNP converges fastest and best; with $x_t \in \mathbb{R}^f$33, only BNP and LayerNorm succeed, and BNP achieves higher final accuracy (Lange et al., 2021).

6. Implementation patterns, computational properties, and recurring limitations

Across the literature, online normalization methods tend to require small persistent state and simple update rules. Online Normalization maintains $x_t \in \mathbb{R}^f$34 state per feature—$x_t \in \mathbb{R}^f$35, $x_t \in \mathbb{R}^f$36, and two accumulators $x_t \in \mathbb{R}^f$37, $x_t \in \mathbb{R}^f$38—and can be implemented in automatic differentiation by adding statistical normalization as a primitive (Chiley et al., 2019). Streaming Normalization adds long-term and short-term running-stat storage for forward statistics and a second running-stat storage for stabilized gradients; in convolutional networks it uses per-channel streaming statistics over all spatial locations of one feature map, and in recurrent networks it uses the same streaming statistics across timesteps (Liao et al., 2016).

Several papers provide explicit hyperparameter recommendations. For Streaming Normalization, the reported defaults are $x_t \in \mathbb{R}^f$39 for long-term statistics and $x_t \in \mathbb{R}^f$40 for short-term statistics with $x_t \in \mathbb{R}^f$41, gradient decay $x_t \in \mathbb{R}^f$42, $x_t \in \mathbb{R}^f$43, $x_t \in \mathbb{R}^f$44, and $x_t \in \mathbb{R}^f$45 in the range $x_t \in \mathbb{R}^f$46 to $x_t \in \mathbb{R}^f$47. The paper also recommends decoupled accumulation and update with $x_t \in \mathbb{R}^f$48 to stabilize training, especially in recurrent nets (Liao et al., 2016). For Online Normalization, $x_t \in \mathbb{R}^f$49 is recommended to be very close to $x_t \in \mathbb{R}^f$50, such as $x_t \in \mathbb{R}^f$51–$x_t \in \mathbb{R}^f$52, while $x_t \in \mathbb{R}^f$53 can be slightly lower, such as $x_t \in \mathbb{R}^f$54–$x_t \in \mathbb{R}^f$55, with $x_t \in \mathbb{R}^f$56 in the range $x_t \in \mathbb{R}^f$57–$x_t \in \mathbb{R}^f$58 (Chiley et al., 2019).

Computationally, the methods differ in where they pay their cost. MixNorm requires exactly the same per-layer storage as BatchNorm—one running mean and variance per channel, plus the affine parameters—but computes one extra forward pass for an augmented view when $x_t \in \mathbb{R}^f$59, doubling the cost in the normalization layers only; all operations remain $x_t \in \mathbb{R}^f$60 per layer and no backpropagation is required (Hu et al., 2021). Continual Normalization reports memory footprint identical to BatchNorm and a $x_t \in \mathbb{R}^f$61 wall-clock overhead over BN on Split CIFAR-100 with ER, whereas SN is reported as $x_t \in \mathbb{R}^f$62 (Pham et al., 2022). SWBN requires $x_t \in \mathbb{R}^f$63 storage for $x_t \in \mathbb{R}^f$64 and $x_t \in \mathbb{R}^f$65, so its conditioning benefit comes with higher memory than diagonal normalizers (Zhang et al., 2021). The Apache Storm method for adaptive normalization in streaming data has $x_t \in \mathbb{R}^f$66 time per window, $x_t \in \mathbb{R}^f$67 memory for the window buffer plus $x_t \in \mathbb{R}^f$68 for the reference statistics, and an amortized per-tuple cost of $x_t \in \mathbb{R}^f$69 (Gupta et al., 2019).

Several recurring misconceptions are addressed by the cited work. Online normalization is not equivalent to LayerNorm or GroupNorm: those methods normalize across features within a sample, whereas Online Normalization is explicitly designed to normalize the sample dimension without using batches (Chiley et al., 2019). Nor is online normalization necessarily gradient-free: MixNorm is gradient-free at test time, but Online Normalization and Streaming Normalization both define explicit backward mechanisms (Hu et al., 2021, Chiley et al., 2019). Conversely, not every method called online is a training-time replacement for BatchNorm; some, such as MixNorm and the Apache Storm method, are adaptation or preprocessing procedures for non-stationary data streams (Hu et al., 2021, Gupta et al., 2019).

The main open issues are also recurrent. The adaptive streaming-data paper notes that $x_t \in \mathbb{R}^f$70 is static and that a dynamic or self-tuning threshold could further balance precision and latency; it also assumes no extreme outliers and only explores Min–Max normalization (Gupta et al., 2019). MixNorm lists theoretical analysis of convergence and bias/variance trade-off of mixed moments, dynamic adaptation of the mixing weight $x_t \in \mathbb{R}^f$71, and application to video, point clouds, adversarial shift, and temporal drift as future directions (Hu et al., 2021). Online Normalization notes that integrating the method with adaptive optimizers may require careful handling of their own moment estimates (Chiley et al., 2019). Streaming Normalization frames biological plausibility as a design objective and argues that neuron-wise SN, $x_t \in \mathbb{R}^f$72 moments, and shared recurrent statistics are closer to local biological normalization, which suggests a research program that is partly algorithmic and partly neuroscientific (Liao et al., 2016).

In aggregate, online normalization refers to a technically diverse but coherent research area: the replacement of batch-tied normalization by stream-aware estimators, exact or stabilized online gradients, online whitening, or gradient preconditioning. The literature consistently treats the key benefits as applicability to pure online learning, tiny mini-batches, recurrent and mixed architectures, non-stationary data, and test-time or continual adaptation, while differing on the precise estimator, the degree of statistical coupling across samples, and the computational trade-offs.