Neuron-Wise Normalization in Neural Networks

• Neuron-wise normalization is a method that computes per-neuron statistics to stabilize activation dynamics and overcome batch-dependence.
• It includes variants like Layer Norm, Normality Norm, Online Norm, and analytic normalization that tailor mean, variance, and scaling for each neuron.
• This approach improves train-test consistency and convergence speed across diverse architectures including feedforward, recurrent, and spiking neural networks.

Neuron-wise normalization refers to a class of normalization techniques in artificial and spiking neural networks that operate at the per-neuron (feature channel) level, as opposed to per-batch or per-layer statistics. These methods are foundational for ensuring stable activation dynamics, accelerating convergence, and preserving representational expressiveness under diverse network topologies, including deep, recurrent, and spiking architectures. Neuron-wise approaches encompass a spectrum of forward-pass transformations and optimizer-level schemes, and have been driven by the need to overcome the batch-size- and architecture-dependence of classical normalization strategies.

1. Mathematical Foundations and Core Principles

Neuron-wise normalization computes statistics and/or applies transformations individually for each neuron (or, in the case of convolutional layers, each feature map or channel), using only data specific to that neuron and its current context.

A seminal example is Layer Normalization (Ba et al., 2016), which, for a layer with $H$ neurons, computes pre-activations $\mathbf{z} = [z_1, \ldots, z_H]^\top$:

• Mean: $\mu = \frac{1}{H}\sum_{i=1}^H z_i$
• Variance: $\sigma^2 = \frac{1}{H}\sum_{i=1}^H (z_i - \mu)^2$

Each pre-activation is normalized: $$\hat z_i = \frac{z_i - \mu}{\sqrt{\sigma^2 + \varepsilon}}$$ and then scaled/shifted via learnable parameters per neuron: $y_i = \gamma_i \hat z_i + \beta_i$ where $\gamma_i$ and $\beta_i$ are per-neuron parameters.

The essential property is that all necessary statistics for normalization are drawn from the set of activations for individual neurons (sometimes within one example, sometimes over a stream), ensuring per-sample invariance, independence from minibatch size, and applicability to non-i.i.d. or sequence modeling workloads.

2. Methodological Variants and Algorithmic Schemes

Several neuron-wise normalization methods have been introduced, spanning both activation normalization and optimizer-level normalization:

• Layer Normalization (LN): Computes mean and variance over all neurons in a layer for a single training case. Identical computation in both training and testing, and directly extensible to RNNs by applying LN at every timestep (Ba et al., 2016).
• Normality Normalization: Incorporates not only mean/variance standardization but also a learned power transform (Yeo–Johnson) to Gaussianize each neuron's output, followed by additive Gaussian noise injection at training time. The power parameter $\lambda$ is optimized neuron-wise to maximize normality, and the affine scale/bias remain per neuron (Eftekhari et al., 1 May 2025).
• Online Normalization (ON): Maintains running, per-neuron estimates of mean and variance using exponential moving averages, with a forward-pass normalization

$$\hat h^l_t = \frac{h^l_t - \mu^l_t}{\sqrt{(\sigma^l_t)^2 + \varepsilon}}$$

updated per sample, and a two-stage unbiased backward pass that projects out the mean and variance direction for each neuron separately (Chiley et al., 2019).

• Analytic Normalization via Moment Propagation: Resolves each neuron's statistics analytically across the network by computing the expected mean and variance under the network's weights for each neuron, then standardizing with per-neuron learned scale and bias (Shekhovtsov et al., 2018).
• Neuron-wise Weight and Firing Rate Normalization in SNNs: For spiking neural networks, some approaches rescale incoming weights to each neuron to maintain a fixed $\mathbf{z} = [z_1, \ldots, z_H]^\top$0 norm, optionally per neuron type (e.g. excitatory/inhibitory), ensuring homeostatic firing and preventing drift (Kozdon et al., 2019).
• Neuron-wise Normalized Optimization (NorMuon): Implements row-wise normalization of parameter updates after matrix-level orthogonalization, tracking and normalizing each neuron’s (row's) update magnitude using its own EMA of second-order momenta. This prevents neuron domination in parameter updates and stabilizes large-scale optimizer behavior (Li et al., 7 Oct 2025).

A table summarizing key schemes:

Method	Statistic Scope	Normalization Operation
Layer Norm	Per-example, all units	Mean/var over layer, scale/shift per unit
Normality Norm	Per-neuron	Mean/var, power Gaussianization, per-neuron scale
Online Norm	Per-neuron, streaming	EWA mean/var per neuron, train/test match
Analytic Norm	Per-neuron, analytic	Propagated mean/var via weights, scale/shift per unit
SNN Weight Norm	Per-neuron, weights	$\mathbf{z} = [z_1, \ldots, z_H]^\top$1 rescaling of incoming weights (activity norm)
NorMuon Optimizer	Per-neuron, optimizer	EMA row-norm normalization post-orthogonalization

3. Integration with Network Topologies and Extensions

Neuron-wise normalization is inherently architecture-agnostic due to its independence from batch size and spatial groupings.

• Feed-forward and Fully Connected Networks: LN and its variants are trivial to add to each layer, providing per-example stability and robust convergence, unaffected by mini-batch arrangements (Ba et al., 2016).
• Recurrent Neural Networks (RNNs): LN computes statistics per timestep and applies the same affine parameters throughout the sequence, avoiding batch-induced instability and enabling stable deep unrolled networks (Ba et al., 2016, Chiley et al., 2019).
• Convolutional Networks: Normality normalization and online normalization adapt neuron-wise statistics over appropriate spatial axes (e.g., batch × spatial for conv features), maintaining per-channel consistency (Eftekhari et al., 1 May 2025, Chiley et al., 2019).
• Spiking Neural Networks (SNNs): Postsynaptic potential normalization tracks each neuron’s energy via the second raw moment, dynamically adjusting spiking thresholds and stabilizing ultradeep SNNs (Ikegawa et al., 2022). Weight normalization schemes maintain homeostatic activity under plasticity (Kozdon et al., 2019).

4. Functional Benefits and Comparative Analysis

Neuron-wise normalization produces several advantages, especially compared to batch-dependent or global normalization methods:

• Train-test Consistency: LN and analytic norms perform identically in both phases (no running averages or batch-induced distribution shifts), ensuring reproducibility and robustness (Ba et al., 2016, Shekhovtsov et al., 2018, Chiley et al., 2019).
• Batch-size Invariance: Methods require no minimum batch size and work for batch size one or even online learning, critical in language modeling, reinforcement learning, and other memory-constrained or sequential applications (Ba et al., 2016, Chiley et al., 2019).
• Statistical Expressiveness: Normality normalization enforces not only the first two moments but targets Gaussianity, yielding both improved generalization and quantitatively better robustness to input noise and small batch sizes compared to classical batchnorm or layernorm (Eftekhari et al., 1 May 2025).
• Biologically Motivated Stability: For SNNs, normalization by raw second moment and per-neuron $\mathbf{z} = [z_1, \ldots, z_H]^\top$2 weight normalization directly stabilize firing rates, prevent pathological modes (e.g., neuron silence or firing saturation), and sustain performance in long-run or plastic networks (Ikegawa et al., 2022, Kozdon et al., 2019).

A plausible implication is that for heterogeneous or non-i.i.d. network activity, neuron-wise (particularly non-centered or mixture-based) normalization is superior for preserving both learning stability and flexible activation distributions.

5. Implementation and Algorithmic Considerations

Implementation details differ by method:

• LN, Normality Norm, Analytic Norm: Require only per-layer (or per-neuron) statistics; analytic norm needs an additional moment-propagation pass and explicit activation moment formula selection (closed form or approximations) (Ba et al., 2016, Eftekhari et al., 1 May 2025, Shekhovtsov et al., 2018).
• Online Norm: Maintains per-neuron moving statistics and dedicated backward-pass project-and-scale accumulator variables, encapsulated in an autodiff primitive (Chiley et al., 2019).
• NorMuon: Orthogonalizes updates via Newton-Schulz iterations and applies per-row RMS scaling, maintaining a vector of EMAs per parameter matrix with negligible storage cost. Per-shard computation in distributed training is streamlined by FSDP2, minimizing cross-GPU communication (Li et al., 7 Oct 2025).
• SNN Norms: Postsynaptic normalization requires only a streaming or batch EMA of $\mathbf{z} = [z_1, \ldots, z_H]^\top$3 per neuron; weight normalization applies $\mathbf{z} = [z_1, \ldots, z_H]^\top$4 rescaling after each STDP epoch (Ikegawa et al., 2022, Kozdon et al., 2019).

6. Empirical Results and Application Domains

Experimental evidence supports the effectiveness of neuron-wise normalization across a wide range of domains:

• Feedforward/Convnet/LN: Layer Normalization accelerates convergence and is especially robust for small batch sizes; for convolutional networks, classical BatchNorm remains strong, but Normality Norm and Online Norm can outperform when batch sizes decrease or data is heterogeneous (Ba et al., 2016, Eftekhari et al., 1 May 2025, Chiley et al., 2019).
• RNNs/SNNs: LN and ON provide stability and faster convergence in sequence learning. PSP-Norm in SNNs yields up to 98.2% test accuracy on N-MNIST and reduces firing rates by 30–50% over unnormalized models (Ikegawa et al., 2022).
• Optimizer Performance: NorMuon delivered 21.74% fewer required steps to target loss on 1.1B-parameter LLM pretraining, compared to AdamW, while maintaining only a 2.9% step-time overhead (Li et al., 7 Oct 2025).
• Generalization and Robustness: Normality Norm consistently improves top-1 accuracy by 1–4% over prior normalization layers and exhibits 2–5× better attenuation of Gaussian noise during inference (Eftekhari et al., 1 May 2025).

7. Recent Advances and Theoretical Perspectives

Emerging neuron-wise normalization strategies extend statistical and geometric normalization to:

• Gaussian Mixture Normalization: Unsupervised Adaptive Normalization (UAN) adapts per-neuron normalization via mixtures of Gaussians, updating both cluster assignments and distribution parameters online, leading to robust handling of shifting or multimodal activations (Faye et al., 2024).
• Analytic Propagation and Deterministic Norms: Analytic normalization decouples normalization from data batches entirely, relying on propagated statistics deriving from the network weights and assumed data distribution, and is recommended for deterministic test-time pipelines especially when regularization can be supplied externally (Shekhovtsov et al., 2018).
• Robustness Motivations: Information-theoretic arguments motivate Gaussianization and noise injection in neuron-wise normalizers, targeting maximal entropy and adversarial robustness properties as fundamental capacity and generalization goals (Eftekhari et al., 1 May 2025).

The diversity and extensibility of neuron-wise normalization methods ensure their centrality in modern deep learning optimization, architecture scaling, and biologically motivated neural computation.

---

References:

Layer Normalization (Ba et al., 2016); Normality Normalization (Eftekhari et al., 1 May 2025); Online Normalization (Chiley et al., 2019); PSP-Norm (Ikegawa et al., 2022); NorMuon optimizer (Li et al., 7 Oct 2025); Neuron-type SNN normalization (Kozdon et al., 2019); UAN (Faye et al., 2024); Analytic Moment Propagation (Shekhovtsov et al., 2018).