Parallel Layer Normalization in Deep Networks

• Parallel Layer Normalization (PLN) is a neural module that splits neuron activations into groups for independent layer normalization, serving as both activation and normalization.
• The universal approximation theorem for PLN networks confirms that even solely normalization-based nonlinearity achieves expressivity comparable to traditional activation functions.
• Empirical studies show that PLN enhances training stability and performance in deep architectures like CNNs, ResNets, and Transformers by integrating activation and normalization.

Parallel Layer Normalization (PLN) is a neural network module that generalizes standard layer normalization by dividing a layer’s neurons into multiple groups and applying layer normalization independently within each group. This architecture not only maintains the benefits of normalization for optimization and training stability but, crucially, enables PLN to serve as both the source of nonlinearity (“activation”) and normalization in deep networks. Recent theoretical and empirical work has established that networks with parallel layer normalization can achieve the universal approximation property, matching the expressive power of traditional activation functions while retaining normalization characteristics (Ni et al., 19 May 2025).

1. Mathematical Formulation and Operational Definition

PLN operates by splitting the concatenated pre-activations $\rvh$ of a neural layer into $N$ disjoint groups, each of size $d_i \geq 2$:

$\rvh = [\,\rvh_1^\top,\;\rvh_2^\top,\;\dots,\;\rvh_N^\top]^\top, \qquad \rvh_i\in\R^{d_i}$

Standard layer normalization is then applied independently to each group:

$\mathrm{PLN}(\rvh) = [\,LN(\rvh_1)^\top,\;\dots,\;LN(\rvh_N)^\top]^\top$

where, for group $i$,

$LN(\rvh_i)_j = \frac{(h_i)_j - \mu_i}{\sqrt{\frac{1}{d_i}\sum_{k=1}^{d_i}((h_i)_k - \mu_i)^2} + \delta}, \quad \mu_i = \frac{1}{d_i} \sum_{k=1}^{d_i} (h_i)_k,$

and $\delta>0$ ensures numerical stability. When all groups are equal in size $d$, the notation “PLN-$d$” is used. The computational abstraction termed the “Parallel $N$0-Net” instantiates the architecture as

$N$1

where setting $N$2 yields a one-hidden-layer PLN network with blockwise normalization-activation (Ni et al., 19 May 2025).

2. Universal Approximation Theorem for PLN Networks

Networks whose only nonlinearity derives from PLN—in particular, a single-hidden-layer sum of PLN-$N$3 blocks—possess the universal approximation property. For $N$4,

$N$5

is dense in $N$6: for any continuous $N$7 on $N$8 and any $N$9, there exists $d_i \geq 2$0 such that

$d_i \geq 2$1

This result extends classical UATs (which typically require explicit elementwise nonlinearity) by showing that PLN blocks—themselves only normalization layers—are sufficient for universality if group size $d_i \geq 2$2. The theory covers compact input domains, and relies on controlling the normalization barrier $d_i \geq 2$3 for continuity.

In the 1D case for $d_i \geq 2$4-Lipschitz $d_i \geq 2$5, a constructive proof shows that as $d_i \geq 2$6, a two-neuron LN block can realize the sign function, and a set of these blocks can approximate step functions, assembling a staircase approximation to $d_i \geq 2$7 with controlled uniform error.

3. Neuron Complexity and Minimum Width Bounds

The minimum neuron requirement for uniformly approximating $d_i \geq 2$8-Lipschitz functions to error $d_i \geq 2$9 with a single PLN layer is quantitatively established. For group size $\rvh = [\,\rvh_1^\top,\;\rvh_2^\top,\;\dots,\;\rvh_N^\top]^\top, \qquad \rvh_i\in\R^{d_i}$0,

$\rvh = [\,\rvh_1^\top,\;\rvh_2^\top,\;\dots,\;\rvh_N^\top]^\top, \qquad \rvh_i\in\R^{d_i}$1

For PLN-2, this bound is tight: $\rvh = [\,\rvh_1^\top,\;\rvh_2^\top,\;\dots,\;\rvh_N^\top]^\top, \qquad \rvh_i\in\R^{d_i}$2; for ReLU-activated networks, $\rvh = [\,\rvh_1^\top,\;\rvh_2^\top,\;\dots,\;\rvh_N^\top]^\top, \qquad \rvh_i\in\R^{d_i}$3. Thus, all these architectures achieve $\rvh = [\,\rvh_1^\top,\;\rvh_2^\top,\;\dots,\;\rvh_N^\top]^\top, \qquad \rvh_i\in\R^{d_i}$4 scaling. The required network width increases linearly with $\rvh = [\,\rvh_1^\top,\;\rvh_2^\top,\;\dots,\;\rvh_N^\top]^\top, \qquad \rvh_i\in\R^{d_i}$5 in PLN-$\rvh = [\,\rvh_1^\top,\;\rvh_2^\top,\;\dots,\;\rvh_N^\top]^\top, \qquad \rvh_i\in\R^{d_i}$6 networks.

4. Capacity Comparison with Traditional Activations

A summary of minimum widths for various activations and normalizations in the context of uniform approximation for $\rvh = [\,\rvh_1^\top,\;\rvh_2^\top,\;\dots,\;\rvh_N^\top]^\top, \qquad \rvh_i\in\R^{d_i}$7-Lipschitz functions is as follows:

Nonlinear Layer	Lower Bound	Upper Bound	Width Lower Bound	Width Upper Bound
Sigmoid/Tanh	—	$\rvh = [\,\rvh_1^\top,\;\rvh_2^\top,\;\dots,\;\rvh_N^\top]^\top, \qquad \rvh_i\in\R^{d_i}$8	—	Same as upper
ReLU	$\rvh = [\,\rvh_1^\top,\;\rvh_2^\top,\;\dots,\;\rvh_N^\top]^\top, \qquad \rvh_i\in\R^{d_i}$9	$\mathrm{PLN}(\rvh) = [\,LN(\rvh_1)^\top,\;\dots,\;LN(\rvh_N)^\top]^\top$0	Same	Same
PLN-$\mathrm{PLN}(\rvh) = [\,LN(\rvh_1)^\top,\;\dots,\;LN(\rvh_N)^\top]^\top$1	—	$\mathrm{PLN}(\rvh) = [\,LN(\rvh_1)^\top,\;\dots,\;LN(\rvh_N)^\top]^\top$2	—	$\mathrm{PLN}(\rvh) = [\,LN(\rvh_1)^\top,\;\dots,\;LN(\rvh_N)^\top]^\top$3
PLS-$\mathrm{PLN}(\rvh) = [\,LN(\rvh_1)^\top,\;\dots,\;LN(\rvh_N)^\top]^\top$4 (RMSNorm)	—	$\mathrm{PLN}(\rvh) = [\,LN(\rvh_1)^\top,\;\dots,\;LN(\rvh_N)^\top]^\top$5	—	$\mathrm{PLN}(\rvh) = [\,LN(\rvh_1)^\top,\;\dots,\;LN(\rvh_N)^\top]^\top$6
PLN-2	$\mathrm{PLN}(\rvh) = [\,LN(\rvh_1)^\top,\;\dots,\;LN(\rvh_N)^\top]^\top$7	$\mathrm{PLN}(\rvh) = [\,LN(\rvh_1)^\top,\;\dots,\;LN(\rvh_N)^\top]^\top$8	$\mathrm{PLN}(\rvh) = [\,LN(\rvh_1)^\top,\;\dots,\;LN(\rvh_N)^\top]^\top$9	$i$0
PLS-1	$i$1	$i$2	Same	Same

PLN and PLS thus match the $i$3 scaling of standard nonlinearities, introducing only a constant factor in width proportional to group size $i$4 (Ni et al., 19 May 2025).

5. Dual Functionality: Activation and Normalization

PLN serves as both an activation mechanism and a normalization operation. As normalization, PLN inherits the scale-invariance and gradient-stabilizing properties associated with layer normalization, with normalization statistics computed independently in each group. As an activation, PLN exhibits nonlinearity—even in the absence of explicit pointwise nonlinearities: a two-neuron LN group can realize a step function, and multiple PLN groups can build staircase approximators of arbitrary precision. This dual role enables PLN networks to substitute both standard normalization layers and activation functions within deep architectures.

6. Empirical Evaluation and Practical Impact

Empirical analyses demonstrate that PLN substantially improves the stability and expressiveness of deep networks in various domains:

• CNNs (VGG without BatchNorm, CIFAR-10): Substituting activations with Channel-PLN ($i$5) yields a test accuracy of 89.45%, whereas Sigmoid, Tanh, and ReLU fail to train without batch normalization (test accuracy near 10%).
• ResNet (without BN): In ResNet-20/-56/-110, ReLU exhibits severe gradient explosion (backward norms reaching $i$6). PLN-8 maintains stability and achieves higher final test accuracy compared to standard activations.
• CNNs (with BN): PLN-8 underperforms ReLU marginally when batch normalization is present but outperforms Sigmoid and Tanh.
• Transformers (WMT-Multi30K, BLEU score): Replacing LayerNorm with PLN-8 (“norm-size=8”) reliably increases BLEU scores; with GELU activations, BLEU rises from 42.05 (LN) to 42.91 (PLN-8). Combining PLN-8 and ReLU outperforms original LN+ReLU schemes.

These results indicate that PLN can confer measurable gains over standard normalization/activation pairings and sustain stable training in deep or wide neural network regimes (Ni et al., 19 May 2025).

7. Significance and Theoretical Extensions

Parallel Layer Normalization extends universal approximation theory to encompass networks using solely parallel normalization blocks as their nonlinearity source. This structure achieves comparable expressivity to networks utilizing ReLU, Sigmoid, or Tanh activations, while also offering normalization-driven optimization benefits such as improved gradient flow and stability during training. This suggests that PLN architectures may resolve limitations encountered when decoupling activation and normalization in very deep networks. The dual function of PLN, combined with its theoretical and empirical merits, situates it as a foundational primitive for neural architectures beyond traditional linear-activation-normalization pipelines (Ni et al., 19 May 2025).