NFNets: Normalizer-Free Deep Networks

• NFNets are deep neural network architectures that forgo conventional normalization layers by employing analytic signal propagation methods to maintain stable training.
• NFNets use variance-preserving initialization and Scaled Weight Standardization to regulate activation statistics, enabling efficient performance even with small batch sizes.
• NFNets integrate Adaptive Gradient Clipping and advanced regularization strategies to optimize training speed and robustness, often surpassing batch-normalized counterparts.

Normalizer-Free Networks (NFNets) are deep neural network architectures that eschew all forms of activation normalization (such as BatchNorm, LayerNorm, and GroupNorm) while maintaining or exceeding the state-of-the-art accuracy and training stability traditionally dependent on normalization layers. Developed primarily in the context of large-scale visual recognition, NFNets leverage theoretically principled initialization and signal propagation control, enhanced regularization strategies, and a novel optimization technique—Adaptive Gradient Clipping (AGC)—to resolve the training instabilities that previously limited normalization-free models. The ability to match or surpass the performance of batch-normalized models, with improved training throughput, marks a fundamental shift in design philosophy for high-performance deep architectures.

1. Motivation for Normalizer-Free Architectures

The predominant role of normalization layers, especially Batch Normalization (BN), in modern convolutional networks is well established; their empirical benefits include faster convergence, higher accuracy, and easier optimization of deep architectures. However, BN’s reliance on batch statistics introduces several well-recognized limitations:

• Dependence on batch size: BN accuracy degrades with small batch sizes, and instability arises with large ones.
• Training-inference discrepancy: The use of batch statistics at training and moving averages at inference leads to subtle behavioral shifts.
• Memory and computational overhead: BN adds significant operations and requires synchronization overhead in distributed or hardware-accelerated settings.
• Cross-sample coupling: By introducing dependencies between examples within the same batch, BN complicates reproducibility, privacy, and certain application domains (e.g., sequence modeling, contrastive/self-supervised learning).

Earlier attempts at normalization-free training (e.g., Fixup, SkipInit, initial NF-ResNets) fell short in practical scenarios, either failing to achieve SOTA accuracy or introducing instability under large learning rates and stronger augmentations. Thus, a robust recipe for designing, regularizing, and optimizing high-performance CNNs without normalization remained an unsolved challenge (Brock et al., 2021, Brock et al., 2021, Liu et al., 2022).

2. Controlling Signal Propagation Without Normalization

BatchNorm’s central role can largely be attributed to its control of signal propagation—specifically, it keeps the mean and variance of activations from drifting or exploding as depth increases. In the absence of normalization, careful signal tracking and correction mechanisms are required to maintain stable forward and backward passes.

Key mechanisms in NFNets:

• Variance-preserving initialization and explicit scaling: Every residual block in an NFNet applies an analytic variance adjustment:

$$x_{\ell+1} = x_\ell + \alpha f_\ell\left(\frac{x_\ell}{\beta_\ell}\right)$$

where $\beta_\ell$ is precomputed from initialization statistics to neutralize signal growth, and $\alpha$ is a hyperparameter controlling variance accumulation.

• Scaled Weight Standardization (Scaled WS): To suppress cumulative mean shift in channels—otherwise problematic with nonlinearities such as ReLU—weights are reparametrized:

$$\hat{W}_{i,j} = \gamma \frac{W_{i,j} - \mu_{W_{i,\cdot}}}{\sigma_{W_{i,\cdot}}\sqrt{N}}$$

with $\gamma$ chosen to preserve output variance post-nonlinearity; e.g., $$\gamma = 1/\sqrt{\mathrm{Var}(g(x))}, x \sim \mathcal{N}(0,1)$$. This maintains zero mean and controlled variance propagation at arbitrary depths (Brock et al., 2021).

• Signal Propagation Plots (SPPs): Empirical analysis tools (Editor’s term: “SPPs”) track average squared mean and variance through blocks, making abnormal drift observable and correctable during design and debugging.

3. Optimizing and Regularizing NFNets: Adaptive Gradient Clipping (AGC)

Conventional normalization-free models are prone to divergent or unstable training—especially for large batch sizes, high learning rates, or aggressive data augmentations. NFNets address this with Adaptive Gradient Clipping (AGC):

• Principle: Instead of a global update norm constraint, AGC applies filter- or neuron-wise clipping based on the relative scale of the update to its target weight:

$$G^{\ell}_i \rightarrow \begin{cases} \lambda \frac{\|W^{\ell}_i\|^\star_F}{\|G^{\ell}_i\|_F} G^{\ell}_i & \text{if } \frac{\|G^{\ell}_i\|_F}{\|W^{\ell}_i\|^\star_F} > \lambda \ G^{\ell}_i & \text{otherwise} \end{cases}$$

where $G^{\ell}_i$ and $W^{\ell}_i$ are gradients and parameters for unit $i$ at layer $\ell$, $\lambda$ is the threshold, and $||W||^\star_F = \max(||W||_F, \epsilon)$ to avoid division by zero (Brock et al., 2021).

• Benefits: AGC enables stable training in regimes previously requiring normalization, diminishes hyperparameter sensitivity, and is compatible with large batch training and strong augmentation pipelines.
• Complementary regularization: Further empirical gains are achieved by employing advanced augmentation (RandAugment, CutMix), stochastic depth, and dropout.

4. Architectural Details and Generalization Scope

NFNets generalize the normalization-free recipe across a broad range of architectures and tasks:

• Backbone: Based on pre-activation SE-ResNeXt-D, integrating squeeze-and-excitation and grouped convolutions.
• Structural modifications: Architectural scaling (depth, width), bottleneck modifications, and variance corrections for components such as Squeeze-and-Excitation and pooling are applied, ensuring analytic signal propagation properties hold.
• Extension to diverse models: The approach was successfully applied to EfficientNet-style scaling in RegNets, demonstrating near-parity with normalized EfficientNets for image classification FLOPs and accuracy (Brock et al., 2021).
• Downstream tasks: NF-ResNets displayed competitive or superior transfer to semantic segmentation (Pascal VOC) and depth estimation (NYUv2), as well as improved robustness to small batch sizes, where BN-based models typically deteriorate.

5. Empirical Benchmarks and Performance Analysis

NFNets demonstrate competitive or superior accuracy and throughput compared to batch-normalized and other normalization-free models.

Model	Top-1 Accuracy (ImageNet)	Relative Training Speed
EfficientNet-B7	84.7%	Baseline
NFNet-F1	84.7%	8.7× faster
NFNet-F5 (+SAM)	86.0–86.5%	Similar, higher accuracy
NFNet-F4+ (pre-trained)	89.2%	SOTA with extra data

• Speed: NFNet-F1 matches EfficientNet-B7 accuracy with 8.7× faster training.
• State-of-the-art validation: NFNet-F5 achieves 86.0% top-1, and NFNet-F6 with SAM attains 86.5% (no extra data).
• Pretraining advantage: After large-scale pre-training (300M images), fine-tuned NFNet-F4+ achieves 89.2% top-1, exceeding batch-normalized counterparts by up to 1% in transfer scenarios.
• Small batch robustness: Accuracy is maintained with small batch training, where BN typically struggles.

6. Implications, Limitations, and Broader Context

Eliminating normalization layers has notable theoretical and practical consequences:

• Benefits: Increased training throughput, hardware compatibility, and batch-size-agnostic behavior improve large-scale training reproducibility and deployment portability.
• Removal of batch size effects: Training and inference become decoupled from sample aggregation, enhancing privacy and distributed system performance.
• Generalization of AGC: Adaptive Gradient Clipping is broadly applicable to other model types and domains typified by normalization-related instability.
• Limitations: Depthwise convolutions as in EfficientNet may pose difficulties for Scaled WS, making grouped convolutions preferable for complete normalization-free application (Brock et al., 2021).
• Alternative approaches: NoMorelization introduces a distinct method, using two trainable scalars and a noise injector to explicitly mimic the regularization effect of normalization, with superior speed-accuracy trade-off in some settings (Liu et al., 2022).

7. Conclusion and Future Research Directions

NFNets establish that high test accuracy, stable and efficient optimization, and scalable deployment are achievable without normalization layers. The methodological combination of analytic signal tracking, Scaled Weight Standardization, careful residual scaling, and Adaptive Gradient Clipping closes the historical performance and stability gap between normalized and unnormalized deep networks. The results suggest model design can shift away from reliance on BN and related techniques, pointing towards deeper investigation of signal propagation mechanisms and regularization strategies as new frontiers for architecture and optimization research.

Reference implementations and training code are available via the DeepMind Github repository: https://github.com/deepmind/deepmind-research/tree/master/nfnets (Brock et al., 2021).