---
title: Group Normalization (GN)
url: https://www.emergentmind.com/topics/group-normalization-gn
type: topic
---

# Group Normalization (GN)

Group Normalization (GN) is a feature normalization technique for deep neural networks, designed to address the limitations of batch-dependent normalization layers such as Batch Normalization (BN). GN operates by partitioning the feature channels of activations into groups and computing normalization statistics (mean and variance) independently within each group, which enables consistent model behavior regardless of batch size. The method provides stable training and inference in scenarios with small batch sizes or significant data heterogeneity, proving its versatility in vision, biomedical, generative, and multimodal learning tasks.

## 1. Formal Definition and Mechanism

Consider an activation tensor $x \in \mathbb{R}^{N \times C \times H \times W}$ where $N$ is batch size, $C$ is the number of channels, and $H \times W$ is the spatial resolution. GN divides the $C$ channels into $G$ disjoint groups, each containing $C/G$ channels. For each group $g$ and sample $n$, the mean $\mu_{n,g}$ and variance $\sigma_{n,g}^2$ are computed over all elements belonging to the group (across its $C/G$ channels and all spatial locations):

\[
\mu_{n,g} = \frac{1}{m} \sum_{i \in S_{n,g}} x_i 
\]
\[
\sigma_{n,g}^2 = \frac{1}{m} \sum_{i \in S_{n,g}} (x_i - \mu_{n,g})^2
\]

where $m = (C/G) \cdot H \cdot W$ and $S_{n,g}$ indexes all elements in group $g$ for sample $n$.

Normalization is then applied to each element $i \in S_{n,g}$:

\[
\hat{x}_i = \frac{x_i - \mu_{n,g}}{\sqrt{\sigma_{n,g}^2 + \epsilon}}
\]

A learned per-channel scale $\gamma_c$ and bias $\beta_c$ restore representational flexibility:

\[
y_i = \gamma_c \hat{x}_i + \beta_c
\]

When $G = 1$, GN reduces to Layer Normalization; when $G = C$, it becomes Instance Normalization [1803.08494].

## 2. Comparison with Other Normalization Methods

The principal distinction between GN and BN is the axis over which normalization statistics are computed and the resulting batch-size dependency:

| Normalization Method | Normalized Axes                      | Batch Dependence               |
|----------------------|--------------------------------------|-------------------------------|
| Batch Norm (BN)      | (N, H, W) per channel                | Yes (performance varies with batch size) |
| Layer Norm (LN)      | (C, H, W) per sample                 | No                            |
| Instance Norm (IN)   | (H, W) per sample, per channel       | No                            |
| Group Norm (GN)      | (C/G, H, W) per group per sample     | No                            |

BN’s reliance on batch statistics introduces instability in small-batch regimes, whereas GN’s per-sample, per-group estimation maintains accuracy across all batch sizes [1803.08494, 2404.00946]. Unlike BN, GN does not require accumulation or synchronization of running statistics between training and inference.

## 3. Theoretical Perspectives and Spherical Normalization Framework

GN can be expressed within the "spherical normalization" framework, which interprets a group’s pre-activations as a vector in $\mathbb{R}^m$ projected onto a sphere of radius $\sqrt{m}$, removing the scale and mean [2006.09104]. The affine parameters $(\gamma_g, \beta_g)$ then "re-embed" the standardized vector. GN is invariant to groupwise scaling and shifting of pre-activations:

- For any $\alpha \ne 0, t \in \mathbb{R}$,
  \[
  N(\alpha v + t e_m) = N(v)
  \]
  where $e_m$ is the all-ones vector.

This invariance constrains optimization to a compact manifold (sphere), leading to scale-invariant gradients and, if unchecked, monotonic weight norm growth during training. This phenomenon can increase adversarial vulnerability unless regularized with weight decay [2006.09104].

## 4. Empirical Results and Use Cases

GN was demonstrated to outperform or match BN in multiple settings, particularly when batch sizes are small or variable [1803.08494, 2404.00946]:

- **ImageNet Classification (ResNet-50):** With batch size 2, GN achieves 24.1% top-1 error, while BN degrades to 34.7%. For standard batch sizes ($N=32$), GN remains within 0.5% of BN.
- **COCO Detection/Segmentation:** GN achieves higher AP$_{\text{bbox}}$ (40.8 vs. 38.6) and AP$_{\text{mask}}$ (36.1 vs. 34.5) compared to BN in Mask R-CNN, especially when $N=1$ per GPU.
- **Medical Imaging:** In U-Net training for 2D biomedical semantic segmentation, fine-grained grouping (e.g., GN with $G=C$, equivalent to IN) yields highest Dice coefficients, suggesting improved generalization over BN and LN in the presence of significant data heterogeneity and small batches [1809.03783].
- **Few-Shot and Conditional Learning:** Conditional GN, in which the affine parameters become functions of auxiliary conditioning variables, supports systematic generalization and domain-shift robustness in visual question answering and meta-learning benchmarks [1908.00061].
- **Multimodal and RL Settings:** Difficulty-aware GN (Durian) uses sample difficulty metrics (visual entropy, reasoning uncertainty) for regrouping, improving stability in multimodal reinforcement learning and yielding substantial gains over standard group-relative normalization [2602.21743].

## 5. Practical Implementation and Hyperparameter Selection

Implementing GN typically involves selecting an appropriate group count $G$. The consensus is:

- Use $G=32$ by default for $C \geq 32$.
- If $C < 32$, select $G$ as a divisor of $C$, for example, $G = C/2$ or $G = 1$ (Layer Norm).
- GN’s computational overhead is minimal (within 5% of BN), requiring only group-wise mean and variance computations plus per-channel affine transformation [1803.08494, 2404.00946].
- Training GN models often accommodates higher learning rates than when using BN.
- GN does not use or maintain population statistics for inference; the same normalization is applied at train and test time [2404.00946].
- Code availability in frameworks: `torch.nn.GroupNorm(num_groups=G, num_channels=C)`.

## 6. Limitations and Hybrid Approaches

While GN resolves batch-size dependence, several limitations exist:

- **Lack of BN’s Stochastic Regularization:** GN’s deterministic computation can lead to less regularization compared to BN’s batch-statistics-induced noise, which is particularly beneficial in generative models (e.g., GANs) [1908.00061].
- **Training Instability and Sensitivity:** GN exhibits greater sensitivity to injected noise and weight-decay regularization, and does not consistently stabilize optimization throughout all training phases. Specifically, early-phase “loss landscape” flatness and gradient predictiveness can lag behind BN, with GN only providing a smoothing effect in the training mid-stage [2207.01972].
- **Hybrid Normalization:** To address such instabilities, GN + BN hybrid layers (e.g., “GN-first sequential,” where BN is applied after GN and outputs are fused by a learned gate) combine the batch-size invariance of GN with BN’s regularization. These hybrids improve robustness and reduce performance variance over batch size, yielding accuracy gains on diverse datasets [2207.01972].

## 7. Extensions and Specialized Variants

GN’s framework is adaptable beyond the standard vision domain:

- **Difficulty-Aware GN:** For reinforcement learning and multimodal LLMs, GN can be made difficulty-aware by regrouping samples via perceptual or reasoning metrics and normalizing within groups of homogeneous difficulty. This addresses collapse of variance when dealing with bimodal or highly polarized samples and stabilizes policy optimization [2602.21743].
- **Conditional GN (CGN):** CGN conditions the affine parameters $(\gamma, \beta)$ on external meta-data (e.g., task embedding or class label), supporting domain adaptation, generalization to novel conditions, and controlled modulation of feature statistics [1908.00061].

## References

- Wu & He, "Group Normalization" [1803.08494]
- Sun et al., "New Interpretations of Normalization Methods in Deep Learning" [2006.09104]
- Zhou et al., "Normalization in Training U-Net for 2D Biomedical Semantic Segmentation" [1809.03783]
- "Exploring the Efficacy of Group-Normalization in Deep Learning Models for Alzheimer's Disease Classification" [2404.00946]
- "Understanding and Improving Group Normalization" [2207.01972]
- "An Empirical Study of Batch Normalization and Group Normalization in Conditional Computation" [1908.00061]
- "Enhancing Multi-Modal LLMs Reasoning via Difficulty-Aware Group Normalization" [2602.21743]

Source: https://www.emergentmind.com/topics/group-normalization-gn