ConvNeXt V2 Huge Encoder

• The paper presents a fully convolutional architecture integrating GRN in masked autoencoder pre-training to prevent feature collapse.
• It features a deep hierarchical design with stage-wise channel scaling, totaling 659M parameters and achieving 88.9% top-1 accuracy on ImageNet-1K.
• Co-designed with FCMAE, the model leverages efficient sparse-to-dense training and GRN-based normalization for improved feature diversity.

ConvNeXt V2 Huge Encoder is a fully convolutional architecture representing advances in convnet-based visual recognition by co-designing architectural innovations and self-supervised learning via masked autoencoders. The model, with approximately 659 million parameters, is characterized by stage-wise channel scaling, deep hierarchical structure, and the integration of Global Response Normalization (GRN) specifically to address feature collapse in masked pre-training scenarios. When pre-trained and fine-tuned with only public data, the Huge variant attains 88.9% top-1 accuracy on ImageNet-1K, surpassing previous pure convnet benchmarks (Woo et al., 2023).

1. Architectural Configuration

The ConvNeXt V2 Huge encoder is structured in four hierarchical stages, each utilizing a stem, downsampling mechanisms, depthwise convolution, channel expansion, and the inclusion of GRN for feature normalization. The network accepts 224×224 RGB inputs and systematically reduces their spatial resolution to 7×7 while expanding channel dimensionality.

Stage	Input Res→Output Res	Channels	#Blocks	Downsampling
1	224×224→112×112	352	3	stem conv 4×4, stride 2
2	112×112→56×56	704	3	conv, stride 2
3	56×56→28×28	1408	27	none
4	28×28→14×14→7×7	2816	3	conv, stride 2 + pooling

Each block within these stages employs a 7×7 depthwise convolution (groups=channel dimension), LayerNorm, MLP expansion with ratio $r=4$, application of GRN, MLP projection, and residual addition. This design enables expressive feature transformations while mitigating representational collapse.

2. Global Response Normalization (GRN) Mechanism

GRN is a central architectural addition, inserted after each MLP expansion and preceding the 1×1 projection. For a feature tensor $X \in \mathbb{R}^{H \times W \times C}$, GRN consists of:

1. Global Aggregation (G):

$$G(X)_i = \|X_i\|_2 = \left( \sum_{h=1}^H \sum_{w=1}^W X_i(h,w)^2 \right)^{1/2}$$

2. Channel-wise Divisive Normalization (N):

$$N_i = \frac{G(X)_i}{\sum_{j=1}^C G(X)_j + \epsilon}$$

3. Feature Recalibration and Residual:

$$X'_i = \gamma_i \cdot (X_i \cdot N_i) + \beta_i + X_i$$ ($\gamma$, $\beta$ are learnable; $\epsilon$ ensures numerical stability.)

GRN introduces no additional layers except scale and bias for stable initialization, directly promoting inter-channel competition and restoring channel diversity lost in high-ratio masked autoencoder pre-training.

3. Comparison with ConvNeXt V1 and Design Rationale

ConvNeXt V2 modifies the block structure of its predecessor by integrating GRN post-MLP-expand, whereas ConvNeXt V1 used LayerScale for channel-wise re-scaling. In V1, blocks exposed to high masking ratios during masked autoencoder pre-training exhibited channel saturation (feature collapse). GRN addresses this collapse by rebalancing channel responses, enhancing the ability for stable training without LayerScale. The ordering of operations in the block is DepthwiseConv → LayerNorm → MLP expand → GRN → MLP project → residual addition.

A plausible implication is that channel diversity restoration via normalization can generalize to other convolutional architectures facing similar collapse phenomena.

4. Co-design with Fully Convolutional Masked Autoencoder (FCMAE)

The Huge encoder is jointly developed with a fully convolutional masked autoencoder pre-training framework. Images are divided into non-overlapping 32×32 patches, of which 60% are randomly masked. Unmasked patches are represented as sparse tensors, processed only during pre-training with submanifold sparse convolution. Sparse convolutions are replaced by dense variants during fine-tuning.

The decoder comprising a single ConvNeXt block (512 channels, one layer) operates on the encoder output concatenated with mask tokens, predicting pixel-level logits over masked regions. Optimization uses mean squared error applied only to the reconstructed RGB values of masked patches.

Critical hyper-parameters include AdamW optimization (β₁=0.9, β₂=0.95), learning rate schedules (linear scaling, cosine decay), large batch sizes (pre-training: 4096; fine-tuning: 1024), and extensive training durations (800 epochs pre-training, typically 50 for fine-tuning). Augmentations like RandAug, mixup (0.8), cutmix (1.0), and label smoothing (0.1) are employed during fine-tuning.

5. Scaling Strategy to Huge Variant

Scaling from ConvNeXt V2-Large (198M parameters; C=192, B=(3,3,27,3)) to Huge involves width-only scaling. The base channel dimension is amplified by approximately 1.83, leading to C₁=352, with blocks per stage unchanged. Channel dimensions thus become [352, 704, 1408, 2816], and overall parameter count achieves ∼659M, preserving the model’s hierarchical depth and computational structure.

This scaling approach maintains the underlying architectural principles while significantly increasing representational capacity, optimized for performance on large-scale benchmarks.

6. Performance and Benchmarking

ConvNeXt V2 Huge encoder, co-designed with FCMAE and GRN, establishes state-of-the-art performance for pure convnets using only public data. When pre-trained and intermediate-fine-tuned on ImageNet-22K and subsequently on ImageNet-1K, the model achieves 88.9% top-1 accuracy, outperforming all prior ConvNet-based models in public benchmarks (Woo et al., 2023).

This suggests that architectural modifications targeting channel diversity and efficient masked encoding are essential for scaling convnets, particularly in the regime of self-supervised representation learning.