PoolFormer: Pooling-Based Neural Architecture

• PoolFormer is a neural architecture that replaces traditional attention or MLP token mixers with parameter-free pooling operations within a MetaFormer framework.
• It achieves competitive results in vision tasks by reducing computational costs, exemplified by efficient ImageNet classifications and enhanced object detection metrics.
• The design supports long-sequence modeling by combining recurrent gating and multi-scale pooling, offering robust performance in audio and sequence-to-sequence applications.

PoolFormer denotes a class of neural architectures that replace traditional attention- or MLP-based token mixers with parameter-free pooling operations embedded within a residual, channel-MLP, and normalization "MetaFormer" skeleton. Two main research strands under this name exist: a vision-centric architecture that instantiates MetaFormer with simple pooling in lieu of self-attention (Yu et al., 2021), and a long-sequence modeling architecture for sequence-to-sequence tasks, especially audio, which recursively combines recurrence and pooling to alleviate the costs of self-attention in extremely long contexts (Fernández, 2 Oct 2025).

1. MetaFormer Skeleton and Pooling-Based Token Mixer

The core design principle behind PoolFormer is to isolate and examine the effect of the general Transformer encoder "skeleton"—consisting of residual connections, channel-wise MLPs, and normalization—independently of the specific token mixing mechanism. The hypothesis is that much of the Transformer’s power arises from this skeleton rather than the details of attention or spatial MLPs (Yu et al., 2021).

In PoolFormer, the token mixer is instantiated as a parameter-free average pooling operation: $$\mathrm{Pool}(X)_{b,c,i,j} = \frac{1}{K^2} \sum_{p=1}^K\sum_{q=1}^K X_{b,c,i + p - \lfloor K/2 \rfloor,\;j + q - \lfloor K/2 \rfloor}$$ with $K=3$ by default, and the pooling-based mixing is expressed as

$\mathrm{PoolMix}(X) = \mathrm{Pool}(X) - X.$

Each PoolFormer block uses LayerNorm (or Modified LayerNorm) and a two-layer channel MLP with expansion ratio $r$.

The overall block forward equations are: $$Y = X + \mathrm{PoolMix}(\mathrm{LN}(X)), \qquad Z = Y + \mathrm{MLP}(\mathrm{LN}(Y)),$$ where $\mathrm{MLP}(U) = W_2\,\sigma(W_1\,U)$, with $$W_1 \in \mathbb{R}^{rC \times C}, W_2 \in \mathbb{R}^{C\times rC}$$.

2. PoolFormer Architectures: Vision and Sequence Modeling

Vision PoolFormer: The architecture follows a 4-stage hierarchical, "pyramid" design: each stage applies spatial downsampling (3×3 stride-2 patch embedding) followed by a repeated sequence of PoolFormer blocks. Representative model configurations are:

Model	[Blocks per Stage]	[Dims]	Params (M)	MACs (G)	Top-1 (%)
PoolFormer-S12	[2, 2, 6, 2]	[64,128,320,512]	11.9	1.8	77.2
PoolFormer-S24	[4, 4,12, 4]	[64,128,320,512]	21.4	3.4	80.3
PoolFormer-S36	[6, 6,18, 6]	[64,128,320,512]	30.8	5.0	81.4
PoolFormer-M36	[6, 6,18, 6]	[96,192,384,768]	56.1	8.8	82.1
PoolFormer-M48	[8, 8,24, 8]	[96,192,384,768]	73.4	11.6	82.5

Long-Sequence PoolFormer: Addressing limitations of self-attention for extremely long sequences, the architecture replaces self-attention with recurrent gated linear units (RG-LRU) and introduces recursive multi-scale pooling/upsampling via the "SkipBlock" module. Each SkipBlock combines stacks of pre-/post-pool residual blocks, downsampling by 1D convolutions, recursive nesting or deep residual processing, transposed convolution upsampling, and skip/residual connections. This design enables effective modeling of both local and global dependencies while containing computational costs (Fernández, 2 Oct 2025).

3. Computational Complexity and Empirical Results

Vision Tasks: PoolFormer (e.g., M36: 56M params, 8.8G MACs, 82.1% top-1) outperforms or matches strong attention- and MLP-based vision baselines with fewer parameters and reduced MACs. For instance, on ImageNet-1K, PoolFormer-M36 outperforms DeiT-B by 0.3% with 35% fewer params and 50% fewer MACs (Yu et al., 2021). On COCO object detection (Mask R-CNN), PoolFormer-S24 backbone produces higher AP (38.9) and Mask AP (40.1) than ResNet-50. On ADE20K segmentation, PoolFormer-S24 and M36 exceed ResNet-50 and PVT-Small in mIoU.

Sequence Modeling: On SC09 (16k tokens), Poolformer (7.27M params) achieves NLL = 1.854, FID = 0.46, IS = 6.46, surpassing SaShiMi and matching or exceeding Mamba in both log-likelihood and perceptual metrics. Pooling increases training speed (11.45 vs. 6.11 epochs/hour compared to no pooling), improves FID and IS, and reduces overfitting (Fernández, 2 Oct 2025).

Dataset	PoolFormer NLL	PoolFormer FID	PoolFormer IS	SaShiMi NLL	SaShiMi FID	SaShiMi IS
SC09	1.854	0.46	6.46	1.891	1.81	3.89
Beethoven	0.915	—	—	0.946	—	—
YouTubeMix	1.391	—	—	1.294	—	—

4. Training and Implementation Protocols

Vision PoolFormer: Trained on ImageNet-1K with AdamW (weight decay 0.05), batch size 4096, cosine LR decay (300 epochs, 5-epoch warm-up), and augmentations (RandAugment, MixUp, CutMix, Label Smoothing). The normalization is Modified LayerNorm over $C\times H\times W$. Stochastic Depth and layer scale are optional regularization components. Implementation utilizes PyTorch/TIMM for classification and MMDetection/MMSeg for detection/segmentation; experiments are performed on TPU v3 and A100 clusters (Yu et al., 2021).

Sequence PoolFormer: Uses the SkipBlock recursion and stacking, residual block thermostats, RG-LRU for temporal mixing, 1D convolutions for pooling, and grouped operations to allow efficient computation at varied temporal resolutions. Proper kernel/stride initialization ensures identity path dominance during upsampling. Empirical studies demonstrate regularization and improved convergence (Fernández, 2 Oct 2025).

5. Analysis, Insights, and Theoretical Implications

MetaFormer Principle: Ablation studies reveal that the "MetaFormer" skeleton—residual architecture with channel-wise MLPs and normalization—is the dominant source of representational power, with token mixers exerting only secondary influence provided they enable minimal cross-token interaction. Pooling, despite being parameter-free, suffices to launch competitive models at scale: identity mixing still attains 74.3% top-1, random global mixing 75.8%, while depthwise convolution and larger $K$ yield incremental improvements. Removing either normalization or residual connections leads to catastrophic degradation (down to 46.1% or effectively random guess, respectively) (Yu et al., 2021).

Vision Representations: Grad-CAM visualizations indicate that PoolFormer learns tight, object-focused features resembling CNNs, as opposed to the often spatially-dispersed activations typical of pure MLP/attention-based vision models.

Depth and Multi-Scale Modeling: In the sequence PoolFormer, analysis of recurrence gating parameters shows that deep layers (processing highly downsampled sequences) capture long-range dependencies ($|\mathbb{E}[a]| \approx 1$), while shallow layers (high-resolution) extract local patterns ($K=3$0 significantly lower). This multi-scale organization enables the model to maintain global coherence without incurring quadratic complexity (Fernández, 2 Oct 2025).

6. Extensions and Future Research Directions

Anticipated directions for PoolFormer-related research include:

• Examination of pooling-based MetaFormer architectures in self-supervised and few-shot scenarios.
• Application of the pooling-mixer paradigm to NLP, where efficiency for long inputs is critical.
• Multi-modal modeling that unifies dense visual representations (e.g., hundreds of patch embeddings) and long text inputs within a pooled, recurrent framework.
• Hybrid architectures that incorporate pooling in early stages to maximize efficiency and deploy attention or richer token mixing only in later stages for finer modeling (e.g., vision: 81.0% top-1 with 16.5M params, 2.5G MACs using such hybrids).
• Investigation of pooling as a regularization technique to prevent overfitting and enhance generalization, as validated empirically in audio modeling (Fernández, 2 Oct 2025).

A plausible implication is that PoolFormer may serve as both a robust baseline and a low-complexity alternative for domains requiring scalable modeling over long horizons and large-resolution data, thereby prioritizing architectural skeleton refinement over continual development of increasingly complex token mixers (Yu et al., 2021, Fernández, 2 Oct 2025).