MetaFormer: Transformer Block Abstraction

• MetaFormer is a generalized Transformer block that separates token mixing from channel transformation using normalization and residual connections.
• It supports various token mixers such as pooling, convolution, and static-key attention, achieving competitive performance on benchmarks like ImageNet.
• Its modular design has been adapted for diverse applications including medical imaging, reinforcement learning, and industrial systems, proving its versatility.

MetaFormer is, in much of the recent literature, an architectural abstraction of the Transformer in which the decisive structure is not a specific self-attention operator but a repeated block that separates token mixing from channel-wise transformation, typically with normalization and residual connections (Yu et al., 2021, Yu et al., 2022). In this formulation, attention is only one possible token mixer among many; pooling, convolution, random mixing, identity mapping, deformable convolution, static-key attention, and polynomial operators have all been instantiated within the same scaffold. The concept has consequently become a broad design template for hierarchical vision backbones and, by extension, for medical imaging, reinforcement learning, event-based perception, OCR, digital-twin forecasting, and large-scale recommender systems (Tran et al., 1 Jan 2026, Kim et al., 2023).

1. Conceptual definition and historical framing

The foundational MetaFormer claim is that the general Transformer-like block structure can matter more than the particular token mixer placed inside it. In its canonical form, an input image $I$ is embedded into a token sequence $X$, and each block applies token mixing followed by a channel MLP:

$X = \mathrm{InputEmb}(I),$

$Y = \mathrm{TokenMixer}(\mathrm{Norm}(X)) + X,$

$Z = o(\mathrm{Norm}(Y)W_1)W_2 + Y.$

This is the formulation used to argue that a Transformer is only one instantiation of a more general template, while MLP-like and pooling-based models can be understood as parallel instantiations rather than separate families (Yu et al., 2021). A later formulation preserved the same decomposition in notation closer to vision-transformer practice:

$$Z_1 = Z + \mathrm{TokenMixer}(\mathrm{Norm}(Z)), \qquad Z_2 = Z_1 + \mathrm{FFN}(\mathrm{Norm}(Z_1)),$$

again emphasizing that the block skeleton, not self-attention alone, is the defining feature (Tran et al., 1 Jan 2026).

This reframing was sharpened by PoolFormer, which replaced attention with simple spatial pooling and still achieved competitive results on ImageNet-1K and downstream tasks. PoolFormer-M36 reached 82.1% top-1 accuracy on ImageNet-1K, surpassing DeiT-B by 0.3% and ResMLP-B24 by 1.1%, while using fewer parameters and MACs; ablations further showed that removing normalization, the channel MLP, or residual connections caused severe degradation, thereby supporting the view that MetaFormer itself is the critical structural ingredient (Yu et al., 2021).

2. Canonical block structure and baseline families

MetaFormer baselines for vision established a systematic family of models in which the macro-architecture is fixed and only the token mixer varies. These models use a hierarchical 4-stage design, with convolutional downsampling between stages; the first downsampling uses $7 \times 7$, stride 4, and later downsampling uses $3 \times 3$, stride 2 (Yu et al., 2022). The same study argued that MetaFormer “ensures solid lower bound of performance,” “works well with arbitrary token mixers,” and can deliver state-of-the-art accuracy with conventional operators rather than specially designed attention mechanisms (Yu et al., 2022).

Model	Token mixer	Reported ImageNet-1K top-1
IdentityFormer-M48	Identity mapping	80.4%
RandFormer-M48	Frozen random mixing	81.4%
ConvFormer-S18	Depthwise separable convolution	83.0%
CAFormer-B36	Convolution in lower stages, self-attention in higher stages	85.5%

These results are notable because IdentityFormer performs without meaningful token mixing, and RandFormer improves on it despite using a frozen random matrix as mixer (Yu et al., 2022). ConvFormer, a pure CNN-style MetaFormer using inverted separable convolution with kernel size 7 and expansion ratio 2, was reported to outperform ConvNeXt; CAFormer, which uses depthwise separable convolutions in the bottom two stages and vanilla self-attention in the top two, reached 85.5% top-1 at $224^2$ resolution under normal supervised training without external data or distillation (Yu et al., 2022).

The same baseline work also introduced StarReLU, a modified squared-ReLU activation defined as $s \cdot (\mathrm{ReLU}(x))^2 + b$. It was reported to reduce activation FLOPs by 71% compared with GELU while slightly improving performance in ConvFormer-S18 and CAFormer-S18 (Yu et al., 2022). This is consistent with a broader MetaFormer tendency: once the block scaffold is treated as primary, mixer choice, activation choice, and stage allocation become modular design variables rather than identity-defining commitments.

3. Token-mixer pluralism and the expansion of the design space

Subsequent work treated MetaFormer as a controlled backbone template for testing increasingly heterogeneous token mixers. SPANet kept MetaFormer stage layouts and embedding dimensions but replaced the mixer with SPAM, a frequency-domain module designed to balance low- and high-frequency representations through spectral masking. The resulting SPANet-S, SPANet-M, and SPANet-B achieved 83.1%, 83.5%, and 84.0% top-1 on ImageNet-1K, while SPANet-S and SPANet-M reached 45.4 and 46.2 mIoU on ADE20K (Yun et al., 2023). In that line of work, MetaFormer functions as a backbone design pattern whose mixer slot can be replaced by a frequency-aware operator without altering the rest of the model.

A separate line of work examined how much of attention’s internal parameterization is actually necessary. “Static Key Attention in Vision” replaced dynamically computed keys with trainable static keys, and its convolutional variant, CSKA, was integrated into a MetaFormer backbone as an intermediate-stage mixer. On ImageNet-1K, CSKAFormer-S18 slightly exceeded CAFormer-S18, 83.4 versus 83.3, while CSKAFormer-B36 slightly exceeded CAFormer-B36, 85.7 versus 85.5; ablations further indicated that placing CSKA in stage 2 of a 4-stage hierarchy was most effective (Hu et al., 2024). This suggests that MetaFormer can accommodate intermediate operators that lie between depthwise convolution and full self-attention in both inductive bias and complexity.

The MetaFormer scaffold has also been used to probe the necessity of conventional nonlinearities. PolyNeXt replaced standard activations in MLPs, convolutions, and attention with Hadamard-product-based polynomial modules while preserving a four-stage MetaFormer-style hierarchy. APolyNeXt-L reached 85.2% on ImageNet-1K, matching CAFormer-M36, and CPolyNeXt-S reached 50.6 mIoU on ADE20K, exceeding ConvFormer-S18 and CAFormer-S18 baselines in that setting (Wang et al., 20 May 2026). The paper’s ablation that replacing the Hadamard product with addition caused a catastrophic $X$0 drop on Tiny CPolyNeXt further reinforced the claim that MetaFormer is a modular structural template rather than a commitment to any single source of nonlinearity (Wang et al., 20 May 2026).

Efficiency-oriented reinterpretations follow the same pattern. MicroViT explicitly adopts a multi-stage MetaFormer architecture for edge devices, using depth-wise convolution in early stages and Efficient Single Head Attention (ESHA) in the final stage. Its abstract reports 3.6 faster inference speed and 40% higher efficiency than the MobileViT series, with the model framed throughout as a stage-wise MetaFormer instantiation rather than a standard ViT (Setyawan et al., 9 Feb 2025).

4. Medical imaging and biomedical adaptation

Medical imaging papers frequently use MetaFormer as a way to decouple the general encoder scaffold from the locality or globality demanded by specific biomedical tasks. DeformableFormer, for pre-staining endoscopic ultrasound-guided biopsy classification in pancreatic disease, replaced the TokenMixer with deformable convolution to better handle sparse, spatially localized tissue fragments. On a Nagoya University dataset of 145 available and 28 unavailable images, with 28-fold cross-validation and class-balanced augmentation, the proposed model achieved 88.44% accuracy, 93.10% precision, 93.10% recall, and 64.29% specificity, outperforming medical specialists, ResNet34, contrastive learning, and PoolFormer; the specificity improvement was particularly emphasized because it measures correct identification of clinically critical unavailable samples (Kurami et al., 2023).

In medical segmentation, several papers use MetaFormer to replace or lighten conventional transformer encoders. MFEnNet incorporates MetaFormer only in the encoder of a U-Net backbone, replacing self-attention with pooling-based token mixing, using Swish in the FFN and spatial pyramid pooling at the bottleneck. It reports IoU $X$1, Dice $X$2 on CASIA Iris Interval and IoU $X$3, Dice $X$4 on ISBI 2016 skin lesion segmentation, at 11.14M parameters and 17.13 GFLOPs (Tran et al., 1 Jan 2026). M$X$5UNet combines a MetaFormer-based encoder with a Multi-scale Upsampling decoder block for polyp segmentation, reaching $X$6 and $X$7 on Kvasir-SEG and showing monotonic gains in ablations as repeated upsampling and MU blocks are added (Trinh et al., 2023).

MetaSeg extends the MetaFormer idea beyond the backbone into the decoder. It uses a CNN-based MSCAN encoder for local information and MetaFormer-based decoder blocks, called Global Meta Blocks, with Channel Reduction Attention for efficient global context aggregation. Reported results include 42.4 mIoU on ADE20K, 80.1 mIoU on Cityscapes, and 39.7 mIoU on COCO-Stuff for MetaSeg-T at 5.5 GFLOPs, as well as 82.78% DSC on the Synapse medical benchmark for MetaSeg-B (Kang et al., 2024). A plausible implication is that MetaFormer’s abstraction remains useful even when token mixing is reassigned from encoder to decoder, provided the task benefits from explicitly splitting local detail extraction from global context recovery.

The first systematic comparison of token mixers for medical imaging reinforced this task dependence. Across five classification datasets and three segmentation datasets, the study found that for classification, low-complexity mixers such as grouped convolution, pooling, or even identity are often sufficient, whereas for segmentation the local inductive bias of convolutional token mixers is essential; grouped convolution emerged as the preferred choice because it reduces runtime and parameter count while channel-MLPs already supply cross-channel interaction (Keuth et al., 7 Oct 2025).

Biomedical forecasting provides another adaptation. A high-throughput digital twin framework for neurite deterioration uses a SimVP-like MetaFormer temporal translator with gated spatiotemporal attention. Given 10 input frames, it predicts the next 20, and reports average errors of 1.9641% on synthetic deterioration data and 6.0339% on experimental data (Qian et al., 2024). Here MetaFormer is neither a classifier nor a segmenter, but a spatiotemporal predictor whose mixer is chosen to model long-range temporal dependencies and fine morphological change.

5. Sequential, event-based, and industrial systems

Outside biomedical imaging, MetaFormer has been specialized to sequential decision making, neuromorphic vision, OCR, and recommender systems. Decision ConvFormer replaced the attention mechanism of Decision Transformer with a 1D causal depthwise convolution inside a MetaFormer block, motivated by the claim that offline RL trajectories are governed primarily by local Markovian dependencies rather than dense long-range interactions. On D4RL MuJoCo and AntMaze, the paper reports state-of-the-art performance with fewer resources; on hopper-medium, Decision ConvFormer reached 92.5 versus 68.4 for Decision Transformer, and on Atari the mean score was 86.2 for DC versus 62.2 for DT, while training time dropped from 764 s to 193 s and GPU memory from 3.7 GB to 1.8 GB (Kim et al., 2023).

Event-based perception has produced two distinct MetaFormer adaptations. SDTrack introduces a Spiking MetaFormer backbone for event-based tracking, combining SNN Conv Blocks with SNN Transformer Blocks in a spike-driven pipeline. On FE108, SDTrack-Tiny achieved 59.0 AUC / 91.3 PR with 19.61M parameters and 8.16 mJ energy, while SDTrack-Base reached 59.9 AUC / 91.5 PR with 107.26M parameters and 30.52 mJ (Shan et al., 9 Mar 2025). EMF, by contrast, uses MetaFormer-like convolutional components tailored to event volumes for real-time traffic object detection; on Gen1 it achieved 49.1 mAP with 9.1 ms inference time and is presented as the fastest DNN-based architecture in the domain, outperforming RVT-B in both accuracy and speed (Khan et al., 5 Apr 2025). In both cases, the MetaFormer idea is retained while the token mixer is redesigned around event sparsity and temporal progression rather than inherited directly from RGB transformers.

Industrial document and recommendation systems show the same decoupling. PP-OCRv6 reorganizes its OCR stack around a unified MetaFormer-style primitive, LCNetV4Block, which explicitly separates depthwise spatial token mixing from pointwise channel mixing and applies structural reparameterization only to the token mixer. PP-OCRv6_medium reports 83.2% recognition accuracy and 86.2% detection Hmean, while the tiny tier achieves 3.9$X$8 faster inference than PP-OCRv5_mobile on Intel Xeon CPU (Zhang et al., 11 Jun 2026). RankUp, in recommender systems, keeps a MetaFormer backbone but addresses representation collapse using randomized permutation splitting, multi-embedding, a global token, crossed pretrained embedding tokens, and task-specific token decoupling. It was fully deployed in production across Weixin Video Accounts, Official Accounts, and Moments, yielding GMV improvements of 3.41%, 4.81%, and 2.21%, respectively (Chen et al., 20 Apr 2026). In that setting, MetaFormer appears not as a vision backbone but as a token-mixer-plus-FFN principle for large-scale sequential sparse-feature ranking.

6. Nomenclature, distinct usages, and recurrent misconceptions

Not every paper titled “MetaFormer” uses the term in the same sense. “MetaFormer: A Unified Meta Framework for Fine-Grained Recognition” uses the name for a hybrid vision-transformer framework that fuses vision tokens with meta-information tokens such as spatio-temporal priors, attributes, and text descriptions. It reports 92.3% on CUB-200-2011 and 92.7% on NABirds, and on iNaturalist 2017 and 2018 states that adding meta-information exceeds current state of the art by 5.9% and 5.3%, respectively (Diao et al., 2022). In this usage, “meta” refers to auxiliary information rather than to a generalized token-mixing architecture.

Similarly, the ASD classification model called METAFormer stands for “Multi-Atlas Enhanced Transformer.” It uses three atlas-specific transformer encoders over vectorized functional connectomes from AAL, CC200, and DOS160, together with masked reconstruction pretraining on the same training folds. On ABIDE I it reports 83.7% accuracy and 0.832 AUC, compared with 0.628 accuracy and 0.623 AUC without pretraining (Mahler et al., 2023). This is a transformer framework with multi-atlas fusion and self-supervised pretraining, not the token-mixer-agnostic MetaFormer abstraction of the vision-backbone literature.

Two misconceptions are therefore recurrent. The first is that MetaFormer is synonymous with self-attention; the baseline and application literature repeatedly contradicts this by showing strong performance with pooling, convolution, deformable convolution, static-key attention, or polynomial mixing (Yu et al., 2021, Yu et al., 2022). The second is that MetaFormer denotes a single standardized model; the literature instead uses it either as a general architectural scaffold or as a task-specific model name with different semantics (Diao et al., 2022, Mahler et al., 2023). This suggests that “MetaFormer” is best understood as a family resemblance term whose dominant contemporary meaning is architectural abstraction, but whose local meaning still depends on the paper and domain.