CAFormer: Hybrid Conv-Attention Vision Backbone

• The paper demonstrates that CAFormer, a key MetaFormer instantiation, splits token mixing by using depthwise separable convolutions early and vanilla self-attention in later stages.
• CAFormer’s four-stage hierarchical design efficiently downscales features with novel components like StarReLU to reduce computational cost while maintaining high accuracy.
• CAFormer achieves state-of-the-art ImageNet-1K performance and improved interpretability via Threshold-Grad-CAM, underscoring its practical versatility in vision tasks.

CAFormer most commonly denotes a hybrid MetaFormer vision backbone introduced in “MetaFormer Baselines for Vision,” where depthwise separable convolutions are used in the bottom two stages and vanilla self-attention in the top two stages, i.e., $$T=(\mathrm{Conv}, \mathrm{Conv}, \mathrm{Attn}, \mathrm{Attn})$$ (Yu et al., 2022). In later literature, the same name has also been reused for a transformer-based mechanism for combinatorial auctions (Pham et al., 31 Jan 2025) and, with different capitalization, for a causal time-series framework titled “Caformer” (Zhang et al., 2024). The term therefore refers to distinct architectures across domains; within computer vision, however, CAFormer specifically designates a hierarchical MetaFormer backbone that couples local convolutional token mixing with global attention-based token mixing.

1. MetaFormer provenance and the CAFormer design rationale

CAFormer was introduced as one of the principal instantiations in the MetaFormer framework, which abstracts the Transformer into a generic block structure without committing to a specific token mixer. Given an input image $I$, MetaFormer first forms tokens,

$X = \mathrm{InputEmbedding}(I),$

and then repeatedly applies

$$X' = X + \mathrm{TokenMixer}\!\left(\mathrm{Norm_1}(X)\right),$$

$$X'' = X' + \sigma\!\left(\mathrm{Norm_2}(X')W_1\right)W_2.$$

Within this template, CAFormer is defined by stage-wise heterogeneity in the token mixer rather than by a new block skeleton: convolutional token mixing is used early, and self-attention is used late (Yu et al., 2022).

The paper positions CAFormer as evidence for two related claims. First, much of modern vision-backbone performance comes from the MetaFormer scaffold itself rather than from any single specialized mixer. Second, “old-fashioned” operators can remain competitive when inserted into that scaffold appropriately. In that framing, CAFormer operationalizes a division of labor: early stages use depthwise separable convolutions when token counts are large and local inductive bias is useful, whereas late stages use vanilla self-attention when token counts are smaller and global reasoning is more useful (Yu et al., 2022).

This placement is also part of a broader MetaFormer argument. The same study reports that IdentityFormer, which uses identity mapping as the token mixer, achieves $>80\%$ accuracy on ImageNet-1K, and RandFormer, which uses a frozen random matrix, achieves $>81\%$. CAFormer is therefore not presented as an isolated architecture, but as a particularly strong point in a wider design space where the MetaFormer interface permits interchangeable token mixers while retaining competitive performance (Yu et al., 2022).

2. Architectural specification of the vision backbone

CAFormer adopts a standard 4-stage hierarchical architecture of the kind used by ResNet, PoolFormer, and Swin. Each stage $i$ has $L_i$ blocks and feature dimension $D_i$. Downsampling is implemented by convolutions: the first downsampling uses kernel size $I$0 and stride $I$1, and the remaining three use kernel size $I$2 and stride $I$3. The classifier head is

$I$4

Its defining stage allocation is explicit: $I$5 Thus, the first two stages use depthwise separable convolutions and the last two stages use vanilla self-attention (Yu et al., 2022).

Variant	Channels $I$6	Depths $I$7
CAFormer-S18	$I$8	$I$9
CAFormer-S36	$X = \mathrm{InputEmbedding}(I),$0	$X = \mathrm{InputEmbedding}(I),$1
CAFormer-M36	$X = \mathrm{InputEmbedding}(I),$2	$X = \mathrm{InputEmbedding}(I),$3
CAFormer-B36	$X = \mathrm{InputEmbedding}(I),$4	$X = \mathrm{InputEmbedding}(I),$5

For the convolutional token mixer, the paper uses the inverted separable convolution from MobileNetV2,

$X = \mathrm{InputEmbedding}(I),$6

with kernel size $X = \mathrm{InputEmbedding}(I),$7 and expansion ratio $X = \mathrm{InputEmbedding}(I),$8. The attention stages use vanilla self-attention from the original Transformer as a global token mixer (Yu et al., 2022).

A second architectural component tied closely to CAFormer is the activation function StarReLU. The paper first discusses

$X = \mathrm{InputEmbedding}(I),$9

and then defines

$$X' = X + \mathrm{TokenMixer}\!\left(\mathrm{Norm_1}(X)\right),$$0

where $$X' = X + \mathrm{TokenMixer}\!\left(\mathrm{Norm_1}(X)\right),$$1 and $$X' = X + \mathrm{TokenMixer}\!\left(\mathrm{Norm_1}(X)\right),$$2 are scalar scale and bias terms, shared across channels and optionally learnable. The motivation is computational as well as empirical: GELU is described as about $$X' = X + \mathrm{TokenMixer}\!\left(\mathrm{Norm_1}(X)\right),$$3 FLOPs, while StarReLU uses $$X' = X + \mathrm{TokenMixer}\!\left(\mathrm{Norm_1}(X)\right),$$4 FLOPs, or $$X' = X + \mathrm{TokenMixer}\!\left(\mathrm{Norm_1}(X)\right),$$5 if only one of $$X' = X + \mathrm{TokenMixer}\!\left(\mathrm{Norm_1}(X)\right),$$6 or $$X' = X + \mathrm{TokenMixer}\!\left(\mathrm{Norm_1}(X)\right),$$7 is used, corresponding to about a $$X' = X + \mathrm{TokenMixer}\!\left(\mathrm{Norm_1}(X)\right),$$8 reduction in activation FLOPs relative to GELU (Yu et al., 2022).

The same study reports that replacing StarReLU in CAFormer-S18 with ReLU reduces top-1 accuracy from $$X' = X + \mathrm{TokenMixer}\!\left(\mathrm{Norm_1}(X)\right),$$9 to $$X'' = X' + \sigma\!\left(\mathrm{Norm_2}(X')W_1\right)W_2.$$0, while Squared ReLU and GELU each yield $$X'' = X' + \sigma\!\left(\mathrm{Norm_2}(X')W_1\right)W_2.$$1. This places StarReLU not merely as a cheaper substitute, but as the default activation actually used in the published CAFormer configurations (Yu et al., 2022).

3. Supervised training regime, empirical performance, and backbone behavior

The published ImageNet-1K results for CAFormer are obtained under normal supervised training, without external data or distillation. The training setup is unusually explicit: ImageNet-1K at resolution $$X'' = X' + \sigma\!\left(\mathrm{Norm_2}(X')W_1\right)W_2.$$2, $$X'' = X' + \sigma\!\left(\mathrm{Norm_2}(X')W_1\right)W_2.$$3 epochs, LAMB optimizer for CAFormer, training batch size $$X'' = X' + \sigma\!\left(\mathrm{Norm_2}(X')W_1\right)W_2.$$4, and fine-tuning at $$X'' = X' + \sigma\!\left(\mathrm{Norm_2}(X')W_1\right)W_2.$$5 for $$X'' = X' + \sigma\!\left(\mathrm{Norm_2}(X')W_1\right)W_2.$$6 epochs with batch size $$X'' = X' + \sigma\!\left(\mathrm{Norm_2}(X')W_1\right)W_2.$$7 and EMA. Regularization and augmentation include RandAugment, Mixup, CutMix, Random Erasing, weight decay, Label Smoothing, and Stochastic Depth; repeated augmentation, external data, and distillation are not used. The paper notes that CAFormer slightly drops in performance under AdamW with very large batch size and therefore switches to LAMB as a more large-batch-friendly optimizer (Yu et al., 2022).

Under that regime, CAFormer-B36 is reported to achieve $$X'' = X' + \sigma\!\left(\mathrm{Norm_2}(X')W_1\right)W_2.$$8 top-1 accuracy on ImageNet-1K at $$X'' = X' + \sigma\!\left(\mathrm{Norm_2}(X')W_1\right)W_2.$$9, described as a new record under normal supervised training without external data or distillation. The paper also states that this surpasses MViTv2-L at $>80\%$0, with $>80\%$1 fewer parameters and $>80\%$2 fewer MACs. At $>80\%$3, CAFormer-B36 reaches $>80\%$4 (Yu et al., 2022).

Across smaller and medium scales, the reported top-1 accuracies are $>80\%$5 for CAFormer-S18, $>80\%$6 for CAFormer-S36, and $>80\%$7 for CAFormer-M36. The same tables compare these against contemporary backbones: CAFormer-S18 exceeds ConvFormer-S18 ($>80\%$8), Swin-T ($>80\%$9), ConvNeXt-T ($>81\%$0), iFormer-S ($>81\%$1), and CoAtNet-0 ($>81\%$2); CAFormer-S36 exceeds ConvFormer-S36 ($>81\%$3), Swin-S ($>81\%$4), ConvNeXt-S ($>81\%$5), and CoAtNet-1 ($>81\%$6); CAFormer-M36 exceeds ConvFormer-M36 ($>81\%$7), Swin-B ($>81\%$8), ConvNeXt-B ($>81\%$9), MViTv2-B ($i$0), CoAtNet-2 ($i$1), and iFormer-L ($i$2) (Yu et al., 2022).

The paper further states that CAFormer compares favorably on throughput and robustness, and that it performs strongly on COCO detection and segmentation as well as ADE20K segmentation. This suggests that its conv-attention stage split functions not merely as an ImageNet classification design choice, but as a general-purpose backbone configuration within the MetaFormer family (Yu et al., 2022).

4. Interpretability studies and the Grad-CAM failure mode

A later study examined CAFormer in the context of interpretability alignment with human visual strategies using ClickMe heatmaps and Spearman rank correlation as the metric. The evaluated model was CAFormer_s18 together with a DCLS variant, CAFormer_s18_dcls, obtained by drop-in replacing the model’s depthwise separable convolutions with Dilated Convolution with Learnable Spacings (DCLS) layers. CAFormer is a central case in that study because vanilla Grad-CAM is reported to fail on it, producing heatmaps that are essentially random or uninformative (Chamas et al., 2024).

For standard Grad-CAM, the appendix gives

$i$3

$i$4

where $i$5 is the score for target class $i$6, $i$7 is the feature map of channel $i$8, $i$9 is the activation at spatial location $L_i$0, $L_i$1 is the channel importance weight, and $L_i$2 is the number of spatial locations. The authors attribute the CAFormer failure mode to the behavior of internal activations, especially StarReLU, whose learned activation dynamics can cause the standard Grad-CAM summation followed by ReLU to suppress useful signals (Chamas et al., 2024).

To address this, the paper introduces Threshold-Grad-CAM. Its construction is

$L_i$3

$L_i$4

$L_i$5

$L_i$6

This amounts to applying ReLU before summation rather than after, normalizing, and thresholding weak activations (Chamas et al., 2024).

The quantitative shift for CAFormer is substantial. CAFormer_s18 has Top-1 accuracy $L_i$7, Grad-CAM score $L_i$8, and Threshold-Grad-CAM score $L_i$9. CAFormer_s18_dcls has Top-1 accuracy $D_i$0, Grad-CAM score $D_i$1, and Threshold-Grad-CAM score $D_i$2. On this evidence, the study concludes that vanilla Grad-CAM suggests failure, Threshold-Grad-CAM reveals a meaningful alignment, and DCLS improves CAFormer’s Threshold-Grad-CAM alignment from $D_i$3 to $D_i$4 (Chamas et al., 2024).

The broader significance is methodological. Low interpretability scores for CAFormer under vanilla Grad-CAM need not indicate that the model itself is poorly aligned with human attention; they may instead indicate that the explanation method is mismatched to the architecture. In that sense, CAFormer functions in the DCLS study as a stress test for explanation pipelines rather than simply as another backbone in a benchmark table (Chamas et al., 2024).

5. Relation to later MetaFormer-style work

Subsequent MetaFormer-style research has used CAFormer as a reference architecture for probing which parts of the design are essential. “Activation-Free Backbones for Image Recognition: Polynomial Alternatives within MetaFormer-Style Vision Models” replaces standard pointwise activations and softmax exponentials with Hadamard-product polynomial modules, while retaining the same modular backbone logic. Its APolyNeXt variant is the closest polynomial analogue to CAFormer because it uses PolyConv in stages 1–2 and PolyAttn in stages 3–4, mirroring CAFormer’s hybrid conv-attention structure (Wang et al., 20 May 2026).

The polynomial replacements are defined explicitly. PolyMLP is

$D_i$5

PolyConv uses a pointwise-conv to depthwise-conv to pointwise-conv structure with multiplicative fusion of a coarse dilated branch and a fine $D_i$6 branch,

$D_i$7

and PolyAttn replaces softmax attention with

$D_i$8

followed by $D_i$9 row normalization, with $I$00 (Wang et al., 20 May 2026).

The empirical comparisons are framed directly against MetaFormer baselines including CAFormer. On ImageNet-1K, APolyNeXt-L matches CAFormer-M36 at $I$01, and APolyNeXt-S exceeds CAFormer-S18, $I$02 versus $I$03. On ADE20K semantic segmentation with UperNet under the ConvNeXt recipe, CAFormer-S18 records $I$04 mIoU, while CPolyNeXt-S reaches $I$05 mIoU and APolyNeXt-S reaches $I$06 mIoU (Wang et al., 20 May 2026).

These comparisons are informative because they isolate a design question that CAFormer had left open: whether the strength of a MetaFormer hybrid is fundamentally tied to GELU/ReLU/softmax-style nonlinearities. The polynomial study argues that the principal difficulty is optimization and stability rather than representational power, and addresses it with Sigmoid-Scale residuals,

$I$07

multi-input skip connections, and a preference for deeper and narrower models (Wang et al., 20 May 2026). A plausible implication is that CAFormer’s enduring influence lies less in any single primitive than in its stage-wise allocation of local and global mixers inside the MetaFormer scaffold.

6. Reuse of the name in auctions and time-series modeling

Outside computer vision, “CAFormer” has been reused for an unrelated transformer-based neural architecture for combinatorial auctions. In “Advancing Differentiable Economics: A Neural Network Framework for Revenue-Maximizing Combinatorial Auction Mechanisms,” CAFormer is the paper’s transformer-style mechanism network for learning approximately incentive-compatible randomized mechanisms under combinatorial feasibility constraints. The input is a bid matrix $I$08, where rows are bidders and columns are bundles. The architecture combines exchangeable layers with stacked attention layers over item-wise, bundle-wise, and agent-wise views; the allocation matrix is constructed so that feasibility is enforced by design through a decomposition into item-to-bundle and bundle-to-agent components. Training follows a regret-penalized revenue objective, and the paper reports that CAFormer achieves negligible regret, typically around $I$09 to $I$10, while outperforming heuristic baselines in revenue; for example, it reaches revenue $I$11 in the symmetric $I$12 case versus CANet’s $I$13, and $I$14 in the symmetric $I$15 case versus CANet’s $I$16 (Pham et al., 31 Jan 2025).

The same auction architecture is then applied to cyber defense planning in “Strategic Cyber Defense via Reinforcement Learning-Guided Combinatorial Auctions.” There, CAFormer consumes host-specific valuations over bundles of defensive actions such as Analyze, Remove, and Restore, with valuations derived from reinforcement-learning Q-values. The mechanism outputs an allocation of action bundles to hosts and a payment vector, and is trained with a combined revenue-and-regret loss. In the DARPA CAGE Challenge 2 Scenario 2 setting, the paper reports that regret approaches zero after about $I$17 iterations, that truthful allocations concentrate on Remove and $I$18, and that aggregate allocation scores correlate strongly with Blue activity, with Pearson $I$19 and $I$20 (Pham et al., 13 Sep 2025).

A further homograph appears in time-series analysis as “Caformer,” short for causal Transformer. This model is unrelated to the MetaFormer backbone and instead comprises a Dependency Learner, Dynamic Learner, and Environment Learner. Its causal graph is

$I$21

and it uses back-door adjustment,

$I$22

to mitigate spurious correlation induced by environmental factors. It is evaluated on long- and short-term forecasting, imputation, classification, and anomaly detection, with reported best average accuracy $I$23 on 10 UEA classification datasets and best average F1-score $I$24 on five anomaly-detection datasets (Zhang et al., 2024).

Taken together, these usages make “CAFormer” a polysemous research term. In vision it denotes a specific conv-attention MetaFormer backbone; in automated mechanism design it denotes a transformer-based auction mechanism; and in time-series analysis the near-homograph “Caformer” denotes a causal sequence model. For technical discourse, domain qualification is therefore essential.