EfficientFormer: Mobile Vision Transformer

• EfficientFormer is a family of vision transformers that achieves MobileNet-class mobile latency while maintaining competitive ImageNet-1K accuracy through a dimension-consistent pure transformer design.
• It splits computation into a 4D partition for efficient local pooling and a 3D partition for global multi-head self-attention, minimizing reshapes and leveraging compiler-friendly operations.
• The architecture employs latency-driven slimming and hardware-aware search to optimize block scheduling and operator efficiency, ensuring strong transfer learning performance on edge devices.

EfficientFormer is a family of vision transformers introduced to achieve MobileNet-class on-device inference speed while retaining competitive ImageNet-1K accuracy through a latency-centered architectural design rather than a FLOP-centered one. The defining idea is a dimension-consistent pure transformer paradigm: most computation is carried out in a long 4D partition that avoids repeated tensor reshaping and exploits compiler-friendly operators, while global multi-head self-attention is deferred to a short 3D partition at low spatial resolution, where the token count is small and the quadratic attention cost is modest (Li et al., 2022). In subsequent literature, EfficientFormer has been used both as a strong mobile baseline for hybrid and efficient vision backbones and as a low-cost feature extractor in transfer settings such as texture analysis (Zhang et al., 2022, Scabini et al., 2024).

1. Historical position and design objective

EfficientFormer was proposed against the background that Vision Transformers (ViTs) had become accurate but remained slow on mobile and edge hardware because of architectural and operator-level inefficiencies, including quadratic attention cost in token count $N$, large hidden dimensions, memory traffic caused by repeated reshapes and transposes between $B \times C \times H \times W$ and $B \times N \times d$, and unfavorable operator choices for mobile compilers such as large-kernel patch embedding convolutions, dynamic normalizations, and frequent embedding-dimension changes across stages (Li et al., 2022). The central question posed by the original work was whether a transformer could run as fast as MobileNet while maintaining strong performance.

The answer offered by EfficientFormer was not to replace attention with an approximation, nor to build a MobileNet hybrid with transformer add-ons, but to reorganize the backbone so that latency on the target device becomes the primary design criterion. The fastest reported model, EfficientFormer-L1, achieves $79.2\%$ top-1 accuracy on ImageNet-1K with $1.6$ ms latency on iPhone 12 compiled with CoreML, matching the reported latency of MobileNetV2$\times 1.4$ while improving accuracy from $74.7\%$ to $79.2\%$; the largest reported model, EfficientFormer-L7, reaches $83.3\%$ at $7.0$ ms (Li et al., 2022). Later survey literature presents EfficientFormer as an instance of hardware-aware lightweight transformer design for edge deployment, emphasizing that its speedup derives from dimension consistency and latency-driven slimming rather than from changing the fundamental $B \times C \times H \times W$0 self-attention complexity (Samson, 5 Jan 2026).

A recurrent point of clarification is that EfficientFormer is described as a pure transformer in the original paper despite extensive use of 4D conv-style operators. In the paper’s terminology, “pure transformer” means the architecture does not rely on MobileNet blocks; its 4D partition implements MetaFormer-like transformer skeletons with pooling token mixing, $B \times C \times H \times W$1 convolutions, BatchNorm, and GeLU, while the final 3D partition contains conventional transformer blocks with global MHSA (Li et al., 2022). This resolves an apparent contradiction between transformer identity and conv-style implementation.

2. Dimension-consistent architecture

The architecture is split into two long partitions to minimize reshape overhead and align computation with compiler and accelerator preferences. The 4D partition, denoted $B \times C \times H \times W$2, operates entirely on $B \times C \times H \times W$3 tensors and uses BatchNorm and GeLU with a lightweight local pooling token mixer. The 3D partition, denoted $B \times C \times H \times W$4, operates on flattened tokens $B \times C \times H \times W$5 with LayerNorm, linear projections, and global MHSA. The transition from 4D to 3D occurs only once, near the end of the network, after token resolution has been reduced to $B \times C \times H \times W$6 so that $B \times C \times H \times W$7 is small (Li et al., 2022).

The macro-architecture uses four stages with downsampling between stages via $B \times C \times H \times W$8 stride-2 convolutional patch embeddings. The input first passes through a convolutional stem of two $B \times C \times H \times W$9 stride-2 layers to reach $B \times N \times d$0, and each subsequent stage transition halves spatial resolution and increases channels with another $B \times N \times d$1 stride-2 convolution. This choice directly reflects latency profiling: the paper reports that replacing large-kernel non-overlapping patch embedding with the conv stem reduces latency by about $B \times N \times d$2 and improves accuracy by $B \times N \times d$3 in an L3-like configuration, from $B \times N \times d$4 ms/$B \times N \times d$5 to $B \times N \times d$6 ms/$B \times N \times d$7 (Li et al., 2022).

Within the 4D partition, an $B \times N \times d$8 block has the structure: Pool token mixer (local pooling, e.g. $B \times N \times d$9, stride 1) followed by $79.2\%$0 convolution + BN + GeLU, then $79.2\%$1 convolution + BN, and a residual addition. Within the 3D partition, an $79.2\%$2 block applies LN, linear QKV projection, MHSA with learned attention bias $79.2\%$3, linear projection, residual addition, then LN and a standard FFN with expansion ratio $79.2\%$4 (Li et al., 2022). The overall consequence is that most of the network remains in a representation regime where BN can be folded at inference and 4D kernels are well optimized, while attention is reserved for the smallest-resolution stage.

The attention itself follows the standard formulation. Given tokens $79.2\%$5, EfficientFormer uses

$79.2\%$6

$79.2\%$7

$79.2\%$8

with $79.2\%$9 heads and head dimension $1.6$0 (Li et al., 2022). The key point is not a new attention operator, but the placement of standard global MHSA only where its latency is acceptable.

3. Latency-driven slimming and hardware-aware search

EfficientFormer’s model family is derived through a latency-driven slimming procedure that directly targets mobile-device latency rather than FLOPs. The supernet defines, for each block, a MetaPath choice: in stages 1–2, $1.6$1; in stages 3–4, $1.6$2. Stage widths $1.6$3, depths $1.6$4, and the locations of the last $1.6$5 blocks are part of the design space (Li et al., 2022).

Architecture parameters $1.6$6 are learned once by Gumbel-Softmax sampling:

$1.6$7

with $1.6$8 and temperature $1.6$9 (Li et al., 2022). A latency lookup table is then built on the target device, specifically iPhone 12 with CoreML, for $\times 1.4$0 and $\times 1.4$1 blocks at candidate widths. Slimming proceeds greedily under a latency budget by selecting among three actions: removing the least important block by setting it to Identity, removing the first $\times 1.4$2 block, or reducing the width of the least important stage by 16 channels. Each step uses expected latency from the lookup table and an accuracy-drop proxy derived from $\times 1.4$3 to optimize improvement in accuracy per millisecond (Li et al., 2022).

This procedure is central to EfficientFormer’s identity. The original paper argues that optimizing FLOPs or GPU throughput correlates poorly with true mobile latency, and the later edge-device survey reiterates this point, stating that EfficientFormer’s hardware-aware NAS yields models that are reported as $\times 1.4$4–$\times 1.4$5 faster than FLOP-optimized designs at equivalent accuracy (Samson, 5 Jan 2026). A plausible implication is that EfficientFormer should be understood less as a new attention mechanism than as a mobile systems co-design exercise in which operator choice, tensor layout, normalization, and block scheduling are jointly optimized against a concrete deployment stack.

The ablation studies reinforce this interpretation. In the original experiments, BN in the 4D partition outperforms GN and LN in latency-sensitive settings: on an L3-like configuration, BN yields $\times 1.4$6 ms/$\times 1.4$7, GN yields $\times 1.4$8 ms/$\times 1.4$9, and LN yields $74.7\%$0 ms/$74.7\%$1. GeLU is reported as efficient on CoreML and much faster than HardSwish, with the same configuration showing $74.7\%$2 ms/$74.7\%$3 for GeLU versus $74.7\%$4 ms/$74.7\%$5 for HardSwish (Li et al., 2022). The design principle is therefore operator realism: fast models emerge from compiler-compatible primitives and stable tensor formats, not from abstract complexity arguments alone.

4. Variants, scaling behavior, and empirical performance

The published family includes EfficientFormer-L1, EfficientFormer-L3, and EfficientFormer-L7 as final variants with explicit architecture specifications, parameter counts, FLOPs, and latency measurements on iPhone 12, CPU, and A100 TensorRT (Li et al., 2022). All reported ImageNet-1K experiments use $74.7\%$6 inputs, AdamW, a 5-epoch warmup, cosine decay, and the DeiT training recipe; the teacher for distillation is RegNetY-16GF at $74.7\%$7 top-1 (Li et al., 2022).

Variant	Params / FLOPs	ImageNet-1K / iPhone 12 NPU
EfficientFormer-L1	12.3M / 1.3 GMACs	79.2%, 1.6 ms
EfficientFormer-L3	31.3M / 3.9 GMACs	82.4%, 3.0 ms
EfficientFormer-L7	82.1M / 10.2 GMACs	83.3%, 7.0 ms

EfficientFormer-L1 uses a stem with channels $74.7\%$8, then stage widths $74.7\%$9, $79.2\%$0, $79.2\%$1, and $79.2\%$2, with stage depths $79.2\%$3, $79.2\%$4, $79.2\%$5, and a final stage composed of $79.2\%$6 followed by $79.2\%$7. EfficientFormer-L3 scales this to widths $79.2\%$8, then $79.2\%$9, $83.3\%$0, $83.3\%$1, $83.3\%$2, with stage depths $83.3\%$3, $83.3\%$4, $83.3\%$5, and a final stage of $83.3\%$6 then $83.3\%$7. EfficientFormer-L7 uses widths $83.3\%$8, then $83.3\%$9, $7.0$0, $7.0$1, $7.0$2, with stage depths $7.0$3, $7.0$4, $7.0$5, and a last stage of $7.0$6 and no $7.0$7 blocks in stage 4 (Li et al., 2022). All final models use eight attention heads in $7.0$8 and $7.0$9 per head.

The speed–accuracy tradeoff is central to the model family’s positioning. On iPhone 12 NPU, EfficientFormer-L1 matches MobileNetV2$B \times C \times H \times W$00 latency at $B \times C \times H \times W$01 ms while improving top-1 accuracy by $B \times C \times H \times W$02; EfficientFormer-L3 reaches $B \times C \times H \times W$03 at $B \times C \times H \times W$04 ms, outperforming EfficientNet-B0 at $B \times C \times H \times W$05 and $B \times C \times H \times W$06 ms; EfficientFormer-L7 reaches $B \times C \times H \times W$07 at $B \times C \times H \times W$08 ms, compared with EfficientNet-B5 at $B \times C \times H \times W$09 and $B \times C \times H \times W$10 ms (Li et al., 2022). Against transformer baselines, EfficientFormer-L3 is reported as $B \times C \times H \times W$11 over DeiT-Small and about $B \times C \times H \times W$12 faster on NPU, and $B \times C \times H \times W$13 over PoolFormer-S36 while about $B \times C \times H \times W$14 faster (Li et al., 2022).

The original paper also reports downstream transfer results. On COCO 2017 with Mask R-CNN and a $B \times C \times H \times W$15 schedule at $B \times C \times H \times W$16, EfficientFormer-L1 achieves $B \times C \times H \times W$17 and $B \times C \times H \times W$18, EfficientFormer-L3 reaches $B \times C \times H \times W$19, and EfficientFormer-L7 reaches $B \times C \times H \times W$20 (Li et al., 2022). On ADE20K with Semantic FPN and input size $B \times C \times H \times W$21, the corresponding mIoU values are $B \times C \times H \times W$22, $B \times C \times H \times W$23, and $B \times C \times H \times W$24 (Li et al., 2022). These figures indicate that the architecture is not restricted to mobile ImageNet classification, although the original optimization target remains latency on mobile inference hardware.

5. Comparison with FcaFormer and the question of efficiency

EfficientFormer later served as a salient baseline in work on hybrid and efficient transformer design. In particular, FcaFormer classifies EfficientFormer as a recent hybrid ConvNet–Transformer baseline and contrasts EfficientFormer’s prevailing “efficiency via sparsification” trend with its own strategy of densifying cross-block attention through Forward Cross Attention (Zhang et al., 2022). The comparison is informative because it isolates two different notions of efficiency: minimizing latency through dimension consistency and device-aware slimming in EfficientFormer, versus improving parameter efficiency through additional cross-level token interaction in FcaFormer.

Two ImageNet-1K head-to-head comparisons are reported. At the lower-compute regime, FcaFormer-L1 without knowledge distillation achieves $B \times C \times H \times W$25 top-1 with $B \times C \times H \times W$26M parameters and $B \times C \times H \times W$27G MACs, whereas EfficientFormer-L1 with knowledge distillation is reported at $B \times C \times H \times W$28 with $B \times C \times H \times W$29M parameters and $B \times C \times H \times W$30G MACs; a non-distilled reimplementation of EfficientFormer-L1 is reported at $B \times C \times H \times W$31 with the same $B \times C \times H \times W$32M parameters and $B \times C \times H \times W$33G MACs (Zhang et al., 2022). At the higher regime, FcaFormer-L2 without knowledge distillation achieves $B \times C \times H \times W$34 with $B \times C \times H \times W$35M parameters and about $B \times C \times H \times W$36G MACs, compared with EfficientFormer-L3 with knowledge distillation at $B \times C \times H \times W$37, $B \times C \times H \times W$38M parameters, and $B \times C \times H \times W$39G MACs (Zhang et al., 2022).

These comparisons should be read carefully. They do not show that EfficientFormer is uniformly inferior, because the FcaFormer paper does not report EfficientFormer latency or memory in those experiments, and the central claim of EfficientFormer is specifically about on-device latency, not only accuracy per parameter or accuracy per MAC (Zhang et al., 2022, Li et al., 2022). What they do show is that, under fixed ImageNet classification settings and especially without distillation, cross-block token reuse can improve the accuracy–parameter tradeoff relative to EfficientFormer. A plausible implication is that “efficient transformer” is not a single optimization axis: EfficientFormer prioritizes deployment latency and operator efficiency, whereas FcaFormer emphasizes richer token interaction with only a modest additional compute cost.

This comparison also clarifies a common misconception. EfficientFormer does not primarily seek to reduce attention complexity by sparse or local attention. Instead, it keeps standard global MHSA, but restricts it to the lowest-resolution stage and reshapes the rest of the network around mobile-friendly execution (Li et al., 2022). FcaFormer, by contrast, increases token interaction density across blocks while holding output token length fixed (Zhang et al., 2022). The two designs therefore occupy different points in the design space and are only partially comparable.

6. Transfer use, deployment practice, and limitations

In the comparative survey of ViTs for texture analysis, EfficientFormer is evaluated as a low-cost alternative for fixed-feature transfer learning. The study uses EfficientFormer-L1 and EfficientFormer-L3 from timm v0.6.7 with supervised ImageNet-1K pre-training, removes the classification head, freezes the backbone, extracts the final feature representation, and trains three linear classifiers—KNN with $B \times C \times H \times W$40, LDA with least-squares solver and Ledoit–Wolf shrinkage, and linear SVM with $B \times C \times H \times W$41—reporting average accuracy across the three heads (Scabini et al., 2024). In that study, EfficientFormer-L1 produces a $B \times C \times H \times W$42-dimensional feature vector with $B \times C \times H \times W$43 GFLOPs and $B \times C \times H \times W$44M parameters, while EfficientFormer-L3 produces a $B \times C \times H \times W$45-dimensional feature vector with $B \times C \times H \times W$46 GFLOPs and $B \times C \times H \times W$47M parameters (Scabini et al., 2024).

The survey reports that EfficientFormer-L1 exceeds ResNet50 throughput on both CPU and GPU for all tested batch sizes. On CPU, L1 achieves $B \times C \times H \times W$48 images/s at batch size 1, $B \times C \times H \times W$49 at batch size 8, and $B \times C \times H \times W$50 at batch size 16; on GPU, it reaches $B \times C \times H \times W$51, $B \times C \times H \times W$52, and $B \times C \times H \times W$53 images/s at batch sizes 1, 8, and 16, respectively (Scabini et al., 2024). Accuracy is competitive for its cost: for example, EfficientFormer-L3 achieves $B \times C \times H \times W$54 on DTD and $B \times C \times H \times W$55 on FMD, compared with ResNet50 at $B \times C \times H \times W$56 and $B \times C \times H \times W$57 (Scabini et al., 2024). At the same time, the survey shows that larger or stronger-pretrained ViTs such as ViT-B/16 with DINO, BeiTv2-B/16, and Swin-B generally deliver higher absolute accuracy, particularly on Outex14 and in-the-wild texture tasks (Scabini et al., 2024).

The deployment guidance in the original EfficientFormer paper and later edge survey is consistent. On Apple platforms, CoreML deployment on the Neural Engine is the reference path; GeLU is preferred, HardSwish should be avoided unless validated, BN folding should be exploited, and unnecessary reshapes and transposes should be eliminated (Li et al., 2022). The edge-device survey further recommends mixed FP16/INT8 quantization for vision transformers in general, with FP16 for sensitive layers and INT8 for dense layers, stating that INT8 typically incurs less than $B \times C \times H \times W$58 ImageNet top-1 degradation and that per-channel quantization is essential to remain under $B \times C \times H \times W$59 degradation, although EfficientFormer-specific post-quantization numbers are not reported there (Samson, 5 Jan 2026).

Limitations emerge from the same sources. EfficientFormer’s latency advantage is hardware- and compiler-dependent, as shown by the strong activation-function effect on CoreML and by the emphasis on on-device lookup tables and target-specific slimming (Li et al., 2022). The method also relies on keeping MHSA at small token counts; increasing input resolution raises the token count in the last stage and weakens the assumption that attention is cheap (Li et al., 2022). The 2026 survey adds a systems-level caveat: models below about $B \times C \times H \times W$60M parameters tend to be memory-bandwidth limited at batch size 1, and models in the $B \times C \times H \times W$61–$B \times C \times H \times W$62M range often achieve better hardware utilization, implying that EfficientFormer-L1 may leave utilization headroom unused even while delivering excellent latency (Samson, 5 Jan 2026). This suggests that model selection within the family depends on whether the objective is absolute latency, utilization efficiency, or accuracy under a fixed power and memory budget.

In the literature as a whole, EfficientFormer occupies a specific and influential niche: it demonstrates that a transformer backbone can achieve mobile-class latency not by abandoning global attention, but by restricting where attention is used, maintaining dimension consistency over long stretches of the network, and optimizing architecture decisions directly against measured device latency (Li et al., 2022). Subsequent comparisons and surveys preserve that characterization, even when they identify other models with better parameter efficiency or stronger transfer accuracy under larger-scale pre-training (Zhang et al., 2022, Scabini et al., 2024, Samson, 5 Jan 2026).