UniFormer: Unified Visual Backbone

• UniFormer is a hierarchical visual backbone that unifies convolutional local aggregation and transformer-style global self-attention to efficiently process visual data.
• It uses a four-stage design with shallow layers for local context and deep layers for capturing global spatiotemporal dependencies.
• Variants such as UniFormerV2 integrate image-pretrained ViTs with UniFormer modules, achieving state-of-the-art performance on classification, detection, segmentation, and video tasks.

UniFormer is a hierarchical visual backbone that unifies convolutional local aggregation and self-attentive global aggregation within a transformer-style architecture for image and video understanding. It was introduced to address the joint problem of large local redundancy and complex global dependency in visual data, and it organizes computation accordingly: shallow stages emphasize local relation aggregation, while deep stages emphasize global spatiotemporal self-attention. In the original formulations, UniFormer is positioned both as a model for efficient spatiotemporal representation learning and as a general visual backbone spanning classification, video recognition, detection, segmentation, and pose estimation (Li et al., 2022, Li et al., 2022).

1. Problem setting and design rationale

UniFormer is motivated by a specific asymmetry in visual representation learning. Neighboring pixels, patches, and adjacent video frames are highly correlated, so computing global pairwise relations on low-level features is inefficient. At the same time, discriminative recognition often depends on long-range interactions across distant regions or frames. The original papers frame this as a tension between local redundancy and global dependency: CNNs reduce redundancy efficiently through small receptive fields but are limited in long-range modeling, whereas ViTs capture long-range relations with self-attention but perform blind global token comparisons even when early-layer structure is predominantly local (Li et al., 2022, Li et al., 2022).

Within that framing, UniFormer’s central claim is architectural rather than merely empirical. The model does not attempt to replace convolution with attention everywhere. Instead, it casts both operations as instances of a common relation-aggregation mechanism and then deploys them stage-wise: local relation aggregation in shallow layers and global relation aggregation in deep layers. This local-to-global schedule is reported to provide the best trade-off relative to all-local or all-global alternatives, which respectively lose global dependency modeling or incur inefficient and overfitting-prone early attention (Li et al., 2022, Li et al., 2022).

A common misconception is to treat UniFormer as a standard ViT with minor convolutional additions. The original description is more specific: UniFormer is a four-stage hierarchical backbone in which the shallow relation aggregators are convolution-like operators instantiated as pointwise-depthwise-pointwise 3D convolutions, while the deep relation aggregators are joint spatiotemporal self-attention blocks. In that sense, the model is explicitly hybrid at the operator level rather than transformer-only in the conventional ViT sense (Li et al., 2022).

2. Core architecture and mathematical formulation

A UniFormer block follows a pre-normalized residual structure with three components: Dynamic Position Embedding (DPE), Multi-Head Relation Aggregator (MHRA), and a feed-forward network. For input $X_{in} \in \mathbb{R}^{C\times T\times H\times W}$, the block is described as

$X = \mathrm{DPE}(X_{in}) + X_{in},$

$Y = \mathrm{MHRA}(\mathrm{Norm}(X)) + X,$

$Z = \mathrm{FFN}(\mathrm{Norm}(Y)) + Y.$

DPE is a depthwise convolution with zero padding: $3\times 3\times 3$ for video and $3\times 3$ in the image-oriented formulation. It provides positional information without relying on fixed absolute position embeddings, and the video-oriented paper emphasizes that this design works for arbitrary clip length (Li et al., 2022, Li et al., 2022).

The unifying abstraction is the relation aggregator. For token sequence $X \in \mathbb{R}^{L\times C}$ and head $n$,

$$R_n(X) = A_n V_n(X), \qquad \mathrm{MHRA}(X) = \mathrm{Concat}(R_1(X);\dots;R_N(X))\,U.$$

The distinction between shallow and deep stages lies in the affinity matrix $A_n$.

In shallow stages, UniFormer uses local relation aggregation. For anchor token $X = \mathrm{DPE}(X_{in}) + X_{in},$0 and neighbor $X = \mathrm{DPE}(X_{in}) + X_{in},$1 inside a local 3D tube $X = \mathrm{DPE}(X_{in}) + X_{in},$2,

$X = \mathrm{DPE}(X_{in}) + X_{in},$3

This is equivalent to convolution-like aggregation with learnable relative-position weights, and in practice the shallow video stages use tube size $X = \mathrm{DPE}(X_{in}) + X_{in},$4. The implementation is a MobileNet-style PWConv-DWConv-PWConv stack, with head count equal to the channel count in local stages (Li et al., 2022). In the broader visual-recognition formulation, the same mechanism is presented as the convolutional special case of MHRA, with local neighborhoods of size $X = \mathrm{DPE}(X_{in}) + X_{in},$5 for images and $X = \mathrm{DPE}(X_{in}) + X_{in},$6 for videos (Li et al., 2022).

In deep stages, UniFormer switches to global relation aggregation by joint spatiotemporal self-attention. With projected queries, keys, and values, the attention operator is

$X = \mathrm{DPE}(X_{in}) + X_{in},$7

For the video-oriented UniFormer, attention is joint over space and time, the head dimension is $X = \mathrm{DPE}(X_{in}) + X_{in},$8, and relative positional bias $X = \mathrm{DPE}(X_{in}) + X_{in},$9 is not used because DPE provides positional information (Li et al., 2022). This joint spatiotemporal attention is explicitly contrasted with divided spatial-temporal attention, and the papers report that joint attention transfers better to motion-centric benchmarks (Li et al., 2022, Li et al., 2022).

The backbone is hierarchical and four-stage. Channels are $Y = \mathrm{MHRA}(\mathrm{Norm}(X)) + X,$0, and the standard stage assignment is local-local-global-global. The video patch stem is a $Y = \mathrm{MHRA}(\mathrm{Norm}(X)) + X,$1 convolution with stride $Y = \mathrm{MHRA}(\mathrm{Norm}(X)) + X,$2, followed by $Y = \mathrm{MHRA}(\mathrm{Norm}(X)) + X,$3 convolutions with stride $Y = \mathrm{MHRA}(\mathrm{Norm}(X)) + X,$4 between stages, so temporal downsampling happens before stage 1 and not thereafter (Li et al., 2022). Normalization is mixed by operator type: BN for local MHRA and LN for global MHRA, with an extra LN in downsampling layers (Li et al., 2022).

3. Variants, optimization, and benchmark results

The original work defines multiple backbone scales. In the video-oriented description, UniFormer-S uses block depths $Y = \mathrm{MHRA}(\mathrm{Norm}(X)) + X,$5 and UniFormer-B uses $Y = \mathrm{MHRA}(\mathrm{Norm}(X)) + X,$6, both with channels $Y = \mathrm{MHRA}(\mathrm{Norm}(X)) + X,$7 (Li et al., 2022). In the broader visual-recognition description, Small, Base, and Large variants are reported, with the Large model using channels $Y = \mathrm{MHRA}(\mathrm{Norm}(X)) + X,$8, about $Y = \mathrm{MHRA}(\mathrm{Norm}(X)) + X,$9M parameters, and $Z = \mathrm{FFN}(\mathrm{Norm}(Y)) + Y.$0 GFLOPs at $Z = \mathrm{FFN}(\mathrm{Norm}(Y)) + Y.$1 (Li et al., 2022).

Training recipes are dataset-specific but structurally consistent. UniFormer is pretrained on ImageNet-1K for the original video experiments, with 2D convolutions inflated to 3D for video. Optimization uses AdamW with cosine learning-rate decay and warmup; the video paper reports base learning rates scaled by batch size, weight decay $Z = \mathrm{FFN}(\mathrm{Norm}(Y)) + Y.$2, and stochastic depth schedules differentiated by dataset and model scale (Li et al., 2022). For Something-Something, the paper specifies uniform frame sampling and no horizontal flip; for Kinetics, dense sampling is used (Li et al., 2022).

The model’s headline results established its original visibility in video recognition. With only ImageNet-1K pretraining, UniFormer achieves $Z = \mathrm{FFN}(\mathrm{Norm}(Y)) + Y.$3 top-1 accuracy on Kinetics-400/Kinetics-600 using UniFormer-B at $Z = \mathrm{FFN}(\mathrm{Norm}(Y)) + Y.$4 testing, and $Z = \mathrm{FFN}(\mathrm{Norm}(Y)) + Y.$5 top-1 on Something-Something V1/V2 with UniFormer-B at $Z = \mathrm{FFN}(\mathrm{Norm}(Y)) + Y.$6 (Li et al., 2022). In the broader visual-recognition formulation, UniFormer reaches $Z = \mathrm{FFN}(\mathrm{Norm}(Y)) + Y.$7 top-1 accuracy on ImageNet-1K classification, $Z = \mathrm{FFN}(\mathrm{Norm}(Y)) + Y.$8 box AP and $Z = \mathrm{FFN}(\mathrm{Norm}(Y)) + Y.$9 mask AP on COCO object detection, $3\times 3\times 3$0 mIoU on ADE20K semantic segmentation, and $3\times 3\times 3$1 AP on COCO pose estimation, all without extra training data beyond ImageNet-1K pretraining (Li et al., 2022).

Variant or setting	Scale information	Reported result
UniFormer-S	21.4M params, 167 GFLOPs	80.8% top-1 on Kinetics-400
UniFormer-B	49.8M params, 1036 GFLOPs	82.9% top-1 on Kinetics-400
UniFormer-B	32×3×1 testing	60.9% / 71.2% on Sth-Sth V1 / V2
UniFormer-L	about 100M params, 12.6 GFLOPs	86.3 top-1 on ImageNet-1K

The ablations clarify what the model is and is not. All-local stacking is efficient but weaker on long-range reasoning; all-global stacking is computationally heavy and drops accuracy. DPE contributes measurable gains, reported as $3\times 3\times 3$2 top-1 on ImageNet and $3\times 3\times 3$3 on Kinetics-400 in the video-oriented study. Replacing shallow UniFormer blocks with pure MobileNet-style PW-DW-PW layers reduces accuracy, indicating that the transformer-style FFN remains functionally important even in the local regime (Li et al., 2022).

4. UniFormerV2 and the integration of pretrained image ViTs

UniFormerV2 is a related but distinct family that reuses the UniFormer design logic while changing the entry point. Rather than requiring a separate image-pretraining phase for a bespoke video backbone, it starts from open-sourced, image-pretrained ViTs and arms them with efficient UniFormer-style relation aggregators for video (Li et al., 2022).

The architecture introduces two new components. A Local UniBlock inserts local temporal MHRA before each standard image-ViT block. For input $3\times 3\times 3$4, the local temporal branch is

$3\times 3\times 3$5

followed by inherited global spatial MHRA from the image ViT and then an FFN. The local temporal affinity is defined over a temporal tube $3\times 3\times 3$6,

$3\times 3\times 3$7

so temporal redundancy is reduced while the pretrained ViT’s per-frame spatial attention is preserved (Li et al., 2022).

The second component is the Global UniBlock, which summarizes all space-time tokens into a compact video token using query-based cross-attention. With learnable query token $3\times 3\times 3$8,

$3\times 3\times 3$9

and the per-head cross-attention affinity is

$3\times 3$0

This reduces the complexity of long-range spatiotemporal modeling from $3\times 3$1 to $3\times 3$2 with respect to the number of tokens in the global summarization path (Li et al., 2022).

Empirically, UniFormerV2 is presented as a state-of-the-art video-recognition family on eight benchmarks. The most prominent reported milestone is that UniFormerV2-L/14, with CLIP-400M pretraining plus K710 post-pretraining, reaches $3\times 3$3 top-1 and $3\times 3$4 top-5 on Kinetics-400 at $3\times 3$5 frames and $3\times 3$6 px resolution, described as the first model to achieve $3\times 3$7 top-1 on Kinetics-400 to the authors’ knowledge (Li et al., 2022). This should not be conflated with the original UniFormer: UniFormerV2 is not merely a larger version, but a paradigm for adapting pretrained image ViTs with UniFormer-inspired local and global video modules.

5. Extensions built on UniFormer: frame flexibility, robustness, and forensics

Subsequent work has treated UniFormer as a backbone to be adapted rather than as a fixed endpoint. One line of work addresses Temporal Frequency Deviation, the reported performance drop when a video model trained at one frame count is evaluated at another. “Frame Flexible Network” introduces a single shared model trained jointly on multiple temporal frequencies and reports UniFormer-specific gains on Something-Something V1. For UniFormer, the baseline high-frequency-only model yields $3\times 3$8, $3\times 3$9, and $X \in \mathbb{R}^{L\times C}$0 top-1 at 4, 8, and 16 frames, whereas Separated Training yields $X \in \mathbb{R}^{L\times C}$1, $X \in \mathbb{R}^{L\times C}$2, and $X \in \mathbb{R}^{L\times C}$3, and UniFormer-FFN yields $X \in \mathbb{R}^{L\times C}$4, $X \in \mathbb{R}^{L\times C}$5, and $X \in \mathbb{R}^{L\times C}$6 at the same frame counts (Zhang et al., 2023). The mechanism combines Multi-Frequency Alignment,

$X \in \mathbb{R}^{L\times C}$7

with frequency-specific normalization and lightweight Weight Alteration adapters, inserted in UniFormer’s transformer blocks following ST-Adapter placement (Zhang et al., 2023).

A second line of work studies robustness under realistic video distribution shifts. In “Adversarial Augmentation Training Makes Action Recognition Models More Robust to Realistic Video Distribution Shifts,” UniFormer-S is one of three evaluated backbones, using ImageNet pretrained weights, $X \in \mathbb{R}^{L\times C}$8 inputs, and 8-frame sparse sampling. The reported curriculum adversarial augmentation raises UniFormer-S from $X \in \mathbb{R}^{L\times C}$9 to $n$0 top-1 on the matched-class cross-dataset setting UCF-65 $n$1 Kinetics-65, and from $n$2 to $n$3 on HMDB-28 $n$4 Kinetics-28 (Kim et al., 2024). In prototype-based cosine evaluation, adversarial augmentation and curriculum variants also improve transfer performance, which suggests that UniFormer’s penultimate features can be made more robust under source-target distribution mismatch without altering the model’s basic architecture (Kim et al., 2024).

UniFormer has also appeared as an encoder in video forensics. “Video Inpainting Localization with Contrastive Learning” states that a 3D Uniformer encoder is applied to the video noise residual for learning effective spatiotemporal forensic features, with a lightweight convolution decoder producing a pixel-wise inpainting localization map (Lou et al., 2024). The abstract-level description supports the broader interpretation that UniFormer’s local-global spatiotemporal hierarchy is applicable beyond action recognition.

6. Scope of the term and later name reuse

In the literature after the original visual-recognition work, the name “UniFormer” is reused for architectures outside the original vision-backbone lineage. “UniFormer: Unified and Efficient Transformer for Reasoning Across General and Custom Computing” defines a cross-platform dual-branch attention architecture for GPUs and FPGAs, centered on a local exact-attention branch and a global linear-attention branch, and reports ImageNet throughput and FPGA latency results rather than visual-backbone benchmarks (Ran et al., 11 Nov 2025). “UniFormer: Efficient and Unified Model-Centric Scaling for Industrial Recommendation” introduces a recommendation architecture built from Feature-space Interaction Modules and Task-space Interaction Modules, with semantic-based tokenization and request-level inference acceleration, and reports online A/B gains in Kuaishou and Kuaishou Lite (Chen et al., 25 Jun 2026).

These later uses are not continuations of the 2022 visual backbone in the strict architectural sense. A plausible implication is that the title “UniFormer” functions as a reusable label for “unified transformer” designs across domains, rather than denoting a single evolving model family. In citation practice, this makes arXiv identifiers especially important: in visual representation learning, “UniFormer” most commonly refers to the 2022 backbone that unifies convolutional local relation aggregation and global self-attention for images and videos (Li et al., 2022, Li et al., 2022).