Video Vision Transformer Overview

• Video Vision Transformers are transformer-based architectures that convert videos into sequences of spatial-temporal tokens for global self-attention.
• They apply innovative tokenization methods such as tubelet and strip embeddings to capture complex video features while optimizing computational efficiency.
• These models achieve state-of-the-art results in tasks like classification, segmentation, and anomaly detection, while driving research on efficient temporal modeling and self-supervision.

A Video Vision Transformer (Video ViT) is a transformer-based deep neural architecture designed to model spatiotemporal dynamics in video sequences through global self-attention over learned video tokens. Unlike traditional 2D CNNs or 3D convolutional models, Video ViTs represent videos as sequences of embedded tokens that encode both spatial (per-frame) and temporal (across-frame) structure, processed by cascades of transformer blocks. These models achieve state-of-the-art results across various video analysis tasks including classification, segmentation, restoration, and anomaly detection, and introduce multiple innovations for improved computational efficiency, data efficiency, and temporal context modeling.

1. Tokenization: Patch, Tubelet, and Strip Embeddings

Video ViT architectures begin by decomposing a video into discrete tokens in space and time. Early forms such as ViViT (Arnab et al., 2021, Singh et al., 2022) use tubelet embedding, where a video of shape $T\times H\times W\times C$ is split into spatio-temporal cubes (tubelets) of size $t \times h \times w$, producing $$N = \lfloor T/t \rfloor \lfloor H/h \rfloor \lfloor W/w \rfloor$$ tubelets. Each tubelet is flattened and projected into a $d$-dimensional token. A learnable class token and positional embeddings are prepended and added, yielding a token sequence passed as input to the transformer encoder.

Variants such as ViStripformer (Tsai et al., 2023) introduce strip-shaped tokenization, decomposing video frames into horizontal and vertical strips to form tokens, enabling efficient modeling of orientation-specific degradations and reducing memory usage for high-resolution video restoration. In convolutional ViT hybrids (e.g., Deepfake detection (Wodajo et al., 2021)), per-frame features are extracted by a CNN before patch or strip embedding.

2. Transformer Backbone: Space-Time Attention and Factorization

The transformer backbone in Video ViTs consists of stacked encoder blocks, each composed of multi-head self-attention (MHSA) and MLP sublayers, both wrapped with residual connections and layer normalization. These blocks enable global receptive fields, allowing each token to attend to every other token—across space and time—for rich dependency modeling.

Due to the quadratic complexity of naive MHSA over $N$ video tokens, efficient factorization strategies are essential. ViViT (Arnab et al., 2021) introduces:

• Joint space-time attention: Full attention over all tokens, high expressivity, prohibitive complexity for long videos.
• Factorized encoder: Separate spatial attention per frame, followed by temporal attention on aggregated per-frame representations.
• Divided space-time attention: Alternates spatial MSA on per-frame tokens with temporal MSA on per-location tokens.
• Factorized dot-product attention: Splits attention heads into spatial and temporal subsets, concatenating their outputs.

Attention can be further optimized using linear kernels; for example, the Linear Video Transformer (Lu et al., 2022) replaces standard softmax attention with kernelized attention that computes $Y = \rho(Q)[\rho(K)^T V]$ for linear complexity, coupled with modules such as feature fixation (channel-wise reweighting) and neighborhood association (spatiotemporal feature shifting).

3. Temporal Modeling Innovations

Video Vision Transformers must capture long-range, multi-scale temporal patterns. Several key mechanisms have emerged:

• Memory-Augmented Attention: MeMViT (Wu et al., 2022) caches compressed keys and values from previous clips at each transformer layer, so queries attend to both current and past context. This pipelined memory design extends temporal support by $\times$30 with minimal compute overhead, enabling cross-clip reasoning for action recognition, localization, and anticipation tasks.
• Messenger-Shift Mechanism: TeViT (Yang et al., 2022) introduces messenger tokens that are shifted across frames in the backbone, propagating temporal information without quadratic cost. In the head, spatiotemporal query interaction alternates shared spatial and temporal MHSA on instance queries for efficient video instance segmentation.
• Iterative Refinement and Token Selection: SSViT (Geng et al., 9 Nov 2025) uses local-complexity scoring to select only intricate region tokens for full Transformer processing, offloading basic tokens to optical flow-based warping. This selective strategy halves memory footprint while maintaining SOTA modulo video recovery accuracy.
• Late Fusion and Feature Shift: IV-ViT (Shimizu et al., 2023) aggregates per-frame transformer outputs via average pooling for joint image–video learning, but acknowledges limitations in capturing deep temporal correlations with blunt aggregation.

4. Training Protocols, Data Efficiency, and Self-Supervision

Video ViTs require specialized training to overcome weak inductive bias, data scarcity, and computational challenges:

• Aggressive Data Augmentation: Augmenting each frame with blur, rotation, color jitter, and flips is critical for reducing overfitting in small-dataset video tasks (e.g., violence detection (Singh et al., 2022), AU detection (Vu et al., 2023)).
• Transfer Learning: Initialization from large-scale image-pretrained ViTs (e.g., ImageNet, JFT) is standard. Tubelet and positional embeddings are adapted via filter inflation and temporal tiling (Arnab et al., 2021).
• Self-Supervised Objectives: SVT (Ranasinghe et al., 2021) trains Transformers to match features of multiple spatiotemporal views of the same video (global/slow-fast, local), using a BYOL/DINO-style EMA-teacher without negatives, yielding competitive transfer with orders-of-magnitude fewer pretraining epochs.
• Forward Video Prediction: GSViT (Schmidgall et al., 2024) pre-trains on surgical video via next-frame reconstruction, inducing temporal priors into a pure 2D transformer architecture. This approach achieves SOTA single-frame phase annotation with high efficiency.
• Multi-Contextual Self-Supervision: For anomaly detection (Lee et al., 2022), ViTs are trained to reconstruct masked frames, predict from whole/partial sequences, and jointly model optical flow, amplifying reconstruction error for abnormal events.

5. Applications and Quantitative Impact

Video ViTs have demonstrated marked improvements in:

• Video Classification: ViViT (Arnab et al., 2021), SSViT, and Linear Video Transformer (Lu et al., 2022) reach 79–85% Top-1 accuracy on Kinetics-400/600, outperforming leading 3D ConvNets and prior transformer baselines at lower computational cost.
• Object Segmentation and Detection: TransVOS (Mei et al., 2021), TeViT (Yang et al., 2022), and memory-augmented ViTs yield state-of-the-art results on DAVIS, YouTube-VOS, and YouTube-VIS, with innovations in spatiotemporal attention and memory extension.
• Low-Level Restoration: ViStripformer (Tsai et al., 2023) achieves PSNR/SSIM superiority in deblurring, deraining, and demoireing, with token-efficient block-structured attention scaling to HD video.
• Specialized Domains: GSViT (Schmidgall et al., 2024) delivers real-time surgical phase detection, open-source pre-training, and parameter efficiency on 680 hours of surgical videos.
• Generalization to Omnidirectional Video: SalViT360 (Cokelek et al., 2023) adapts transformer attention to spherical coordinates, outperforming prior saliency predictors on 360° video datasets.

6. Computational Efficiency and Scaling

Managing quadratic attention cost is a central challenge:

Model	Attention Type	Complexity	Efficiency Strategies
ViViT	Full, Factored	$O(N^2)$, $O(n_t^2)$	Tubelet embedding, factorization
ViStripformer	Strip attention	$O((H^2+W^2)T)$	Directional strip factorization
MeMViT	Pooling + memory	$O(MN\bar N)$	Hierarchical pooling, memory
SSViT	Selective tube	$O(K^2)$ ($K\ll N$)	Token selection, optical flow
Linear ViT	Linear kernel	$O(ND^2)$	Feature fixation, NA enhancement

Even as model complexity grows, innovations such as token-pruning, localized or linear attention, memory banks, and hardware-efficient backbones (sandwich design, CGA) enable deployment on real-time streams and extensive video corpora.

7. Limitations, Open Challenges, and Future Directions

Despite strong empirical results, several limitations and research avenues remain:

• Temporal Aggregation: Simple late fusion or averaging as in IV-ViT limits temporal representational depth; more expressive, learned sequence-level attention or memory modules are preferred for long-term reasoning.
• Spatial-Temporal Tradeoff: Many factorizations sacrifice spatial or temporal resolution. Joint attention or selective mechanisms seek a balance; design choices are task-dependent.
• Data Efficiency: Transformers’ inductive bias is weak, especially for small datasets; strong augmentation, pretraining, and hybrid CNN–transformer designs (RegNetY in AU detection) address this, but further advances in self-supervised or unsupervised schemes (SVT, GSViT) are needed.
• Generalization Across Domains: Models trained on natural videos may not generalize to spherical (SalViT360), modulo, or domain-specific videos without architecture or tokenization adaptations.
• Resource Constraints: Real-time, low-latency, and low-memory architectures (GSViT, ViStripformer) are critical for industrial and medical deployment.
• Unified Multi-Task Modeling: The vision articulated in recent surveys (Han et al., 2020) is a parameter-efficient, multi-task video ViT that can jointly tackle detection, segmentation, captioning, retrieval, and restoration via unified spatiotemporal attention and dynamic token selection.

A plausible implication is that future research will focus on further reducing memory/computational footprints—potentially via dynamic token routing, hybrid spatial/temporal windows, or continual memory architectures—while expanding the generality and accuracy of Video Vision Transformers across diverse tasks and environments.