---
title: 'ViViTs: Video Vision Transformers'
url: https://www.emergentmind.com/topics/video-vision-transformers-vivits
type: topic
---

# ViViTs: Video Vision Transformers

Video Vision Transformers (ViViTs) are deep learning architectures that generalize the Vision Transformer (ViT) paradigm from image to video, operating directly on spatio-temporal token sequences derived from video clips. By leveraging various forms of spatio-temporal tokenization and transformer-based attention, ViViT models achieve state-of-the-art results on video understanding tasks such as action recognition, video classification, fake video detection, and temporally dense encoding. The ViViT approach encompasses design variants based on attention factorization, patch embedding, and regularization, as well as adaptations for computational efficiency and parameter efficiency across supervised, low-label, and adversarial settings.

## 1. Spatio-Temporal Tokenization and Model Architectures

ViViTs operate on video clips represented as tensors $V \in \mathbb{R}^{T \times H \times W \times C}$, where $T$ is the number of frames, $H \times W$ is the spatial resolution, and $C$ is the channel number (typically 3 for RGB) [2103.15691]. Key to ViViT is the process of partitioning video into spatio-temporal units:

- **Tubelet embedding**: Non-overlapping cubes (tubelets) of size $t \times h \times w$ are flattened and projected via a learned linear map $E_{3D}$, yielding tokens $z_i = E_{3D} \cdot \text{vec}(\text{tubelet}_i)$. The total number of patches is $N = n_t \cdot n_h \cdot n_w$, with $n_t= \lfloor T / t \rfloor$, and analogously for $n_h, n_w$ [2103.15691, 2209.03561].
- **Token sequence and positional encoding**: A learnable classification token is prepended, and positional embeddings (2D, 3D, or factorized) are added to token vectors: $y = [y_{\text{cls}}, J x_1, ..., J x_N] + P$.

Four principal ViViT encoder variants have been established [2103.15691]:

1. **Joint Spatio-Temporal Attention**: Applies global self-attention to all $N$ tokens per layer.
2. **Factorized Encoder**: Applies independent spatial attention blocks per frame, followed by temporal attention over frame representations.
3. **Divided Self-Attention**: Each layer applies spatial attention across patch tokens within frames, then temporal attention across the same spatial location over time.
4. **Split-Head Attention**: Attention heads are divided between spatial-only and temporal-only neighborhoods within each layer.

Empirically, factorized and divided attention designs offer more favorable compute/parameter trade-offs on long video sequences. Recent work has further developed one-stream spatio-temporal transformer blocks, lightweight parameterization (e.g., DualPath adapters), and interleaved efficient inference schemes (e.g., ResidualViT) [2209.03561, 2303.09857, 2509.13255].

## 2. Regularization, Initialization, and Optimization Strategies

To ensure competitive generalization, especially on modest-sized video datasets, ViViT architectures depend on strong regularization, augmentation, and transfer from large-scale image pretraining [2103.15691, 2106.05058, 2209.03561]:

- **Data augmentation**: Includes random cropping, horizontal/vertical flipping, color jitter, Mixup, CutMix, random rotation, Gaussian blur, and temporal jitter. For small datasets (e.g., violence detection), aggressive frame-level augmentation is necessary to compensate for weaker inductive bias [2209.03561].
- **Image-to-video weight transfer**: Tubelet embeddings are initialized via inflation or central frame copying from ImageNet/Vision Transformer weights; positional embeddings are tiled or interpolated to match the video token count [2103.15691, 2106.05058].
- **Regularization**: Utilizes stochastic depth, RandAugment, label smoothing, and dropout.
- **Optimization**: Follows AdamW or synchronous SGD with cosine learning-rate schedules, often with linear warm-up phases. Typical batch sizes are $32-128$ and training durations range between $30-100$ epochs depending on task and dataset.

Careful pretraining and strong regularization are critical—standard ViT training without these measures results in overfitting and poor transfer to video [2103.15691, 2209.07474].

## 3. Specialized ViViT Variants: Efficiency and Adaptation

Several derivative architectures have been proposed to address limitations of vanilla ViViT in efficiency, data efficiency, and domain adaptation:

- **ResidualViT** [2509.13255]: Employs an interleaved encoding scheme, where every $(N\!+\!1)$-th frame is fully encoded by a frozen ViT backbone ($E_V$), and intermediate frames are processed with a lightweight encoder sharing transformer weights but with only a residual token (derived from previous I-frame) and a subset of spatial tokens (retained via random, uniform, center, or motion-based selection). ResidualViT reduces computational cost by $50$–$60$\% and achieves comparable accuracy to full ViT across temporally dense video tasks.
- **DualPath adaptation** [2303.09857]: Introduces separate spatial and temporal lightweight adapters into each transformer block of a frozen backbone. Spatial path adapters are parallel (after attention, MLP), while temporal path adapters process an artificial "grid" of downscaled consecutive frames for improved temporal modeling. DualPath achieves strong results on Kinetics-400 and other benchmarks, requiring only $10$–$33$M trainable parameters (out of total $96$–$336$M).
- **Parameter-efficient adaptation**: Most DualPath parameters are dedicated to bottleneck adapters ($W_\text{down} \in \mathbb{R}^{d \times r}, W_\text{up} \in \mathbb{R}^{r \times d}$), with $r \ll d$ (e.g., $r=128$ for $d=768, 1024$).
- **Empirical speed and accuracy**: ResidualViT achieves $2$–$2.5\times$ speedups at inference with $<3\%$ drop in accuracy on natural language video grounding, activity localization, and audio description [2509.13255]. DualPath matches or exceeds full fine-tuning methods at $<10$\% parameter cost on action recognition and temporally dynamic benchmarks [2303.09857].

## 4. Application Domains

ViViTs have demonstrated state-of-the-art or highly competitive results on diverse video tasks:

- **Action Recognition**: On Kinetics-400, ViViT-L/16 (factorized FE) achieves $81.7\%$ top-1 (IN21k pretrain), exceeding 3D CNNs such as SlowFast and X3D-XXL. On EpicKitchens-100, fine-tuned ViViTs achieve $44.0\%$ action, $66.4\%$ verb, and $56.8\%$ noun accuracy [2103.15691].
- **Violence Detection**: A single-stream, spatio-temporal augmented ViViT (8 transformer layers, $d=128$) achieves $97.14\%$ accuracy on Hockey Fight and $98.46\%$ on Violent Crowd, outperforming specialized CNN+LSTM, SVM, and Fisher Vector systems [2209.03561].
- **Fake Video Detection**: A framework leveraging a frozen video ViT backbone with a lightweight Q-Former head/globally pooled prototype scoring achieves robust detection under heavy compression and generalizes across diverse generative video pipelines, maintaining AUC $>0.85$ under CRF$=50$ [2504.20669].
- **Low-Labeled Regimes**: ViViT models (with or without attention localization) outperform semi-supervised CNNs leveraging unlabeled data, e.g., achieving $24.91\%$ (ViViT-FE) vs $19.79\%$ (C2D+NL-R50) on Kinetics-400 with only $1\%$ labeled data [2209.07474].
- **Temporally Dense Encoding**: ResidualViT secures up to $60\%$ computational savings on frame-level temporal video reasoning at minimal accuracy loss [2509.13255].

## 5. Analysis, Ablation Studies, and Empirical Findings

Empirical results highlight the following core observations:

- **Inductive biases and pretraining**: While ViViT lacks strong local bias of 3D CNNs, spatiospatial pretraining (e.g., ImageNet-1K/21k) compensates, especially in low-labeled and transfer settings [2209.07474].
- **Attention structure trade-offs**: Factorized attention (spatial then temporal) lowers compute and parameter count without significantly degrading performance versus joint global attention, especially for longer or higher-resolution inputs [2103.15691].
- **Data augmentation**: Heavy augmentation is essential on small datasets for both generalization and preventing representation collapse. Aggressive strategies (color jitter, rotation, Gaussian blur, temporal jitter) are key for specialized domains like violence detection [2209.03561].
- **Parameter and FLOPs efficiency**: Adapter-based methods (DualPath, ST-Adapter) and token-reduction schemes (ResidualViT) substantially cut training and inference costs (e.g., DualPath on ViT-L/14 requires only $27$M trainable out of $330$M total parameters, with $87.7\%$ K400 top-1) [2303.09857].
- **Ensembling**: For action recognition, fusion with advanced convnets (e.g., SlowFast, ir-CSN-152) yields the best test-time metrics, compensating for ViViT's modest verb prediction weakness with convolutional temporal features [2106.05058].

Key ablations confirm (1) DualPath's spatial and temporal adapters are strongly complementary; (2) increased input resolution and stronger augmentations benefit "noun" accuracy more than "verb" (i.e., entity/object vs motion), and (3) mid-size tubelets (e.g., $8 \times 8 \times 8$ for violence detection) balance accuracy and compute [2303.09857, 2209.03561].

## 6. Limitations, Open Problems, and Future Directions

Despite their success, current ViViT designs exhibit several limitations:

- **Compute and memory**: Global attention over very long token sequences ($N > 10^3$) remains resource intensive. Factorized and local-windows variants mitigate but do not fully resolve this for high-fps video or long clips [2103.15691, 2509.13255].
- **Temporal granularity**: Frame stacking or grid "frameset" methods (as in DualPath) trade spatial resolution for temporal coverage, and extreme grid sizes may degrade fine-grained temporal reasoning [2303.09857].
- **Verb accuracy/motion modeling**: On action datasets requiring subtle motion discrimination, even advanced ViViTs tend to lag state-of-the-art CNNs on "verb" prediction, while excelling on "noun" (entity) identification [2106.05058].
- **Domain adaptation**: Performance on synthetic/real video mixtures or new generative methods is sensitive to the backbone architecture and pretraining data. Ongoing work explores more robust backbone/fusion and domain-adaptive strategies for evolving video generation distributions [2504.20669].
- **Scalability**: While parameter-efficient (adapter-based) fine-tuning and token selection help scale ViViTs to resource-constrained settings, further work is needed on sparse attention and cross-modal video-language adaptation.

## 7. Conclusion and Impact

ViViT and derived Video Vision Transformers have established a new paradigm in video understanding by extending the scalability, generalization, and non-local receptive field of pure transformers to spatio-temporal domains. Through flexible tokenization, attention factorization, and careful regularization, ViViT achieves or surpasses state-of-the-art results on standard, low-label, adversarial, and efficient-dense encoding tasks. Evolutions—including DualPath and ResidualViT—illustrate how adaptation layers and token-efficient inference assure broad practical applicability, while empirical studies clarify both strengths (few-shot learning, compression robustness, semantic generalization) and ongoing challenges (temporal modeling, compute cost, high-fidelity domain shifts). This corpus of work continues to influence research across surveillance, activity recognition, video grounding, synthetic media detection, and foundation model adaptation [2103.15691, 2106.05058, 2209.03561, 2209.07474, 2303.09857, 2504.20669, 2509.13255].

Source: https://www.emergentmind.com/topics/video-vision-transformers-vivits