---
title: 'VidEoMT: Encoder-Only Mask Transformer for Video'
url: https://www.emergentmind.com/topics/video-encoder-only-mask-transformer-videomt
type: topic
---

# VidEoMT: Encoder-Only Mask Transformer for Video

The Video Encoder-only Mask Transformer (VidEoMT) is an architectural paradigm for video scene understanding and prediction that leverages the representational power of large Vision Transformers (ViTs) in a purely encoder-only regime. Unlike most traditional video models, which employ complex decoder heads or explicit tracking modules, VidEoMT demonstrates that—with sufficient pre-training and carefully designed query propagation and masking mechanisms—a ViT encoder alone suffices for tasks including action recognition, online video segmentation, and video prediction, while achieving significant acceleration in inference throughput. The core of VidEoMT is its ability to jointly process spatial-temporal patch tokens in a single encoder, augmented by lightweight modules for efficient temporal modeling and redundancy removal.

## 1. Encoder-only Architecture and Tokenization

VidEoMT models operate directly on video volumes by slicing the input $V\in\mathbb{R}^{T\times H\times W\times3}$ into non-overlapping space-time "cubes" of fixed shape $c\times p_1\times p_2$ (e.g., $c=2$ frames, $p_1=p_2=16$ pixels). Each cube is linearly projected into a $d$-dimensional embedding space (typically $d=768$ for ViT-B, $d=1024$ for ViT-L) and augmented with learned position encoding, producing a sequence of $N=(T/c)\cdot (H/p_1)\cdot (W/p_2)$ tokens. These tokens serve as the input to a stack of $L$ encoder-only Transformer blocks jointly attending over all space-time positions.

For video prediction tasks (e.g., MaskViT), a VQ-GAN or dVAE is first used to discretize each video frame into a grid of tokens before transformer processing. In all cases, the transformer operates without any separate decoder, pixel head, or external tracking module, and all learnable parameters reside in the encoder stack and, where relevant, in token query embeddings [2206.11894, 2602.17807, 2402.18577].

## 2. Temporal Modeling: Query Propagation and Fusion

A central challenge in video segmentation and online video processing is maintaining object identity and tracking across frames without explicit tracking heads. VidEoMT addresses this by leveraging a two-stage ViT encoder split after $L_1$ blocks:

- In the first stage, only patch tokens are processed.
- In the second (last $L_2$) stage, a set of $N$ object queries is introduced and processed jointly with patch tokens.

At $t=0$ (the first frame), a learned set of queries $\mathbf{Q}_{\text{lrn}}\in\mathbb{R}^{N\times D}$ is injected. For $t>0$, temporal consistency is enforced by propagating the output queries from the previous step and fusing them with $\mathbf{Q}_{\text{lrn}}$:

\[
\mathbf{Q}_t^F = W\,\mathbf{Q}^S_{t-1} + \mathbf{Q}_\text{lrn}
\]

where $W$ is a learnable linear transformation and $\mathbf{Q}^S_{t-1}$ the previous frame's output queries. This fusion preserves track identity while allowing detection of new objects. The output queries not only provide segmentation and classification for the current frame but are also propagated as "track queries" to the next frame. This design eliminates the need for explicit cross-frame attention, separate tracking modules, or identity heads, resulting in dramatically simplified and faster inference pipelines [2602.17807].

## 3. Redundancy Reduction: Motion Guided Token Compression (MGTC)

VidEoMT incorporates Motion Guided Token Compression (MGTC) to address the $O(N^2)$ computational complexity of self-attention over dense video tokens, especially at high FPS. MGTC operates by computing patch-to-patch variance across consecutive frames:

\[
D_t^i = \|P_{t+1}^i - P_t^i\|_2^2
\]
or, in feature space,
\[
D_t^i = \frac{1}{d} \|\mathbf{C}_{t+1}^i - \mathbf{C}_t^i\|_2^2
\]
where $\mathbf{C}_t^i$ is the feature embedding of the $i$th patch at time $t$.

MGTC then masks exactly a ratio $r$ of tokens with the lowest motion score, dropping redundancy while preferentially retaining regions with significant temporal change. Masking thresholds are enforced per-video by selecting the $rN$th smallest $D^i$, ensuring stable compute and performance scaling with FPS. For example, masking $r=0.25$ at 6 FPS or $r=0.10$ at 9/12 FPS yields up to $31$\% GFLOPs reduction (1436→988 GFLOPs on Kinetics-400) while maintaining or improving accuracy [2402.18577].

## 4. Training Procedures and Hyperparameters

VidEoMT models generally rely on large-scale pre-training regimes, frequently leveraging DINOv2 (or DINOv3) on image or video data for feature initialization. Subsequent fine-tuning is performed for the specific downstream task, using per-frame instance mask annotation (e.g., COCO) before optional stage-2 fine-tuning with temporal supervision (as in DVIS++ protocol for segmentation tasks).

Key architectural choices (segmentation case, [2602.17807]):
- Queries: $N=200$, $D=1024$ (ViT-L)
- Transformer split: $L_2=4$ blocks for joint query-token attention
- Optimizer: AdamW with polynomial lr decay; layer-wise lr decay $=0.6$
- Loss: weighted sum of binary cross-entropy, Dice, and cross-entropy classification ($\lambda_\text{bce}=\lambda_\text{dice}=5.0$, $\lambda_\text{ce}=2.0$)
- Video masking (MGTC): $r=0.25$ at 6 FPS; $r=0.10$ at 9/12 FPS for fixed compute budget

For video prediction (MaskViT [2206.11894]), training uses variable masking ratios ($r\sim [0.5,1)$) and alternating spatial and spatiotemporal window attention, with Transformer block width $768$–$1024$, $6$–$8$ blocks in total.

## 5. Complexity, Efficiency, and Empirical Performance

The encoder-only design with MGTC and/or query propagation yields exceptional computational efficiency. In video segmentation, VidEoMT achieves $160$ FPS (ViT-L) with $566$ GFLOPs, a $10\times$ speedup over tracker-based CAVIS ($15$ FPS, $838$ GFLOPs), and within $0.3$ AP of state-of-the-art segmentation accuracy (AP $68.6$ vs $68.9$ on YouTube-VIS 2019) [2602.17807].

In video classification (Kinetics-400, UCF101, HMDB51), elevating input FPS from $6$ to $12$ yields substantial accuracy boosts ($+1.6\%$ on K400 and UCF, $+4.0\%$ on HMDB51), a phenomenon attributed to finer motion capture. Applying MGTC then reduces compute by $>31\%$ and further improves accuracy $(+0.1\%$ to $+0.2\%)$, validating the token pruning approach. This demonstrates that, under a fixed computational budget, higher-FPS input combined with motion-aware masking delivers both reduced inference cost and higher accuracy [2402.18577].

Video prediction benchmarks (MaskViT [2206.11894]) show competitive or superior performance versus prior methods, with up to $512\times$ fewer Transformer calls at inference and parameter counts smaller than many autoregressive models.

| Setting                 | Params | GFLOPs | FPS  | AP / Acc |
|-------------------------|--------|--------|------|----------|
| CAVIS (segmentation)    | 358 M  | 838    | 15   | 68.9     |
| VidEoMT (ViT-L, seg.)   | 318 M  | 566    | 160  | 68.6     |
| VidEoMT (K400, 12FPS)   | 87 M   | 451    | n.a. | 81.6     |
| VidEoMT+MGTC (K400)     | 87 M   | 392    | n.a. | 81.8     |

## 6. Ablation Studies and Model Variants

Stepwise ablations reveal the individual contributions of EoMT, tracker removal, query propagation, and fusion:

- Removal of context-aware features, re-id, and dedicated tracker modules increases FPS (15 → 162) but reduces AP (68.9 → 61.3).
- Adding pure query propagation restores partial AP (61.3 → 63.9).
- Full fusion (VidEoMT) regains almost all accuracy (AP 68.6), confirming the necessity of combining historical and learned queries.

Alternative fusion schemes, periodic query resets, or contrastive re-id heads can be introduced for robustness in long sequences or occlusion-heavy regimes [2602.17807]. For video prediction, variable masking and windowed attention outperform fixed masking or global attention in both speed and fidelity [2206.11894].

## 7. Extensions, Limitations, and Further Directions

VidEoMT’s architectural simplicity enables practical deployment and extensibility. Proposed extensions include adaptive query gating, multi-scale query injection, and attention-based fusion. Notably, VidEoMT's strong results on large ViT backbones (DINOv2-pretrained) decrease as backbone capacity is reduced, indicating a capacity–performance tradeoff. A plausible implication is that the degree of pre-training and model width is critical for encoder-only video transformers to close the gap with complex tracker-based systems. Datasets with more pronounced occlusion or longer temporal horizons may further benefit from contrastive identity loss or query reset mechanisms.

The encoder-only paradigm with motion-guided compression and propagative/fusion-based temporal modeling redefines efficiency and capacity in video scene understanding, prediction, and segmentation—offering a new baseline for transformer-based video models [2402.18577, 2602.17807, 2206.11894].

Source: https://www.emergentmind.com/topics/video-encoder-only-mask-transformer-videomt