VidEoMT: Encoder-Only Mask Transformer for Video

• VidEoMT is an encoder-only video transformer that processes spatiotemporal tokens with query propagation and masking to efficiently understand and predict video scenes.
• The architecture employs motion-guided token compression to reduce computational costs by up to 31%, ensuring faster inference without sacrificing accuracy.
• Extensive pre-training and fusion of historical and learned queries enable VidEoMT to perform robust action recognition, segmentation, and video prediction close to state-of-the-art results.

The Video Encoder-only Mask Transformer (VidEoMT) is an architectural paradigm for video scene understanding and prediction that leverages the representational power of large Vision Transformers (ViTs) in a purely encoder-only regime. Unlike most traditional video models, which employ complex decoder heads or explicit tracking modules, VidEoMT demonstrates that—with sufficient pre-training and carefully designed query propagation and masking mechanisms—a ViT encoder alone suffices for tasks including action recognition, online video segmentation, and video prediction, while achieving significant acceleration in inference throughput. The core of VidEoMT is its ability to jointly process spatial-temporal patch tokens in a single encoder, augmented by lightweight modules for efficient temporal modeling and redundancy removal.

1. Encoder-only Architecture and Tokenization

VidEoMT models operate directly on video volumes by slicing the input $V\in\mathbb{R}^{T\times H\times W\times3}$ into non-overlapping space-time "cubes" of fixed shape $c\times p_1\times p_2$ (e.g., $c=2$ frames, $p_1=p_2=16$ pixels). Each cube is linearly projected into a $d$-dimensional embedding space (typically $d=768$ for ViT-B, $d=1024$ for ViT-L) and augmented with learned position encoding, producing a sequence of $N=(T/c)\cdot (H/p_1)\cdot (W/p_2)$ tokens. These tokens serve as the input to a stack of $L$ encoder-only Transformer blocks jointly attending over all space-time positions.

For video prediction tasks (e.g., MaskViT), a VQ-GAN or dVAE is first used to discretize each video frame into a grid of tokens before transformer processing. In all cases, the transformer operates without any separate decoder, pixel head, or external tracking module, and all learnable parameters reside in the encoder stack and, where relevant, in token query embeddings (Gupta et al., 2022, Norouzi et al., 19 Feb 2026, Feng et al., 2024).

2. Temporal Modeling: Query Propagation and Fusion

A central challenge in video segmentation and online video processing is maintaining object identity and tracking across frames without explicit tracking heads. VidEoMT addresses this by leveraging a two-stage ViT encoder split after $L_1$ blocks:

• In the first stage, only patch tokens are processed.
• In the second (last $c\times p_1\times p_2$0) stage, a set of $c\times p_1\times p_2$1 object queries is introduced and processed jointly with patch tokens.

At $c\times p_1\times p_2$2 (the first frame), a learned set of queries $c\times p_1\times p_2$3 is injected. For $c\times p_1\times p_2$4, temporal consistency is enforced by propagating the output queries from the previous step and fusing them with $c\times p_1\times p_2$5:

$c\times p_1\times p_2$6

where $c\times p_1\times p_2$7 is a learnable linear transformation and $c\times p_1\times p_2$8 the previous frame's output queries. This fusion preserves track identity while allowing detection of new objects. The output queries not only provide segmentation and classification for the current frame but are also propagated as "track queries" to the next frame. This design eliminates the need for explicit cross-frame attention, separate tracking modules, or identity heads, resulting in dramatically simplified and faster inference pipelines (Norouzi et al., 19 Feb 2026).

3. Redundancy Reduction: Motion Guided Token Compression (MGTC)

VidEoMT incorporates Motion Guided Token Compression (MGTC) to address the $c\times p_1\times p_2$9 computational complexity of self-attention over dense video tokens, especially at high FPS. MGTC operates by computing patch-to-patch variance across consecutive frames:

$c=2$0

or, in feature space,

$c=2$1

where $c=2$2 is the feature embedding of the $c=2$3th patch at time $c=2$4.

MGTC then masks exactly a ratio $c=2$5 of tokens with the lowest motion score, dropping redundancy while preferentially retaining regions with significant temporal change. Masking thresholds are enforced per-video by selecting the $c=2$6th smallest $c=2$7, ensuring stable compute and performance scaling with FPS. For example, masking $c=2$8 at 6 FPS or $c=2$9 at 9/12 FPS yields up to $p_1=p_2=16$0\% GFLOPs reduction (1436→988 GFLOPs on Kinetics-400) while maintaining or improving accuracy (Feng et al., 2024).

4. Training Procedures and Hyperparameters

VidEoMT models generally rely on large-scale pre-training regimes, frequently leveraging DINOv2 (or DINOv3) on image or video data for feature initialization. Subsequent fine-tuning is performed for the specific downstream task, using per-frame instance mask annotation (e.g., COCO) before optional stage-2 fine-tuning with temporal supervision (as in DVIS++ protocol for segmentation tasks).

Key architectural choices (segmentation case, (Norouzi et al., 19 Feb 2026)):

• Queries: $p_1=p_2=16$1, $p_1=p_2=16$2 (ViT-L)
• Transformer split: $p_1=p_2=16$3 blocks for joint query-token attention
• Optimizer: AdamW with polynomial lr decay; layer-wise lr decay $p_1=p_2=16$4
• Loss: weighted sum of binary cross-entropy, Dice, and cross-entropy classification ($p_1=p_2=16$5, $p_1=p_2=16$6)
• Video masking (MGTC): $p_1=p_2=16$7 at 6 FPS; $p_1=p_2=16$8 at 9/12 FPS for fixed compute budget

For video prediction (MaskViT (Gupta et al., 2022)), training uses variable masking ratios ($p_1=p_2=16$9) and alternating spatial and spatiotemporal window attention, with Transformer block width $d$0–$d$1, $d$2–$d$3 blocks in total.

5. Complexity, Efficiency, and Empirical Performance

The encoder-only design with MGTC and/or query propagation yields exceptional computational efficiency. In video segmentation, VidEoMT achieves $d$4 FPS (ViT-L) with $d$5 GFLOPs, a $d$6 speedup over tracker-based CAVIS ($d$7 FPS, $d$8 GFLOPs), and within $d$9 AP of state-of-the-art segmentation accuracy (AP $d=768$0 vs $d=768$1 on YouTube-VIS 2019) (Norouzi et al., 19 Feb 2026).

In video classification (Kinetics-400, UCF101, HMDB51), elevating input FPS from $d=768$2 to $d=768$3 yields substantial accuracy boosts ($d=768$4 on K400 and UCF, $d=768$5 on HMDB51), a phenomenon attributed to finer motion capture. Applying MGTC then reduces compute by $d=768$6 and further improves accuracy $d=768$7 to $d=768$8, validating the token pruning approach. This demonstrates that, under a fixed computational budget, higher-FPS input combined with motion-aware masking delivers both reduced inference cost and higher accuracy (Feng et al., 2024).

Video prediction benchmarks (MaskViT (Gupta et al., 2022)) show competitive or superior performance versus prior methods, with up to $d=768$9 fewer Transformer calls at inference and parameter counts smaller than many autoregressive models.

Setting	Params	GFLOPs	FPS	AP / Acc
CAVIS (segmentation)	358 M	838	15	68.9
VidEoMT (ViT-L, seg.)	318 M	566	160	68.6
VidEoMT (K400, 12FPS)	87 M	451	n.a.	81.6
VidEoMT+MGTC (K400)	87 M	392	n.a.	81.8

6. Ablation Studies and Model Variants

Stepwise ablations reveal the individual contributions of EoMT, tracker removal, query propagation, and fusion:

• Removal of context-aware features, re-id, and dedicated tracker modules increases FPS (15 → 162) but reduces AP (68.9 → 61.3).
• Adding pure query propagation restores partial AP (61.3 → 63.9).
• Full fusion (VidEoMT) regains almost all accuracy (AP 68.6), confirming the necessity of combining historical and learned queries.

Alternative fusion schemes, periodic query resets, or contrastive re-id heads can be introduced for robustness in long sequences or occlusion-heavy regimes (Norouzi et al., 19 Feb 2026). For video prediction, variable masking and windowed attention outperform fixed masking or global attention in both speed and fidelity (Gupta et al., 2022).

7. Extensions, Limitations, and Further Directions

VidEoMT’s architectural simplicity enables practical deployment and extensibility. Proposed extensions include adaptive query gating, multi-scale query injection, and attention-based fusion. Notably, VidEoMT's strong results on large ViT backbones (DINOv2-pretrained) decrease as backbone capacity is reduced, indicating a capacity–performance tradeoff. A plausible implication is that the degree of pre-training and model width is critical for encoder-only video transformers to close the gap with complex tracker-based systems. Datasets with more pronounced occlusion or longer temporal horizons may further benefit from contrastive identity loss or query reset mechanisms.

The encoder-only paradigm with motion-guided compression and propagative/fusion-based temporal modeling redefines efficiency and capacity in video scene understanding, prediction, and segmentation—offering a new baseline for transformer-based video models (Feng et al., 2024, Norouzi et al., 19 Feb 2026, Gupta et al., 2022).