- The paper introduces a unified observation ratio sampling strategy that enables a single model to robustly predict actions at various levels of video observation.
- It employs a joint forecasting and classification approach with a ViT-based encoder and forecasting decoder, achieving significant performance gains over state-of-the-art methods.
- Difference-based token masking reduces GPU memory and training time by 50%, enhancing efficiency while maintaining high prediction accuracy.
Summary of "EAST: Early Action Prediction Sampling Strategy with Token Masking" (2604.18367)
Introduction and Motivation
Early action prediction (EAP) aims to recognize human actions from only partially observed video sequences, which is essential in contexts requiring rapid response, such as surveillance, autonomous driving, and human-robot interaction. Prior approaches typically struggle with the challenge of limited contextual cues, and most methods require training separate models for each observation ratio, leading to inefficient resource utilization and impractical deployment. The paper introduces EAST, an end-to-end framework targeting efficient and generalizable EAP by leveraging a novel randomized sampling scheme and memory-efficient token masking in video transformers (ViTs).
Methodology
Unified Observation Ratio Sampling
A key contribution is the randomized temporal sampling strategy: during training, the model receives incomplete (present) and full (oracle) video clips by randomly sampling an observation ratio p. This enables a single model to generalize across all observation ratios at inference, covering p from 0.1 to 0.9. Empirical analysis demonstrates that training individual models at fixed ratios leads to poor generalization at unseen ratios, while the proposed method delivers robust prediction for all ratios with a single model.
Joint Forecasting and Classification Objective
EAST jointly optimizes classification over both observed (partial) and oracle (full) features. The encoder is a ViT with spatio-temporal tubelets as input tokens. For EAP, the model uses a forecasting decoder to process present tokens and predict future (unobserved) features. Two sets of classification logits are produced per sample—one from decoder-forecasted features (EAP prediction) and one from oracle full-clip encoder features (standard recognition). Both are supervised with cross-entropy losses, which yields discriminative, temporally aligned representations. The direct transformer decoder, concatenating present and masked tokens, empirically outperforms autoregressive structures.
Difference-based Token Masking for Efficient Training
Token masking targets redundancy by removing spatial tubelet patches with minimal differences across contiguous frames (via L1 norm). This is inspired by feature stability criteria (Moravec/Harris detector class). By retaining only the most dynamically relevant 50% of tokens, the approach halves GPU memory and training time with negligible accuracy degradation. The masking is performed independently on present and future clips to prevent information leakage, maintaining compatibility with MAE-based pre-training. The strategy outperforms both random and running-cell masking baselines.
Experimental Results
Datasets and Protocols
EAST is extensively validated on standard activity recognition datasets: Something-Something v2 (both full and subset), NTU RGB+D, UCF101, and Epic-Kitchens-100. A unified protocol is adopted: a single model is trained and evaluated across all commonly used observation ratios, with accuracy measured at each ratio and averaged.
EAST achieves significant improvements over previous methods:
- NTU60: Outperforms TemPr by up to 19.2 percentage points (pp) at p=0.3, with a 6.8 pp average improvement across ratios.
- SSv2: Achieves a 28.3 pp average improvement over TemPr, with consistent gains at all ratios; surpasses Early-ViT by 10.1 pp on SSsub21.
- UCF101: With the MoViNet backbone, improves over ERA and TemPr by 1.3 pp and 3.9 pp, respectively, establishing that gains are not backbone-dependent.
- Epic-Kitchens-100: Provides substantial gains at low observation ratios, with a 13 pp improvement on action noun verbs at p=0.1.
Notably, EAST achieves these results while requiring only one model per dataset, versus prior work necessitating multiple models per observation ratio.
Efficiency and Analysis
- Training time per epoch is reduced by over 2x compared to TemPr.
- Inference speed is increased sixfold.
- Ablation studies confirm that the proposed masking and joint loss are essential for accuracy and resource efficiency.
- Training a single model yields robust generalization across all observation ratios, whereas per-ratio models degrade rapidly when applied outside their training range.
- Qualitative examples highlight that EAST successfully predicts correct actions at the lowest observation ratios where VideoMAE or conventional finetuning fail.
Theoretical and Practical Implications
By training with random observation ratios and token masking, EAST sets a new standard for scalable and generalizable early action prediction. The insights regarding joint supervision over both present and oracle features reinforce the value of discriminative forecasting decoders in ViT-based architectures. Difference-based masking connects action dynamics with visual saliency, yielding efficient backbones applicable to extended video sequences.
The framework addresses practical limitations of existing methods:
- Deployment: A single model per task is practical and easy to maintain.
- Resource Utilization: The approach enables SOTA performance with significantly reduced hardware requirements, being trainable on two 20GB GPUs.
- Generalization: Performance does not degrade for unseen observation ratios, crucial for real-world streaming or asynchronous inference scenarios.
However, there are remaining practical limitations: while training is efficient, real-time deployment with ViTs on commodity hardware is still constrained by transformer inference cost. The requirement for windowed rather than fully causal inference further limits streaming applications on current benchmarks.
Future Directions
Potential avenues for future work include unsupervised sequence modeling for EAP, achieving unified approaches for both anticipation and early action prediction, and extending efficient token selection and training strategies to fully online or streaming transformer architectures.
Conclusion
EAST introduces a new paradigm for early action prediction via a randomized sampling-based training protocol and token difference-based masking, achieving state-of-the-art accuracy and computational efficiency across multiple major datasets. The method establishes the viability of unified, end-to-end ViT-based models for scalable action prediction and provides a template for future research on efficient sequence modeling and practical deployment in safety-critical, real-time systems.