JFAA: Technical Report for the EPIC-KITCHENS-100 Action Anticipation Challenge at EgoVis 2026

Abstract: We propose JFAA, a JEPA-based Future Action Anticipation method for the EPIC-KITCHENS-100 (EK-100) Action Anticipation task. Inspired by the representation learning and future prediction ability of V-JEPA 2.1, JFAA uses a frozen encoder and predictor to extract observed context features and near-future latent tokens. A lightweight attentive probe is then trained to predict verb, noun, and action logits with separate task queries. To improve robustness, we further build a field-aware ensemble over selected epoch-level predictions, allowing each output field to benefit from its most reliable candidates. Experimental results on the official challenge server show that JFAA achieves first place in the EgoVis 2026 EK-100 Action Anticipation Challenge. Our code will be released at https://github.com/CorrineQiu/JFAA.

• The paper introduces a JEPA-based architecture that uses a frozen ViT-G/384 encoder and lightweight attentive probes to anticipate actions from egocentric videos.
• It employs separate prediction streams for verbs and nouns, integrating motion and object cues while using sigmoid focal loss to mitigate class imbalances.
• Experimental results demonstrate robust performance with an overall action MT5R of 27.95, showcasing improved generalization across varied kitchen scenarios.

JEPA-Based Future Action Anticipation for Egocentric Video: JFAA Technical Report

Introduction and Problem Context

Egocentric video action anticipation presents distinctive challenges due to frequent ego-motion, partial object occlusions, and subjective observation bias. The EPIC-KITCHENS-100 (EK-100) dataset is an established benchmark, targeting the prediction of the next action (verb, noun, action tuple) before it becomes visually observable, strictly using only the preceding context window. This task is practically relevant for Embodied AI systems requiring timely and accurate pre-emptive understanding of human intentions and object interactions under strong camera motion and restricted viewpoint constraints.

Methodology

JFAA introduces a JEPA-based future action anticipation framework optimized for the EK-100 challenge protocol. The pipeline leverages V-JEPA 2.1, using a frozen ViT-G/384 encoder-predictor backbone to extract both observed spatiotemporal features and near-future latent tokens from sampled RGB frames. This backbone is kept frozen throughout, avoiding full backbone fine-tuning while maximizing robust representation capacity. On top of these features, JFAA deploys a lightweight attentive probe consisting of four blocks with 16 attention heads and three dedicated task queries for verb, noun, and action prediction.

The classifier architecture maintains separation between verb and noun prediction streams, exploiting motion cues for verbs and object/scene cues for nouns, per challenge best-practices. For robustness, 20 probe heads are trained under varied learning-rate and weight-decay settings, providing a diverse candidate pool without retraining the encoder. Sigmoid focal loss is employed to counteract the long-tailed action distribution, and predictions are ensembled through a field-aware strategy at the epoch level, allowing independent selection and weighting of the most reliable verb, noun, and action predictions.

Figure 1: JFAA extracts frozen V-JEPA 2.1 features from the observed clip, forecasts near-future latent tokens, and employs attentive probes and field-aware ensemble inference.

Experimental Results

Validation on the EK-100 protocol demonstrates that optimal epochs diverge for verb, noun, and action. Separate epoch selection substantiates the field-aware ensemble approach. Quantitative evaluation on the official challenge server positions JFAA as the top submission, achieving an overall action MT5R (Mean Top-5 Recall) of 27.95, outperforming all contemporaneous challenge participants. JFAA also exhibits high accuracy on unseen participants and tail classes, underscoring its generalization and robustness across user and class distributions.

Qualitative Case Analysis

Successful qualitative cases reveal that JFAA effectively leverages pre-action cues—such as hand motion and object visibility—to anticipate upcoming actions in kitchen scenarios.

Figure 2: JFAA accurately anticipates take plate in a pre-action context with clear visual evidence of the plate and hand trajectory.

Failure analysis highlights persistent shortcomings in noun disambiguation when contexts are visually similar and contain multiple candidate objects.

Figure 3: JFAA erroneously predicts wash plate instead of wash cloth, revealing noun confusion within visually analogous sink-based scenes.

Implications, Theoretical and Practical

JFAA's architecture validates that frozen future-predictive video foundation models, complemented by attentive task probes and field-aware ensemble inference, achieve high action anticipation performance in egocentric domains. Maintaining backbone parameters frozen is shown not to degrade anticipatory capability, reducing computation and setup costs. The task-separation principle demonstrates that disentangled semantic cues enhance prediction in the presence of partial evidence. The results are theoretically significant for action forecasting architectures in embodied settings and practically impactful for real-time assistive systems.

Future research trajectories may include adaptive backbone reconfiguration, integration with multimodal prompt-based models, and more aggressive temporal context augmentation. Novel approaches could explore uncertainty modeling in noun prediction and compositionally structured intention forecasting.

Conclusion

JFAA advances the state-of-the-art for EK-100 action anticipation via JEPA-based feature extraction, attentive probing, and field-aware ensemble inference. Its architectural choices and empirical results reinforce the utility of foundation video models in anticipatory tasks, with attested robustness across user and class distributions. The methodology is extensible to broader egocentric domains, and its findings inform both practical deployment and theoretical modeling of next-action anticipation in embodied video understanding.

(2605.20904)