- The paper presents a first-place EgoVis 2026 system that combines a COCO-pretrained Faster R-CNN still-image detector with frozen V-JEPA 2.1 temporal features for joint object, action, box, and time-to-contact prediction.
- VISTA injects temporal context through FPN-level FiLM modulation and ROI-level feature fusion, improving Overall Top-5 mAP to 5.40 and Noun+Verb mAP to 16.15 versus 5.12 and 13.29 for the StillFast baseline.
- The results show that temporal video representations improve joint semantic anticipation but do not consistently improve time-to-contact estimation, while small, occluded objects and visually salient distractors remain key failure cases.
Overview
VISTA is a technical report describing the first-place entry in the EgoVis 2026 Ego4D Short-Term Object Interaction Anticipation (STA) Challenge (2605.20901). The STA task requires a model, given an untrimmed egocentric video and a timestamp t, to output hypotheses about the next human-object interaction: the future active object's bounding box, noun category, verb category, time-to-contact (TTC), and confidence score. Because the official metric requires box, noun, verb, and TTC predictions to be correct simultaneously, the task couples spatial localization, semantic classification, and temporal regression under a strict joint-matching criterion. VISTA addresses this by extending the StillFast paradigm (2605.20901) with a frozen self-supervised video encoder as an additional temporal context source.
Architecture
The system follows a two-branch design. The still branch is a COCO-pretrained Faster R-CNN with ResNet-50 FPN (2605.20901), applied to the last observed high-resolution frame. The COCO classification layer is replaced with STA-specific heads while retaining the RPN, ROIAlign pooling, and detector box head. At inference, up to 300 proposals survive RPN filtering; rather than collapsing them early, each proposal is encoded independently and ranked later by the STA heads.
The temporal branch uses a frozen V-JEPA 2.1 ViT-G encoder at 384 resolution (2605.20901). Eight observed frames are sampled at 2 FPS; global V-JEPA features are cached to reduce inference cost, and a lightweight attentive probe compresses the cached sequence into a single temporal token summarizing recent egocentric context.
Temporal context is injected into the detection pathway at two levels: a FiLM-style projection generates per-channel scale and bias terms that modulate FPN features before proposal generation, and the projected token is concatenated with each ROI feature through a small context MLP whose residual output augments the local ROI representation. This dual-level fusion preserves detector spatial precision while making proposals aware of recent hand-object dynamics.
Prediction heads and inference
Fused ROI features feed four parallel heads predicting noun logits over Ego4D object classes, class-specific box refinements, verb logits over future action classes, a non-negative TTC via softplus regression, and an interaction quality score for ranking. Training averages the four losses over 16 epochs at learning rate 1.0×10−4. At inference, retained proposals are expanded into top noun/verb hypotheses, ranked by objectness, interaction quality, noun probability, and verb probability, filtered by class-aware NMS, and exported as the top-100 predictions. The final submission additionally ensembles complementary predictions, grouping hypotheses by noun, verb, box overlap, and TTC proximity, then reweighting and merging within groups based on confidence and cross-head agreement.
Results
On the official test split evaluated by the challenge server, VISTA achieves an Overall Top-5 mAP of 5.40, ranking first:
| Rank |
Participant |
Overall |
Noun |
Noun+Verb |
Noun+TTC |
| 1 |
corrine (VISTA) |
5.40 |
27.26 |
16.15 |
8.95 |
| 2 |
sun0710 |
5.13 |
23.83 |
14.52 |
8.07 |
| 3 |
StillFast Baseline V2 |
5.12 |
25.06 |
13.29 |
9.14 |
| 4 |
Faster R-CNN + SlowFast Baseline V2 |
3.61 |
26.15 |
9.45 |
8.69 |
Relative to the StillFast baseline, VISTA improves Overall mAP from 5.12 to 5.40 and Noun+Verb mAP substantially, from 13.29 to 16.15 — indicating that frozen V-JEPA temporal context most benefits joint semantic prediction. Notably, the Noun+TTC score (8.95) is slightly below the StillFast baseline (9.14), so the win on the primary metric comes from better balancing all matching conditions rather than uniformly dominating every sub-metric. Qualitative analysis shows correct anticipation of interactions such as grasping a wooden plank, but also a failure mode where the model attends to a visually salient paper region and misses a smaller knife, revealing sensitivity to small active objects and distractors.
Limitations and open questions
The report concedes several constraints. First, because the final model was trained on the official training split plus most validation annotations, no separate validation results are reported, so ablations of individual components (FiLM modulation versus ROI fusion versus ensembling) cannot be isolated from the leaderboard numbers alone. Second, the slight regression on Noun+TTC relative to the StillFast baseline suggests TTC estimation may not benefit from the added temporal fusion as much as semantic prediction does, and the mechanism behind this trade-off is not analyzed. Third, the qualitative failure case indicates that small or occluded active objects remain difficult, pointing toward stronger hand-object region modeling and intention reasoning as unresolved needs. Finally, the ensemble strategy is described only at a high level, leaving its individual contribution unquantified.
Conclusion
VISTA demonstrates that combining a COCO-pretrained still-image detector with a frozen V-JEPA 2.1 temporal encoder, fused via FPN-level modulation and ROI-level context injection, yields state-of-the-art performance on Ego4D short-term object interaction anticipation, achieving first place with an Overall Top-5 mAP of 5.40. The result supports the broader finding that generic self-supervised video representations can serve as effective, training-free temporal context for object-centric egocentric anticipation tasks.