Persistent object tracking across frames in egocentric videos

Establish methods that enable multimodal foundation models to maintain persistent tracking of objects across frames in egocentric videos, thereby supporting a stable world-state memory rather than purely view-dependent evidence, as required by the Spatial Memory tasks in the SAW (Situated Awareness in the Real World) benchmark.

Background

Within the SAW benchmark’s Spatial Memory tasks, the authors observe that models frequently infer that objects no longer exist when they exit the camera’s field of view, indicating reliance on view-dependent evidence rather than a persistent world-state representation.

This behavior leads to incorrect conclusions about object presence across time during egocentric motion, motivating the need for models that can maintain object persistence across frames—a capability the paper explicitly identifies as an open challenge.

References

Persistent tracking of objects across frames remains an open challenge across models.

— Learning Situated Awareness in the Real World  (2602.16682 - Li et al., 18 Feb 2026) in Finding 3, Subsection "Failure to maintain persistent object memory", Section 5 (Analysis)

Therefore, maintaining identity consistency to alleviate perception instability under deteriorating observation conditions remains an unsolved problem.

— UCON: Uncertainty-aware Navigation with Historical Re-association in Dynamic Environments  (2609.29419 - Sun et al., 24 Sep 2026) in Section II-A, “Attempts to Mitigate Perception Instability”

Graphs do not survive ego-motion. The detectors and trackers that populate graph nodes were tuned on framed third-person video, and the head motion, blur, and truncation of first-person capture (Section~\ref{sec:challenges}) corrupt node identity and edge assignment directly. That both SAMJAM and FocusGraph resort to heavy foundation models or optical-flow heuristics just to hold their graphs together across motion is the symptom; robust, lightweight egocentric graph construction is the unmet need.

— Vision-Language Models for Egocentric Video: From Hand-Object Interaction to Embodied AI  (2608.18671 - Zamani et al., 19 Aug 2026) in Section 9.6, “Open Challenges in Graph-based Reasoning” (Sec. graph-challenges); reiterated in Section 11.3

However, handling situations where two similar small size that occludes each other (a vase occludes a napkin nearby box) remains future work.

— Touvigation: Embodied Adaptive Object Acquisition for Blind and Low-Vision Users in Unfamiliar Indoor Environments  (2609.21828 - Wang et al., 18 Sep 2026) in Section 4, subsection “Associating Visual Detections with 3D Entities”

Persistent cross-chunk identity association and memory consolidation for hour- or day-scale streams also remain open problems.

— MEMO: Multi-Level Entity-Aware Memory for Streaming Video Understanding  (2609.38900 - Li et al., 30 Sep 2026) in Section 6, “Limitations”

We leave for future work the simple extension to associate new objects with previously removed ones to handle displacement explicitly.

— FOCUS: Fine-Grained Open-Vocabulary Change Detection for Uncertainty-Aware Semi-Static Scenes  (2610.05639 - Xu et al., 5 Oct 2026) in Section 3.1, “Problem Formulation and Assumptions” (labeled Sec. problem)

However, comprehensive 3D spatial reasoning, the cognitive ability to navigate changing environments by tracking dynamic objects and transforming first-person observations into consistent global representations, remains largely unsolved.

— FloorSAV: Elucidating Spatial Audio-Visual Context with 2D Floormap for AV-LLMs  (2610.11310 - Kim et al., 8 Oct 2026) in Section 1, Introduction

These results support combining execution context with selective visual access, while failures under cup swaps expose persistent identity tracking as an unresolved requirement.

— Dual-Stream Memory for Vision-Language-Action Policies  (2610.13262 - Wang et al., 9 Oct 2026) in Abstract; Section DISCUSSION AND CONCLUSION