Assessing the necessity of long-horizon memory for video understanding

Ascertain the extent to which long-horizon memory is necessary for video understanding tasks as opposed to approaches that rely on limited temporal cues such as short frame bursts or motion signals like optical flow, and determine appropriate memory granularity across different tasks.

Background

The survey emphasizes memory as a core challenge for scaling video understanding to hour-level or streaming inputs, calling for architectures that balance latency, compute, and fidelity and that maintain both internal and external scene states.

However, the authors point out uncertainty about when long-horizon memory is essential versus when shorter temporal cues suffice, framing memory granularity as a key design dimension.