Using historical context without harming present-scene perception in streaming video understanding

Develop streaming video understanding techniques for vision–language models operating under causal online constraints that leverage historical video context while preserving current-scene understanding; specifically, determine how to integrate past frames or memory representations without degrading real-time perception performance.

Background

The paper introduces SimpleStream, a minimal recent-context baseline that answers each streaming query with only the last N frames and an off-the-shelf vision–LLM. Across OVO-Bench and StreamingBench, SimpleStream outperforms recent streaming systems while using less memory and maintaining competitive latency.

Analyses show that adding longer or retrieved historical context is not uniformly beneficial and often induces a perception–memory trade-off: memory-oriented gains frequently coincide with degraded current-scene perception. This motivates an explicit open problem of exploiting history without harming present-scene understanding.

References

The central open problem is not how to add more memory, but how to use history without degrading current-scene understanding.

— A Simple Baseline for Streaming Video Understanding  (2604.02317 - Shen et al., 2 Apr 2026) in Conclusion (Section 7)

This observation is consistent with MemFusionMap, which conjectures that excessive temporal fusion dilutes the current-frame feature, and MambaMap, which conjectures that an excessively long memory window introduces redundancy and noise.

— MapLightning: Online Vectorized HD Map Construction with 1D Map Tokens  (2610.01905 - Zheng et al., 1 Oct 2026) in Appendix, Section "Ablation Studies"

Despite these efforts, a fundamental open problem remains: how should a streaming VLM remember?

— Memory as Weights: Internalizing Long-Term History for Streaming Videos  (2610.22070 - Liu et al., 15 Oct 2026) in Section 1, Introduction

Since MasW is not trained with agentic tool-use data, extending it to such agentic settings is left for future work.

— Memory as Weights: Internalizing Long-Term History for Streaming Videos  (2610.22070 - Liu et al., 15 Oct 2026) in Appendix, Section Additional Analysis, subsection Additional Qualitative Results