LiveStarPro: Proactive Streaming Video Understanding with Hierarchical Memory for Long-Horizon Streams

Abstract: Despite the remarkable progress of Video LLMs (Video-LLMs), current online architectures still struggle to simultaneously process continuous video streams, decide autonomously when to respond, and preserve long-horizon contextual memory. These obstacles undermine real-time responsiveness and cause severe forgetting throughout prolonged interactions. In this work, we introduce LiveStarPro, a live streaming assistant that is designed for proactive video understanding over long-horizon streams. The design of LiveStarPro rests on three complementary components. The first component is Streaming Verification Decoding (SVeD), an inference framework that identifies the appropriate response timing through single-pass perplexity verification, thereby eliminating the dependency on explicit silence tokens. The second component is Streaming Causal Attention Masks (SCAM), a training strategy that enforces incremental video-language alignment over variable-length streams. The third component is Tree-Structured Hierarchical Memory (TSHM), a recursive memory architecture that organizes evicted historical information into event chains and consequently enables efficient retrieval from effectively unbounded video streams. To facilitate a comprehensive evaluation under realistic online conditions, we further present OmniStarPro, a large-scale benchmark that spans 15 diverse real-world scenarios and that extends to hour-scale streams for the assessment of long-term recall. Extensive experiments demonstrate that LiveStarPro consistently surpasses existing methods, attaining a 28.9% improvement in semantic correctness and an 18.2% reduction in timing error, while its streaming key-value cache further yields a 1.58x inference speedup over the same model without caching. The model and the code are publicly available at https://github.com/sotayang/LiveStarPro.

• The paper introduces a verification-based streaming inference method (SVeD) that enhances response timing and accuracy in Video-LLMs.
• The paper presents a tree-structured hierarchical memory system for sublinear retrieval and robust long-horizon context management.
• The OmniStarPro benchmark and experiments demonstrate significant gains in semantic correctness and memory recall over prior methods.

Proactive Long-Horizon Streaming Video Understanding with LiveStarPro

Introduction and Motivation

LiveStarPro advances the state of online Video LLMs (Video-LLMs) by directly targeting the intrinsic limitations of existing online architectures: the inability to efficiently and proactively decide response timing, preservation of context over unbounded streams, and data-imbalance and training objective misalignment induced by silence-token based approaches. In contrast to previous EOS-token and silence-centric streaming paradigms, LiveStarPro introduces a verification-based framework that divorces silence detection from generation, while introducing a fundamentally new hierarchical memory system designed to enable robust long-horizon reasoning.

Streaming Verification Decoding and the Response-Silence Paradigm

A central innovation of LiveStarPro is the Streaming Verification Decoding (SVeD) mechanism. SVeD reconceptualizes streaming inference for Video-LLMs, replacing EOS-token prediction with a single-pass verification process. Silence is rendered a byproduct of semantic continuity, not an explicit supervised signal. At each frame step, the model computes the perplexity of the prior output under the updated visual-linguistic context; if the perplexity deviates significantly from its historical baseline, this triggers new decoding. Otherwise, output is suppressed and context is logically extended via a buffer swap operation. This approach directly remedies the four main pathologies of explicit silence supervision: data imbalance, temporal inconsistency, objective mismatch, and vocabulary pollution. SVeD is operationally efficient due to a tailored streaming key-value cache that supports both append and non-causal update operations, producing a 1.58x throughput improvement with negligible accuracy loss over standard autoregressive streaming.

Streaming Causal Attention Masks and Training Consistency

The Streaming Causal Attention Masks (SCAM) training protocol is introduced to further optimize the verification-centric inference logic. SCAM masks all prior intra-event language tokens during both pretraining and finetuning, ensuring that the model grounds its predictions in visual evidence instead of degenerate linguistic copying. The interleaved frame-caption sequence formulation, augmented with stochastic paraphrase sampling, enforces model robustness to semantic redundancy and translation invariance. By aligning both training and inference with the streaming, verification-based paradigm, LiveStarPro sustains high semantic consistency and avoids the convergence and generalization issues that plague EOS-based and sliding-window approaches.

Tree-Structured Hierarchical Memory for Unbounded Streams

To address the critical limitation of finite context windows and catastrophic forgetting, LiveStarPro proposes a Tree-Structured Hierarchical Memory (TSHM). Recent history is aggressively compressed through Peak-End filtering, prioritizing semantically salient keyframes and summary captions within a strict token budget. Beyond the active context window, evicted context is recursively partitioned into event subtrees, with parent embeddings continuously updated via momentum aggregation. This enables sublinear retrieval (O(log n)) under reasonable tree branching, compared to O(n) for flat memory banks. During any reasoning or generation, relevant event chains and summary descriptors are dynamically injected into the current context window, thus enabling high-precision recall even when supporting evidence lies far outside the model’s active window.

The OmniStarPro Benchmark

Recognizing the absence of a comprehensive and realistic online video understanding benchmark, the authors introduce OmniStarPro: a rigorously curated dataset spanning fifteen broad real-world scenarios, including both short-horizon (≤6 minutes) and long-horizon (10–60+ minutes) streams. Expert annotation and multi-stage filtering ensure high-quality video-text alignment, while new memory-centric tasks (long-range memory recall, cross-event difference query, temporal backtracking) probe the memory and persistent reasoning capabilities that static benchmarks and scripted QA tools fail to diagnose. Most queries—especially in the long-form partition—necessitate access to evidence far beyond the current context, cementing OmniStarPro as the only existing resource for principled evaluation of long-horizon video understanding.

Experimental Results and Quantitative Impact

Extensive empirical evaluation demonstrates that LiveStarPro sets a new state of the art across a spectrum of metrics on OmniStarPro: a relative improvement of 28.9% in semantic correctness and an 18.2% absolute reduction in timing error over the strongest prior online Video-LLMs, while also outpacing all sliding-window and dropout-based memory strategies for long-horizon recall. On memory-centric tasks where prior models' accuracy collapses towards chance as the evidence window grows (e.g., LMR long bucket accuracy of 6.4% for VideoLLM-online), LiveStarPro achieves nearly sixfold higher recall at 37.2%. Theoretical analysis and ablation results confirm that these gains are directly attributable to TSHM’s sublinear retrieval latency and aggressive context pruning, as well as the alignment-aware streaming protocol.

Furthermore, LiveStarPro maintains competitive accuracy on conventional offline video understanding benchmarks, exhibiting only modest degradation compared to its Intern Video2.5 backbone and trailing only the largest closed and open-source generalist LVLMs.

Implications and Future Directions

The introduction of SVeD and SCAM constitutes a paradigm shift for online multimodal understanding, removing reliance on structure-misaligned objectives such as EOS tokens and enabling a practical, calibration-friendly deployment protocol. The TSHM memory design effectively bridges the gap between finite-context LLMs and the requirements of streaming, always-on perceptual systems. LiveStarPro demonstrates that, with the appropriate training and inference framework, catastrophic forgetting and context window limitations can be mitigated even on hour-scale video.

Practically, these advances set the stage for robust, real-time assistants for domains such as surveillance, autonomous driving, live event understanding, and augmented reality, where truly unbounded context and proactive temporal reasoning are a necessity. Theoretically, the tree-structured memory and verification-based strategies provide an architectural template for future Video-LLMs and Multimodal LLMs, highlighting the benefits of hierarchical, retrieval-augmented generative models over uniform or naive sliding-window solutions. Open questions remain regarding the extension of these methods to other modalities (e.g., audio, continuous conversational history), as well as to open-ended generative scenarios and reinforcement learning from real-world interaction.

Conclusion

LiveStarPro (2606.17798) establishes a new standard for long-horizon streaming video understanding by introducing a verification-based, response-silence inference protocol and a tree-structured hierarchical memory. These architectural contributions, supported by the large-scale OmniStarPro benchmark and strong empirical results, demonstrate that proactive, contextually aware, and persistent video reasoning is achievable within the operational constraints of modern multimodal LLMs. The framework and benchmark together provide a robust foundation for future research in online, scalable, and memory-augmented video-language intelligence.