Hierarchical Streaming Video Understanding

• Hierarchical streaming video understanding is the analysis of continuous video streams using multi-scale representations that capture frame-level details up to narrative-level context.
• It integrates online temporal segmentation, dynamic memory allocation, and causal reasoning to maintain temporal coherence and real-time responsiveness under bounded computation.
• The approach leverages methods like hierarchical supervision, graph-based memory, and adaptive token compression to enhance long-horizon video understanding and system efficiency.

Hierarchical streaming video understanding denotes the causal analysis of continuous video streams with bounded computation and memory, using multi-scale representations that range from frame-level perception to longer events, scenes, episodes, and narrative abstractions. In current work, the central challenge is not merely long-context encoding, but the simultaneous maintenance of temporal coherence, semantic abstraction, and real-time responsiveness as frames arrive continuously and the total duration is unknown. The topic therefore spans online temporal segmentation, hierarchical memory construction, event-level and narrative-level reasoning, and evaluation protocols that explicitly distinguish offline long-video understanding from true online streaming comprehension (Zheng et al., 23 Dec 2025, Wu et al., 31 Mar 2025).

1. Historical formation and conceptual scope

Early work approached the problem through hierarchical segmentation rather than language-conditioned reasoning. The streaming graph-based hierarchical video segmentation lineage processed videos as sequences of subsequences $v=\{V_1,V_2,\dots,V_m\}$, approximated the full hierarchical segmentation as $S\approx\{S_1,S_2,\dots,S_m\}$, and built multiple nested levels of abstraction through supervoxels and motion layers computed in a streaming fashion. Its contribution was to show that a streaming system could preserve bounded temporal context while still maintaining a hierarchy over space-time structure, rather than treating each frame independently (Tripathi et al., 2014).

A second precursor formalized hierarchy at the target level. A model trained on Something-Something V2 predicted coarse action groups, fine action categories, and captions jointly, with the fine classifier conditioned on the coarse prediction and the captioner conditioned on the finer prediction. The paper’s main claim was that models exploiting targets at different levels of granularity achieve better performance on all levels, which established coarse-to-fine supervision as a distinct design principle for video understanding, even though the model itself was not streaming (Mahdisoltani et al., 2018).

A more explicit bridge to streaming appears in the framework for video story understanding. That work defined each question as a point in a three-dimensional hierarchy

$q \mapsto (M(q), L(q), D(q)),$

where $M$ is memory capacity, $L$ is logical complexity, and $D$ is the DIKW level. Its memory axis ranges from frame to shot, scene, sequence, and entire story; its logical axis ranges from simple recall to creative thinking; and its DIKW axis ranges from data to information, knowledge, and wisdom. The paper does not explicitly use the term “streaming,” but its emphasis on long scenes, multi-scene sequences, and entire episodes makes the framework directly relevant to streaming and long-term video understanding, because it treats temporal span, reasoning depth, and semantic abstraction as coupled dimensions of difficulty rather than as isolated tasks (Heo et al., 2019).

This historical trajectory suggests that hierarchical streaming video understanding should not be reduced either to flat clip classification or to a single notion of “long context.” A plausible implication is that the field is best understood as the convergence of three lines of work: hierarchical supervision, hierarchical temporal representation, and hierarchical memory control.

2. Temporal hierarchy, event structure, and narrative abstraction

A central move in recent work is to replace the flat frame sequence with nested events. In PARSE, this appears as an event partonomy, where each level $L_i=\{e_j^{(i)}\}$ is a set of disjoint temporal intervals and higher-level events contain sets of lower-level events through temporal containment. PARSE learns this multiscale structure directly from streaming video without labels by stacking recurrent predictors across temporal granularities, using prediction error spikes as boundary evidence, and adding attention-based top-down feedback so that higher layers stabilize lower-level interpretation. The result is an unsupervised, causal hierarchy in which lower layers model short-term dynamics and higher layers integrate longer-term context, with state-of-the-art performance among streaming methods on Breakfast Actions, 50 Salads, and Assembly 101 in both H-GEBD and structural metrics such as TED and hF1 (Chen et al., 3 Dec 2025).

VideoScaffold instantiates a closely related idea in multimodal LLMs. It defines streaming video understanding as the setting in which frames arrive continuously, the total length is unknown, and the model must respond causally using bounded computation and memory. Its Elastic-Scale Event Segmentation performs prediction-guided segmentation, and its Hierarchical Event Consolidation aggregates semantically related segments into multi-level abstractions. The hierarchy is elastic: slow-changing scenes produce longer events, rapidly changing scenes produce shorter events, and the system can transition from fine-grained frame understanding to abstract event reasoning as the video stream unfolds. The paper reports state-of-the-art performance across both offline and streaming benchmarks, while preserving temporally coherent structure that static sparse sampling or offline clustering often destroys (Zheng et al., 23 Dec 2025).

An adjacent but offline blueprint appears in hierarchical self-supervised learning for movie understanding. There, a low-level Slow-only 3D CNN encodes short clips, while a high-level transformer contextualizer operates over sequences of event features and is pretrained with event mask prediction. The separation between low-level backbone and higher-level contextualizer is important: it shows how local perceptual features and longer-horizon event context can be pretrained with different objectives and data sources. Although the model is not causal, it offers a concrete template for streaming systems that maintain clip-level tokens online and update a higher-level contextual state over those tokens (Xiao et al., 2022).

Another structurally rich line models evolving interactivities through graphs. HIG defines scene graph representations over Appearance-Situation-Position-Interaction-Relation predicates and introduces a Hierarchical Interlacement Graph with a unified layer and graph within a hierarchical structure. On ASPIRe, which contains 1,488 videos, about 1.6 million frames, 833 object categories, and 4,549 interactivity categories, HIG aggregates dense human-object interactivities over time through recursively applied graph layers. Although it is not presented as a causal streaming system, this suggests a graph-state formulation for streaming settings in which node identities, attributes, and relations are updated hierarchically rather than recomputed from scratch (Nguyen et al., 2023).

3. Hierarchical memory under bounded compute and token budgets

Once the stream is potentially unbounded, hierarchical representation becomes inseparable from hierarchical memory. One influential approach is to reinterpret the transformer’s internal state as a memory hierarchy. HERMES treats the KV cache itself as hierarchical memory: shallow layers behave as sensory memory with strong recency bias, middle layers as working memory, and deep layers as long-term memory with sparse frame-level anchor tokens. Compression policies therefore differ by layer, combining recency-based decay in shallow layers with attention-based importance in deeper layers and cross-layer smoothing to preserve coherence. The system requires no auxiliary computations upon the arrival of user queries, guarantees real-time responses for continuous interactions, achieves $10\times$ faster TTFT compared to prior SOTA, and attains superior or comparable accuracy even when reducing video tokens by up to 68% compared with uniform sampling (Zhang et al., 21 Jan 2026).

A second family builds explicit external memory structures. StreamChat organizes memory into short-term visual memory $M_s$, long-term visual memory tree $M_l$, and dialogue memory $S\approx\{S_1,S_2,\dots,S_m\}$0, and couples them with selective frame stacking and a three-thread scheduling strategy. Because $S\approx\{S_1,S_2,\dots,S_m\}$1 preserves recent high-resolution details, $S\approx\{S_1,S_2,\dots,S_m\}$2 stores clustered and captioned summaries over long horizons, and $S\approx\{S_1,S_2,\dots,S_m\}$3 indexes prior question-answer turns, the system supports real-time multi-round interaction without retraining the base Video-LLM. On StreamBench, the Fast configuration reaches 32 FPS with 0.85 s request processing delay, while the ablations show distinct gains from each memory component on short-term, long-term, and conversational tasks (Xiong et al., 23 Jan 2025).

OASIS reframes the problem more aggressively as temporal routing rather than memory maximization. It maintains a short window, a medium-resolution buffer, and a multi-resolution Event Forest whose nodes are temporal intervals with keyframes, text summaries, embeddings, and hierarchical depth. Coarse reasoning first determines whether the answer can be resolved from short context; retrieval is triggered only when uncertainty remains, and the retrieval query is a high-level intent rather than the raw user question. This on-demand refinement keeps token cost bounded and request delay low while improving long-horizon and compositional reasoning over multiple backbones (Liang et al., 18 Apr 2026).

A third family emphasizes training-free routing and compression of visual input itself. CurveStream assigns each incoming frame to Clear Memory, Blurred Memory, or Discard based on a Curvature Score and an online K-Sigma dynamic threshold, motivated by the observation that high-curvature regions in feature trajectories align with critical semantic transitions. The framework consistently yields absolute performance gains of over 10%, including 10.69% on StreamingBench and 13.58% on OVOBench, under a strict token budget (Wang et al., 20 Mar 2026). STC pushes the same logic into two stages: STC-Cacher reuses ViT features for temporally similar frames, and STC-Pruner removes low-novelty visual tokens before LLM pre-filling. On ReKV, it retains up to 99% of accuracy while reducing ViT encoding latency and LLM pre-filling latency by 24.5% and 45.3%, respectively (Wang et al., 30 Nov 2025).

These designs differ in where hierarchy is imposed—inside the cache, in an external tree, in retrieval policy, or in the visual token stream—but they share a common thesis: long-horizon streaming understanding is fundamentally a memory-allocation problem across multiple temporal and semantic scales.

4. Reasoning, synchronized generation, and online interaction

Hierarchical streaming understanding is not only about storing the past; it is also about deciding when and how to speak, reason, or retrieve. LyraV addresses this through Streaming Video-Language Synchrony, a paradigm in which a Video-LLM should perceive while speaking by interleaving incoming frames with generated word tokens at frame-level granularity. Its control stack is explicitly hierarchical: the Frame-Driven Transition Controller is a training-free finite-state machine that decides whether to stay silent, continue an utterance, or start a new one, while the Streaming Token Pacer dynamically sets how many tokens may be emitted within the per-frame latency budget. This yields per-frame incremental, sub-budget decoding, 98.29% synchrony with video playback, and a real-time processing speed of 3.89 FPS, while preserving the backbone’s general understanding ability across online and offline benchmarks (Yang et al., 5 Jun 2026).

LiveStarPro generalizes the same concern to long-horizon proactive assistance. Its Streaming Verification Decoding replaces explicit silence-token prediction with single-pass perplexity verification, Streaming Causal Attention Masks enforce incremental video-language alignment during training, and Tree-Structured Hierarchical Memory organizes evicted history into recursive event chains for long-range retrieval. On OmniStarPro, the system surpasses existing methods with a 28.9% improvement in semantic correctness and an 18.2% reduction in timing error; its streaming key-value cache yields a 1.58x inference speedup over the same model without caching (Yang et al., 16 Jun 2026).

Question-aware reasoning introduces another kind of hierarchy: dynamic control over which evidence is needed and which agents or modules should handle it. HiCrew, although not implemented as a streaming system, preserves temporal topology through a Hybrid Tree built from shot boundaries and within-shot hierarchical clustering, then adds Question-Aware Captioning and a Planning Layer that adaptively selects roles and execution paths based on question complexity. Its strongest gains occur in temporal and causal reasoning on EgoSchema and NExT-QA. This suggests that future streaming systems may benefit from combining causal online memory with question-aware hierarchical planning rather than relying on a fixed reasoning workflow (Zhu et al., 23 Apr 2026).

A related but distinct semantic direction is to represent the stream as an evolving graph of entities and predicates rather than as a sequence of captions or tokens. HIG’s interactivity-centered graph hierarchy indicates that streaming video understanding can also be framed as continual scene-graph maintenance over tracked subjects, especially when the task requires persistent object relations, human-object interaction, or predicate-level explanation (Nguyen et al., 2023).

5. Benchmarks, task taxonomies, and empirical diagnostics

Benchmark design has become a first-order part of the field because streaming evaluation cannot be reduced to offline QA on sampled clips. H2VU is the most explicit attempt to formalize hierarchical and holistic evaluation. It contains 5,902 videos and 10,183 QA pairs, spans durations from 1 s to 90 min, and organizes assessment into three ability levels with 47 leaf abilities. The benchmark unifies offline general video comprehension with online streaming video comprehension and includes 2,710 first-person streaming videos, along with task families such as countercommonsense comprehension and state trajectory tracking. Its results show that current MLLMs remain substantially weaker on countercommonsense and long-horizon tracking than on standard perception and reasoning, which makes those tasks especially informative for hierarchical streaming systems (Wu et al., 31 Mar 2025).

StreamBench narrows the focus to streaming reasoning and multi-round dialogue. It contains 306 videos totaling 24.8 hours and 1.8k high-quality QA pairs, with six task types: Object Search, Long-term Memory Search, Short-term Memory Search, Conversational Interaction, Knowledge-based QA, and Simple Factual. Its metrics explicitly include semantic score, accuracy, coherence, request processing delay, and FPS. Because each video is paired with a six-question dialogue spanning different temporal offsets, StreamBench measures not only memory span but also whether a system can coordinate visual memory and dialogue memory under online constraints (Xiong et al., 23 Jan 2025).

OmniStarPro extends evaluation toward realistic long-horizon proactive assistance. It spans 15 diverse real-world scenarios and separates short-horizon live tasks such as Real-time Narration Generation, Online Temporal Grounding, Frame-level Dense QA, Contextual Online QA, and Multi-turn Interactive QA from long-horizon tasks such as Long-range Memory Recall, Cross-event Difference Query, and Temporal Backtracking. The long partition has an average memory span of 18.6 minutes, with 73.4% of queries lying beyond the active window, making it particularly diagnostic for hierarchical long-term memory rather than short-window optimization (Yang et al., 16 Jun 2026).

Synchrony has also become an evaluation axis in its own right. LyraV introduces metrics such as Sync Rate, semantic score, narrative fluency, and real-time FPS, thereby separating streaming competence from static answer correctness. This indicates that hierarchical streaming understanding now includes not only what is remembered and reasoned about, but also whether perception and language remain synchronized under real-time budgets (Yang et al., 5 Jun 2026).

6. Limitations, misconceptions, and open research directions

A persistent misconception is to equate hierarchical streaming video understanding with either hierarchical label prediction or generic long-video subsampling. The literature indicates that neither is sufficient. Hierarchical label spaces improve multi-granularity supervision, but they do not by themselves solve causal processing or memory growth (Mahdisoltani et al., 2018). Static sparse sampling, frame compression, and offline clustering may work in offline settings, but they often produce fragmented or over-compressed outputs when applied to continuous streams, and they can break temporal order or merge distinct events (Zheng et al., 23 Dec 2025).

Several limitations recur across current systems. Boundaries and hierarchy levels remain partly heuristic: the story-understanding framework itself notes subjective boundaries between memory and reasoning levels (Heo et al., 2019); VideoScaffold remains sensitive to the segmentation threshold $S\approx\{S_1,S_2,\dots,S_m\}$4 (Zheng et al., 23 Dec 2025); PARSE relies on non-differentiable post-hoc boundary extraction (Chen et al., 3 Dec 2025); CurveStream depends on hand-designed curvature thresholds and FIFO eviction (Wang et al., 20 Mar 2026). Even strong memory systems remain visually dominant: VideoScaffold does not explicitly use audio or subtitles (Zheng et al., 23 Dec 2025), CurveStream is visual-only (Wang et al., 20 Mar 2026), and LiveStarPro largely ignores audio-heavy settings (Yang et al., 16 Jun 2026). Another recurrent limitation is that many methods remain query-agnostic until late in the pipeline, while question-aware systems such as HiCrew are not yet fully causal (Zhu et al., 23 Apr 2026).

Open directions are comparatively consistent across papers. One is adaptive hierarchy discovery: learning variable-depth or content-dependent temporal abstraction rather than fixing levels or thresholds in advance (Chen et al., 3 Dec 2025). A second is multimodal boundary and memory construction, integrating audio, ASR, or other signals into segmentation and retrieval (Zheng et al., 23 Dec 2025, Wang et al., 20 Mar 2026). A third is query-aware control, where segmentation, retrieval, or memory refresh becomes conditioned on active goals rather than only on generic saliency (Liang et al., 18 Apr 2026, Zhu et al., 23 Apr 2026). A fourth is benchmark-driven debiasing: H2VU’s countercommonsense and state-trajectory tasks suggest that many current gains on short-context perception do not transfer cleanly to evidence-grounded long-horizon reasoning (Wu et al., 31 Mar 2025).

Taken together, the literature indicates that hierarchical streaming video understanding is best viewed not as a single architecture class but as a systems-level doctrine. It requires causal temporal abstraction, explicit management of finite memory across multiple timescales, mechanisms for selective refinement under uncertainty, and evaluation protocols that test perception, reasoning, synchronization, and long-range recall jointly rather than in isolation.