VideoStir: Understanding Long Videos via Spatio-Temporally Structured and Intent-Aware RAG

Abstract: Scaling multimodal LLMs (MLLMs) to long videos is constrained by limited context windows. While retrieval-augmented generation (RAG) is a promising remedy by organizing query-relevant visual evidence into a compact context, most existing methods (i) flatten videos into independent segments, breaking their inherent spatio-temporal structure, and (ii) depend on explicit semantic matching, which can miss cues that are implicitly relevant to the query's intent. To overcome these limitations, we propose VideoStir, a structured and intent-aware long-video RAG framework. It firstly structures a video as a spatio-temporal graph at clip level, and then performs multi-hop retrieval to aggregate evidence across distant yet contextually related events. Furthermore, it introduces an MLLM-backed intent-relevance scorer that retrieves frames based on their alignment with the query's reasoning intent. To support this capability, we curate IR-600K, a large-scale dataset tailored for learning frame-query intent alignment. Experiments show that VideoStir is competitive with state-of-the-art baselines without relying on auxiliary information, highlighting the promise of shifting long-video RAG from flattened semantic matching to structured, intent-aware reasoning. Codes and checkpoints are available at Github.

• The paper introduces a retrieval-augmented generation framework that leverages spatio-temporal graph modeling and intent-aware evidence selection.
• The method employs multi-hop graph traversal and a LoRA-tuned scorer to enhance retrieval accuracy and coherence in complex video narratives.
• Experimental results demonstrate performance gains over baseline RAG methods, emphasizing the importance of structured retrieval for long-video reasoning.

VideoStir: Spatio-Temporally Structured and Intent-Aware RAG for Long-Video Understanding

Introduction

VideoStir introduces a new retrieval-augmented generation (RAG) paradigm for long-video understanding that directly addresses two major bottlenecks in visual reasoning with MLLMs: the contextual fragmentation induced by flattened segmentations, and the myopic reliance on explicit semantic similarity for evidence retrieval. The framework tightly integrates structured spatio-temporal modeling and intent-oriented evidence selection, aiming to optimize the relevance and coherence of visual cues provided to downstream multimodal LLMs.

The approach is motivated by limitations observed in prior long-video models, including those employing enlarged context windows or dense uniform sampling. Such strategies are susceptible to losing fine-grained cues under subsampling and creating redundant evidence or overwhelming noise. Additionally, state-of-the-art RAG solutions typically operate over flattened video segments utilizing contrastive models (e.g., CLIP), which leads to the decoupling of intrinsic spatio-temporal context and the omission of cues latent to the query’s intent.

Figure 1: Paradigm shift from flat semantic matching to structured, intent-aware long-video RAG. VideoStir structures videos as spatio-temporal graphs, overcoming the pitfalls of context flattening.

Method

Spatio-Temporal Topology Modeling

The initial transformation in VideoStir converts video streams into spatio-temporal graphs, where nodes correspond to clips extracted via event boundary detection (using techniques such as PELT-based change-point analysis over vision-language embeddings), and edges encode both chronological (temporal) and semantic (spatial) proximity. This graph structure ensures that contextually pertinent evidence—possibly separated by significant temporal gaps but semantically linked—is retrievable by explicit multi-hop traversal.

Graph-Based Multi-Hop Clip Retrieval

Upon receiving a query, the system embeds it into the shared multimodal space and selects anchor nodes by standard similarity measures. Then, multi-hop graph traversal is performed to aggregate not only those clips with surface-level semantic alignment, but also spatio-temporally and semantically adjacent clips, filtered by edge weights and hop constraints. This method re-entangles interleaved events, critical for reconstructing complex narratives that are characteristically missed in flat retrieval.

Figure 2: Overview of VideoStir: from spatio-temporal graph construction (a), anchor-based multi-hop retrieval (b), to intent-aware frame scoring (c).

Figure 3: Structured retrieval generates more contextually relevant, cleaner evidence compared to flat semantic clip matching, enhancing downstream reasoning.

Intent-Aware Frame Retrieval

To further filter the aggregated clips, VideoStir introduces an intent-relevance scorer, a LoRA-tuned, lightweight MLLM. This module is fine-tuned with pseudo-labeled data from the newly curated IR-600K dataset, where high-capacity MLLMs annotate frame-query pairs with relevance assessments on a 1–5 ordinal scale. The scorer outputs a probability-weighted expected relevance score for each frame, directly optimizing for frames that support the underlying reasoning necessary to answer the query, not merely those exhibiting high textual or visual overlap.

The IR-600K Dataset

IR-600K is the first large-scale resource explicitly targeting intent-level frame-query alignment, encompassing over 600K query–frame pairs with relevance labels distilled from teacher MLLMs. Unlike saliency-driven datasets (e.g., TVSum), this corpus is engineered to bridge the semantic–intent gap and is instrumental in tuning scorers for high-fidelity evidence selection.

Figure 4: Distribution of intent relevance scores in IR-600K, demonstrating balanced coverage across relevance levels in train and validation splits.

Experimental Results

Across LV-Bench, MLVU, Video-MME-Long, and EgoSchema, VideoStir demonstrates dominant or near-dominant performance versus state-of-the-art RAG baselines and agentic systems, all while relying only on native visual inputs—excluding any form of auxiliary textual or multimodal augmentation. Reported absolute gains are generally in the 1–7% range over baseline RAG systems, and system-level comparisons show VideoStir performing at or above the level of agentic approaches employing expert-captioned or OCR-augmented signals.

Component-level ablations reveal marked drops in retrieval accuracy and end-task performance upon removal of the intent-relevance scorer (even when using robust vision-language encoders), weighted expectation in scorer output, or the spatio-temporal graph structure. This substantiates the criticality of both structure-aware retrieval and intent supervision over naive segment matching or pure contrastive retrieval.

Implications and Future Directions

The primary contribution of VideoStir is evidencing that the limitations of current long-video MLLM comprehension pipelines are fundamentally retrieval-centric: improving the structure and intent-alignment of selected visual cues enables maximal leverage of MLLM downstream reasoning, independent of further context window scaling or contrastive model pretraining. The IR-600K resource and the LoRA-based scorer also constitute valuable foundations and benchmarks for future intent-sensitive retrieval systems.

A clear implication is that further improvements in long-video question answering, reasoning, and action understanding are likely to be retrieval- and structure-driven, rather than merely based on expanding LLM or MLLM scale. Architectural directions such as more expressive graph-based neighborhood sampling, integration with symbolic or event-centric representations, or adaptive context modeling conditioned on task intent are all directly motivated by VideoStir’s empirical findings.

On the practical side, structured retrieval incurs a latency overhead relative to flattened pipelines, but the system design shows a preferable tradeoff in accuracy and evidence quality. Addressing system latency remains a priority for real-time or agentic long-video applications.

Conclusion

VideoStir provides an explicit framework for spatio-temporally structured and intent-aware retrieval-augmented generation on long videos. By addressing structure decoupling and intent misalignment, it demonstrates that upstream retrieval and evidence organization represent principal levers for scaling video–language reasoning. The release of IR-600K and an open-source implementation further solidify this work as a blueprint for next-generation intent-centric video RAG systems.