LongVideo-R1: Efficient Hierarchical Video QA

Updated 5 July 2026

LongVideo-R1 is an active, reasoning-equipped multimodal model that applies chain-of-thought reasoning for efficient long-video question answering.
It uses a tree-structured video decomposition to enable hierarchical navigation and selective evidence acquisition, reducing computational cost.
The system is fine-tuned with supervised learning and refined via reinforcement learning to balance accuracy and compute efficiency.

Searching arXiv for LongVideo-R1 and closely related long-video understanding work to ground the article with current references. LongVideo-R1 is an active, reasoning-equipped multimodal LLM agent for low-cost long video understanding. It is designed for settings in which exhaustive traversal of hour-scale video is computationally prohibitive, and it replaces uniform scan with hierarchical navigation guided by chain-of-thought-with-tool reasoning. The system is fine-tuned on top of Qwen-3-8B, operates over a tree-structured decomposition of the input video, and interleaves a reasoning-and-planning module with captioning and leaf-level video question answering tools. Its central claim is that long-video QA should be optimized as a Pareto trade-off between answer accuracy and computational cost rather than as pure accuracy under unconstrained search (Qiu et al., 24 Feb 2026).

1. Problem setting and optimization objective

LongVideo-R1 is motivated by a practical constraint in long-video QA: prior agent-based systems can achieve strong performance, but their total cost grows linearly with video duration because they repeatedly invoke expensive captioning and QA tools on many sub-clips. The framework therefore formalizes inference cost at the episode level as

$C_{\rm total} = C_1\,T_1 + C_2\,T_2 + C_3\,T_3,$

where $C_1$ is the number of reasoning-model calls, $C_2$ the number of video caption calls, and $C_3$ the number of video QA calls. The objective is to maximize expected QA accuracy while minimizing $C_{\rm total}$ (Qiu et al., 24 Feb 2026).

This formulation distinguishes LongVideo-R1 from methods that assume exhaustive retrieval or dense captioning. A plausible implication is that the method treats navigation policy itself as a first-class optimization target, rather than merely a preprocessing step. In the long-video regime, this matters because the dominant bottleneck is often not only representation quality but also the number of expensive tool invocations required before answer generation.

2. Hierarchical representation and agent architecture

The model represents an input video $\mathbb{V}[0,T]$ as a $D$ -level tree, with $D=3$ in practice. Each non-leaf node is split into

$K\approx (T/16\mathrm{s})^{1/D}$

equal-length children, so that leaf nodes cover roughly $16$ s each. A clip at depth $C_1$ 0 is denoted $C_1$ 1. This hierarchy enables the agent to begin from coarse summaries and progressively narrow its search without enumerating all leaves (Qiu et al., 24 Feb 2026).

At inference time, LongVideo-R1 interleaves a reasoning module $C_1$ 2, based on Qwen-3-8B, with two external tools:

$C_1$ 3, which samples $C_1$ 4 frames and returns a text description $C_1$ 5
$C_1$ 6, callable only on leaf nodes, which returns a candidate answer

Each episode is modeled as a chain-of-thought-with-tool trajectory,

$C_1$ 7

where $C_1$ 8 is the agent’s natural-language reasoning at step $C_1$ 9, $C_2$ 0 is the tool output, and $C_2$ 1 is the final answer. The architectural emphasis is thus not on a monolithic end-to-end video encoder, but on a reasoning policy that decides which clip to inspect next and when to terminate.

The navigation algorithm begins with a top-level caption: $C_2$ 2 The initial history is set to $C_2$ 3, after which the reasoning model produces $C_2$ 4. The loop then continues until the reasoning trace contains an answer: the agent parses from its own reasoning which tool to call next and on which node, executes that tool, appends the result to the history, and invokes reasoning again (Qiu et al., 24 Feb 2026).

Operationally, the agent can “zoom in” to children, “slide” across siblings, or terminate early. The decision criterion is not hand-coded as a separate search heuristic; it emerges from the learned chain-of-thought policy, which is trained to balance expected information gain against extra cost. This is a substantive design choice: termination is part of the learned policy, not an externally fixed budget rule.

A common misconception is that long-video QA requires either uniform dense sampling or exhaustive clip traversal. LongVideo-R1 directly contests that assumption. Its navigation policy starts from top-level visual summaries and iteratively refines focus, immediately halting exploration upon acquiring sufficient knowledge to answer the query. The method therefore frames long-video understanding as selective evidence acquisition.

4. Training data, supervision, and reinforcement learning

LongVideo-R1 is trained in two stages on top of a pretrained Qwen-3-8B backbone: supervised fine-tuning followed by reinforcement learning. For supervised training, the system uses CG-Bench’s $C_2$ 5 K long videos and $C_2$ 6 K QA pairs, with $C_2$ 7 videos and $C_2$ 8 K QA pairs used to generate chain-of-thought-with-tool data. GPT-5 is prompted zero-shot, with minor fall-back hints drawn from timestamp annotations, to produce complete reasoning trajectories from top-level captions to final answers. After filtering and minimal hint insertion in approximately $C_2$ 9 of cases, the dataset contains $C_3$ 0 K verified trajectories averaging $C_3$ 1 steps each, yielding $C_3$ 2 K training samples (Qiu et al., 24 Feb 2026).

The SFT stage uses explicitly tagged trajectories with 〈think〉, 〈tool〉, and 〈answer〉 markers, teaching the model when to reason, when to call which tool, and when to stop. The remaining $C_3$ 3 videos and $C_3$ 4 K QA pairs are held out for RL. Reinforcement learning uses Group Relative Policy Optimization to refine the policy $C_3$ 5 with the composite reward

$C_3$ 6

where $C_3$ 7 rewards answer correctness, $C_3$ 8 measures a segment-level F1 between visited nodes and ground-truth intervals, and $C_3$ 9 penalizes revisiting the same clip (Qiu et al., 24 Feb 2026).

The location reward is particularly important because it supervises navigation quality rather than only final correctness. This suggests that the method is not merely learning answer production from sparse rewards; it is also learning a latent notion of efficient evidence localization.

5. Empirical performance and efficiency profile

The reported results emphasize the accuracy-efficiency trade-off rather than isolated accuracy. On LVBench, which contains $C_{\rm total}$ 0 videos with average duration $C_{\rm total}$ 1 s and $C_{\rm total}$ 2 QA pairs, LongVideo-R1 attains $C_{\rm total}$ 3 accuracy and does so with only approximately $C_{\rm total}$ 4 rounds of navigation per question, corresponding to about $C_{\rm total}$ 5 min. An updated version, denoted “new†”, using Qwen3-VL-32B for captions reaches $C_{\rm total}$ 6 overall. On MLVU and Video-MME-Long, LongVideo-R1 reaches $C_{\rm total}$ 7 and $C_{\rm total}$ 8 without subtitles, respectively, and remains competitive even when the setting is less favorable to selective navigation (Qiu et al., 24 Feb 2026).

A core efficiency comparison is against Ego-R1: Ego-R1 captions every $C_{\rm total}$ 9 s, for approximately $\mathbb{V}[0,T]$ 0 calls, whereas LongVideo-R1 averages $\mathbb{V}[0,T]$ 1 tool calls, yielding a $\mathbb{V}[0,T]$ 2– $\mathbb{V}[0,T]$ 3 reduction in runtime. The reported Pareto curve further indicates that one can trade only $\mathbb{V}[0,T]$ 4 pt of accuracy to reduce processing time from $\mathbb{V}[0,T]$ 5 min to $\mathbb{V}[0,T]$ 6 min per QA (Qiu et al., 24 Feb 2026).

Ablation studies identify three points. More SFT data, comparing $\mathbb{V}[0,T]$ 7 K against $\mathbb{V}[0,T]$ 8 K samples, improves both raw and RL-fine-tuned performance. The F1-style location reward $\mathbb{V}[0,T]$ 9 is critical to navigation efficiency. Varying the maximum reasoning rounds or caption-model scale produces a smooth trade-off between accuracy and time. These findings indicate that LongVideo-R1’s gains are inseparable from its policy-training recipe; the method is not only a search heuristic layered over a fixed backbone.

6. Position within long-video understanding research

LongVideo-R1 belongs to a broader transition from exhaustive processing toward selective or adaptive long-video reasoning. DrVideo converts long-video QA into long-document understanding by retrieving and augmenting text descriptions of relevant frames (Ma et al., 2024). AdaVideoRAG routes queries among no retrieval, naïve clip-level retrieval, and graph-based retrieval according to query complexity (Xue et al., 16 Jun 2025). VideoRouter addresses the visual-token bottleneck through query-adaptive dual routing and budgeted evidence allocation (Lin et al., 7 May 2026). LOVE-R1 introduces a slow-fast adaptive frame sampling mechanism and explicit zoom-in reasoning over high-resolution clips (Fu et al., 29 Sep 2025). QuickVideo, by contrast, targets real-time throughput through parallel decoding, KV-cache pruning, and CPU-GPU overlap rather than agent policy learning (Schneider et al., 22 May 2025).

Benchmark design in this area has increasingly foregrounded long-range retrieval and reasoning difficulty. LongVideoBench defines referring reasoning over interleaved frames and subtitles up to an hour long and reports that performance improves only when models can process more frames (Wu et al., 2024). LongVQUBench further shows hierarchical degradation from local event quality understanding to global quality understanding in long-term perceptual reasoning (Nema et al., 1 Jul 2026). Against this backdrop, LongVideo-R1 can be read as a navigation-centric response: instead of only increasing frame budgets or compressing token streams, it learns when and where to look.

This suggests that LongVideo-R1 occupies a specific branch of the long-video literature: agentic hierarchical navigation under explicit compute constraints. Its closest conceptual relatives are systems that make retrieval, zoom, or routing decisions contingent on the question, but its distinctive contribution is to train the decision process itself with verified chain-of-thought-with-tool trajectories and RL.

7. Limitations, failure modes, and prospective directions

The paper identifies several limitations. LongVideo-R1 is less dominant on questions whose answers depend on broad global context, where uniform sampling can be advantageous. Reported failure modes include distraction by visually similar but irrelevant clips, although simple human hints can steer the model back on track (Qiu et al., 24 Feb 2026).

Future directions proposed for the framework include expanding the toolset, for example with instance recognition, exploring multi-question amortized inference, and co-training video description tools alongside the reasoning agent. These directions follow directly from the agentic design: once long-video understanding is formulated as controlled tool use over a hierarchy, improvements can come either from better policies, better tools, or tighter joint optimization between them.

In that sense, LongVideo-R1 is not only a single model but also a problem formulation. It recasts long-video understanding as sequential decision-making over a hierarchical visual context, with answer accuracy and compute budget treated as co-equal constraints.