Interactive Video Reasoning
- Interactive video reasoning is an approach where systems iteratively rewatch, refocus, and verify visual clues to overcome static representation limitations.
- It employs methods like cue localization, tool-interleaved search, and iterative evidence acquisition to refine understanding dynamically.
- Empirical results show improvements in accuracy and reduced hallucination through adaptive strategies and process-aware reward mechanisms.
Interactive video reasoning denotes a class of video-understanding systems that interleave inference with active evidence acquisition, rather than encoding a video once and conducting all subsequent reasoning in text. Recent work characterizes this shift as a move from passive video understanding to active visual reasoning: models are expected to rewatch, refocus, zoom, seek clues, retrieve candidate videos, ground relevant segments, or verify intermediate observations before answering. The motivation is consistent across the literature: static encode-and-reason pipelines create a semantic bottleneck, frozen perception, and susceptibility to hallucination when reasoning drifts away from the actual video content (Rasheed et al., 28 Nov 2025, Tan et al., 22 Apr 2026, Zeng et al., 30 Jan 2026).
1. From structured video understanding to agentic evidence seeking
The intellectual background of interactive video reasoning lies in earlier attempts to treat video as more than a flat sequence of frames. HOSTR represented a video as a dynamic stream of interacting objects, maintained object lifelines in a hierarchically nested spatio-temporal graph, and separated intra-object temporal aggregation from inter-object interaction, thereby making VideoQA more object-oriented, interpretable, and aligned with compositional reasoning (Dang et al., 2021). In parallel, iReason combined videos and natural language captions to infer causal relationships between events, selected canonical frames, and added a causal rationalization module for explanation, interpretability, error analysis, and bias detection (Chadha et al., 2021).
More recent work argues that such structuring is still insufficient if perception remains static during inference. Video-CoM states that most MLLMs “think about videos” rather than “think with videos”: once a video is encoded into a static representation, the model can no longer rewatch, refocus, zoom, or verify evidence, and its intermediate reasoning may drift toward world knowledge or generic priors (Rasheed et al., 28 Nov 2025). Video-ToC sharpens the same diagnosis by arguing that Video LLMs often improve textual reasoning style without learning perception-aware adaptation to the input video, so smaller student models may “think” in language space, miss relevant visual evidence, and hallucinate (Tan et al., 22 Apr 2026).
The result is a conceptual redefinition of video reasoning. In this newer view, reasoning is not only deliberation over latent tokens; it is a perception-action process in which the model must decide where to look, what to inspect more closely, when to retrieve more evidence, and when to stop. Long-video systems such as Video-o3 make this explicit through native interleaved tool invocation, while training-free systems such as AVI and CAVIA impose structured retrieve-perceive-review or reasoning-perception loops that turn inference into iterative exploration rather than single-turn answer generation (Zeng et al., 30 Jan 2026, Gao et al., 18 Nov 2025, Dong et al., 25 Aug 2025).
2. Core interaction paradigms
Current systems instantiate “interaction” in several distinct but related ways. Some localize cues within a given video; some manipulate the visual state directly; some orchestrate external tools; some treat retrieval itself as part of reasoning; and some use human visual prompts to determine what the model should reason about.
| Paradigm | Core mechanism | Representative work |
|---|---|---|
| Perception-aware cue localization | Coarse-to-fine traversal from whole video to evidence-bearing clips | Video-ToC (Tan et al., 22 Apr 2026) |
| Manipulation-driven reasoning | Iterative find-segment, find-frame, spatial-zoom |
Video-CoM (Rasheed et al., 28 Nov 2025) |
| Native tool-interleaved long-video search | Clue seeking, crop-and-inspect, dynamic visual quota, adaptive stopping | Video-o3 (Zeng et al., 30 Jan 2026) |
| End-to-end agentic tool use | Temporal grounding, frame selection, trim, tracking, spatial grounding | Weaver (Shi et al., 5 Feb 2026) |
| Training-free reasoning-perception coordination | Retrieve–Perceive–Review and confidence-driven iterative synthesis | AVI (Gao et al., 18 Nov 2025), CAVIA (Dong et al., 25 Aug 2025) |
| Inter-video plus intra-video reasoning | Search-engine interaction, verification, Soft Query Refinement | VideoSearch-R1 (Lee et al., 1 Jul 2026) |
| User-guided object-centric interaction | Visual referring input, grounded output, subject-conditioned relation discovery | RGA3 (Wang et al., 25 Jul 2025), Click2Graph (Ruschel et al., 20 Nov 2025) |
Video-ToC organizes reasoning as a tree-guided search over semantically coherent clips. A video is segmented into a tree whose leaves are clips and whose higher nodes are progressively coarser groupings; reasoning then traverses from the root toward key leaf nodes, so every step corresponds to localizing a more informative visual cue (Tan et al., 22 Apr 2026). Video-CoM instead turns the video into an active cognitive workspace. At each step the model observes textual context and current visual input, emits both reasoning tokens and a manipulation, executes that manipulation, receives an updated visual state, and continues; the full trajectory explicitly alternates exploratory reasoning, visual manipulation, and observation (Rasheed et al., 28 Nov 2025).
Long-video systems emphasize shared multi-turn context. Video-o3 appends each cropped clip returned by a tool call back into the conversation and lets the model repeatedly seek additional clues, inspect fine-grained details in local segments, and adaptively stop once sufficient evidence has been acquired (Zeng et al., 30 Jan 2026). Weaver generalizes this to a tool library containing Temporal Grounding, Frame Selection, Temporal Count, Trim, Spatial Tracking, and Spatial Grounding, with the policy model learning when and how to invoke them during reasoning (Shi et al., 5 Feb 2026). AVI constrains this process into Retrieve, Perceive, and Review phases, while CAVIA uses hierarchical localization, gap identification, targeted multimodal prompting, and iterative caption enhancement to create a closed loop in which reasoning continuously guides visual extraction (Gao et al., 18 Nov 2025, Dong et al., 25 Aug 2025).
Interactive reasoning can also occur across video corpora. VideoSearch-R1 rejects retrieval-as-preprocessing and instead alternates inter-video retrieval with intra-video verification and temporal grounding. If a retrieved video is a mismatch, the model refines the search through Soft Query Refinement in continuous latent space and tries again (Lee et al., 1 Jul 2026). In another direction, RGA3 and Click2Graph make the interaction explicitly user-guided: a human can indicate an object or subject with a visual prompt, after which the system tracks it, reasons about its temporal behavior, grounds outputs, or discovers interacting objects and predicts temporally grounded scene-graph triplets (Wang et al., 25 Jul 2025, Ruschel et al., 20 Nov 2025).
3. Supervision, datasets, and annotation pipelines
A common theme in this literature is that ordinary video-question-answer triples do not teach a model how to search, manipulate, verify, or stop. Consequently, recent work has invested heavily in synthetic and semi-automatic trajectory construction. Video-ToC builds Video-ToC-SFT-1k and Video-ToC-RL-2k from LLaVA-Video-178K using clip segmentation, clip captioning, key-clip selection, tree backtracking, cue-description generation, and filtering for sufficiency of the final cues (Tan et al., 22 Apr 2026). Video-CoM constructs Video-CoM-Instruct, an 18K instruction-tuning dataset with 15K for SFT and 3K for RA-GRPO, designed specifically for multi-step manipulation reasoning and built from about 9K videos for SFT plus 70 manually curated high-complexity YouTube videos for RL (Rasheed et al., 28 Nov 2025). Video-o3 introduces Seeker-173K, about 173K high-quality tool-interaction trajectories synthesized through clue localization, validity verification, trajectory generation, and logical consistency checking (Zeng et al., 30 Jan 2026).
Two other large-scale pipelines are particularly influential. VideoRFT constructs VideoRFT-CoT-102K for SFT and VideoRFT-RL-310K for RL by first producing structured textual video representations, then eliciting cognition-inspired preliminary CoTs from a reasoning LLM, and finally revising those CoTs with an MLLM conditioned on the actual video to reduce visual hallucinations (Wang et al., 18 May 2025). ReWatch builds ReWatch-Caption-10k, ReWatch-QA-170k, and ReWatch-CoT-135k through hierarchical timestamped captioning, contrastive QA generation that filters out summary and text bias, and a Multi-Agent ReAct framework that simulates a human-like re-watching process with explicit segment retrieval and verification (Zhang et al., 28 Sep 2025).
Interactive video reasoning has also driven new benchmark design. VideoInfer contains 1,620 videos and 28,811 object-centric QA pairs, uses eight visual prompt types, and is curated under the rules of indirectness and dynamism so that questions require reasoning rather than single-frame lookup (Wang et al., 25 Jul 2025). LiViBench extends the setting to interactive livestreams by providing 3,168 videos, 3,175 multiple-choice questions, 24 tasks, and synchronized video, audio, speech, and real-time comments, with a standardized semi-automatic workflow that uses multiple MLLMs and human-in-the-loop refinement (Wang et al., 21 Jan 2026). I-VQA creates a stricter implicit reasoning condition by removing explicit evidence segments and retaining 5,549 QA samples that must be answered from contextual clues (Chen et al., 9 Jun 2025). RISE-Video broadens the scope further: it is a reasoning-oriented benchmark for Text-Image-to-Video synthesis with 467 human-annotated samples spanning eight categories and evaluated by Reasoning Alignment, Temporal Consistency, Physical Rationality, and Visual Quality (Liu et al., 5 Feb 2026).
4. Optimization objectives and reward design
The training literature largely agrees that sparse answer-only reward is inadequate for interactive video reasoning. Video-CoM argues that final-answer reward cannot distinguish grounded manipulation paths from lucky or spurious ones, while VideoRLVR observes that sparse success-only reward is too weak in low-success-rate settings such as FlowFree and Sokoban (Rasheed et al., 28 Nov 2025, Zhu et al., 14 May 2026). The response has been a rapid diversification of process-aware, grounded, or verifiable reward schemes.
Video-ToC introduces a reasoning-demand reward. Reasoning demand is estimated by asking the model to answer directly over multiple trials without thinking, measuring how often it succeeds, and then using that demand to rescale GRPO advantages so that harder questions receive stronger optimization pressure when answered correctly (Tan et al., 22 Apr 2026). Video-CoM’s RA-GRPO adds a step-level reasoning reward that gives partial credit when predicted manipulations match annotated valid segments, frames, or regions; for spatial-zoom, correctness is determined by an IoU threshold of 0.3 (Rasheed et al., 28 Nov 2025). Video-o3’s Verifiable Trajectory-Guided Reward combines answer correctness, formatting validity, a Hybrid Clue Score based on interval alignment, and a Turn Decay Factor that penalizes unnecessary exploration beyond the annotated reference limit (Zeng et al., 30 Jan 2026).
Other systems reward the interaction protocol itself. Weaver’s tool-augmented GRPO uses a final reward weighted as correctness, format, and tool reward with coefficients 0.7, 0.2, and 0.1, respectively (Shi et al., 5 Feb 2026). VideoRFT adds a semantic-consistency reward that measures whether the video-description portion of the reasoning trace semantically matches sampled video frames, and activates that reward only when answer accuracy is non-zero (Wang et al., 18 May 2025). ReWatch-R1’s Observation & Reasoning reward explicitly judges whether intermediate observations are supported by the video and whether the resulting action-observation sequence is sufficient to recover the correct answer, thereby penalizing hallucinated or irrelevant CoTs (Zhang et al., 28 Sep 2025).
Verifiability has also been extended to video generation. VideoRLVR models reverse denoising as an MDP, trains with SDE-GRPO, uses dense decomposed rewards tailored to Maze, FlowFree, and Sokoban, and introduces Early-Step Focus, which restricts optimization to the first 10 of 20 denoising steps and reduces training latency by about 40% while preserving performance (Zhu et al., 14 May 2026). VersaVid-R1 advances a different position: it argues that direct RL can outperform SFT-then-RL for video reasoning, uses no KL penalty in its main method, and assigns task-specific rewards across DarkEventInfer, MixVidQA, MCQA, and captioning, including a three-level judge reward for masked-event inference and an AutoDQ-plus-keyword reward for captioning (Chen et al., 10 Jun 2025).
5. Empirical performance and application domains
The empirical results show that interactive mechanisms tend to help most on reasoning-heavy, perception-heavy, or long-video settings. Video-ToC improves average accuracy over the baseline Qwen2.5-VL-7B and Video-R1 across six benchmarks at 16, 32, and 64 frames; at 64 frames it reaches 59.9 average accuracy versus 58.8 for Video-R1 and 55.6 for the baseline. On the hallucination benchmark VideoHallucer, it achieves 51.9 average accuracy versus 45.4 for Video-R1 and 50.5 for the baseline, with especially large improvements on object-relation and semantic-detail hallucination categories (Tan et al., 22 Apr 2026).
Manipulation-centric training shows similar effects. Video-CoM reports an average reasoning score of 40.5, a Video-CoM-Bench score of 68.7, and an average generic score of 63.9, while using only 25K SFT and 3K GRPO video samples. Its ablations show a progression from SFT at 64.0 on Video-CoM-Bench and 37.5 average reasoning, to GRPO at 66.7 and 38.9, to RA-GRPO at 68.7 and 40.5 (Rasheed et al., 28 Nov 2025). VideoRFT reports state-of-the-art performance on six reasoning benchmarks, with best reported scores of 36.8 on VSI-Bench, 51.1 on VideoMMMU, 68.5 on MMVU, 62.1 on MVBench, 73.7 on TempCompass, and 59.8 on VideoMME (Wang et al., 18 May 2025). ReWatch-R1 reaches 35.51 average on five challenging video reasoning benchmarks at 192 frames, improves long-video reasoning on videos longer than 20 minutes to 27.46%, and maintains 54.15 average on four general video understanding benchmarks (Zhang et al., 28 Sep 2025).
Long-video agentic systems report strong gains from iterative evidence acquisition. Video-o3 achieves 72.1% on MLVU and 46.5% on Video-Holmes, and its appendix reports 10.2 seconds per sample on MLVU while outperforming decoupled iterative baselines (Zeng et al., 30 Jan 2026). Weaver reaches 75.4 on LVReason, 65.3 on VideoMME, 43.0 on LVBench, 54.5 on MLVU, 51.3 on VideoMMMU, 40.3 on VSIBench, and 67.7 on MVBench (Shi et al., 5 Feb 2026). AVI reports 61.4 on LVBench, 59.8 on VideoMME-Long, 62.8 on LongVideoBench-Long, and 60.0 mIoU on Charades-STA, while CAVIA reaches 65.7 on EgoSchema, 76.1 on NExT-QA, and 73.8 on IntentQA, with especially strong gains on temporal and causal categories (Gao et al., 18 Nov 2025, Dong et al., 25 Aug 2025).
The paradigm is also spreading beyond closed-set QA on a given clip. VideoSearch-R1 reports state-of-the-art performance on Video Corpus Moment Retrieval, including 16.5, 13.4, and 8.2 on Charades-FIG for VCMR 0.3/R@1, 0.5/R@1, and 0.7/R@1; 33.3, 30.2, and 19.7 on DiDeMo-FIG; and 33.8, 22.3, and 12.3 on ActivityNet-FIG. Its Soft Query Refinement uses 8 soft tokens, compared with about 26.8 generated tokens for hard text-level query refinement on ActivityNet-FIG (Lee et al., 1 Jul 2026). In object-centric and relational settings, RGA3 reports 74.0 average on VideoRefer-Bench and 58.0 overall J&F on ReVOS, while Click2Graph establishes interactive PVSG on OpenPVSG and shows that subject-conditioned DIDM substantially outperforms a heuristic prompt generator (Wang et al., 25 Jul 2025, Ruschel et al., 20 Nov 2025). In livestream video understanding, LiVi-LLM-7B reaches 64.4 overall on LiViBench and 60.9 on livestream-specific tasks, supporting the claim that real-time comments and interaction-specific retrieval materially affect reasoning quality (Wang et al., 21 Jan 2026).
6. Scope, misconceptions, and open problems
A persistent misconception is that longer textual chain-of-thought alone is sufficient for video reasoning. Several papers argue the opposite. Video-ToC states that models can improve textual reasoning style while still missing fine-grained spatio-temporal evidence, and Weaver argues that text-centric video CoT suffers from representational mismatch and frozen perception because the model cannot fetch missing visual evidence at the right moment (Tan et al., 22 Apr 2026, Shi et al., 5 Feb 2026). Another misconception is that retrieval is merely preprocessing; VideoSearch-R1 directly contests this by treating retrieval and temporal grounding as a single multi-turn reasoning loop (Lee et al., 1 Jul 2026).
The scope of interactive video reasoning is also broader than question answering over a fixed clip. IRM shows that explicit visual evidence may be deliberately unavailable, so the model must reason from dual streams of contextual actions and intent clues, with iterative clue verification and visual enhancement (Chen et al., 9 Jun 2025). LiViBench shows that interactive meaning can be conditioned by audio, speech, and massive comment streams, and that raw comments may hurt whereas retrieved relevant comments help (Wang et al., 21 Jan 2026). In generation, VideoRLVR treats a video as a visual reasoning trajectory whose correctness is objectively verifiable, and RISE-Video evaluates TI2V models as world simulators that must obey implicit world rules rather than merely produce realistic frames (Zhu et al., 14 May 2026, Liu et al., 5 Feb 2026). This suggests that “interactive” now covers model-video, model-tool, model-corpus, and human-model interaction, as well as verifiable trajectory generation.
Several limitations remain recurrent. Tool quality and planning remain bottlenecks: Weaver depends on a curated library of strong frozen expert tools, and wrong planning can still lead to incorrect evidence retrieval (Shi et al., 5 Feb 2026). VideoSearch-R1 treats episodes with no valid match within the allowed turns as failures, implying that retrieval quality and verification reliability remain central constraints (Lee et al., 1 Jul 2026). CAVIA reports higher computational overhead than single-pass methods and dependence on initial caption quality, while RGA3 exhibits a failure on a video with over 1,000 frames when only 16 sampled frames are used and the key event falls in skipped frames (Dong et al., 25 Aug 2025, Wang et al., 25 Jul 2025). A plausible implication is that the central unresolved problem is no longer whether video models can generate elaborate rationales, but whether they can couple evidence acquisition, memory, and adaptive termination tightly enough to remain grounded under severe context, latency, and search-budget constraints.