---
title: Interactive Video Reasoning
url: https://www.emergentmind.com/topics/interactive-video-reasoning
type: topic
---

# Interactive Video Reasoning

Interactive video reasoning denotes a class of video-understanding systems that interleave inference with active evidence acquisition, rather than encoding a video once and conducting all subsequent reasoning in text. Recent work characterizes this shift as a move from passive video understanding to active visual reasoning: models are expected to rewatch, refocus, zoom, seek clues, retrieve candidate videos, ground relevant segments, or verify intermediate observations before answering. The motivation is consistent across the literature: static encode-and-reason pipelines create a semantic bottleneck, frozen perception, and susceptibility to hallucination when reasoning drifts away from the actual video content [2511.23477][2604.20473][2601.23224].

## 1. From structured video understanding to agentic evidence seeking

The intellectual background of interactive video reasoning lies in earlier attempts to treat video as more than a flat sequence of frames. HOSTR represented a video as a dynamic stream of interacting objects, maintained object lifelines in a hierarchically nested spatio-temporal graph, and separated intra-object temporal aggregation from inter-object interaction, thereby making VideoQA more object-oriented, interpretable, and aligned with compositional reasoning [2106.13432]. In parallel, iReason combined videos and natural language captions to infer causal relationships between events, selected canonical frames, and added a causal rationalization module for explanation, interpretability, error analysis, and bias detection [2107.10300].

More recent work argues that such structuring is still insufficient if perception remains static during inference. Video-CoM states that most MLLMs “think about videos” rather than “think with videos”: once a video is encoded into a static representation, the model can no longer rewatch, refocus, zoom, or verify evidence, and its intermediate reasoning may drift toward world knowledge or generic priors [2511.23477]. Video-ToC sharpens the same diagnosis by arguing that Video LLMs often improve textual reasoning style without learning perception-aware adaptation to the input video, so smaller student models may “think” in language space, miss relevant visual evidence, and hallucinate [2604.20473].

The result is a conceptual redefinition of video reasoning. In this newer view, reasoning is not only deliberation over latent tokens; it is a perception-action process in which the model must decide where to look, what to inspect more closely, when to retrieve more evidence, and when to stop. Long-video systems such as Video-o3 make this explicit through native interleaved tool invocation, while training-free systems such as AVI and CAVIA impose structured retrieve-perceive-review or reasoning-perception loops that turn inference into iterative exploration rather than single-turn answer generation [2601.23224][2511.14446][2508.17932].

## 2. Core interaction paradigms

Current systems instantiate “interaction” in several distinct but related ways. Some localize cues within a given video; some manipulate the visual state directly; some orchestrate external tools; some treat retrieval itself as part of reasoning; and some use human visual prompts to determine what the model should reason about.

| Paradigm | Core mechanism | Representative work |
|---|---|---|
| Perception-aware cue localization | Coarse-to-fine traversal from whole video to evidence-bearing clips | Video-ToC [2604.20473] |
| Manipulation-driven reasoning | Iterative `find-segment`, `find-frame`, `spatial-zoom` | Video-CoM [2511.23477] |
| Native tool-interleaved long-video search | Clue seeking, crop-and-inspect, dynamic visual quota, adaptive stopping | Video-o3 [2601.23224] |
| End-to-end agentic tool use | Temporal grounding, frame selection, trim, tracking, spatial grounding | Weaver [2602.05829] |
| Training-free reasoning-perception coordination | Retrieve–Perceive–Review and confidence-driven iterative synthesis | AVI [2511.14446], CAVIA [2508.17932] |
| Inter-video plus intra-video reasoning | Search-engine interaction, verification, Soft Query Refinement | VideoSearch-R1 [2607.00446] |
| User-guided object-centric interaction | Visual referring input, grounded output, subject-conditioned relation discovery | RGA3 [2507.19599], Click2Graph [2511.15948] |

Video-ToC organizes reasoning as a tree-guided search over semantically coherent clips. A video is segmented into a tree whose leaves are clips and whose higher nodes are progressively coarser groupings; reasoning then traverses from the root toward key leaf nodes, so every step corresponds to localizing a more informative visual cue [2604.20473]. Video-CoM instead turns the video into an active cognitive workspace. At each step the model observes textual context and current visual input, emits both reasoning tokens and a manipulation, executes that manipulation, receives an updated visual state, and continues; the full trajectory explicitly alternates exploratory reasoning, visual manipulation, and observation [2511.23477].

Long-video systems emphasize shared multi-turn context. Video-o3 appends each cropped clip returned by a tool call back into the conversation and lets the model repeatedly seek additional clues, inspect fine-grained details in local segments, and adaptively stop once sufficient evidence has been acquired [2601.23224]. Weaver generalizes this to a tool library containing Temporal Grounding, Frame Selection, Temporal Count, Trim, Spatial Tracking, and Spatial Grounding, with the policy model learning when and how to invoke them during reasoning [2602.05829]. AVI constrains this process into Retrieve, Perceive, and Review phases, while CAVIA uses hierarchical localization, gap identification, targeted multimodal prompting, and iterative caption enhancement to create a closed loop in which reasoning continuously guides visual extraction [2511.14446][2508.17932].

Interactive reasoning can also occur across video corpora. VideoSearch-R1 rejects retrieval-as-preprocessing and instead alternates inter-video retrieval with intra-video verification and temporal grounding. If a retrieved video is a mismatch, the model refines the search through Soft Query Refinement in continuous latent space and tries again [2607.00446]. In another direction, RGA3 and Click2Graph make the interaction explicitly user-guided: a human can indicate an object or subject with a visual prompt, after which the system tracks it, reasons about its temporal behavior, grounds outputs, or discovers interacting objects and predicts temporally grounded scene-graph triplets [2507.19599][2511.15948].

## 3. Supervision, datasets, and annotation pipelines

A common theme in this literature is that ordinary video-question-answer triples do not teach a model how to search, manipulate, verify, or stop. Consequently, recent work has invested heavily in synthetic and semi-automatic trajectory construction. Video-ToC builds Video-ToC-SFT-1k and Video-ToC-RL-2k from LLaVA-Video-178K using clip segmentation, clip captioning, key-clip selection, tree backtracking, cue-description generation, and filtering for sufficiency of the final cues [2604.20473]. Video-CoM constructs Video-CoM-Instruct, an 18K instruction-tuning dataset with 15K for SFT and 3K for RA-GRPO, designed specifically for multi-step manipulation reasoning and built from about 9K videos for SFT plus 70 manually curated high-complexity YouTube videos for RL [2511.23477]. Video-o3 introduces Seeker-173K, about 173K high-quality tool-interaction trajectories synthesized through clue localization, validity verification, trajectory generation, and logical consistency checking [2601.23224].

Two other large-scale pipelines are particularly influential. VideoRFT constructs VideoRFT-CoT-102K for SFT and VideoRFT-RL-310K for RL by first producing structured textual video representations, then eliciting cognition-inspired preliminary CoTs from a reasoning LLM, and finally revising those CoTs with an MLLM conditioned on the actual video to reduce visual hallucinations [2505.12434]. ReWatch builds ReWatch-Caption-10k, ReWatch-QA-170k, and ReWatch-CoT-135k through hierarchical timestamped captioning, contrastive QA generation that filters out summary and text bias, and a Multi-Agent ReAct framework that simulates a human-like re-watching process with explicit segment retrieval and verification [2509.23652].

Interactive video reasoning has also driven new benchmark design. VideoInfer contains 1,620 videos and 28,811 object-centric QA pairs, uses eight visual prompt types, and is curated under the rules of indirectness and dynamism so that questions require reasoning rather than single-frame lookup [2507.19599]. LiViBench extends the setting to interactive livestreams by providing 3,168 videos, 3,175 multiple-choice questions, 24 tasks, and synchronized video, audio, speech, and real-time comments, with a standardized semi-automatic workflow that uses multiple MLLMs and human-in-the-loop refinement [2601.15016]. I-VQA creates a stricter implicit reasoning condition by removing explicit evidence segments and retaining 5,549 QA samples that must be answered from contextual clues [2506.07811]. RISE-Video broadens the scope further: it is a reasoning-oriented benchmark for Text-Image-to-Video synthesis with 467 human-annotated samples spanning eight categories and evaluated by Reasoning Alignment, Temporal Consistency, Physical Rationality, and Visual Quality [2602.05986].

## 4. Optimization objectives and reward design

The training literature largely agrees that sparse answer-only reward is inadequate for interactive video reasoning. Video-CoM argues that final-answer reward cannot distinguish grounded manipulation paths from lucky or spurious ones, while VideoRLVR observes that sparse success-only reward is too weak in low-success-rate settings such as FlowFree and Sokoban [2511.23477][2605.15458]. The response has been a rapid diversification of process-aware, grounded, or verifiable reward schemes.

Video-ToC introduces a reasoning-demand reward. Reasoning demand is estimated by asking the model to answer directly over multiple trials without thinking, measuring how often it succeeds, and then using that demand to rescale GRPO advantages so that harder questions receive stronger optimization pressure when answered correctly [2604.20473]. Video-CoM’s RA-GRPO adds a step-level reasoning reward that gives partial credit when predicted manipulations match annotated valid segments, frames, or regions; for spatial-zoom, correctness is determined by an IoU threshold of 0.3 [2511.23477]. Video-o3’s Verifiable Trajectory-Guided Reward combines answer correctness, formatting validity, a Hybrid Clue Score based on interval alignment, and a Turn Decay Factor that penalizes unnecessary exploration beyond the annotated reference limit [2601.23224].

Other systems reward the interaction protocol itself. Weaver’s tool-augmented GRPO uses a final reward weighted as correctness, format, and tool reward with coefficients 0.7, 0.2, and 0.1, respectively [2602.05829]. VideoRFT adds a semantic-consistency reward that measures whether the video-description portion of the reasoning trace semantically matches sampled video frames, and activates that reward only when answer accuracy is non-zero [2505.12434]. ReWatch-R1’s Observation & Reasoning reward explicitly judges whether intermediate observations are supported by the video and whether the resulting action-observation sequence is sufficient to recover the correct answer, thereby penalizing hallucinated or irrelevant CoTs [2509.23652].

Verifiability has also been extended to video generation. VideoRLVR models reverse denoising as an MDP, trains with SDE-GRPO, uses dense decomposed rewards tailored to Maze, FlowFree, and Sokoban, and introduces Early-Step Focus, which restricts optimization to the first 10 of 20 denoising steps and reduces training latency by about 40% while preserving performance [2605.15458]. VersaVid-R1 advances a different position: it argues that direct RL can outperform SFT-then-RL for video reasoning, uses no KL penalty in its main method, and assigns task-specific rewards across DarkEventInfer, MixVidQA, MCQA, and captioning, including a three-level judge reward for masked-event inference and an AutoDQ-plus-keyword reward for captioning [2506.09079].

## 5. Empirical performance and application domains

The empirical results show that interactive mechanisms tend to help most on reasoning-heavy, perception-heavy, or long-video settings. Video-ToC improves average accuracy over the baseline Qwen2.5-VL-7B and Video-R1 across six benchmarks at 16, 32, and 64 frames; at 64 frames it reaches 59.9 average accuracy versus 58.8 for Video-R1 and 55.6 for the baseline. On the hallucination benchmark VideoHallucer, it achieves 51.9 average accuracy versus 45.4 for Video-R1 and 50.5 for the baseline, with especially large improvements on object-relation and semantic-detail hallucination categories [2604.20473].

Manipulation-centric training shows similar effects. Video-CoM reports an average reasoning score of 40.5, a Video-CoM-Bench score of 68.7, and an average generic score of 63.9, while using only 25K SFT and 3K GRPO video samples. Its ablations show a progression from SFT at 64.0 on Video-CoM-Bench and 37.5 average reasoning, to GRPO at 66.7 and 38.9, to RA-GRPO at 68.7 and 40.5 [2511.23477]. VideoRFT reports state-of-the-art performance on six reasoning benchmarks, with best reported scores of 36.8 on VSI-Bench, 51.1 on VideoMMMU, 68.5 on MMVU, 62.1 on MVBench, 73.7 on TempCompass, and 59.8 on VideoMME [2505.12434]. ReWatch-R1 reaches 35.51 average on five challenging video reasoning benchmarks at 192 frames, improves long-video reasoning on videos longer than 20 minutes to 27.46%, and maintains 54.15 average on four general video understanding benchmarks [2509.23652].

Long-video agentic systems report strong gains from iterative evidence acquisition. Video-o3 achieves 72.1% on MLVU and 46.5% on Video-Holmes, and its appendix reports 10.2 seconds per sample on MLVU while outperforming decoupled iterative baselines [2601.23224]. Weaver reaches 75.4 on LVReason, 65.3 on VideoMME, 43.0 on LVBench, 54.5 on MLVU, 51.3 on VideoMMMU, 40.3 on VSIBench, and 67.7 on MVBench [2602.05829]. AVI reports 61.4 on LVBench, 59.8 on VideoMME-Long, 62.8 on LongVideoBench-Long, and 60.0 mIoU on Charades-STA, while CAVIA reaches 65.7 on EgoSchema, 76.1 on NExT-QA, and 73.8 on IntentQA, with especially strong gains on temporal and causal categories [2511.14446][2508.17932].

The paradigm is also spreading beyond closed-set QA on a given clip. VideoSearch-R1 reports state-of-the-art performance on Video Corpus Moment Retrieval, including 16.5, 13.4, and 8.2 on Charades-FIG for VCMR 0.3/R@1, 0.5/R@1, and 0.7/R@1; 33.3, 30.2, and 19.7 on DiDeMo-FIG; and 33.8, 22.3, and 12.3 on ActivityNet-FIG. Its Soft Query Refinement uses 8 soft tokens, compared with about 26.8 generated tokens for hard text-level query refinement on ActivityNet-FIG [2607.00446]. In object-centric and relational settings, RGA3 reports 74.0 average on VideoRefer-Bench\(^Q\) and 58.0 overall J&F on ReVOS, while Click2Graph establishes interactive PVSG on OpenPVSG and shows that subject-conditioned DIDM substantially outperforms a heuristic prompt generator [2507.19599][2511.15948]. In livestream video understanding, LiVi-LLM-7B reaches 64.4 overall on LiViBench and 60.9 on livestream-specific tasks, supporting the claim that real-time comments and interaction-specific retrieval materially affect reasoning quality [2601.15016].

## 6. Scope, misconceptions, and open problems

A persistent misconception is that longer textual chain-of-thought alone is sufficient for video reasoning. Several papers argue the opposite. Video-ToC states that models can improve textual reasoning style while still missing fine-grained spatio-temporal evidence, and Weaver argues that text-centric video CoT suffers from representational mismatch and frozen perception because the model cannot fetch missing visual evidence at the right moment [2604.20473][2602.05829]. Another misconception is that retrieval is merely preprocessing; VideoSearch-R1 directly contests this by treating retrieval and temporal grounding as a single multi-turn reasoning loop [2607.00446].

The scope of interactive video reasoning is also broader than question answering over a fixed clip. IRM shows that explicit visual evidence may be deliberately unavailable, so the model must reason from dual streams of contextual actions and intent clues, with iterative clue verification and visual enhancement [2506.07811]. LiViBench shows that interactive meaning can be conditioned by audio, speech, and massive comment streams, and that raw comments may hurt whereas retrieved relevant comments help [2601.15016]. In generation, VideoRLVR treats a video as a visual reasoning trajectory whose correctness is objectively verifiable, and RISE-Video evaluates TI2V models as world simulators that must obey implicit world rules rather than merely produce realistic frames [2605.15458][2602.05986]. This suggests that “interactive” now covers model-video, model-tool, model-corpus, and human-model interaction, as well as verifiable trajectory generation.

Several limitations remain recurrent. Tool quality and planning remain bottlenecks: Weaver depends on a curated library of strong frozen expert tools, and wrong planning can still lead to incorrect evidence retrieval [2602.05829]. VideoSearch-R1 treats episodes with no valid match within the allowed turns as failures, implying that retrieval quality and verification reliability remain central constraints [2607.00446]. CAVIA reports higher computational overhead than single-pass methods and dependence on initial caption quality, while RGA3 exhibits a failure on a video with over 1,000 frames when only 16 sampled frames are used and the key event falls in skipped frames [2508.17932][2507.19599]. A plausible implication is that the central unresolved problem is no longer whether video models can generate elaborate rationales, but whether they can couple evidence acquisition, memory, and adaptive termination tightly enough to remain grounded under severe context, latency, and search-budget constraints.

Source: https://www.emergentmind.com/topics/interactive-video-reasoning