Video CoM Instruct: Interactive Video Reasoning

• Video CoM Instruct is an instruction-tuning dataset that trains models to actively manipulate video evidence, transforming passive sequences into dynamic cognitive workspaces.
• It employs three atomic operations—find-segment, find-frame, and spatial-zoom—to extract temporally localized and spatially detailed evidence for refined reasoning.
• The dataset’s 18K QA pairs, reinforced with a GRPO module, support multi-step interactive reasoning and demonstrate improved performance across diverse video benchmarks.

Video CoM Instruct is an instruction-tuning dataset introduced as the data foundation of the Video-CoM framework, a system for interactive video reasoning in which a multimodal LLM reasons by actively manipulating video evidence rather than treating video as a fixed encoded context (Rasheed et al., 28 Nov 2025). Its defining purpose is to teach a model not only to answer questions about videos, but to decide when to revisit a temporal segment, which frame to isolate, where to zoom spatially, and how to chain these operations into a grounded multi-step reasoning trajectory. In this formulation, the video becomes a dynamic cognitive workspace rather than a passive input, and the dataset supervises that behavior through structured Chain-of-Manipulations traces, question-answer pairs, and, for a reinforcement-learning subset, dense step-level temporal and spatial annotations (Rasheed et al., 28 Nov 2025).

1. From passive video reasoning to interactive video reasoning

The paper positions Video-CoM-Instruct within a broader critique of conventional video MLLMs. In the prevailing passive paradigm, a video is encoded once and subsequent reasoning unfolds almost entirely in language space, so the model effectively “thinks about videos” rather than “thinks with videos” (Rasheed et al., 28 Nov 2025). This creates what the authors call a semantic bottleneck: because the model cannot rewatch, refocus, or verify evidence during inference, it is prone to drift toward generic world knowledge and weakly grounded textual reasoning when tasks demand fine-grained temporal localization, precise spatial inspection, or multi-step evidence aggregation.

Video-CoM addresses this limitation through Interactive Video Reasoning, in which the model performs visual actions during reasoning. At reasoning step $i$, given text context $T_i$ and current visual input $\mathcal{V}_i$, the model generates

$$\mathcal{O}_i = f_{\text{MLLM}}(T_i, \mathcal{V}_i),$$

where $\mathcal{O}_i$ includes both reasoning tokens and a predicted manipulation $m_i$. The manipulation updates the visual input according to

$\mathcal{V}_{i+1} = g(\mathcal{V}_i, m_i).$

Over multiple turns, this yields a trajectory

$$\tau = \{(T_1, \mathcal{V}_1, m_1), (T_2, \mathcal{V}_2, m_2), \ldots, (T_N, \mathcal{V}_N, m_N)\}.$$

Video-CoM-Instruct is designed specifically to make such trajectories learnable (Rasheed et al., 28 Nov 2025).

This orientation distinguishes the framework from earlier instruction-conditioned video comprehension systems that rely on event descriptions, captions, or compact evidence representations but do not make manipulation itself the central supervised object. A closely related precursor is VidCoM, which already treated video comprehension as instruction-conditioned event reasoning with tool outputs and temporal refinement, but did so through scene graphs, captions, and event alignment rather than through explicit frame/segment/zoom manipulations (Qi et al., 2023). Video-CoM-Instruct extends the instruction-following perspective into a manipulation-centric regime (Rasheed et al., 28 Nov 2025).

2. Chain of Manipulations as the operative reasoning interface

The core operational concept taught by Video-CoM-Instruct is the Chain of Manipulations (CoM), defined as the sequence of visual operations selected during reasoning: $CoM = \{m_1, m_2, \ldots, m_{N-1}\}.$ The manipulation space contains three atomic operations (Rasheed et al., 28 Nov 2025).

Find-Segment revisits a short temporal clip from the original video $\mathcal{V}_0$ using a segment index $T_i$0: $T_i$1 Its role is temporal revisitation: locating brief events, rewatching critical moments, or restricting reasoning to an instruction-relevant temporal window.

Find-Frame isolates a single frame from the original video using frame index $T_i$2: $T_i$3 This operation is used when evidence is fleeting but becomes legible once frozen, such as object attributes, counts, small text, or transient visual states.

Spatial-Zoom magnifies a localized region of a frame using frame index $T_i$4 and bounding box $T_i$5: $T_i$6 It is intended for fine-grained inspection of signs, prices, logos, scores, or other small regions that are visually present but not easily readable at full scale (Rasheed et al., 28 Nov 2025).

These primitives induce a characteristic reasoning grammar. The paper identifies common chains such as find-frame $T_i$7 spatial-zoom for reading small text, find-segment for counting or localizing repeated transient actions, find-segment $T_i$8 find-frame, and find-segment $T_i$9 find-frame $\mathcal{V}_i$0 spatial-zoom for multi-stage evidence gathering (Rasheed et al., 28 Nov 2025). The dataset is explicitly curated to expose the model to these patterns and to teach when each manipulation is warranted.

The CoM notion also clarifies what “instruction” means in this setting. The question-answer task is not sufficient; the model must infer a reasoning program over the video. Thus, Video-CoM-Instruct supervises a model to decide which evidence is missing, select an operation to acquire it, observe the result, and continue. In the appendix description, every trace interleaves three components: Exploratory-Reasoning, Visual-Manipulation, and Observation (Rasheed et al., 28 Nov 2025). This structure makes the reasoning process inspectable rather than leaving it implicit in free-form text.

3. Dataset design, composition, and curation pipeline

Video-CoM-Instruct contains 18K QA pairs curated for multi-step manipulation reasoning, with 15K used for supervised fine-tuning and 3K reserved for GRPO-based reinforcement learning (Rasheed et al., 28 Nov 2025). The dataset combines 16.5K examples sampled from existing video sources and 1.5K examples from a manually curated set of high-complexity videos. The existing-source videos are drawn from the Video-LLaVA-178K ecosystem, including ActivityNet, YouCookII, Charades, HD-VILA, InternVid, VidOR, LanguageBind sources, Kinetics, and Ego4D; the appendix further specifies that the selected 9K existing-source videos are distributed approximately as ActivityNet: 28%, Charades: 19%, VidOR: 17%, and YouCookII: 14%, with the remainder from YouTube videos collected by LLaVA-178K, PerceptionTest, CLEVRER, and Ego4D (Rasheed et al., 28 Nov 2025).

The dataset is produced by a five-stage pipeline: curating videos for active reasoning, designing chain-of-manipulations, interactive video reasoning data generation, data verification, and dense spatio-temporal annotation (Rasheed et al., 28 Nov 2025). Video selection is guided by information density. The authors first generate captions with Qwen2.5-VL-72B when needed, then use Qwen3-235B-A22B to score videos on a 1–10 scale emphasizing scene diversity, number of activities, object interactions, visible text or small details, and changes over time. Approximately 9K high-information-density videos are retained after filtering (Rasheed et al., 28 Nov 2025).

For QA generation, the authors use Gemini-2.5-Pro to create 3–5 QA pairs per video, yielding about 25K candidate QA pairs in total; for the manually curated 70 high-complexity videos, the appendix notes up to 20 QA pairs per video (Rasheed et al., 28 Nov 2025). The generation prompt requires questions that cannot be answered by passive viewing, but instead require rewatching segments, freezing precise frames, zooming into small regions, or combinations thereof. Each generated item includes a question, a short answer, 5 multiple-choice options (A–E) with exactly one correct answer, and a brief reasoning plan listing the sequence of tools (Rasheed et al., 28 Nov 2025).

Quality control is stringent. Each candidate QA is re-evaluated by Gemini-2.5-Pro in both multiple-choice and open-ended formats; both the reasoning trace and final answer must agree with the intended answer. This removes ambiguous, inconsistent, or unanswerable examples. The retention rate is about 75%, reducing the initial 25K candidates to 18K final examples (Rasheed et al., 28 Nov 2025).

After filtering, each retained sample includes the question, answer, MCQ options, and a CoM reasoning trace (Rasheed et al., 28 Nov 2025). That trace is the central annotation object: it teaches the model a manipulation trajectory, not merely a final label. For the 3K GRPO subset, the supervision becomes denser still: the paper states that it includes all valid segments for each find-segment step, all valid frames for each find-frame step, and manually annotated bounding boxes propagated into tubelets with SAM2.1 for spatial grounding (Rasheed et al., 28 Nov 2025).

4. Supervised fine-tuning and reasoning-aware reinforcement learning

Video-CoM uses Qwen2.5-VL-7B-Instruct as its base model and trains it in two stages: Stage I: Supervised Fine-Tuning (SFT) and Stage II: RA-GRPO (Rasheed et al., 28 Nov 2025). Video-CoM-Instruct is central to both stages.

In SFT, the model is trained on the 15K SFT split of Video-CoM-Instruct, which provides the high-quality manipulation reasoning traces. This supervision is augmented with 9K ActivityNet samples without manipulations to strengthen temporal understanding and frame localization, and 180K Visual-CoT image samples with spatial grounding to improve spatial-zoom capability (Rasheed et al., 28 Nov 2025). The paper summarizes the resulting SFT corpus as 25K video samples and 180K image samples. The appendix adds that the 15K Video-CoM-Instruct subset is concatenated twice, yielding 30K effective samples, though the main text continues to refer to 15K high-quality samples (Rasheed et al., 28 Nov 2025). The SFT setup uses DeepSpeed ZeRO-3, bf16, context length 32K, global batch size 32, learning rate $\mathcal{V}_i$1, cosine decay, 3% warm-up, and max reasoning turns: 5 (Rasheed et al., 28 Nov 2025).

The paper argues that SFT alone is insufficient because multiple manipulation trajectories can lead to the same answer, and because the correct trajectory is strongly task-dependent (Rasheed et al., 28 Nov 2025). It therefore adds Group Relative Policy Optimization (GRPO) over the 3K densely annotated Video-CoM-Instruct subset. The RL stage uses global batch size 64, learning rate $\mathcal{V}_i$2, group size 8, KL coefficient 0.04, max turns 5, and spatial IoU threshold $\mathcal{V}_i$3 (Rasheed et al., 28 Nov 2025).

The distinctive feature of this stage is reasoning-aware reward design. For each predicted manipulation $\mathcal{V}_i$4, correctness $\mathcal{V}_i$5 is defined as

$\mathcal{V}_i$6

where $\mathcal{V}_i$7 is the set of valid segments, $\mathcal{V}_i$8 the set of valid frames, and $\mathcal{V}_i$9 a valid bounding box from the annotated set $$\mathcal{O}_i = f_{\text{MLLM}}(T_i, \mathcal{V}_i),$$0 (Rasheed et al., 28 Nov 2025).

The step-level reasoning reward is then

$$\mathcal{O}_i = f_{\text{MLLM}}(T_i, \mathcal{V}_i),$$1

while the final answer reward is

$$\mathcal{O}_i = f_{\text{MLLM}}(T_i, \mathcal{V}_i),$$2

The total reward is

$$\mathcal{O}_i = f_{\text{MLLM}}(T_i, \mathcal{V}_i),$$3

This reward makes intermediate evidence gathering part of the optimization target. A trajectory that reaches the right answer by ungrounded shortcuts is not equivalent to one that performs the right manipulations (Rasheed et al., 28 Nov 2025). This is a central methodological claim of the framework.

5. Empirical performance and what the benchmark results imply

Video-CoM is evaluated on nine video benchmarks, including Video-CoM-Bench, ScienceVid, VideoMath, MMVU-Val, Minerva, VideoMME, TempComp, and MLVU, with the table in the paper also reporting VideoMMMU / MMMU-Perc / MMMU-Avg variants (Rasheed et al., 28 Nov 2025). The abstract states that Video-CoM improves average performance by 3.6% over recent state-of-the-art models across these nine benchmarks while training on only 25K SFT and 3K GRPO video samples (Rasheed et al., 28 Nov 2025).

The most direct evidence for the value of Video-CoM-Instruct is on Video-CoM-Bench, the manipulation-specific evaluation set. There, Video-CoM reaches 68.7, clearly above the compared baselines listed in the paper, including scores of 60.2, 59.9, 57.1, and 56.3 for the competing systems shown in Table 1 (Rasheed et al., 28 Nov 2025). On the reasoning benchmark average, Video-CoM obtains 40.5, compared with 39.3 and 38.8 for the strongest listed comparators in the same table; on the generic benchmark average it remains competitive at 63.9 (Rasheed et al., 28 Nov 2025).

The ablations show the role of Video-CoM-Instruct and its supervision signal more specifically. On Video-CoM-Bench, the progression is:

• SFT only: 64.0
• GRPO with accuracy-only rewards: 66.7
• RA-GRPO: 68.7 (Rasheed et al., 28 Nov 2025)

On the average reasoning benchmarks, the progression is:

• SFT only: 37.5
• GRPO: 38.9
• RA-GRPO: 40.5 (Rasheed et al., 28 Nov 2025)

The paper also reports reasoning-quality metrics on Video-CoM-Bench:

• SFT: Acc. 64.0 / Reason 49.2 / Acc@IoU 38.1
• GRPO: 66.7 / 51.0 / 37.6
• RA-GRPO: 68.7 / 53.8 / 39.2 (Rasheed et al., 28 Nov 2025)

Here, the “Reason” metric is defined as the fraction of trajectories with manipulation accuracy $$\mathcal{O}_i = f_{\text{MLLM}}(T_i, \mathcal{V}_i),$$4, so the gains imply that the dataset and reward design improve both final answering and the quality of the manipulation trajectory itself (Rasheed et al., 28 Nov 2025).

A further ablation over the manipulation set supports the dataset design. The best results are achieved when all three manipulations are enabled, with the paper noting that find-segment gives the largest gain, find-frame strengthens spatial reasoning at key moments, and spatial-zoom is critical for small-region inspection (Rasheed et al., 28 Nov 2025). This result is consistent with the dataset’s curation strategy, which explicitly targeted videos requiring segment revisitation, frame isolation, and zoom-based inspection.

6. Interpretation, examples, and limitations

The paper’s examples clarify what Video-CoM-Instruct is trying to teach. One recurring case involves reading prices on cups of ice in a convenience-store video: the model must first localize the right temporal moment, then isolate the correct frame, then zoom into the freezer-door region to read the fine text (Rasheed et al., 28 Nov 2025). Another appendix example temporally localizes the instant when an on-screen timing graphic is fully visible, then zooms spatially to inspect split times. A third localizes a brief tunnel explosion sequence and the subsequent worker inspection with a flashlight (Rasheed et al., 28 Nov 2025). These are precisely the kinds of tasks that are difficult for passive video encoders but natural for a manipulation-based reasoning policy.

The interpretability claim of Video-CoM-Instruct is correspondingly strong. Because each example includes Exploratory-Reasoning, Visual-Manipulation, and Observation, the learned model’s inference trace can, in principle, be examined as a sequence of evidence-gathering actions rather than as an opaque answer string (Rasheed et al., 28 Nov 2025). The paper further notes attention visualizations suggesting that Video-CoM maintains attention to visual tokens across reasoning rounds, unlike passive baselines that concentrate more heavily on text tokens (Rasheed et al., 28 Nov 2025).

The limitations are equally important. The appendix identifies spatial localization as the hardest part of dataset construction and model behavior, especially for text, numbers, and small regions that do not correspond to standard object categories (Rasheed et al., 28 Nov 2025). Even strong MLLMs such as Gemini-2.5-Pro and InternVL3-78B struggle with direct spatial localization in video, which is why the 3K GRPO subset required manual spatial annotation before propagation with SAM2.1 (Rasheed et al., 28 Nov 2025). More broadly, manipulation-centric reasoning only becomes necessary when the video has sufficient internal variability; uniform videos with little scene change do not justify iterative interaction, which makes high-quality data harder to scale (Rasheed et al., 28 Nov 2025).

A plausible implication is that Video-CoM-Instruct is most valuable when questions require temporally localized, frame-specific, or small-region evidence. It is less obviously transformative for coarse whole-video classification or captioning. Another plausible implication is that the dataset’s strongest contribution is not merely scale—18K is modest by contemporary foundation-model standards—but the supervision of when to manipulate, which operation to invoke, and how to chain operations under explicit reasoning traces (Rasheed et al., 28 Nov 2025).

In the broader landscape of arXiv work on instruction-conditioned video systems, Video-CoM-Instruct occupies a distinct position. Earlier systems such as VidCoM concentrated on event-centric video comprehension with tool outputs and temporal refinement (Qi et al., 2023), while related efforts in coherent 4D or video editing focus on instruction-guided generation and propagation rather than reasoning-time evidence manipulation (Mou et al., 2024). Video-CoM-Instruct instead treats manipulation itself as the substrate of reasoning. That is its defining contribution: it converts interactive evidence gathering from an architectural intuition into a supervised and reinforcement-learned behavior (Rasheed et al., 28 Nov 2025).