---
title: 'Video CoM Instruct: Interactive Video Reasoning'
url: https://www.emergentmind.com/topics/video-com-instruct
type: topic
---

# Video CoM Instruct: Interactive Video Reasoning

Video CoM Instruct is an instruction-tuning dataset introduced as the data foundation of the Video-CoM framework, a system for **interactive video reasoning** in which a multimodal large language model reasons by actively manipulating video evidence rather than treating video as a fixed encoded context [2511.23477]. Its defining purpose is to teach a model not only to answer questions about videos, but to decide **when** to revisit a temporal segment, **which** frame to isolate, **where** to zoom spatially, and **how** to chain these operations into a grounded multi-step reasoning trajectory. In this formulation, the video becomes a **dynamic cognitive workspace** rather than a passive input, and the dataset supervises that behavior through structured Chain-of-Manipulations traces, question-answer pairs, and, for a reinforcement-learning subset, dense step-level temporal and spatial annotations [2511.23477].

## 1. From passive video reasoning to interactive video reasoning

The paper positions Video-CoM-Instruct within a broader critique of conventional video MLLMs. In the prevailing **passive paradigm**, a video is encoded once and subsequent reasoning unfolds almost entirely in language space, so the model effectively “thinks about videos” rather than “thinks with videos” [2511.23477]. This creates what the authors call a **semantic bottleneck**: because the model cannot rewatch, refocus, or verify evidence during inference, it is prone to drift toward generic world knowledge and weakly grounded textual reasoning when tasks demand fine-grained temporal localization, precise spatial inspection, or multi-step evidence aggregation.

Video-CoM addresses this limitation through **Interactive Video Reasoning**, in which the model performs visual actions during reasoning. At reasoning step \(i\), given text context \(T_i\) and current visual input \(\mathcal{V}_i\), the model generates
\[
\mathcal{O}_i = f_{\text{MLLM}}(T_i, \mathcal{V}_i),
\]
where \(\mathcal{O}_i\) includes both reasoning tokens and a predicted manipulation \(m_i\). The manipulation updates the visual input according to
\[
\mathcal{V}_{i+1} = g(\mathcal{V}_i, m_i).
\]
Over multiple turns, this yields a trajectory
\[
\tau = \{(T_1, \mathcal{V}_1, m_1), (T_2, \mathcal{V}_2, m_2), \ldots, (T_N, \mathcal{V}_N, m_N)\}.
\]
Video-CoM-Instruct is designed specifically to make such trajectories learnable [2511.23477].

This orientation distinguishes the framework from earlier instruction-conditioned video comprehension systems that rely on event descriptions, captions, or compact evidence representations but do not make manipulation itself the central supervised object. A closely related precursor is VidCoM, which already treated video comprehension as instruction-conditioned event reasoning with tool outputs and temporal refinement, but did so through scene graphs, captions, and event alignment rather than through explicit frame/segment/zoom manipulations [2310.10586]. Video-CoM-Instruct extends the instruction-following perspective into a manipulation-centric regime [2511.23477].

## 2. Chain of Manipulations as the operative reasoning interface

The core operational concept taught by Video-CoM-Instruct is the **Chain of Manipulations (CoM)**, defined as the sequence of visual operations selected during reasoning:
\[
CoM = \{m_1, m_2, \ldots, m_{N-1}\}.
\]
The manipulation space contains three atomic operations [2511.23477].

**Find-Segment** revisits a short temporal clip from the original video \(\mathcal{V}_0\) using a segment index \(s\):
\[
\mathcal{V}_{i+1} = g_{\text{clip}}(\mathcal{V}_0, s).
\]
Its role is temporal revisitation: locating brief events, rewatching critical moments, or restricting reasoning to an instruction-relevant temporal window.

**Find-Frame** isolates a single frame from the original video using frame index \(f\):
\[
\mathcal{V}_{i+1} = g_{\text{frame}}(\mathcal{V}_0, f).
\]
This operation is used when evidence is fleeting but becomes legible once frozen, such as object attributes, counts, small text, or transient visual states.

**Spatial-Zoom** magnifies a localized region of a frame using frame index \(f\) and bounding box \(b\):
\[
\mathcal{V}_{i+1} = g_{\text{zoom}}(\mathcal{V}_i, f, b).
\]
It is intended for fine-grained inspection of signs, prices, logos, scores, or other small regions that are visually present but not easily readable at full scale [2511.23477].

These primitives induce a characteristic reasoning grammar. The paper identifies common chains such as **find-frame \(\rightarrow\) spatial-zoom** for reading small text, **find-segment** for counting or localizing repeated transient actions, **find-segment \(\rightarrow\) find-frame**, and **find-segment \(\rightarrow\) find-frame \(\rightarrow\) spatial-zoom** for multi-stage evidence gathering [2511.23477]. The dataset is explicitly curated to expose the model to these patterns and to teach when each manipulation is warranted.

The CoM notion also clarifies what “instruction” means in this setting. The question-answer task is not sufficient; the model must infer a **reasoning program** over the video. Thus, Video-CoM-Instruct supervises a model to decide which evidence is missing, select an operation to acquire it, observe the result, and continue. In the appendix description, every trace interleaves three components: **Exploratory-Reasoning**, **Visual-Manipulation**, and **Observation** [2511.23477]. This structure makes the reasoning process inspectable rather than leaving it implicit in free-form text.

## 3. Dataset design, composition, and curation pipeline

Video-CoM-Instruct contains **18K QA pairs** curated for **multi-step manipulation reasoning**, with **15K** used for supervised fine-tuning and **3K** reserved for GRPO-based reinforcement learning [2511.23477]. The dataset combines **16.5K** examples sampled from existing video sources and **1.5K** examples from a manually curated set of high-complexity videos. The existing-source videos are drawn from the Video-LLaVA-178K ecosystem, including ActivityNet, YouCookII, Charades, HD-VILA, InternVid, VidOR, LanguageBind sources, Kinetics, and Ego4D; the appendix further specifies that the selected 9K existing-source videos are distributed approximately as **ActivityNet: 28%**, **Charades: 19%**, **VidOR: 17%**, and **YouCookII: 14%**, with the remainder from YouTube videos collected by LLaVA-178K, PerceptionTest, CLEVRER, and Ego4D [2511.23477].

The dataset is produced by a five-stage pipeline: **curating videos for active reasoning**, **designing chain-of-manipulations**, **interactive video reasoning data generation**, **data verification**, and **dense spatio-temporal annotation** [2511.23477]. Video selection is guided by **information density**. The authors first generate captions with **Qwen2.5-VL-72B** when needed, then use **Qwen3-235B-A22B** to score videos on a **1–10 scale** emphasizing scene diversity, number of activities, object interactions, visible text or small details, and changes over time. Approximately **9K** high-information-density videos are retained after filtering [2511.23477].

For QA generation, the authors use **Gemini-2.5-Pro** to create **3–5 QA pairs per video**, yielding about **25K candidate QA pairs** in total; for the manually curated 70 high-complexity videos, the appendix notes up to **20 QA pairs per video** [2511.23477]. The generation prompt requires questions that cannot be answered by passive viewing, but instead require rewatching segments, freezing precise frames, zooming into small regions, or combinations thereof. Each generated item includes a question, a short answer, **5 multiple-choice options (A–E)** with exactly one correct answer, and a brief reasoning plan listing the sequence of tools [2511.23477].

Quality control is stringent. Each candidate QA is re-evaluated by **Gemini-2.5-Pro** in both **multiple-choice** and **open-ended** formats; both the reasoning trace and final answer must agree with the intended answer. This removes ambiguous, inconsistent, or unanswerable examples. The retention rate is about **75%**, reducing the initial **25K** candidates to **18K** final examples [2511.23477].

After filtering, each retained sample includes the **question**, **answer**, **MCQ options**, and a **CoM reasoning trace** [2511.23477]. That trace is the central annotation object: it teaches the model a manipulation trajectory, not merely a final label. For the **3K GRPO subset**, the supervision becomes denser still: the paper states that it includes **all valid segments** for each find-segment step, **all valid frames** for each find-frame step, and manually annotated bounding boxes propagated into **tubelets** with **SAM2.1** for spatial grounding [2511.23477].

## 4. Supervised fine-tuning and reasoning-aware reinforcement learning

Video-CoM uses **Qwen2.5-VL-7B-Instruct** as its base model and trains it in two stages: **Stage I: Supervised Fine-Tuning (SFT)** and **Stage II: RA-GRPO** [2511.23477]. Video-CoM-Instruct is central to both stages.

In **SFT**, the model is trained on the **15K** SFT split of Video-CoM-Instruct, which provides the high-quality manipulation reasoning traces. This supervision is augmented with **9K ActivityNet** samples without manipulations to strengthen temporal understanding and frame localization, and **180K Visual-CoT** image samples with spatial grounding to improve spatial-zoom capability [2511.23477]. The paper summarizes the resulting SFT corpus as **25K video samples** and **180K image samples**. The appendix adds that the 15K Video-CoM-Instruct subset is concatenated twice, yielding **30K effective samples**, though the main text continues to refer to 15K high-quality samples [2511.23477]. The SFT setup uses DeepSpeed ZeRO-3, bf16, context length **32K**, global batch size **32**, learning rate \(1\times 10^{-5}\), cosine decay, **3% warm-up**, and **max reasoning turns: 5** [2511.23477].

The paper argues that SFT alone is insufficient because multiple manipulation trajectories can lead to the same answer, and because the correct trajectory is strongly task-dependent [2511.23477]. It therefore adds **Group Relative Policy Optimization (GRPO)** over the **3K** densely annotated Video-CoM-Instruct subset. The RL stage uses global batch size **64**, learning rate \(1\times 10^{-6}\), group size **8**, KL coefficient **0.04**, max turns **5**, and spatial IoU threshold \(\tau_b = 0.3\) [2511.23477].

The distinctive feature of this stage is **reasoning-aware reward design**. For each predicted manipulation \(m_i'\), correctness \(c_i\) is defined as
\[
c_i =
\begin{cases}
1, & \begin{cases}
s_i' \in \mathcal{S}^*, & \text{for } find\text{-}segment, \\
f_i' \in \mathcal{F}^*, & \text{for } find\text{-}frame, \\
\text{IoU}(b_i', b^*) \ge \tau_b, & \text{for } spatial\text{-}zoom,
\end{cases}\\
0, & \text{otherwise,}
\end{cases}
\]
where \(\mathcal{S}^*\) is the set of valid segments, \(\mathcal{F}^*\) the set of valid frames, and \(b^*\) a valid bounding box from the annotated set \(\mathcal{B}^*\) [2511.23477].

The step-level reasoning reward is then
\[
R_{\text{reason}} = \frac{1}{N} \sum_{i=1}^{N} c_i,
\]
while the final answer reward is
\[
R_{\text{accuracy}} =
\begin{cases}
1, & \text{if } \hat{y} = y^*,\\
0, & \text{otherwise.}
\end{cases}
\]
The total reward is
\[
R = R_{\text{accuracy}} + R_{\text{reason}}.
\]
This reward makes intermediate evidence gathering part of the optimization target. A trajectory that reaches the right answer by ungrounded shortcuts is not equivalent to one that performs the right manipulations [2511.23477]. This is a central methodological claim of the framework.

## 5. Empirical performance and what the benchmark results imply

Video-CoM is evaluated on **nine video benchmarks**, including **Video-CoM-Bench**, **ScienceVid**, **VideoMath**, **MMVU-Val**, **Minerva**, **VideoMME**, **TempComp**, and **MLVU**, with the table in the paper also reporting **VideoMMMU / MMMU-Perc / MMMU-Avg** variants [2511.23477]. The abstract states that Video-CoM improves average performance by **3.6%** over recent state-of-the-art models across these nine benchmarks while training on only **25K SFT** and **3K GRPO** video samples [2511.23477].

The most direct evidence for the value of Video-CoM-Instruct is on **Video-CoM-Bench**, the manipulation-specific evaluation set. There, Video-CoM reaches **68.7**, clearly above the compared baselines listed in the paper, including scores of **60.2**, **59.9**, **57.1**, and **56.3** for the competing systems shown in Table 1 [2511.23477]. On the reasoning benchmark average, Video-CoM obtains **40.5**, compared with **39.3** and **38.8** for the strongest listed comparators in the same table; on the generic benchmark average it remains competitive at **63.9** [2511.23477].

The ablations show the role of Video-CoM-Instruct and its supervision signal more specifically. On Video-CoM-Bench, the progression is:
- **SFT only**: **64.0**
- **GRPO** with accuracy-only rewards: **66.7**
- **RA-GRPO**: **68.7** [2511.23477]

On the average reasoning benchmarks, the progression is:
- **SFT only**: **37.5**
- **GRPO**: **38.9**
- **RA-GRPO**: **40.5** [2511.23477]

The paper also reports reasoning-quality metrics on Video-CoM-Bench:
- **SFT**: Acc. **64.0** / Reason **49.2** / Acc@IoU **38.1**
- **GRPO**: **66.7** / **51.0** / **37.6**
- **RA-GRPO**: **68.7** / **53.8** / **39.2** [2511.23477]

Here, the “Reason” metric is defined as the fraction of trajectories with manipulation accuracy \(> 0.3\), so the gains imply that the dataset and reward design improve both final answering and the quality of the manipulation trajectory itself [2511.23477].

A further ablation over the manipulation set supports the dataset design. The best results are achieved when **all three manipulations** are enabled, with the paper noting that **find-segment** gives the largest gain, **find-frame** strengthens spatial reasoning at key moments, and **spatial-zoom** is critical for small-region inspection [2511.23477]. This result is consistent with the dataset’s curation strategy, which explicitly targeted videos requiring segment revisitation, frame isolation, and zoom-based inspection.

## 6. Interpretation, examples, and limitations

The paper’s examples clarify what Video-CoM-Instruct is trying to teach. One recurring case involves reading **prices on cups of ice** in a convenience-store video: the model must first localize the right temporal moment, then isolate the correct frame, then zoom into the freezer-door region to read the fine text [2511.23477]. Another appendix example temporally localizes the instant when an on-screen timing graphic is fully visible, then zooms spatially to inspect split times. A third localizes a brief tunnel explosion sequence and the subsequent worker inspection with a flashlight [2511.23477]. These are precisely the kinds of tasks that are difficult for passive video encoders but natural for a manipulation-based reasoning policy.

The interpretability claim of Video-CoM-Instruct is correspondingly strong. Because each example includes **Exploratory-Reasoning**, **Visual-Manipulation**, and **Observation**, the learned model’s inference trace can, in principle, be examined as a sequence of evidence-gathering actions rather than as an opaque answer string [2511.23477]. The paper further notes attention visualizations suggesting that Video-CoM maintains attention to visual tokens across reasoning rounds, unlike passive baselines that concentrate more heavily on text tokens [2511.23477].

The limitations are equally important. The appendix identifies **spatial localization** as the hardest part of dataset construction and model behavior, especially for text, numbers, and small regions that do not correspond to standard object categories [2511.23477]. Even strong MLLMs such as Gemini-2.5-Pro and InternVL3-78B struggle with direct spatial localization in video, which is why the 3K GRPO subset required manual spatial annotation before propagation with SAM2.1 [2511.23477]. More broadly, manipulation-centric reasoning only becomes necessary when the video has sufficient internal variability; uniform videos with little scene change do not justify iterative interaction, which makes high-quality data harder to scale [2511.23477].

A plausible implication is that Video-CoM-Instruct is most valuable when questions require **temporally localized**, **frame-specific**, or **small-region** evidence. It is less obviously transformative for coarse whole-video classification or captioning. Another plausible implication is that the dataset’s strongest contribution is not merely scale—**18K** is modest by contemporary foundation-model standards—but the supervision of **when to manipulate**, **which operation to invoke**, and **how to chain operations** under explicit reasoning traces [2511.23477].

In the broader landscape of arXiv work on instruction-conditioned video systems, Video-CoM-Instruct occupies a distinct position. Earlier systems such as VidCoM concentrated on event-centric video comprehension with tool outputs and temporal refinement [2310.10586], while related efforts in coherent 4D or video editing focus on instruction-guided generation and propagation rather than reasoning-time evidence manipulation [2406.09402]. Video-CoM-Instruct instead treats manipulation itself as the substrate of reasoning. That is its defining contribution: it converts interactive evidence gathering from an architectural intuition into a supervised and reinforcement-learned behavior [2511.23477].

Source: https://www.emergentmind.com/topics/video-com-instruct