---
title: 'VideoInfer: Object-Centric VideoQA Dataset'
url: https://www.emergentmind.com/topics/videoinfer
type: topic
---

# VideoInfer: Object-Centric VideoQA Dataset

VideoInfer is a manually curated, object-centric video instruction dataset for reasoning-heavy video question answering with explicit object referring in the input and visual grounding in the output. It was introduced with the RGA3 framework in “Object-centric Video Question Answering with Visual Grounding and Referring,” where the underlying task is formulated as
\[
\mathcal{A}, \mathcal{M} = \phi_{\theta}(\mathcal{V}, \mathcal{Q}, P_{t}),
\]
with \(\mathcal{V} \in \mathbb{R}^{T \times H \times W \times 3}\) a video, \(\mathcal{Q}\) a textual question, \(P_t \in \mathbb{R}^{H \times W \times 4}\) a single-frame RGBA visual prompt at timestamp \(t\), \(\mathcal{A}\) a textual answer, and \(\mathcal{M} \in \mathbb{R}^{T \times H \times W}\) optional segmentation masks grounding the referred object through time [2507.19599].

## 1. Conceptual scope and task definition

VideoInfer is designed for an object-centric setting in which questions are tied to a specific entity designated by a visual prompt rather than to the scene as a whole. The prompt can be a mask, box, arrow, scribble, point, or related mark on a single frame, and the model is expected to follow that object across the full video while answering open-ended questions and, optionally, emitting spatio-temporal masks. This makes VideoInfer a benchmark for referring VideoQA, reasoning video object segmentation, and joint QA-plus-grounding.

The dataset was created to address several gaps in prior VideoLLM evaluation. Many existing VideoQA benchmarks emphasize holistic scene understanding or short factual queries, while prior object-centric resources often rely on automatically generated question–answer pairs that bias the task toward simple perception rather than temporal, causal, and semantic reasoning. VideoInfer instead centers questions on specific prompted objects and emphasizes questions whose answers are not trivially recoverable from one still image. The annotation principles are explicitly “Indirectness” and “Dynamism”: the answer should not be trivially readable from a single frame, and many questions should require temporal reasoning. Examples given in the source include questions such as “Is he good at jumping rope?”, which require observing an action sequence rather than identifying a static attribute [2507.19599].

## 2. Data sources, prompt construction, and annotation protocol

VideoInfer is built from object-centric video segmentation corpora with dense masks or boxes, including TAO (via BURST), LVOS, LVVIS, VIPSeg, UVO, MOSE, and OVIS. These sources provide diverse scenes involving people, animals, indoor and outdoor environments, and object interactions. Long-form videos from TAO and LVOS are explicitly prioritized to support long-term temporal reasoning and implicit questions.

For each object of interest, a single frame is chosen at random, and one of eight visual prompt types is generated from the ground-truth mask: mask, mask contour, rectangle, ellipse, triangle, scribble, arrow, and point. The prompt is represented as an RGBA image \(P_t\), with a colored mark over an opaque background. This design is intended to mimic interactive user behavior in which a user clicks, draws, or points once and expects the model to continue following the object.

Question–answer annotation is manual. Each prompted object receives one or more QA pairs, written by human annotators, with cross-validation across two annotation groups and discarding of low-scoring or poor-quality QA pairs. The resulting questions cover temporal reasoning, behavior and action analysis, causal and inferential reasoning, attribute and state reasoning, counting or existence over time, and relational reasoning. Open-ended answers are used rather than multiple-choice responses, which makes the benchmark closer to instruction-tuned VideoLLM deployment [2507.19599].

## 3. Corpus composition and linguistic profile

The reported dataset is divided into an instruction-tuning split and a challenging test split. The split statistics are as follows.

| Split | Videos | Objects / QA pairs |
|---|---:|---:|
| Instruction-tuning | 950 | 2,555 objects / 20,320 QA pairs |
| Challenging test | 670 | 1,572 objects / 8,491 QA pairs |
| Total | 1,620 | 28,811 QA pairs |

The supplementary statistics for the test split further characterize the corpus. Question length ranges from 3 to 50 words, with average length about 8.4 words. Answer length ranges from 1 to 75 words, with average length about 8.7 words. Videos range from 7 frames to more than 2000 frames, with average length about 189.5 frames. Objects per video range from 1 to 8, averaging about 2.3. These figures indicate that VideoInfer couples relatively long videos with multi-object scenes while still binding each QA instance to a specific prompted object.

This structure has two consequences. First, it supports instruction-tuning of models that must ground language in a referred entity rather than in scene-level context. Second, it creates a difficult test regime for models that only sample a small number of frames at inference time, because object-centric evidence can be temporally sparse and distributed across long sequences [2507.19599].

## 4. Enabled tasks, metrics, and supervision

VideoInfer supports object-centric open-ended VideoQA with visual prompts and, when a `<SEG>` token is generated, referring or reasoning video object segmentation. In the QA setting, the input is \((\mathcal{V}, \mathcal{Q}, P_t)\), and the expected output is textual answer \(\mathcal{A}\), with optional grounding mask \(\mathcal{M}\). In the segmentation setting, the same example can supervise spatio-temporal mask prediction for the referred object.

For open-ended referring VideoQA, the reported metrics are BLEU-4, CIDEr, ROUGE-L, and GPT-4o-based evaluation, where GPT-4o scores model answers against ground truth in terms of Accuracy (Acc.) and Score. For segmentation, the paper uses the standard \(\mathcal{J}\) region similarity, \(\mathcal{F}\) contour accuracy, and their average \(\mathcal{J}\&\mathcal{F}\). VideoInfer is also used in a multi-task training mixture together with general image QA, video QA, image segmentation, referring segmentation, and video referring or reasoning segmentation datasets.

The training setup combines text generation and mask supervision. The source describes a standard auto-regressive cross-entropy loss for textual outputs, a mask loss combining BCE and DICE, and a total objective that weights textual and mask terms jointly. This joint formulation is central to enabling grounded answering, because the same model is trained to both answer object-centric questions and emit segmentation masks tied to the referred target [2507.19599].

## 5. Role in RGA3, STOM, and grounded multimodal interaction

VideoInfer is central to the RGA3 model family, especially to the Spatial-Temporal Overlay Module (STOM). STOM addresses the problem that the visual prompt is available at only one timestamp. Let \(\mathbf{c} \in \mathbb{R}^{2}\) be the geometric center of the prompt at frame \(t\). STOM uses CoTracker3 to track pixels
\[
\{\mathbf{p} \in \mathbb{R}^2 \mid \lVert \mathbf{p} - \mathbf{c} \rVert_2 \leq r \}
\]
through the video, propagates the prompt to each frame as \(P_i\), and alpha-blends the resulting prompt into the original frame through
\[
\mathcal{V}'_i = \Phi(\mathcal{V}_i, P_i), \quad 1 \leq i \leq T.
\]
The blended sequence \(\mathcal{V}'\) provides a persistent object cue for the downstream VideoLLM.

Grounded output is implemented by coupling Qwen2.5-VL with SAM2. The blended video is encoded by the Qwen2.5-VL visual encoder, dense visual features are extracted by SAM2’s Hiera-L encoder, and when the language model emits `<SEG>`, the corresponding hidden state \(\mathbf{h} \in \mathbb{R}^{C}\) is passed to the SAM2 decoder to produce the final spatio-temporal mask. In this pipeline, VideoInfer supplies precisely the paired prompt, reasoning QA, and dense object supervision required to learn both “referring for input” and “grounding for output.”

The STOM ablation directly demonstrates the dataset’s role. On VideoInfer, a variant without STOM reports GPT-4o Acc./Score of 43.6 / 2.46, while the STOM-equipped model reports 46.7 / 2.60. The same mechanism also improves multiple-choice referring VideoQA on VideoRefer-Bench\(^Q\), which suggests that VideoInfer’s prompt-rich supervision is useful beyond its own benchmark [2507.19599].

## 6. Empirical results and position relative to prior datasets

On VideoInfer’s open-ended referring VideoQA benchmark, RGA3-7B reports BLEU-4 12.2, CIDEr 104.3, ROUGE-L 29.9, and GPT-4o Acc./Score 46.7 / 2.60. The same table reports VideoRefer-7B at 7.4 BLEU-4, 77.8 CIDEr, 25.6 ROUGE-L, and 40.4 / 2.29; GPT-4o (detail: high) reports 7.6 BLEU-4, 64.4 CIDEr, 23.0 ROUGE-L, and 45.5 / 2.50. These numbers establish VideoInfer as a setting in which a specialized object-centric VideoLLM can surpass both a specialist referring baseline and a strong commercial multimodal model.

The benchmark is positioned against several adjacent resources. VideoRefer-700K is described as very large and auto-generated, with multiple-choice QA dominated by perception-level questions. ReVOS and ReasonVOS focus on reasoning object segmentation rather than open-ended QA with visual prompts. Standard VideoQA datasets such as NExT-QA, ActivityNetQA, and PerceptionTest emphasize holistic or clip-level reasoning without explicit tethering to a prompted object. VideoInfer differs by combining object-centric questioning, eight prompt types, manual reasoning QA, longer videos, and joint QA-plus-segmentation supervision.

The same training regime transfers to other tasks. The paper reports that RGA3-7B reaches 74.0% average accuracy on VideoRefer-Bench\(^Q\), outperforming VideoRefer-7B at 71.9% and GPT-4o with Set-of-Masks prompting at 71.3%, and also improves reasoning video object segmentation on ReVOS and ReasonVOS by 10 percentage points in \(\mathcal{J}\&\mathcal{F}\) over previous MLLM-based approaches. This suggests that VideoInfer functions not only as a benchmark but also as a core supervision source for object-centric multimodal transfer [2507.19599].

## 7. Limitations and future directions

The primary limitation discussed for VideoInfer is the mismatch between video length and current VideoLLM inference budgets. Many videos contain hundreds or thousands of frames, while the evaluated RGA3 system processes only 16 frames at inference time. This can create temporal gaps and failure cases in tasks that require precise temporal localization or sustained object tracking. A second limitation is scale: 28.8K QA pairs is moderate by contemporary standards, and the videos are drawn mainly from object segmentation datasets, leaving domains such as egocentric, instructional, and sports video underrepresented. A third limitation is the cost of manual annotation: the same human curation that improves reasoning quality also constrains expansion.

The paper’s future directions therefore emphasize three axes. One is better architectures or tokenization strategies for ultra-long videos that preserve object-centric information without exploding compute. A second is more sophisticated multi-stage training or prompting strategies that balance QA, referring, and segmentation. A third is domain expansion through further VideoInfer-style annotation on longer and more complex narratives. These directions imply that VideoInfer is best understood not as a static benchmark endpoint, but as a template for object-centric, prompt-conditioned, reasoning-focused video supervision [2507.19599].

Source: https://www.emergentmind.com/topics/videoinfer