VTTS-80K: Iterative Multimodal Perception Dataset

• VTTS-80K is a multimodal dataset with about 80K entries integrating QA pairs, spatial/temporal grounding, and chain-of-thought 'Think' sequences.
• It supports iterative perception by enabling MLLMs to repeatedly propose, verify, and refine region selections with diverse spatio-temporal annotations.
• The dataset is used in reinforcement fine-tuning within the VTTS framework, enhancing model performance across various video and image reasoning benchmarks.

Searching arXiv for the specified paper and directly related work to ground the article. VTTS-80K is a multimodal dataset introduced alongside the Visual Test-Time Scaling (VTTS) framework in "VideoChat-R1.5: Visual Test-Time Scaling to Reinforce Multimodal Reasoning by Iterative Perception" (Yan et al., 25 Sep 2025). It is tailored for iterative perception in multimodal LLMs (MLLMs), with supervision spanning question answering, temporal grounding, spatial grounding, object tracking, and chain-of-thought-style "Think" sequences. Within the associated training pipeline, VTTS-80K is not positioned as a conventional benchmark with predefined train/validation/test partitions; instead, all 80K entries are used for reinforcement fine-tuning, after which evaluation is conducted zero-shot or on external held-out benchmarks (Yan et al., 25 Sep 2025).

1. Dataset definition and scope

VTTS-80K contains roughly 80,000 multimodal entries drawn from existing video-QA, image-QA, temporal-grounding, spatial-grounding, and tracking benchmarks (Yan et al., 25 Sep 2025). The dataset is explicitly designed to support iterative perception, a setting in which an MLLM repeatedly proposes, verifies, and refines spatio-temporal regions during inference rather than relying on a single static perceptual pass.

The composition reported for VTTS-80K is heterogeneous rather than monolithic. It includes approximately 50,000 question-answer pairs, approximately 15,000 temporal-interval annotations, approximately 30,000 spatial-box annotations, and approximately 80,000 "Think" sequences. The counts are not additive in the sense of defining disjoint subsets; the dataset description indicates overlap between QA entries and clue annotations, with roughly one annotated spatio-temporal region per QA entry. More specifically, the paper reports about 45,000 region annotations over about 50,000 QA pairs, or approximately 0.9 regions per clip.

The visual content spans both video and image settings. The raw video portion covers clips from short durations of approximately 16 to 23 seconds to medium and long clips extending to hundreds or thousands of seconds, with source material drawn from LLavaVideo, PerceptionTest, Star, Charades, NextGQA, and related upstream resources. No single total-hours statistic or overall average clip length is reported for VTTS-80K itself (Yan et al., 25 Sep 2025). This absence is consequential for dataset comparison: VTTS-80K is better understood as a supervision layer aggregated from ten upstream datasets than as a single raw-media corpus with a unified temporal scale.

2. Internal composition and task coverage

The dataset supports several task families that are normally treated separately in multimodal evaluation. These include video question answering in both multiple-choice and open-ended forms, image reasoning and visual question answering, temporal grounding, spatial grounding, object tracking, and chain-of-thought supervision through "Think" annotations (Yan et al., 25 Sep 2025). In the paper's ablation framing, the principal components are described as S-Grounding, T-Grounding, V-MCQ, and I-MCQ, corresponding respectively to approximately 30K spatial clues, approximately 15K temporal clues, and approximately 50K combined video- and image-QA pairs.

A compact summary of the reported dataset composition is given below.

Component	Reported scale	Role
VTTS-QA	~50,000	Question-answer supervision
VTTS-TEMP	~15,000	Temporal-interval annotations
VTTS-SPATIAL	~30,000	Spatial-box annotations
"Think" sequences	~80,000	Iterative-perception guidance

The QA portion is further described as roughly 30,000 video-QA examples and 20,000 image-QA examples. The clue annotations total 45,000 regions, split into 15K temporal and 30K spatial instances. The domains represented are diverse, including everyday actions, long-form events, and diagrams for image math. The paper does not define a single class distribution; instead, regions may correspond to objects, actions, text overlays, and related visual targets (Yan et al., 25 Sep 2025).

This organization suggests that VTTS-80K is intended less as a narrowly specialized grounding dataset than as a multi-objective reinforcement fine-tuning resource. A plausible implication is that the dataset's value lies in coupling answer supervision with localized perceptual supervision, thereby encouraging iterative region selection rather than answer generation from globally encoded inputs alone.

3. Annotation schema and region representation

VTTS-80K uses a clue-based annotation schema. Every clue annotation is either a temporal segment $[t_{\text{start}}, t_{\text{end}}]$ in a video clip or a two-dimensional bounding box $(x_1, y_1, x_2, y_2)$ in an image or single frame (Yan et al., 25 Sep 2025). When a spatial and temporal specification are combined into a unified spatio-temporal region, the representation is

$$R = (x_1,\, y_1,\, x_2,\, y_2,\, t_{\rm start},\, t_{\rm end}).$$

Alignment between a predicted region and a ground-truth region is measured primarily with Intersection-over-Union (IoU), applied either to spatial boxes or to temporal intervals. The paper writes this as

$$\mathrm{IoU}(R_{\rm pred}, R_{\rm gt}) \;=\; \frac{\mathrm{measure}\bigl(R_{\rm pred}\cap R_{\rm gt}\bigr)} {\mathrm{measure}\bigl(R_{\rm pred}\cup R_{\rm gt}\bigr)}.$$

The schema is notable for placing temporal and spatial supervision under a common formalism. This matters for iterative perception because the model is expected to refine not only where to look but also when to look. In standard spatial grounding, region quality is confined to frame-localized boxes; in VTTS-80K, the same supervisory logic extends to temporal localization and, when available, to their joint representation as a spatio-temporal region.

The chain-of-thought-style "Think" annotations are also integral to the schema. The paper reports 80K such sequences, one for each dataset entry at the reported scale. These sequences are used to guide iterative perception rather than serving merely as post hoc rationales. This suggests a training regime in which intermediate reasoning tokens and region proposals are mutually coupled.

4. Supervision, rewards, and optimization

VTTS-80K is used in the reinforcement fine-tuning stage of VideoChat-R1.5 to train Iterative Perception (ITP) with three reward streams: spatio-temporal alignment, answer correctness, and output-format compliance (Yan et al., 25 Sep 2025). The combined reward is

$$\mathcal{R}(\theta)\;=\; \underbrace{\lambda_{0}\,r_{\rm clue}}_{\substack{\text{spatio-temporal}\\text{alignment}}} + \underbrace{\lambda_{1}\,r_{\rm ans}}_{\text{answer correctness}} + \underbrace{\lambda_{2}\,r_{\rm fmt}}_{\text{output format}}.$$

The constituent rewards are defined as follows:

• $$r_{\rm clue} = \mathrm{IoU}(R_{\rm pred}, R_{\rm gt})$$
• $r_{\rm ans} = 1$ if the final answer matches ground truth, else $0$
• $r_{\rm fmt} = 1$ if the model's output passes a format-check regex, else $0$

The paper also gives an autoregressive-plus-iterative-vision objective,

$(x_1, y_1, x_2, y_2)$0

with

$(x_1, y_1, x_2, y_2)$1

where $(x_1, y_1, x_2, y_2)$2 denotes the ROI cropping step. In practice, however, this objective is optimized via a GRPO-style policy gradient rather than plain cross-entropy (Yan et al., 25 Sep 2025).

The supervisory design is important because VTTS-80K does not only reward correct final answers. It also rewards region alignment and valid structured output. This means that perceptual refinement is directly optimized rather than treated as a latent internal process. A plausible implication is that the dataset's annotation structure is inseparable from the training objective: the utility of the 45K region annotations depends on their incorporation into the reward signal, not merely on their existence as auxiliary labels.

5. Integration with the VTTS inference procedure

Within the broader VTTS framework, VTTS-80K teaches the model how to perform iterative ROI selection and answer refinement under reinforcement signals (Yan et al., 25 Sep 2025). At inference time, VTTS runs $(x_1, y_1, x_2, y_2)$3 perception iterations, typically $(x_1, y_1, x_2, y_2)$4. The reported procedure consists of an initial full-video or full-image pass, followed by repeated cropping and dense resampling of high-IoU regions, conditioned on updated textual predictions.

The paper describes the iteration pattern in three steps:

1. Step 0: run over full video or image, then generate an initial clue and partial answer.
2. Step $(x_1, y_1, x_2, y_2)$5: crop and densely resample only the high-IoU region $(x_1, y_1, x_2, y_2)$6, re-fuse with language context, and update the clue and answer.
3. Final answer: produce the answer after $(x_1, y_1, x_2, y_2)$7 loops, leveraging all "Think" steps.

This architecture places VTTS-80K in a distinctive position relative to static perception datasets. The annotations are not merely consumed in a one-pass encoder-decoder setting. Instead, they supervise a recurrent perceptual policy in which each round of reasoning can alter the next visual crop. This suggests that VTTS-80K operationalizes a specific view of multimodal reasoning: perception and reasoning are interleaved across inference steps, and increased perceptual compute can improve performance without increasing model size.

The associated paper reports that these designs allow an MLLM to enhance performance by increasing perceptual compute, and that VideoChat-R1.5 achieves an average increase of over 5% compared to baselines such as Qwen2.5VL-3B and Qwen2.5VL-7B across more than 15 benchmarks encompassing video conversation, video reasoning, and spatio-temporal perception (Yan et al., 25 Sep 2025). Those results characterize the framework rather than VTTS-80K in isolation, but they clarify the dataset's intended functional role.

6. Evaluation context, interpretation, and common misconceptions

VTTS-80K itself is not evaluated through an internal held-out test split. All 80K entries are used for reinforcement fine-tuning, and downstream assessment is performed on external benchmarks (Yan et al., 25 Sep 2025). The reported metrics span a wide range of tasks: Accuracy for video QA and conversational evaluation; Acc@[email protected], Acc@GQA, mIoP, and IoP@$(x_1, y_1, x_2, y_2)$8 for grounded video QA; mIoU and [email protected]/0.5/0.7 for temporal grounding; Precision@IoU$(x_1, y_1, x_2, y_2)$9 for spatial grounding; AO, [email protected], and [email protected] for object tracking; and task-specific scores for OCR and image reasoning.

The comparison set includes Qwen2.5-VL, InternVideo(2.5), VideoChat-Flash, LLaVA-OneVision, GPT-4/V, Gemini-1.5, MDETR, Grounding-DINO, and SiamFC. Because these baselines span generative MLLMs, grounding models, and trackers, the evaluation context is inherently cross-paradigmatic. VTTS-80K is therefore best interpreted as a unifying training resource for multimodal iterative perception rather than as a standalone dataset intended for leaderboard-style internal comparison.

Several misconceptions are explicitly precluded by the paper's description. VTTS-80K is not reported with predefined train/validation/test splits. It does not come with a single total-hours statistic or a single overall average clip length. It also does not define a unified class taxonomy over all annotations; the supervised regions arise from heterogeneous upstream datasets and may denote objects, actions, text overlays, and other targets (Yan et al., 25 Sep 2025). Likewise, the approximate counts of QA pairs, clue annotations, and "Think" sequences should not be read as mutually exclusive partitions.

From a methodological standpoint, the main significance of VTTS-80K lies in the coupling of answer supervision, spatio-temporal localization, and iterative reasoning traces. This suggests a dataset design principle different from conventional multimodal corpora: supervision is organized around the refinement loop itself. In that sense, VTTS-80K functions as an infrastructure dataset for test-time scaling in visual reasoning, not merely as a repository of labeled visual-language examples.