---
title: 'Video-Thinker-10K: Multimodal Video-QA Dataset'
url: https://www.emergentmind.com/topics/video-thinker-10k
type: topic
---

# Video-Thinker-10K: Multimodal Video-QA Dataset

Video-Thinker-10K is a large-scale, curated corpus of 10,000 video–question–answer instances explicitly designed to advance the capabilities of multimodal large language models (MLLMs) in agentic long-form video reasoning. This dataset operationalizes dynamic tool use—spanning temporal grounding, spatial and temporal zooming, retrieval, and video captioning—within multi-step chain-of-thought (CoT) traces. It is a foundational resource for supervised and reinforcement learning approaches in agentic video LLMs and is built upon synthetic or hindsight-curated annotations to ensure each reasoning trajectory leads unambiguously to the ground-truth answer. Video-Thinker-10K supports both tool-augmented and model-intrinsic (no-tool) video reasoning research, and has been pivotal in the development and benchmarking of state-of-the-art video-language models [2601.15724][2510.23473].

## 1. Dataset Design and Composition

Video-Thinker-10K comprises 10,000 video–question–answer (VQA) pairs, each paired with detailed multimodal chain-of-thought traces. Sample sources differ across instantiations:

- In [2601.15724], samples are drawn from CG-Bench, with over half of constituent videos ranging from 20–40 minutes and 17.1% exceeding 40 minutes, targeting long-form comprehension.
- In [2510.23473], the dataset draws from six public VQA/captioning benchmarks: ActivityNet, TutorialVQA, YouCook2 (caption-labeled); STAR, ScaleLong, LVBench (QA-labeled), ensuring diversity across task and domain. Around 1,500–2,000 samples derive from each source.

Each entry contains:  
- The video $V$ (preprocessed to conserve compute, e.g., up to 16 sampled frames at $128 \times 28 \times 28$ resolution per [2510.23473]).
- The question $Q$, requiring multi-segment temporal reasoning.
- The answer $Y$.
- The chain-of-thought trace $T$ (details below), embedding grounding, captioning, and analytic reasoning steps.

Task splits are balanced, covering activity recognition, instructional procedures, narrative understanding, and complex multi-step temporal reasoning [2510.23473].

## 2. Annotation Methodology and Generation Pipelines

Two principal annotation approaches underlie Video-Thinker-10K:

### Agentic Tool-Driven Synthesis ([2601.15724])
Annotation proceeds in three stages:
1. **Caption-space proxy:** For each video $v_i$, a VideoLLM captioning tool produces a global caption $c_i$. A rich prompt $p(x_i, c_i)$ then describes available tools (ClipRetrieval, SubtitleRetrieval, SubtitleSummary, SubtitleZoom, CaptionZoom).
2. **Agentic tool reasoning:** An LLM (Qwen3-235B) is conditioned to generate multi-step reasoning and tool calls, sampling at temperature 0.7, terminating upon emitting an "Answer" command or $T=20$ steps. Only CaptionZoom (which yields captions for intervals, not frames) is exposed in synthesis.
3. **Grounding to video:** From up to 5 sampled traces per instance, the one matching the ground-truth answer is retained (randomly selected if none match). In the retained trace, CaptionZoom calls are replaced by FrameZoom calls, and captions are swapped for actual frames, yielding a truly multimodal CoT sequence (Algorithm 1 in [2601.15724]).

### Automated Hindsight-Curation ([2510.23473])
Annotation in [2510.23473] follows a fully automated pipeline:
1. **Question generation:** For caption-rich sources, DeepSeek-R1 generates complex multi-segment reasoning questions.
2. **Caption generation:** For QA-rich sources, Gemini-2.5-Flash-Lite conditions per-segment descriptive captions on ground-truth and associated temporal markup.
3. **Chain-of-thought synthesis:** DeepSeek-V3 produces reasoning traces strictly using three tags: `<time>start–end</time>` for grounding, `<caption>…</caption>` for visual description, `<think>…</think>` for analytic steps. Hindsight curation is applied: if Qwen2.5-VL-7B-Instruct cannot recover $Y$ from a candidate trace, up to three regenerations are tried to guarantee correctness.

Annotation guidelines enforce strict demarcation for grounding, captioning, and analytic moves, with explicit tags for each operation.

## 3. Toolset and Reasoning Trace Structure

The dataset encodes tool use either as explicit tool calls (for models trained to interface with external tool APIs, as in [2601.15724]) or as intrinsic tagging within CoT traces (for model-internalized reasoning, as in [2510.23473]). Key tools and their computational roles include:

| Tool/Tag Name         | Function Description                                              | Usage Context          |
|-----------------------|------------------------------------------------------------------|------------------------|
| ClipRetrieval         | Returns top-k 10s clips matching a textual query                 | External-tool pipeline |
| SubtitleRetrieval     | Retrieves transcript segments matching query (via Whisper)        | External-tool pipeline |
| SubtitleSummary       | Query-focused summary from the full transcript                   | External-tool pipeline |
| FrameZoom             | Resamples frames from a designated interval                      | Both                  |
| SubtitleZoom          | Returns subtitle text for interval                               | External-tool pipeline |
| CaptionZoom           | Generates caption for interval (internally calls FrameZoom + LLM)| Proxy/tool-augmented   |
| `<time>start–end</time>`  | Temporal grounding within the reasoning trace                     | Model-intrinsic trace  |
| `<caption>…</caption>`    | Visual description within the trace                               | Model-intrinsic trace  |
| `<think>…</think>`        | Analytical/reflective reasoning step                              | Model-intrinsic trace  |

A typical CoT trajectory utilizes 3–5 tool calls (or structured tags), with approximately 2–3 grounding, captioning, and analytic steps per instance [2510.23473][2601.15724].

## 4. Training, Objectives, and Optimization

Training VideoLLMs on Video-Thinker-10K employs both supervised and reinforcement learning strategies:

- **Supervised Fine-Tuning (SFT):** The model learns to autoregressively generate the concatenated trace $T$ and answer $Y$, minimizing negative log-likelihood:
  $$
  \mathcal{L}_{\rm SFT}(\theta)= -\mathbb{E}_{(V,Q,Y)\sim\mathcal{D}\;}\sum_{t=1}^{|[T;Y]|} \log p_\theta\left([T;Y]_t \mid V,Q,[T;Y]_{<t}\right)
  $$
- **Group Relative Policy Optimization (GRPO):** For model-intrinsic tool usage ([2510.23473]), GRPO is applied after SFT, using a group-based reward structure to optimize answer correctness and adherence to the CoT format, incorporating relative advantage normalization and KL regularization against a reference policy. The optimization objective is:
  $$
  \mathcal{J}_{\rm GRPO}(\theta)
  = \mathbb{E}\Biggl[\frac1G\sum_{i=1}^G
  \left(\min(r_i,\;\mathrm{clip}(r_i,1-\epsilon,1+\epsilon))\,A_i
  -\beta \,\mathrm{KL}(p_\theta\Vert p_{\rm ref})\right)\Biggr]
  $$
  where $r_i$ is the sum of correctness and format adherence, and $A_i$ normalizes group advantage.

For tool-augmented models ([2601.15724]), the student VideoLLM is trained to imitate full multimodal CoT traces via cross-entropy loss.

Typical hyperparameters include learning rates from $1 \times 10^{-5}$ (SFT) to $5 \times 10^{-6}$ (GRPO), batch sizes of 8–16, and 1–3 epochs over the corpus.

## 5. Evaluation Protocols and Metrics

Evaluation covers the domains and capabilities for which the dataset is constructed:

- **Grounding:** Measured via mIoU (mean Intersection-over-Union) and Recall@K (K=0.3, 0.5) for temporal localization.
- **Captioning:** Evaluated with BLEU@1, METEOR, and ROUGE-L metrics.
- **End-to-end reasoning performance:** Assessed on downstream video reasoning benchmarks (LongVideoBench, VideoMME, LVBench, MLVU [2601.15724]; Video-Holmes, CG-Bench-Reasoning, VRBench [2510.23473]) with task-specific accuracy. Comparative results show, for example, VideoThinker yielding LVBench accuracy of 48.9% vs baseline Qwen2.5-VL-7B at 38.3%, comparable to GPT-4o.

A confidence-gated two-stage inference protocol (in [2601.15724]) first attempts direct reasoning on sampled frames; if the predicted confidence $\gamma$ (formulated as an exponentiated average log-likelihood) falls below threshold $\tau=0.7$, a secondary, tool-augmented reasoning pass is triggered. This enables adaptive allocation of compute to uncertain instances.

## 6. Significance and Impact in Video Reasoning Research

Video-Thinker-10K has been instrumental in operationalizing dynamic, agentic video reasoning in both tool-augmented and model-intrinsic settings. By embedding explicit, step-wise tool use or grounding-captioning-analytic sequences within each sample, the dataset overcomes the limitations of static, frame-uniform reasoning protocols. This supports models that can explore, localize, and interpret complex temporal structures in long-form videos.

The synthetic and hindsight-curation annotation paradigms break the circular dependency inherent in training agentic video LLMs—circumventing the need for strong existing long-form comprehension capabilities when constructing training data. The effectiveness of Video-Thinker-10K has been substantiated by substantial improvements in long-form video QA and reasoning benchmarks, with models trained on this dataset reaching or exceeding performance of closed-source and agent-composition baselines in both tool-enhanced and fully end-to-end inference regimes.

A plausible implication is that future benchmarks and datasets inspired by Video-Thinker-10K will further catalyze progress toward more adaptive, interpretable, and data-efficient multimodal LLMs for temporally complex video understanding [2601.15724][2510.23473].

Source: https://www.emergentmind.com/topics/video-thinker-10k