Video-Thinker-10K: Multimodal Video-QA Dataset

• Video-Thinker-10K is a large-scale corpus of 10K video–QA pairs enriched with multimodal chain-of-thought traces for dynamic video reasoning.
• It operationalizes tool-driven techniques like temporal grounding, spatial zooming, and video captioning to ensure precise analysis and answer generation.
• The dataset supports both supervised and reinforcement learning approaches, significantly advancing performance metrics on long-form video understanding benchmarks.

Video-Thinker-10K is a large-scale, curated corpus of 10,000 video–question–answer instances explicitly designed to advance the capabilities of multimodal LLMs (MLLMs) in agentic long-form video reasoning. This dataset operationalizes dynamic tool use—spanning temporal grounding, spatial and temporal zooming, retrieval, and video captioning—within multi-step chain-of-thought (CoT) traces. It is a foundational resource for supervised and reinforcement learning approaches in agentic video LLMs and is built upon synthetic or hindsight-curated annotations to ensure each reasoning trajectory leads unambiguously to the ground-truth answer. Video-Thinker-10K supports both tool-augmented and model-intrinsic (no-tool) video reasoning research, and has been pivotal in the development and benchmarking of state-of-the-art video-LLMs (Li et al., 22 Jan 2026, Wang et al., 27 Oct 2025).

1. Dataset Design and Composition

Video-Thinker-10K comprises 10,000 video–question–answer (VQA) pairs, each paired with detailed multimodal chain-of-thought traces. Sample sources differ across instantiations:

• In (Li et al., 22 Jan 2026), samples are drawn from CG-Bench, with over half of constituent videos ranging from 20–40 minutes and 17.1% exceeding 40 minutes, targeting long-form comprehension.
• In (Wang et al., 27 Oct 2025), the dataset draws from six public VQA/captioning benchmarks: ActivityNet, TutorialVQA, YouCook2 (caption-labeled); STAR, ScaleLong, LVBench (QA-labeled), ensuring diversity across task and domain. Around 1,500–2,000 samples derive from each source.

Each entry contains:

• The video $V$ (preprocessed to conserve compute, e.g., up to 16 sampled frames at $128 \times 28 \times 28$ resolution per (Wang et al., 27 Oct 2025)).
• The question $Q$, requiring multi-segment temporal reasoning.
• The answer $Y$.
• The chain-of-thought trace $T$ (details below), embedding grounding, captioning, and analytic reasoning steps.

Task splits are balanced, covering activity recognition, instructional procedures, narrative understanding, and complex multi-step temporal reasoning (Wang et al., 27 Oct 2025).

2. Annotation Methodology and Generation Pipelines

Two principal annotation approaches underlie Video-Thinker-10K:

Agentic Tool-Driven Synthesis (Li et al., 22 Jan 2026)

Annotation proceeds in three stages:

1. Caption-space proxy: For each video $v_i$, a VideoLLM captioning tool produces a global caption $c_i$. A rich prompt $p(x_i, c_i)$ then describes available tools (ClipRetrieval, SubtitleRetrieval, SubtitleSummary, SubtitleZoom, CaptionZoom).
2. Agentic tool reasoning: An LLM (Qwen3-235B) is conditioned to generate multi-step reasoning and tool calls, sampling at temperature 0.7, terminating upon emitting an "Answer" command or $T=20$ steps. Only CaptionZoom (which yields captions for intervals, not frames) is exposed in synthesis.
3. Grounding to video: From up to 5 sampled traces per instance, the one matching the ground-truth answer is retained (randomly selected if none match). In the retained trace, CaptionZoom calls are replaced by FrameZoom calls, and captions are swapped for actual frames, yielding a truly multimodal CoT sequence (Algorithm 1 in (Li et al., 22 Jan 2026)).

Automated Hindsight-Curation (Wang et al., 27 Oct 2025)

Annotation in (Wang et al., 27 Oct 2025) follows a fully automated pipeline:

1. Question generation: For caption-rich sources, DeepSeek-R1 generates complex multi-segment reasoning questions.
2. Caption generation: For QA-rich sources, Gemini-2.5-Flash-Lite conditions per-segment descriptive captions on ground-truth and associated temporal markup.
3. Chain-of-thought synthesis: DeepSeek-V3 produces reasoning traces strictly using three tags: <time>start–end</time> for grounding, <caption>…</caption> for visual description, > …</think> for analytic steps. Hindsight curation is applied: if Qwen2.5-VL-7B-Instruct cannot recover $Y$ from a candidate trace, up to three regenerations are tried to guarantee correctness.

   Annotation guidelines enforce strict demarcation for grounding, captioning, and analytic moves, with explicit tags for each operation.

   3. Toolset and Reasoning Trace Structure

   The dataset encodes tool use either as explicit tool calls (for models trained to interface with external tool APIs, as in (Li et al., 22 Jan 2026)) or as intrinsic tagging within CoT traces (for model-internalized reasoning, as in (Wang et al., 27 Oct 2025)). Key tools and their computational roles include:

   Tool/Tag Name	Function Description	Usage Context
   ClipRetrieval	Returns top-k 10s clips matching a textual query	External-tool pipeline
   SubtitleRetrieval	Retrieves transcript segments matching query (via Whisper)	External-tool pipeline
   SubtitleSummary	Query-focused summary from the full transcript	External-tool pipeline
   FrameZoom	Resamples frames from a designated interval	Both
   SubtitleZoom	Returns subtitle text for interval	External-tool pipeline
   CaptionZoom	Generates caption for interval (internally calls FrameZoom + LLM)	Proxy/tool-augmented
   <time>start–end</time>	Temporal grounding within the reasoning trace	Model-intrinsic trace
   <caption>…</caption>	Visual description within the trace	Model-intrinsic trace
   <think>…	Analytical/reflective reasoning step	Model-intrinsic trace

A typical CoT trajectory utilizes 3–5 tool calls (or structured tags), with approximately 2–3 grounding, captioning, and analytic steps per instance (Wang et al., 27 Oct 2025, Li et al., 22 Jan 2026).

4. Training, Objectives, and Optimization

Training VideoLLMs on Video-Thinker-10K employs both supervised and reinforcement learning strategies:

• Supervised Fine-Tuning (SFT): The model learns to autoregressively generate the concatenated trace $128 \times 28 \times 28$0 and answer $128 \times 28 \times 28$1, minimizing negative log-likelihood:

$128 \times 28 \times 28$2

• Group Relative Policy Optimization (GRPO): For model-intrinsic tool usage (Wang et al., 27 Oct 2025), GRPO is applied after SFT, using a group-based reward structure to optimize answer correctness and adherence to the CoT format, incorporating relative advantage normalization and KL regularization against a reference policy. The optimization objective is:

$128 \times 28 \times 28$3

where $128 \times 28 \times 28$4 is the sum of correctness and format adherence, and $128 \times 28 \times 28$5 normalizes group advantage.

For tool-augmented models (Li et al., 22 Jan 2026), the student VideoLLM is trained to imitate full multimodal CoT traces via cross-entropy loss.

Typical hyperparameters include learning rates from $128 \times 28 \times 28$6 (SFT) to $128 \times 28 \times 28$7 (GRPO), batch sizes of 8–16, and 1–3 epochs over the corpus.

5. Evaluation Protocols and Metrics

Evaluation covers the domains and capabilities for which the dataset is constructed:

• Grounding: Measured via mIoU (mean Intersection-over-Union) and Recall@K (K=0.3, 0.5) for temporal localization.

• Captioning: Evaluated with BLEU@1, METEOR, and ROUGE-L metrics.
• End-to-end reasoning performance: Assessed on downstream video reasoning benchmarks (LongVideoBench, VideoMME, LVBench, MLVU (Li et al., 22 Jan 2026); Video-Holmes, CG-Bench-Reasoning, VRBench (Wang et al., 27 Oct 2025)) with task-specific accuracy. Comparative results show, for example, VideoThinker yielding LVBench accuracy of 48.9% vs baseline Qwen2.5-VL-7B at 38.3%, comparable to GPT-4o.

A confidence-gated two-stage inference protocol (in (Li et al., 22 Jan 2026)) first attempts direct reasoning on sampled frames; if the predicted confidence $128 \times 28 \times 28$8 (formulated as an exponentiated average log-likelihood) falls below threshold $128 \times 28 \times 28$9, a secondary, tool-augmented reasoning pass is triggered. This enables adaptive allocation of compute to uncertain instances.

6. Significance and Impact in Video Reasoning Research

Video-Thinker-10K has been instrumental in operationalizing dynamic, agentic video reasoning in both tool-augmented and model-intrinsic settings. By embedding explicit, step-wise tool use or grounding-captioning-analytic sequences within each sample, the dataset overcomes the limitations of static, frame-uniform reasoning protocols. This supports models that can explore, localize, and interpret complex temporal structures in long-form videos.

The synthetic and hindsight-curation annotation paradigms break the circular dependency inherent in training agentic video LLMs—circumventing the need for strong existing long-form comprehension capabilities when constructing training data. The effectiveness of Video-Thinker-10K has been substantiated by substantial improvements in long-form video QA and reasoning benchmarks, with models trained on this dataset reaching or exceeding performance of closed-source and agent-composition baselines in both tool-enhanced and fully end-to-end inference regimes.

A plausible implication is that future benchmarks and datasets inspired by Video-Thinker-10K will further catalyze progress toward more adaptive, interpretable, and data-efficient multimodal LLMs for temporally complex video understanding (Li et al., 22 Jan 2026, Wang et al., 27 Oct 2025).