STGR-CoT-30k: Unified Spatio-temporal Reasoning

• STGR-CoT-30k is a supervised dataset that unifies temporal and spatial grounding with structured chain-of-thought reasoning for video question answering.
• It comprises 30k samples with detailed annotations linking questions to precise bounding boxes, timestamps, and textual rationales, ensuring verifiable evidence.
• The corpus drives significant performance gains in video understanding models by enabling clear, evidence-based supervision and robust reinforcement learning.

STGR-CoT-30k is a supervised fine-tuning corpus designed to provide unified spatio-temporal grounding and chain-of-thought (CoT) reasoning for video understanding models. Developed as a pivotal resource for the Open-o3 Video architecture, it addresses the principal limitation of previous video QA datasets: the lack of explicit, joint supervision over both temporal spans and spatial object groundings within reasoning annotations. This unified annotation enables models to produce verifiable, evidence-centered explanations by specifying not only textual rationales but also when and where supporting evidence appears in the video (Meng et al., 23 Oct 2025).

1. Motivation and Rationale

Prior video reasoning benchmarks typically focus on either temporal grounding (identifying relevant time intervals) or spatial annotation (localizing objects in frames), but do not jointly supervise both dimensions within a synchronized reasoning trace. Most datasets also lack structured chains of thought that link question answering to explicit, localized video evidence. STGR-CoT-30k was curated to close these supervision gaps. The unified annotations enable supervised learning of not only general video QA but also fine-grained, evidence-grounded reasoning, supporting advanced use cases such as confidence-aware verification and robust reinforcement learning reward definition (Meng et al., 23 Oct 2025).

2. Dataset Composition and Scale

STGR-CoT-30k consists of approximately 30,000 examples constructed from a diverse mixture of QA and grounding resources, augmented with new spatio-temporal annotations. The dataset composition, as used for supervised fine-tuning in Open-o3 Video, is summarized as follows:

Subset	Number of samples	Annotation type
TVG-Coldstart	4,100	Temporal grounding CoT
TreeVGR-SFT	5,000	Spatial grounding CoT
New spatio-temporal data	5,900	Spatio-temporal CoT
Video-R1-CoT	15,000	General video QA CoT

The new spatio-temporal subset (5,900), generated by the authors’ annotation pipeline, is the main novelty; it systematically aligns timestamped video frames, object bounding boxes, and textual reasoning. Source datasets include ActivityNet, COIN, QueryD, QVHighlight, DiDeMo, and PLM-Rdcap for temporal and spatial grounding diversity. The final mixture comprises roughly 13.7% temporal, 16.7% spatial, 19.7% spatio-temporal, and 50% general QA samples (Meng et al., 23 Oct 2025).

3. Annotation Schema and Structured Reasoning Format

Each STGR-CoT-30k sample includes:

1. A question–answer pair, typically focused on an object or person.
2. One to five key frames sampled from the relevant video segment.
3. Bounding boxes for one to three salient objects within each key frame.
4. A structured chain of thought (reasoning trace) that explicitly grounds each evidence step.

The chain of thought adopts a strict output schema for mention annotation:

<obj>object_name</obj><box>[x_min, y_min, x_max, y_max]</box>at<t>timestamp</t>s

This format fuses entity identity, spatial localization, and temporal context, making annotation tokens directly computable for later reward or evaluation functions. Each reference in the reasoning trace must map unambiguously to both a box and timestamp, ensuring evidence is coherent and verifiable (Meng et al., 23 Oct 2025).

4. Data Curation Pipeline and Quality Control

Creation of the 5,900 new spatio-temporal samples follows a structured three-stage pipeline:

(a) Initial Annotation:

Videos from temporal grounding datasets and PLM-Rdcap are processed with Gemini 2.5 Pro using specialized prompts tailored to extract question–answer pairs, key frames, object labels, bounding boxes, and spatio-temporal reasoning traces. Frame indices from PLM-Rdcap are post-processed into timestamps.

(b) Bounding-Box Filtering:

Noisy or uninformative boxes are removed if they cover more than 80% of the frame. Each object crop is then automatically validated using Qwen2.5-VL-7B, accepting only regions for which the query “Is this a {object_name}?” yields an affirmative response.

(c) Self-Consistency Checking:

Annotation integrity is enforced by eliminating mismatches or redundancies. Every entity mentioned in the reasoning trace must have an associated valid box and timestamp, and all annotation elements must be internally consistent.

This pipeline yields a set of samples where answers, timestamps, bounding boxes, and reasoning chains are tightly aligned, enabling high-precision grounding supervision (Meng et al., 23 Oct 2025).

5. Distinctions from Prior Datasets

STGR-CoT-30k is distinguished from previous datasets on several axes:

• Temporal grounding datasets (e.g., ActivityNet) provide annotated time intervals but rarely localize objects.
• Spatial grounding or region caption datasets (e.g., TreeVGR-SFT, PLM-Rdcap) provide boxes per frame, often lacking temporal alignment.
• Traditional video-CoT datasets (e.g., Video-R1-CoT) yield textual rationales without explicit visual evidence.
• VideoEspresso provides chains of thought but does not synchronize spatial and temporal grounding within the rationale.

STGR-CoT-30k uniquely delivers joint supervision: its reasoning traces are explicitly linked to both spatial (object bounding boxes) and temporal (timestamped frames) evidence. The authors' ablation studies demonstrate that this spatio-temporal subset yields significantly better grounding metrics—improving mAM by +5.4% and mLGM by +10.4% over text-only and coarse-grained grounding datasets (Meng et al., 23 Oct 2025).

6. Use in Open-o3 Video Training and Evaluation

STGR-CoT-30k is employed for cold-start supervised fine-tuning (SFT) of Open-o3 Video, initializing the model from Qwen2.5-VL-7B. The SFT stage uses a learning rate of $1 \times 10^{-6}$ for one epoch over STGR-CoT-30k, imparting:

• Adherence to the structured reasoning format,
• Proficiency in spatio-temporal grounding,
• Ability to link evidence to answer rationale.

This initialization is critical for stabilizing subsequent reinforcement learning (RL), where reward sparsity is mitigated by pretrained competence in grounded reasoning. The dataset’s structured annotations directly facilitate computation of reward components (answer accuracy, temporal and spatial alignment, format correctness), as the RL reward functions require alignment between predicted reasoning and spatio-temporal ground truth. Open-o3 Video, with SFT on STGR-CoT-30k followed by RL, achieves state-of-the-art results on V-STAR (+14.4% mAM, +24.2% mLGM over the Qwen2.5-VL baseline) as well as on VideoMME, WorldSense, VideoMMMU, and TVGBench (Meng et al., 23 Oct 2025).

7. Impact and Significance

The introduction of STGR-CoT-30k has reshaped the methodology for training grounded video reasoning models by providing, for the first time, a dataset where every reasoning step is anchored in both spatial and temporal evidence. This supports improved model transparency, more effective test-time verification, and finer-grained RL reward engineering. Empirical results confirm that the uniquely structured spatio-temporal subset is a major driver of performance gains. A plausible implication is that future video QA and multimodal reasoning tasks will increasingly require similar joint supervision formats to achieve robust, verifiable evidence tracing in dynamic contexts (Meng et al., 23 Oct 2025).