Open-o3 Video: Grounded Video Reasoning

• Open-o3 Video is a unified framework for grounded video reasoning that integrates precise temporal and spatial evidence annotations to support traceable answers.
• The system employs a single-model pipeline that interleaves chain-of-thought outputs with explicit timestamps, bounding boxes, and object-level evidence.
• Training leverages curated spatio-temporal datasets and reinforcement learning with GSPO, significantly boosting benchmark performance and interpretability.

Open-o3 Video is a technical framework for grounded video reasoning that augments conventional video question answering with explicit spatio-temporal evidence localization. The model and methodology, named and formalized in "Open-o3 Video: Grounded Video Reasoning with Explicit Spatio-Temporal Evidence" (Meng et al., 23 Oct 2025), address the key challenge of coupling natural-language reasoning with precise temporal and spatial localization of evidence over dynamic video streams. The system is architected as a non-agentic, single-model pipeline that produces chain-of-thought outputs with interleaved timestamp, bounding box, and object-level provenance—enabling reasoning that is directly grounded in concrete video observations.

1. Motivation and Problem Statement

Traditional video reasoning models focus on textual chain-of-thought generation, without providing explicit traces about when and where the decisive evidence appears. This disconnect undermines verifiability and makes responses susceptible to hallucination, as it is unclear which frames or objects underpin a given answer. Spatio-temporal grounding has largely been addressed in isolation: temporal localization datasets annotate intervals but lack spatial markers, while spatial localization datasets supply bounding boxes without temporal context. The absence of synchronized supervision inhibits unified evidence-centric video reasoning.

Open-o3 Video targets this deficit by requiring that all reasoning steps be explicitly anchored to both temporal and spatial annotations. Each evidence reference in the reasoning trace must specify which timestamps and object bounding boxes were attended to, and in what frame, before producing the final answer.

2. Model Architecture and Output Format

The model operates as a single-stage, non-agentic LLM initialized from Qwen2.5-VL-7B. It accepts a video and a natural language query, and outputs a reasoning trace that alternates between natural language and explicit spatio-temporal evidence tags with the following structured schema:

• > blocks contain text interleaved with <obj>object</obj>, <box>[x_min, y_min, x_max, y_max]</box>, and <t>timestamp</t>s elements, each aligned to a specific moment and region in the video. > > - The final <answer> token produces the model's answer. > > This output format ensures each inference step is explainable, explicitly revealing the evidence for each conclusion and enabling subsequent verification, either automatically or by human auditors. > > ## 3. Training Data: Unified Spatio-Temporal Reasoning Datasets > > Open-o3 Video is trained on two deliberately constructed and curated datasets: > > ### 3.1 STGR-CoT-30k (SFT Stage) > > - 4.1k temporal grounding CoT samples (from TVG-Coldstart) > > - 5k spatial grounding CoT samples (from TreeVGR-SFT) > > - 5.9k newly curated spatio-temporal samples with synchronized frame- and box-level evidence > > - 15k general video QA CoT samples (from Video-R1-CoT) > > Each curated sample ensures that all references to objects or events in reasoning refer to annotated bounding boxes at validated timestamps. > > ### 3.2 STGR-RL-36k (RL Stage) > > - 5.2k temporal grounding (mixed sources) > > - 5k spatial grounding > > - 10.9k spatio-temporal (union of new curation and filtered VideoEspresso) > > - 15k general reasoning samples > > This data construction overcomes the lack of unified spatio-temporal annotation in prior resources. Filtration and self-consistency validation removes objects with excessively large bounding boxes (>80% of frame), or mismatches between referenced objects and their annotated box. > > ## 4. Training Objective and Reinforcement Learning > > Model optimization is staged: > > - Supervised Fine-Tuning (SFT) on STGR-CoT-30k teaches output structural format and initial grounding. > > - Reinforcement Learning (RL) via Group Sequence Policy Optimization (GSPO) on STGR-RL-36k sharpens reward alignment to accuracy, spatio-temporal grounding, and output formatting. > > Rewards are decomposed as: > > - Answer Accuracy: 1 for correct MCQ, ROUGE or vIoU/tIoU for free-form, spatial, or temporal grounding, respectively. > > - Thinking Reward: Temporal (match to ground-truth intervals/points with adaptive Gaussian), and spatial (IoU with ground-truth boxes, gated by proximity to the correct time). > > - Format Reward: 1.0 for full structuring, 0.5 for partial, 0 otherwise. > > Rewards leverage two essential mechanisms: > > - Adaptive Temporal Proximity: Early training assigns broad temporal proximity tolerance, narrowed as training progresses. > > - Temporal Gating: Spatial rewards are computed only for timestamps within a permitted window of ground-truth, enforcing coupling between when and where evidence must align. > > The sequence-level GSPO objective is: > > $$J_{\mathrm{GSPO}}(\theta) = \mathbb{E}_{x, \{y_i\}} \left[ \frac{1}{G} \sum_{i=1}^G \min \left( s_i(\theta) \hat{A}_i, \mathrm{clip}(s_i(\theta), 1-\epsilon, 1+\epsilon) \hat{A}_i \right) \right]$$ > > where $s_i(\theta)$ is the normalized sequence-level importance ratio and $\hat{A}_i$ is the normalized reward advantage. > > ## 5. Grounded Evidence Emission and Test-Time Scaling > > During inference, the model emits structured outputs: > > - Each <think> block grounds objects, boxes, and timestamps. > > - The final <answer> aggregates evidence-based reasoning. > > This enables confidence-aware test-time verification: multiple candidate answers can be resampled, and the model's own predicted evidence regions are automatically cropped and rescored for consistency with the answer, enabling evidence-weighted voting for more reliable responses. > > ## 6. Experimental Results and Benchmark Performance > > Open-o3 Video strongly outperforms baselines in spatio-temporal reasoning and general video QA: > > | Benchmark | Metric | Qwen2.5-VL-7B Baseline | Open-o3 Video | Δ (improvement) | > |-------------------|------------------------|-------------------------|---------------|------------------| > | V-STAR | mAM | 19.3 | 33.7 | +14.4 | > | | mLGM | 22.4 | 46.6 | +24.2 | > | VideoMME | Overall / Long | 62.4 / 50.8 | 63.6 / 54.9 | +1.2 / +4.1 | > | WorldSense | Overall | 36.1 | 37.5 | +1.4 | > | VideoMMMU | Overall | 51.2 | 52.3 | +1.1 | > | TVGBench | mIoU | 16.3 | 20.8 | +4.5 | > > Ablation studies demonstrate: > > - RL with GSPO produces the strongest grounding. > > - Removal of adaptive temporal proximity or temporal gating substantially reduces spatial/temporal localization scores. > > - Addition of curated spatio-temporal training data is essential for high-grounding performance. > > - Test-time evidence-weighted response selection outperforms majority voting on compound tasks. > > ## 7. Analysis, Interpretability, and Limitations > > Open-o3 Video demonstrates that unified, evidence-centric supervision and output structuring drive more transparent and reliable video reasoning. Qualitative examples show precise identification of frames and regions supporting answers to questions involving action recognition, weather detection, and appearance-based tasks. > > The main limitations arise in scenarios involving: > > - Long videos with many small objects (localization difficulty persists as video length grows). > > - Multi-step inferential reasoning beyond direct evidence localization. > > - Lack of audio and speech integration, which are outside the model's current remit. > > The approach's dependence on curated multi-modal annotation also induces a substantial data preparation overhead. > > ## 8. Significance and Future Directions > > By formalizing spatio-temporal evidence tracing and supervision, Open-o3 Video establishes a technical reference for grounded video reasoning. It supports more rigorous verification, reduces hallucinations, and provides a blueprint for integrating evidence-centered reasoning into emerging multimodal LLMs and MLLMs. The structured output format and reinforcement objectives may inform future developments in evidence alignment for audio-visual and grounded multimodal agents. > > Open-o3 Video represents the first large-scale benchmarked effort to transition from "reasoning in text" to "reasoning with evidence" for videos, operationalizing a research agenda that prioritizes interpretability, reproducibility, and community-accessible open-source deployment (Meng et al., 23 Oct 2025).