StreamingCoT: Temporal Multimodal VideoQA

• StreamingCoT is a large-scale dataset and framework for dynamic, multimodal chain-of-thought reasoning in video QA, supporting temporal answer evolution.
• It features a dynamic hierarchical annotation pipeline with per-second dense descriptions and adaptive segmentation to capture evolving video semantics.
• Its multimodal chain-of-thought paradigm integrates visual keyframes and object grounding, ensuring interpretable, auditable temporal logic in reasoning.

StreamingCoT is a large-scale dataset and framework specifically designed to enable temporally dynamic, multimodal Chain-of-Thought (CoT) reasoning in streaming Video Question Answering (VideoQA). It addresses critical limitations in prior video QA datasets by capturing evolving answers in streaming scenarios and providing explicit, step-wise multimodal reasoning traces grounded in spatiotemporal video evidence (Hu et al., 29 Oct 2025). StreamingCoT introduces a dynamic hierarchical annotation pipeline, temporally adaptive segmentation, and rigorous CoT annotation methods to facilitate research in temporal video understanding, causal inference, and interpretable multimodal logic.

1. Motivation and Dataset Overview

Existing VideoQA benchmarks suffer from two central shortcomings: (1) static annotation mechanisms that fail to encode answer evolution across continuous video streams, and (2) lack of interpretable reasoning traces, compelling models to rely on dataset biases or spurious correlations instead of explicit, verifiable logic. In streaming video contexts such as surveillance, procedural tasks, or real-time scene analysis, the correct answer to a query often changes as new events unfold—necessitating datasets where temporal answer evolution is a core property.

StreamingCoT provides:

• Dynamic annotations that track both the emergence and change of facts and answers throughout each video.
• Explicit multimodal CoT traces, yielding intermediate reasoning states instead of single final labels.

The dataset consists of 5,000 short-form videos (avg. 25.6s) stratified by geographic region, with no more than 20 per YouTube channel. It yields 243,185 per-second dense captions, fused into 68,940 temporally-dependent semantic segments. For each video, five dynamic QA pairs spanning six temporally-evolving types are generated, with each question accompanied by three distractors engineered for specific temporal or logical misalignment. In addition, StreamingCoT provides 68,940 multimodal CoT annotations and 206,820 key-object bounding boxes (Hu et al., 29 Oct 2025).

2. Dynamic Hierarchical Annotation Architecture

StreamingCoT employs a multi-stage annotation pipeline tailored to capture temporal dynamics and granular semantic progression:

• Per-Second Dense Descriptions: Using InternVL3, each video second $t$ is assigned a caption $C_t$, aligning the video to a sequence $\{(t, C_t)\}_{t=1}^N$.
• Adaptive Temporal Segmentation (Dynamic Semantic Fusion, DSF): Adjacent seconds are merged into segments if their caption embedding similarity $S_{t-1,t}$ (cosine similarity of $E(C_{t-1})$ and $E(C_t)$) remains above threshold $\theta=0.9$ :

$$S_{t-1,t} = \frac{E(C_{t-1}) \cdot E(C_t)}{\|E(C_{t-1})\|\|E(C_t)\|}$$

• Construction of Temporally-Dependent Semantic Segments: Segments $\{Seg_i\}_{i=1}^T$ typically partition videos into $T \approx 12$ units, each represented by a “dense caption” $C_t$0 via a VLLM merge under historical context:

$C_t$1

This architecture enables fine-grained indexing of changing states, object transitions, and event structures across continuous video frames.

3. Temporally-Constrained Question–Answer Sets

Each video is curated with five candidate QA pairs, each linked to a specific segment span $C_t$2. Question–Answer types include cumulative counting, periodic pattern recognition, sequential step tracking, state duration measurement, object-state recognition, and clue-revealing queries. The answer at each time $C_t$3, $C_t$4, is constrained to only update based on new evidence present in the succeeding segment:

$C_t$5

where $C_t$6 applies evolution constraints grounded in video content.

Distractors are carefully engineered to violate one (and only one) of four validity conditions:

• Temporal misalignment (using data from incorrect segments)
• Partial pattern compliance
• State-transition fallacy (invalid object progression)
• Premature inference

Human annotators verify (1) segment-to-answer temporal alignment, (2) plausibility of distractors, and (3) overall coherence between segment semantics and answer dynamics, ensuring that the annotation protocol respects the temporal evolution of each video (Hu et al., 29 Oct 2025).

4. Multimodal Chain-of-Thought Reasoning Paradigm

StreamingCoT explicitly encodes both intermediate reasoning and evidence grounding, facilitating interpretability and step-wise temporal inference:

• Temporally-Aware CoT Initialization: For each segment, the most representative keyframe is selected via maximum similarity between frame embeddings and dense caption text:

$C_t$7

The initial reasoning chain $C_t$8 is generated by a VLLM using current and preceding segment context and the identified keyframe.

• Spatiotemporal Key-Object Grounding: Objects involved in reasoning are parsed ($C_t$9 objects/segment), and their bounding boxes isolated using GroundingDINO on keyframes:

$\{(t, C_t)\}_{t=1}^N$0

• Multimodal Reasoning Fusion: Final CoTs, $\{(t, C_t)\}_{t=1}^N$1, fuse textual reasoning, visual keyframes, and grounded object boxes, subject to the constraint that every reasoning step $\{(t, C_t)\}_{t=1}^N$2 can be mapped to a spatiotemporal object and bounding box:

$\{(t, C_t)\}_{t=1}^N$3

• Logical Coherence Verification: A human-in-the-loop protocol evaluates (1) spatiotemporal consistency, (2) temporal causality, (3) evidence completeness, and (4) answer derivation soundness. Feedback $\{(t, C_t)\}_{t=1}^N$4 triggers up to three rounds of regeneration and revision:

$\{(t, C_t)\}_{t=1}^N$5

This ensures that deduction paths remain grounded, auditable, and logically consistent across temporally-evolving video content.

5. Toolkit and Workflow

StreamingCoT is accompanied by an open-source repository providing modular data processing and annotation utilities:

• Video collection and stratified sampling
• Multimodal filtering (social, audio, visual criteria)
• Dense per-second captioning and DSF segmentation
• QA generation with distractor design
• CoT synthesis, with integration of LLMs and object-grounding
• Annotation GUI and validation scheduler

A sample code workflow involves filtering source videos, generating per-second captions, fusing segments, synthesizing QA pairs, and constructing multimodal CoT traces:

$\{(t, C_t)\}_{t=1}^N$6 This replicable pipeline supports further research developments and experimentation (Hu et al., 29 Oct 2025).

6. Baseline Performance and Applications

Standard VideoQA models, such as ClipBERT and TempoVQ, attain only 30–45% accuracy on StreamingCoT, with a naive end-to-end transformer (no CoT supervision) dropping below 35% accuracy for state-duration and sequential-step queries. This underscores the necessity of explicit temporal and multimodal chain-of-thought reasoning to address the unique demands of streaming video understanding.

Core applications and prospective extensions include:

• Real-time surveillance and autonomous driving—answering dynamically-updated queries such as pedestrian counting or traffic signal state estimation
• Procedural long-form video understanding (e.g., cooking, assembly instructions)
• Zero-shot multimodal retrieval using streaming CoT traces as logic-aware embeddings
• Enrichment through integration of additional sensor modalities (audio, language, signal streams)

This suggests StreamingCoT is positioned as a benchmark and toolkit for the next generation of interpretable, truly temporal multimodal reasoning and streaming QA systems, providing the infrastructure and resources for both foundational and applied research in this domain (Hu et al., 29 Oct 2025).