VTS-CoT: Video-Spatiotemporal CoT

• VTS-CoT is a series of video chain-of-thought methods that iteratively incorporate key visual and temporal evidence to enhance reasoning.
• These paradigms replace static text-only rationales with interleaved visual cues, improving performance in temporal localization and spatial grounding tasks.
• They utilize innovations like key-video insertion, progress bars, and object-centric subgoals to enable more accurate and interpretable multimodal reasoning.

Searching arXiv for papers on VTS-CoT and related video/visual chain-of-thought reasoning. VTS-CoT, used here as an Editor’s term, denotes a family of video- and spatiotemporal chain-of-thought paradigms in which a model does not rely on a purely textual rationale after a single video encoding pass, but instead revisits temporally or spatially localized evidence during reasoning. The recent literature does not converge on one standardized name; closely related formulations appear as “ViTCoT,” “VTimeCoT,” “VTI-CoT,” “Video-CoT,” and “CoTasks,” alongside adjacent multimodal work such as “Visual CoT” and “TVI-CoT.” Across these papers, the shared premise is that standard video CoT is text-centric, whereas interleaved variants explicitly reintroduce key frames, key-video clips, progress bars, highlighted temporal intervals, localized crops, or object-centric subgoals into the reasoning process itself (Zhang et al., 14 Jul 2025, Zhang et al., 16 Oct 2025, Zhang et al., 4 Jun 2026, Zhang et al., 10 Jun 2025, Wang et al., 18 Jul 2025, Shao et al., 2024, Hu et al., 7 Jun 2026).

1. Conceptual shift from text-only video CoT

The central departure of VTS-CoT-style work is the rejection of “reason once from a latent video summary” as the dominant reasoning pattern. In ViTCoT, vanilla video CoT is formalized as generating a text reasoning process $\mathcal{R}_t$ from video $V$, question $Q$, and prompt $P_t$, while the interleaved variant first produces initial text reasoning $\mathcal{R'}_t$ and then inserts a key-video $V_k$ into a second-stage reasoning process $\mathcal{R}_{t\rightarrow v}$. In essence, the distinction is between a text-only rationale generated after seeing the video once and an iterative multimodal rationale in which key visual evidence is explicitly revisited during the chain itself (Zhang et al., 14 Jul 2025).

This reformulation is not merely stylistic. VTimeCoT argues that standard text-only CoT is notably deficient for video temporal grounding and reasoning because many tasks depend on when an event happens, how events unfold over time, and which segments are relevant. Its response is a visuotemporal CoT process in which temporal position becomes directly visible through a progress bar and relevant moments can be highlighted as part of reasoning. The resulting pipeline is explicitly tool-using, iterative, and training-free rather than a static text-generation routine (Zhang et al., 16 Oct 2025).

A similar diagnosis appears in adjacent multimodal work beyond video. TVI-CoT characterizes prior multimodal CoT as a “vision-blind reasoning” paradigm: after the initial visual encoding, image information becomes inaccessible during later reasoning steps. Its solution is to alternate between textual reasoning and reasoning-conditioned visual access through learnable control tokens <Think>, <Look>, and <Answer>, which makes the broader VTS-CoT trend intelligible as part of a general move from one-shot visual encoding toward repeated visual consultation (Hu et al., 7 Jun 2026).

2. Canonical design patterns

The current literature exhibits several recurring architectural patterns rather than a single canonical system.

Framework	Interleaving unit	Salient mechanism
ViTCoT	Key-video	Initial text reasoning, then video-text interleaved reasoning
VTimeCoT	Progress bar and highlights	THOUGHT/ACTION/TERMINATE loop with tool execution
VTI-CoT	Step-linked frames	Temporally grounded CoT rendered into a single canvas
CoTasks	Object-centric subgoals	Frame localization, tracking, spatial relations, temporal relations
Video-CoT	Spatiotemporal reasoning traces	CoT-SFT over six subtasks

ViTCoT defines the simplest explicit interleaving pattern. The model first generates initial reasoning from the original video, then continues reasoning with both the original video and a selected key-video embedded into the prompt context. The paper treats this as a general reasoning paradigm that can be applied not only to vanilla CoT, but also to Desp-CoT and Plan-and-Solve, which is important because it frames interleaving as a reasoning protocol rather than a model-specific architectural change (Zhang et al., 14 Jul 2025).

VTimeCoT instead turns temporal reasoning into a tool-augmented interaction with the video. Its progress bar integration tool appends a frame-synchronized temporal bar to each frame, expressed as $V'_t = V_t \oplus T_t$, and its highlighting tool uses VideoCLIP-XL retrieval to identify likely relevant temporal moments. The model operates in a closed loop that alternates THOUGHT, ACTION, and TERMINATE, with tool invocations converted into executable Python code and the manipulated frames fed back as updated video memory (Zhang et al., 16 Oct 2025).

VTI-CoT moves interleaving into the supervision channel itself. It first segments the video into coherent intervals using CLIP feature similarity, summarizes each interval with InternVL3-8B, then prompts Qwen2.5-7B to generate reasoning steps of the form $r_t \mid [i_{ts}, i_{te}]$, so that every step is explicitly grounded to a temporal span. To avoid the very long token sequences produced by such supervision, the interleaved text-image chain is rendered into a single canvas image, creating an OCR-style compression regime for multimodal CoT (Zhang et al., 4 Jun 2026).

CoTasks represents a different branch of the same agenda. Instead of interleaving raw visual evidence back into the chain, it decomposes complex video QA into four entity-level foundational tasks: frame localization, entity tracking, spatial relation extraction, and temporal relation extraction. The contribution is not a new architecture but a structured prompting scheme that injects these intermediate object-centric subgoals into inference, forcing the model to reason through explicit grounding stages before producing the final answer (Wang et al., 18 Jul 2025).

3. Datasets, annotation pipelines, and benchmarks

A defining feature of VTS-CoT-style research is that progress has depended heavily on new supervision regimes and benchmark construction. ViTCoT introduces the Video-Text Interleaved Benchmark (ViTIB), built on VideoEspresso. ViTIB contains 1,382 videos and 5,051 key-video frames, with an average of 3.7 frames per key-video. Construction uses Gemini-2.0-Flash for key-frame recognition, assembly of key-videos at FPS = 1, and a human recheck in which each sample is reviewed by three independent reviewers. Only entries where all three reviewers score above 80 are retained; the resulting overall average score is 83.6. The benchmark spans 14 categories, and the paper notes that Narrative Analysis requires 5.0 frames per key-video, larger than the overall average, because this category needs more evidence (Zhang et al., 14 Jul 2025).

VideoCoT and its companion datasets address the annotation bottleneck from a different angle. “VideoCoT: A Video Chain-of-Thought Dataset with Active Annotation Tool” argues that fully manual video CoT annotation is too cumbersome and expensive, while machine-generated CoT is unreliable due to hallucination. Its solution is an active annotation tool combining prompt generation, automatic scoring, and expert refinement. The paper contributes three datasets—VideoCoT, TopicQA, and TopicCoT—built from Kinetics-700 with a 60%/20%/20% split. VideoCoT contains 11,182 CoT-annotated samples, and the scoring pipeline routes samples with normalized quality below 0.9 to human experts. The reported active iterations raise the generated-CoT quality score from 0.07 to 0.97, which suggests that the human-in-the-loop design is not merely advisory but central to data quality (Wang et al., 2024).

Video-CoT scales the supervision regime further toward spatiotemporal grounding. It provides 192,000 fine-grained spatiotemporal question-answer pairs and 23,000 high-quality CoT-annotated samples organized into six subtasks across three major categories: Temporal Video Localization and Video Captioning; Spatial Video Grounding and Spatio-Temporal Video Grounding; and Spatial Relationship Reference and Temporal Video Reference. Its separate benchmark contains 4,500 video question-answer pairs, with 750 images per task. The construction pipeline draws on existing datasets such as VTimeLLM-stage2, MSR-VTT, MSVD, WebVid, HCSTVG-V1, and VidSTG, then applies filtering to ensure persistent objects, consecutive visibility, and explicit temporal spans (Zhang et al., 10 Jun 2025).

VTI-CoT likewise builds supervision because no existing dataset naturally contains visual-textual interleaved CoT at the required granularity. It reports VTI-Video-R1-CoT-165K, built on Video-R1-CoT-165K, with 86,217 videos, and VTI-MovieChat, built on MovieChat, with 4,136 long-form movie videos. The paper emphasizes that these datasets have long average token lengths and many images per sample, which is precisely why OCR-based compression becomes a training efficiency issue rather than an optional refinement (Zhang et al., 4 Jun 2026).

4. Training and inference regimes

The family spans several distinct optimization regimes. At one extreme, VTimeCoT is explicitly training-free: it augments an MLLM with a plug-and-play progress bar integration tool, a highlighting tool based on VideoCLIP-XL retrieval, and an agentic loop with a maximum reasoning depth of $T = 3$. The model does not learn new parameters for temporal reasoning; rather, it externalizes temporal structure into the visual prompt space and updates its memory through repeated tool execution (Zhang et al., 16 Oct 2025).

At another extreme, Video-CoT introduces an explicit CoT-SFT objective. Its answer-supervised formulation predicts $V$0 directly from $V$1, while the CoT-supervised formulation predicts both reasoning chain $V$2 and answer $V$3:

$V$4

The paper further states that CoT-SFT uses curriculum learning, beginning with simple 2-step reasoning and increasing to more complex 5-step chains involving temporal localization, spatial identification, object trajectory analysis, spatial relation deduction, and causal inference (Zhang et al., 10 Jun 2025).

VTI-CoT adds a hybrid language-vision objective tailored to compressed interleaved supervision. A rendered CoT canvas is encoded by the frozen vision encoder, the model predicts a CoT sequence with standard autoregressive cross-entropy, and a projection head aligns hidden states with visual features using an MSE-based vision loss. The total objective is

$V$5

with $V$6, $V$7, and $V$8. This regime makes the CoT structure itself visually encoded, so interleaving becomes both a reasoning format and a supervision compression mechanism (Zhang et al., 4 Jun 2026).

CoTasks occupies a third point in the design space: it does not define a new end-to-end loss or a new model architecture. Instead, the authors use prompting-based inference-time augmentation in which the original question is supplemented with the ground-truth answers of the intermediate CoTasks. This isolates the causal effect of structured intermediate context on final reasoning performance and makes CoTasks closer to a controlled study of structured prompting than to a conventional training framework (Wang et al., 18 Jul 2025).

5. Empirical gains and what they imply

The empirical literature is unusually consistent on one point: interleaving visual or spatiotemporal evidence into the reasoning loop often improves performance relative to text-only CoT. In ViTCoT, Qwen2.5-VL-7B-Instruct improves from 42.8 with Vanilla CoT to 51.4 with ViT CoT and to 52.9 with ViT Desp-CoT, while VideoLLaMA3-7B reaches 57.2 with ViT Plan-and-Solve. The paper highlights that, for Qwen2.5-VL-7B, ViT methods give an average improvement of 5.4% over vanilla reasoning, and across methods interleaved reasoning methods outperform vanilla methods with an average improvement of 3.5%. Two ablations are especially revealing: even when both original video and key-video are provided, ViT reasoning remains better by a 2.8% average margin, and with a rough key-video selected by CLIP similarity, the ViT paradigm still beats vanilla by 1.7% on average. This suggests that the gain is not reducible to “more visual tokens” alone; the reasoning format matters (Zhang et al., 14 Jul 2025).

VTimeCoT reports larger effects when the problem demands temporal localization. On Charades-STA, Qwen2VL-7B improves from mIoU 24.34 to 43.41 and GPT-4o improves from 40.20 to 46.78. On QVHighlights, Qwen2VL-7B improves from 22.77 to 46.21 and GPT-4o from 37.66 to 54.49. On Vript-RR, GPT-4o rises from 70.39 to 83.55 on multi-choice and from 61.18 to 68.42 on open-ended evaluation. The ablation on GPT-4o shows a clear hierarchy: no CoT/no tools yields QVHighlights mIoU 37.66 and Vript-RR 70.39, standard CoT only reaches 41.85 and 73.68, CoT plus progress bar reaches 49.40 and 76.32, and the full method with progress bar plus highlight reaches 54.49 and 83.55. The implication is that text-only CoT helps somewhat, but explicit temporal instrumentation contributes the major gain (Zhang et al., 16 Oct 2025).

CoTasks shows that even inference-time structured prompting can substantially shift model behavior. On NeXT-QA, LLaVA-video-7B improves from 50.2 to 53.5 average GPT-4 evaluation score, while Qwen2.5-VL-3B improves from 27.8 to 45.2, including gains of +14.6 on causal, +10.9 on temporal, and +48.1 on descriptive subcategories. On STAR with Qwen2.5-VL-3B, no prompting/no fine-tuning yields 31.1%, CoTask-style prompting alone yields 51.6%, and fine-tuning alone yields 65.4%. The paper’s subset ablation further indicates complementarity: CoTasks 1–2 mainly help grounding and descriptive categories, CoTasks 3–4 help causal and temporal reasoning, and the full 1–4 sequence performs best (Wang et al., 18 Jul 2025).

Video-CoT presents a similar contrast between answer-only and reasoning-supervised training. For Qwen2.5-VL-3B, Video-Ans-SFT improves TVL from 4.4 to 8.3, whereas Video-CoT-SFT improves it to 19.7; SVG rises from 10.3 to 14.2 with answer supervision and to 17.0 with CoT supervision; STVG sIoU rises from 5.7 to 8.1 and then to 9.2. The paper interprets this as evidence that explicit reasoning supervision is especially effective for temporal localization and longer reasoning chains, and the comparison table reinforces that even strong models still struggle on STVG, where the best reported sIoU in the benchmark is 9.9 for Gemini-1.5-pro (Zhang et al., 10 Jun 2025).

VTI-CoT reports state-of-the-art performance among open-source 7B-class models listed in its table, with MVBench 65.9, TempCompass 74.5, Video-MME 59.6, MMVU 65.3, LongVideoBench 55.0, and LVBench 40.5. Its ablations are notable because rendered CoT outperforms tokenized CoT on all benchmarks, and shuffled interleaving hurts performance, indicating that the method depends on the correct temporal and logical ordering of multimodal evidence rather than mere co-occurrence of text and frames (Zhang et al., 4 Jun 2026).

6. Limitations, misconceptions, and open questions

A recurring misconception is that VTS-CoT-style systems succeed simply because they introduce extra visual input. Several papers explicitly argue otherwise. ViTCoT’s “original video plus key-video” ablation shows that the reasoning paradigm itself matters, not just the additional evidence channel, while CoTasks shows that structured intermediate context can improve performance even without a new architecture (Zhang et al., 14 Jul 2025, Wang et al., 18 Jul 2025).

A second misconception is that interleaving necessarily improves robustness. The strongest cautionary result comes from the adjacent image-domain literature on Visual CoT robustness. “Visual CoT Makes VLMs Smarter but More Fragile” reports that across 96 evaluated settings, Visual CoT has a higher Performance Drop Rate in 70 out of 96 cases, with average PDR 26.3% versus 18.6% for standard VLMs, even though Visual CoT still achieves higher accuracy in 79 out of 96 cases under perturbation. The paper attributes this fragility to the intermediate local image patch: lower bounding-box quality correlates with larger final PDR, and perturbing both the global image and the intermediate crop hurts more than perturbing the global image alone. Its plug-and-play Grounding DINO augmentation, using region proposals above a confidence threshold typically 0.4, improves Visual CoT accuracy across perturbations by about 6% on average. A plausible implication is that video interleaving methods may face analogous error-propagation risks whenever key-video clips, highlighted segments, or localized crops become a single critical bottleneck (Xu et al., 28 Sep 2025).

There are also concrete structural limits. CoTasks requires object-level annotations, its later stages depend on earlier ones, and the reported results are from prompting rather than direct training on the CoTask formulation. VTI-CoT depends on LLM-generated interval descriptions and heuristic temporal segmentation based on CLIP similarity drops, so annotation errors and segmentation errors can propagate into supervision. Video-CoT’s benchmark results show that even GPT-4o and Gemini-1.5-pro remain far from saturated performance, especially on STVG. These findings collectively suggest that explicit reasoning traces do not eliminate the core difficulty of spatiotemporal grounding; they reorganize it into more interpretable subproblems (Wang et al., 18 Jul 2025, Zhang et al., 4 Jun 2026, Zhang et al., 10 Jun 2025).

From a theoretical perspective, the likely importance of such supervision is supported by work on CoT information. Although that theory is not specific to video, it argues that CoT supervision can improve sample complexity because the learner observes intermediate reasoning traces as well as final outputs. In the realizable setting, the sample complexity can scale as $V$9 rather than the standard $Q$0, while the agnostic setting introduces a caveat: CoT supervision can be harmful if the reasoning signal is misaligned with the end-to-end objective. This provides a useful lens for VTS-CoT-style systems, which are typically motivated by richer supervision but are also exposed to richer forms of annotation noise and trajectory mismatch (Altabaa et al., 21 May 2025).

The present state of VTS-CoT is therefore best understood as a transition in video reasoning methodology rather than as a settled recipe. The field is moving from text-only rationales toward temporally and spatially grounded reasoning, from one-shot video encoding toward repeated visual consultation, and from answer-only supervision toward structured intermediate traces. The strongest empirical gains appear when the task genuinely requires temporal localization, object-centric grounding, or long-form causal reasoning; the strongest unresolved issues concern annotation cost, training efficiency, robustness of intermediate evidence, and the still modest absolute performance on the hardest spatiotemporal grounding tasks (Zhang et al., 16 Oct 2025, Zhang et al., 10 Jun 2025, Zhang et al., 4 Jun 2026).