ST-CoT: Spatiotemporal Chain of Thought

• ST-CoT is a structured reasoning framework that explicitly grounds intermediate steps in temporal intervals, spatial locations, and evolving evidence.
• It integrates multi-stage reasoning across domains—video understanding, surgical analysis, and urban simulation—using both textual chains and tool-augmented modules.
• Empirical evaluations show that ST-CoT supervision significantly boosts performance by improving localization accuracy and decision-making in complex spatiotemporal tasks.

Spatiotemporal Chain of Thought (ST-CoT) denotes a structured, multi-step reasoning process in which intermediate inferences are explicitly grounded in temporal intervals, spatial locations, spatial relationships, trajectories, and evolving state, rather than only in high-level semantics. In recent arXiv work, the term is operationalized most directly for video understanding as textual reasoning chains $R=\{r_1,\dots,r_n\}$ tied to timestamps, boxes, relations, and interval-specific descriptions, and used both as supervision and as analysis artifacts (Zhang et al., 10 Jun 2025). Closely related formulations extend the same idea to surgical video reasoning through progressive global-to-local questioning with separate Knowledge and Clue fields (Wang et al., 22 Apr 2026), and to tool-augmented urban behavior simulation through a five-stage cognitive process coupled to temporal, spatial, environmental, memory, social, and evaluation services (Zhang et al., 12 Jun 2025). Taken together, these works suggest that ST-CoT is not a single canonical algorithm, but a family of reasoning designs that force models to localize, track, compare, and decide in a temporally and spatially grounded manner.

1. Conceptual scope

ST-CoT emerged from the observation that large-scale vision-LLMs and video LLMs often perform adequately on coarse video question answering or captioning while remaining weak at fine-grained spatiotemporal reasoning. The motivating failure modes include hallucinated or mislocalized events and objects, difficulty with long sequences, and inability to execute step-by-step grounding over event boundaries, object positions, trajectories, and relations (Zhang et al., 10 Jun 2025). In this framing, a chain of thought for video is not merely an explanatory text; it is a reasoning sequence that explicitly refers to temporal intervals, object locations, spatial relationships, and intermediate decisions such as first localizing, then grounding, then describing.

A related formulation appears in surgical video analysis, where ST-CoT is defined through clinically meaningful reasoning that moves from whole-video understanding to clip-level analysis and finally to frame- or patch-level localization. Here, the crucial distinction from frame-level VQA is that the reasoning chain must track how evidence evolves over time, while the Clue field always refers to temporal windows and spatial ROIs (Wang et al., 22 Apr 2026). This makes localization part of the reasoning process itself rather than a post hoc explanation.

A broader interpretation appears in mechanistic work on CoT that studies where in the network and when in generation explicit reasoning changes computation. That paper does not define a multimodal benchmark, but it presents a prospective ST-CoT view in which the temporal dimension is the sequence of generated tokens and the spatial dimension is the organization of layers and neurons (Yang et al., 28 Jul 2025). This suggests that ST-CoT can be understood both externally, as grounded reasoning over space and time in data, and internally, as structured computation distributed across token phases and network depth.

2. Formal structures and representations

Recent work operationalizes ST-CoT through several distinct but compatible schemas.

Work	Core structure	Grounding target
Video-CoT	$\{V,Q\}\rightarrow\{R,A\}$	intervals, boxes, relations, trajectories, descriptions
SurgCoT	$$Q \rightarrow O \rightarrow K \rightarrow C \rightarrow A$$	whole video, clip, frame/patch evidence
MCP-enhanced CoT	five-stage cognitive framework + six MCP tool categories	activity, route, schedule, context, memory

In Video-CoT, the basic supervised object is explicitly defined as

$$\text{Input: } \{V, Q\} \rightarrow \text{Output: } \{R, A\},$$

where $V \in \mathbb{R}^{T \times H \times W \times C}$ is the video, $Q$ is the question or instruction, $R=\{r_1,\dots,r_n\}$ is the reasoning chain, and $A$ is the final answer (Zhang et al., 10 Jun 2025). The reasoning chain is therefore a sequence of intermediate textual steps that the model must generate before, and condition upon, the final answer. The task design induces several ST-CoT subtypes: temporal CoT for interval reasoning, spatial CoT for localization and relation judgment, spatio-temporal CoT for trajectory reasoning, and descriptive CoT for interval-specific captioning.

SurgCoT formalizes a hierarchical chain in which each answer becomes an immutable condition for the next stage: $$\begin{aligned} &\underbrace{(Q1,\ O1,\ K1,\ C1)}_{\text{Global Video Comprehension}} \Rightarrow A1 \ &\quad\to\ \underbrace{(Q2,\ O2,\ K2,\ C2,\ A1)}_{\text{Conditioned Clip Analysis}} \Rightarrow A2 \ &\quad\quad\to\ \underbrace{(Q3,\ O3,\ K3,\ C3,\ A2)}_{\text{Fine-grained Frame Localization}} \Rightarrow A3. \end{aligned}$$ Its five-tuple structure separates hypothesis space through Option, background priors through Knowledge, case-specific spatiotemporal evidence through Clue, and the adjudicated decision through Answer (Wang et al., 22 Apr 2026). The distinction between Knowledge and Clue is central: Knowledge provides domain priors independent of the particular video, whereas Clue provides temporally and spatially localized evidence.

In urban simulation, the same general logic is instantiated through a five-stage cognitive framework: Situational Awareness and Problem Definition, Constraint Identification and Goal Clarification, Option Generation and Preliminary Screening, Multi-factor Evaluation and Comparison, and Decision Formation and Consequence Prediction (Zhang et al., 12 Jun 2025). The reasoning chain is not represented as a single textual artifact alone; it is interleaved with Model Context Protocol interactions over six specialized tool categories: temporal management, spatial navigation, environmental perception, personal memory, social collaboration, and experience evaluation. This suggests a broader ST-CoT pattern in which reasoning steps are externally grounded by structured tool calls rather than only by generated prose.

3. Benchmarks and task design

Video-CoT provides a large-scale benchmark for fine-grained spatiotemporal understanding with 192,000 fine-grained spatiotemporal QA pairs and 23,000 high-quality CoT-annotated samples. It is organized into 6 tasks under 3 dimensions: Spatio-Temporal Localization and Captioning with TVL and VC; Spatio-Temporal Grounding with SVG and STVG; and Spatio-Temporal Reference with SRR and TVR (Zhang et al., 10 Jun 2025). The dataset mixes short (0–40s) and long (>160s) videos, and the benchmark contains 4,500 video QA pairs with 750 per task, disjoint from training data. Evaluation is task-specific: TVL uses tIoU, VC and TVR use MENTOR, SRR uses Exact Match (EM), SVG uses sIoU, and STVG uses both tIoU and sIoU. CoT rationales are generated with Qwen2.5-VL-72B-Instruct, filtered by answer quality thresholds based on tIoU, sIoU, EM, MENTOR, and then screened by manual expert review.

SurgCoT defines a domain-specific ST-CoT benchmark for surgical videos with 2,841 surgical videos from 7 specialties and 35 procedures, retained from 8,917 initial cases after filtering for completeness, validity, and narration (Wang et al., 22 Apr 2026). It evaluates five reasoning dimensions: Causal Action Ordering (CAO), Cue-Action Alignment (CAA), Affordance Mapping (AM), Micro-Transition Localization (MTL), and Anomaly Onset Tracking (AOT). The benchmark includes 19,345 main questions and 59,177 sub-questions, generated through ontology-driven design and clinician oversight. Its annotations integrate hierarchical cue fusion, millisecond-precise ASR alignment, tissue boxes via YOLOv10, tool segmentation via SAM2, and cross-frame tracking via ByteTrack. Evaluation is answer-based accuracy under three settings: Baseline (BL) without Knowledge or Clue, Knowledge-Enhanced (KE) with Knowledge, and Full-Context (FC) with both Knowledge and Clue.

A noteworthy commonality is that neither benchmark defines an explicit metric for textual CoT quality. In Video-CoT, evaluation remains answer-centric even though CoT annotations support CoT-style training and analysis (Zhang et al., 10 Jun 2025). SurgCoT likewise encodes CoT structurally through K/C and the progressive Q1–Q3 chain, but still measures accuracy on answers rather than on the faithfulness of the reasoning text (Wang et al., 22 Apr 2026). This design choice is important for interpreting reported gains: improved final answers do not, by themselves, prove that the intermediate reasoning steps are correct.

4. Supervision, prompting, and tool-augmented reasoning

Video-CoT introduces two explicit supervised fine-tuning regimes on Qwen2.5-VL-3B. Video-Ans-SFT uses direct answer supervision with

$$\text{Input: } \{V, Q\} \rightarrow \text{Output: } A,$$

whereas Video-CoT-SFT uses joint reasoning-and-answer supervision: $\{V,Q\}\rightarrow\{R,A\}$0 Its loss is written as

$\{V,Q\}\rightarrow\{R,A\}$1

so the model is trained first to generate the reasoning chain and then to generate the answer conditioned on that reasoning (Zhang et al., 10 Jun 2025). The paper further mentions a curriculum learning strategy that begins with simpler 2-step reasoning and gradually moves to more complex 5-step chains.

SurgCoT does not propose a new architecture, but it advances ST-CoT through benchmark- and prompt-level scaffolding. The central techniques are three-stage progressive reasoning (Q1→Q2→Q3), a context-carry mechanism in which each $\{V,Q\}\rightarrow\{R,A\}$2 is passed to stage $\{V,Q\}\rightarrow\{R,A\}$3, Knowledge + Clue prompting, and ontology-driven task design that constrains distractors and disambiguates closely related phenomena such as glare versus bleeding (Wang et al., 22 Apr 2026). The resulting structure enforces reason-then-decide behavior under progressively narrower spatiotemporal scope.

The MCP-enhanced framework extends ST-CoT beyond evaluation into tool-grounded planning. Its overall workflow includes persona configuration and context initialization, daily activity structure planning, activity detail specification, travel route optimization, state updating and dynamic adjustment, and trajectory validation and refinement (Zhang et al., 12 Jun 2025). MCP messages are structured as Query, Response, Action, and Feedback, allowing the LLM to request temporal feasibility checks, POI search, route planning, environmental state, personal memory retrieval, social coordination, and evaluation scores during reasoning. In this design, ST-CoT is not limited to textual decomposition; it becomes a controller over external time-, space-, and context-aware services.

5. Empirical performance and failure modes

On the Video-CoT benchmark, the strongest tested closed-source models still remain weak on fine-grained spatiotemporal reasoning. GPT-4o achieves TVL 21.7, VC 26.9, SRR 14.6, TVR 25.0, SVG 15.9, and STVG best sIoU 9.9, tIoU 15.8; Gemini-1.5-Pro reaches TVL 24.2, VC 29.0, SRR 12.8, TVR 27.3, and SVG 20.9 (Zhang et al., 10 Jun 2025). Open-source models are substantially lower, with Qwen2.5-VL-3B at TVL 4.4, VC 4.6, SRR 10.0, TVR 14.8, SVG 10.3, and STVG 8.5/5.7 tIoU/sIoU. Fine-tuning shows a clear ST-CoT effect. For the same 3B model, Ans-SFT raises TVL from 4.4 to 8.3 and SVG from 10.3 to 14.2, but CoT-SFT raises TVL from 4.4 to 19.7, VC from 4.6 to 15.4, SRR from 10.0 to 16.9, SVG from 10.3 to 17.0, and STVG from 8.5/5.7 to 14.1/9.2. The most prominent result is that CoT-SFT almost quadruples TVL performance from 4.4 to 19.7 tIoU.

SurgCoT exhibits a different but related pattern. Under FC, Claude-Sonnet-4.5 achieves the best average at approximately 87.54%, with GPT-5 and Gemini-2.5-Pro close behind (Wang et al., 22 Apr 2026). Models improve from BL → KE → FC, indicating that both domain priors and explicit spatiotemporal clues matter. The gains are especially diagnostic in weaker models: LLaVA-Med-7B gains about 7% on average from BL → KE, while Qwen2.5-VL-7B gains +8.23% and Claude-Sonnet-4.5 +13.44% from KE → FC. However, the sub-question analysis reveals a persistent ST-CoT consistency gap. For GPT-5, KE, the main Q accuracy is 80.54%, but Q1, Q2, and Q3 are only 54.71%, 55.85%, and 47.60%. This shows that a model can often reach the correct final answer while failing to reconstruct the progressive reasoning steps.

In urban simulation, the MCP-enhanced CoT framework reports $\{V,Q\}\rightarrow\{R,A\}$4 from 7.86 to 8.36 across tested base models, with DeepSeek-R1-7B: 8.36, GLM-4-9B: 8.07, LLAMA-2-13B: 8.22, and Bloom-7B: 7.86 (Zhang et al., 12 Jun 2025). The ablation results are particularly informative about what constitutes effective ST-CoT in this setting. Removing Spatial Navigation Tools reduces quality from approximately 8.07 to 1.85, effectively breaking spatiotemporal coherence. Removing Personal Memory reduces quality to 5.45 and reduces behavior clusters from 11 to 5, indicating severe homogenization. Removing Environmental Perception Tools reduces quality to 6.38. The paper also reports that No CoT (direct generation) yields a 42% quality reduction. On efficiency, parallel processing decreases generation time from 1.30 to 0.17 minutes per sample when scaling from 2 to 12 workers.

These results support a narrow but important conclusion: explicit decomposition of temporal and spatial reasoning helps across very different settings, but the gains differ by domain. Video understanding shows large benefits from reasoning supervision; surgical reasoning shows strong dependence on Knowledge and Clue scaffolding; and urban simulation shows that CoT alone is insufficient without external temporal, spatial, and memory tools.

6. Mechanistic interpretations

A mechanistic account of CoT provides a useful, though still prospective, internal view of ST-CoT. The core claim is that CoT may function as a decoding space pruner that narrows the set of plausible continuations through answer templates, while also modulating neuron engagement in a task-dependent manner (Yang et al., 28 Jul 2025). The paper studies six decoder-only transformers—LLaMA 3.1: 8B, 70B, LLaMA 3.2: 3B, and Gemma 2: 2B, 9B, 27B—and analyzes CoT through three phases of information flow: decoding, projection, and activation.

The formal reasoning structure is written as

$\{V,Q\}\rightarrow\{R,A\}$5

where $\{V,Q\}\rightarrow\{R,A\}$6 are input entities, $\{V,Q\}\rightarrow\{R,A\}$7 are operations or predicates, $\{V,Q\}\rightarrow\{R,A\}$8 are intermediate derived entities, and $\{V,Q\}\rightarrow\{R,A\}$9 is the final answer statement such as “So the answer is …” (Yang et al., 28 Jul 2025). Empirically, higher template adherence strongly correlates with higher accuracy. The paper also finds lower entropy at decision points under CoT and higher concentration of probabilities around the answer sequence, supporting the decoding-space-pruning interpretation.

At the activation level, the analysis tracks positive FFN activations across layers and generated tokens. The principal finding is that the largest CoT-versus-standard activation differences appear in the final ~1/3 of layers. CoT reduces late-layer neuron engagement on open-domain tasks such as GSM8K and Bamboogle, but increases late-layer engagement on closed-domain tasks such as AQuA, Sports, and Coin Flip (Yang et al., 28 Jul 2025). The authors interpret this as pruning in open-domain settings and amplification in closed-domain settings.

For ST-CoT, the plausible implication is that explicit temporal phase structure in reasoning text—setup, reasoning steps, answer statement—interacts with spatially localized computation inside the model. This does not yet amount to a circuit-level theory of spatiotemporal multimodal reasoning, but it does provide a vocabulary for analyzing where and when structured reasoning exerts its strongest effect.

7. Limitations, misconceptions, and open directions

A common misconception is that the presence of CoT annotations or scaffolding automatically implies faithful reasoning. The current benchmarks do not support that conclusion. Video-CoT evaluates answers rather than the correctness of reasoning steps (Zhang et al., 10 Jun 2025), and SurgCoT shows explicitly that models can score substantially higher on final answers than on intermediate Q1–Q3 reasoning steps (Wang et al., 22 Apr 2026). ST-CoT, as currently operationalized, is therefore best regarded as a method for structuring supervision and evaluation, not as proof that a model’s internal reasoning is correct.

Several limitations recur across the literature. Video-CoT inherits domain bias from source datasets, uses CoT annotations that are model-generated then filtered, and remains purely textual rather than symbolic or graph-based (Zhang et al., 10 Jun 2025). SurgCoT is domain-specific to surgery, requires expensive three-stage Q–O–K–C–A annotation with dual-pass human-in-the-loop validation, relies on specific pipelines such as YOLOv10, SAM2, and ByteTrack, and still lacks an explicit metric for textual CoT fidelity (Wang et al., 22 Apr 2026). The mechanistic study is limited by aggregate activation counts, correlation rather than causal intervention, and short-text reasoning tasks rather than long-context or multimodal settings (Yang et al., 28 Jul 2025). The urban simulation framework is tested in a single district and one typical workday, and its evaluation emphasizes aggregate distribution matching, which the authors note may miss microscopic logical issues (Zhang et al., 12 Jun 2025).

The future directions are correspondingly concrete. Video-CoT points toward verified and faithful ST-CoT, self-consistency and ensemble ST-CoT, cross-modal ST-CoT, interactive ST-CoT agents, and hybrid reasoning representations that combine textual CoT with symbolic logic or spatial graphs (Zhang et al., 10 Jun 2025). SurgCoT suggests training and inference schemes that exploit Q–O–K–C–A supervision directly, along with new metrics for fidelity, temporal grounding, and alignment with Knowledge and Clue (Wang et al., 22 Apr 2026). The mechanistic work motivates per-token activation trajectories, head-level and neuron-level analysis, and causal manipulations of activation profiles (Yang et al., 28 Jul 2025). The MCP-enhanced framework points toward real-time adaptation, multimodal fusion, richer social interaction, and application-oriented deployment in smart city planning and transportation forecasting (Zhang et al., 12 Jun 2025).

Taken together, these directions indicate that ST-CoT is evolving along three axes at once: richer external grounding in time and space, stronger structural supervision of intermediate reasoning, and more precise analysis of how reasoning unfolds across tokens, layers, tools, and decision stages.