---
title: Visual Chain-of-Thought Insights
url: https://www.emergentmind.com/topics/visual-chain-of-thought
type: topic
---

# Visual Chain-of-Thought Insights

Visual Chain-of-Thought refers to a family of multimodal reasoning paradigms in which intermediate reasoning is represented not only as text but also as visually grounded states, regions, images, trajectories, or other visual artifacts that can be inspected, reused, and sometimes generated by the model itself. Across recent work, the common departure from standard text-only Chain-of-Thought is that the model does not reason only over linguistic descriptions of an image or video; instead, it externalizes or conditions on intermediate visual evidence such as bounding boxes, keyframes, optical flow, comics, object-centric crops, edited images, or grounded scene descriptions, and uses those intermediate states to support downstream reasoning or generation [2505.15510].

## 1. Conceptual foundations and definitions

The central idea appears in several distinct but convergent formulations. In video generation, a chain-of-visual-thought is defined as a sparse sequence of keyframes,
\[
chain_{\text{vis}} = [\mathbf{img}_0,\mathbf{img}_1,\dots,\mathbf{img}_{N-1}],
\]
paired with textual thoughts,
\[
chain_{\text{txt}} = [\mathbf{txt}_0,\mathbf{txt}_1,\dots,\mathbf{txt}_{N-1}],
\]
where each pair is a critical intermediate visual state of a scenario [2510.05094]. In image reasoning, visual CoT is defined as a sequence of region selections plus text reasoning, where each step focuses on a specific image region that is causally important for the final answer [2504.18397]. In dataset-centric formulations such as VisReason, the model outputs a sequence of actions \(Y=(a_0,a_1,\dots,a_T)\), with each action containing a textual reasoning snippet and a bounding box predicting where to look next, so that the model literally “thinks by zooming” [2511.17731]. In TVI-CoT, the chain is explicitly interleaved by control tokens such as \(\langle\text{Think}\rangle\), \(\langle\text{Look}\rangle\), and \(\langle\text{Answer}\rangle\), allowing repeated switching between textual reasoning and visual grounding [2606.08464].

Several papers define the contrast with standard textual CoT in closely related terms. Textual CoT is a step-by-step natural-language rationale; visual CoT instead makes the steps themselves visually grounded operations, visual states, or visual outputs [2504.18397]. In the VChain formulation, classical CoT “never commits to concrete visual states or images,” whereas the multimodal model is required to reason about consequences and instantiate them as images at discrete time points [2510.05094]. In TVI-CoT, conventional multimodal CoT is described as “vision-blind reasoning,” because the image is encoded once and subsequent reasoning proceeds in text space only [2606.08464]. In the earlier “Description then Decision” paradigm, the intermediate chain is still textual, but it is explicitly a linguistic chain-of-thought about visual content: the model first describes image information relevant to the question, then reasons over that description to decide among alternatives [2311.09193].

A broader theoretical synthesis is given by “Visual Thoughts,” which argues that both textual-MCoT and interleaved-MCoT work because they inject visual thoughts into the reasoning process, and that the improvements depend on the clarity and conciseness of those visual thoughts rather than on output modality alone [2505.15510]. This suggests that “visual chain-of-thought” is best understood not as one single output format, but as a general mechanism for carrying task-relevant visual information into deeper reasoning.

## 2. Forms of visual thought representation

The literature now spans several distinct representational forms. A first class is region-grounded reasoning. UV-CoT models visual CoT as a chain over bounding boxes and region-conditioned responses, where the model samples multiple candidate regions, evaluates them, and constructs preference pairs over the resulting reasoning trajectories [2504.18397]. VisReason and VisReason-Pro represent each reasoning step by a scene description, a normalized RoI bounding box \([x_1,y_1,x_2,y_2]\in[0,1]^4\), and a rationale explaining why that region is relevant or sufficient [2511.17731]. S-Chain formalizes grounded medical reasoning as a structured four-stage sequence \(Y=(Y_1,Y_2,Y_3,Y_4)\), where \(Y_1\) is ROI localization, \(Y_2\) lesion description, \(Y_3\) lesion grading, and \(Y_4\) the final diagnosis [2510.22728].

A second class is explicit visual-state generation. VChain uses sparse keyframes as “snapshots” of a scenario’s evolution, generated by GPT-4o and gpt-image-1, then uses those keyframes to guide sparse inference-time tuning of a video generator [2510.05094]. GVCoT for image editing generates a visual thought map \(\mathbf{x}_{cot}\) that highlights where editing should occur and then conditions final editing on that thought map, so the intermediate reasoning is itself an image [2603.01893]. Shape-of-Thought generates interleaved textual plans and rendered intermediate object states, so the chain is a visible progressive assembly of an object rather than a hidden latent rationale [2601.21081].

A third class uses higher-level visual sequences. Thinking with Comics externalizes intermediate reasoning into a sequence of comic panels with temporal and causal structure, so each panel is an intermediate reasoning state and the ordered layout becomes a visual chain of thought [2602.02453]. FlowVLA treats optical flow as an intermediate motion representation, forcing the model to first predict \(f_t\) and then predict \(v_{t+1}\), yielding a causal chain \(v_t\rightarrow f_t\rightarrow v_{t+1}\) [2508.18269]. In SSV-CoT, the image is decomposed into question-relevant regions and reasoning unfolds as a sequence of region-selection actions over a structured visual bank \(\mathcal{E}=\{e_1,\dots,e_N,e_{\text{global}}\}\) [2603.26737].

A fourth class is latent or prompt-space visual CoT. “Chain of Thought Prompt Tuning in Vision Language Models” does not produce human-readable intermediate images or boxes, but creates a sequence of prompt embeddings \(t_1^i,t_2^i,\dots,t_N^i\) for each class, combined recursively with dynamic weights \(\lambda_j\), so that reasoning is distributed across prompt steps in latent space [2304.07919]. End-to-End Visual Chain-of-Thought for chart summarization similarly treats visual CoT as an internal latent reasoning process induced by instruction fine-tuning rather than as an explicit output sequence [2502.17589].

The “Visual Thoughts” framework places these formats under a unified taxonomy: Natural Language, Structured Language, Edited Image, and Generative Image [2505.15510]. That taxonomy is useful because it makes explicit that visual chain-of-thought may be expressed as text that caches visual content, as structured symbolic descriptions, or as image-form reasoning artifacts.

## 3. Mechanisms and architectures

The architectural mechanisms vary, but several recurring patterns are visible. One pattern is decomposition into reasoning and execution. VChain is a three-stage inference-time framework: Visual Thought Reasoning, Sparse Inference-Time Tuning, and Video Sampling. A large multimodal model reasons about consequences and key moments; those keyframes are then used to fine-tune Wan via LoRA; finally, the tuned generator samples the full video conditioned on concatenated textual thoughts [2510.05094]. GVCoT also factorizes the task:
\[
\mathbf{x}_{cot}=f_\theta(\mathbf{x}_{src},\mathbf{t}),\qquad
\mathbf{x}_{edit}=f_\theta(\mathbf{x}_{src},\mathbf{t},\mathbf{x}_{cot}),
\]
so the model first generates a spatial cue and then performs the edit [2603.01893]. FlowVLA adopts an analogous decomposition for world modeling:
\[
v_t\rightarrow f_t\rightarrow v_{t+1},
\]
with optical flow as the explicit intermediate motion plan [2508.18269].

A second pattern is iterative, step-conditioned visual access. TVI-CoT encodes the image once into visual tokens \(\mathbf{v}\in\mathbb{R}^{N\times d}\), but when \(\langle\text{Look}\rangle\) is emitted, it computes reasoning-conditioned attention over those tokens:
\[
\alpha_t=\text{softmax}\left(\frac{\mathbf{h}_t\mathbf{W}_q(\mathbf{v}\mathbf{W}_k)^\top}{\sqrt{d}}\right),
\]
then injects the attended visual representation back into the context [2606.08464]. SSV-CoT similarly builds a structured set of salient regions and learns a policy over them, with an explicit STOP action, so visual access becomes sequential and goal-driven rather than static [2603.26737]. DaP-ICoT addresses the same issue from a token-efficiency perspective: Dynamic Visual Thought Integration inserts visual thoughts only when confidence is low, and Precise Visual Thought Guidance uses SAM2 and cross-modal similarity to select object-level, semantically coherent visual thoughts instead of broken patch snippets [2603.21754].

A third pattern is explicit grounding through generated or selected visual entities. UV-CoT treats region selection as part of the CoT trajectory and optimizes it with Score-DPO so that the model prefers chains whose regions lead to better current and next-step answers [2504.18397]. VisReason frames visual CoT as a sequence of actions \(a_t=(r_t,b_{t+1})\), with RoIs serialized into tokens so the model predicts both reasoning and coordinates in one autoregressive stream [2511.17731]. S-Chain couples ROI localization and clinical reasoning so that the chain is not merely explanatory but causally structured: localization precedes description, grading, and diagnosis [2510.22728].

A fourth pattern is explicit use of visual intermediate states as working memory. Shape-of-Thought uses previously generated rendered states as context for future textual plans and future rendered states, creating a multimodal autoregressive trace \(\mathcal{T}=\{(z_1,v_1),\dots,(z_N,v_N)\}\) [2601.21081]. The paper shows that removing visual history sharply lowers trace stability, which supports the claim that visual intermediate states act as a working memory rather than as decoration [2601.21081]. A related idea appears in VChain, where previously generated keyframes are stitched together and shown to GPT-4o for next-frame reasoning, so consequence prediction is conditioned on the whole prior visual trace [2510.05094].

## 4. Training and supervision paradigms

Visual chain-of-thought has been learned under several supervision regimes. One regime is supervised fine-tuning on explicit visual reasoning traces. VisReason provides 489K examples and VisReason-Pro 165K examples, each with multi-round visual CoT and explicit RoIs; Qwen2.5-VL is fine-tuned with standard cross-entropy over the full output sequence of rationales, bounding boxes, and answers [2511.17731]. S-Chain provides 12,000 expert-annotated medical images and over 700k multilingual VQA pairs; its structured visual CoT supervision is also cast as autoregressive sequence prediction with cross-entropy over localization, lesion description, grading, and diagnosis [2510.22728]. Shape-of-Thought supervises both text and images in one sequence and optimizes a sum of next-token cross-entropy and rectified-flow MSE over image latents [2601.21081].

A second regime uses weak or unsupervised signals. UV-CoT eliminates human bounding-box annotations by generating candidate boxes, using an evaluator model to rank the resulting answers, and then optimizing a score-dependent DPO objective over preferred and dispreferred chains [2504.18397]. SSV-CoT learns sequential visual access without region-level annotations; it first bootstraps a pseudo-trajectory from question-conditioned saliency and then refines the visual policy with GRPO using answer correctness, formatting, reasoning length, and visual-budget penalties [2603.26737]. The “Description then Decision” strategy requires no explicit rationale supervision at all; the description phase is induced purely through prompting of pretrained vision-language models [2311.09193].

A third regime is inference-time adaptation. VChain fine-tunes only LoRA adapters of Wan 2.1 T2V per prompt using 3–6 keyframes, each treated as a 1-frame video, and reuses Wan’s original flow-matching objective:
\[
\mathcal{L}_{\text{vchain}}(\theta)=
\mathbb{E}_{x_0,x_1,c,t}\left\|u_\theta(x_t,t,c)-v_t\right\|^2
\]
with Visual Thoughts as the sparse supervision set [2510.05094]. DaP-ICoT also operates entirely at inference time: a confidence score
\[
C_t=\frac{1}{|T_t|}\sum_{i=1}^{|T_t|}\left(\ell_{i,w^{(1)}}-\ell_{i,w^{(2)}}\right)
\]
controls whether a new visual thought is inserted [2603.21754].

A fourth regime combines supervised fine-tuning and reinforcement learning. GVCoT first uses two SFT stages to teach mask-form or zoom-in-form visual thoughts, then uses Flow-GRPO with rewards for format, IoU, CoT-edit consistency, and image quality to jointly improve reasoning traces and edits [2603.01893]. TVI-CoT uses supervised fine-tuning on interleaved traces plus an auxiliary grounding loss,
\[
\mathcal{L}=\mathcal{L}_{\text{SFT}}+\lambda\mathcal{L}_{\text{ground}},
\]
so the model learns not only to emit \(\langle\text{Look}\rangle\) tokens but also to attend to the correct regions when it does so [2606.08464]. S-Chain adds two alignment losses beyond SFT: a margin-based ROI anchoring loss that pushes CoT embeddings toward ROI tokens and away from non-ROI tokens, and a supervised contrastive loss that separates CoT embeddings by disease class [2510.22728].

A fifth regime treats visual CoT as latent rather than explicit. The chart summarization work on V-CoT induces a stepwise visual analysis procedure through carefully designed instructions such as “first identifying the chart type, then describing the axes and scales, and finally highlighting the major trends and data points,” but trains only on final summaries with a standard negative log-likelihood objective [2502.17589]. Prompt-tuning work similarly builds a chain across prompt embeddings and visual biases, but the intermediate reasoning remains latent in the prompt sequence rather than explicit in the output [2304.07919].

## 5. Empirical evidence across tasks

Empirical findings are strong but heterogeneous, and they reveal that visual CoT is most beneficial when the task requires explicit visual state tracking, localized evidence, or repeated visual verification. In video generation, VChain reports large gains on reasoning-related human dimensions: Physics, Commonsense, and Causal Reasoning rise from roughly 32–33% for the T2V baseline and 38–42% for prompt augmentation to 58–62% for the full method, while perceptual quality metrics remain comparable [2510.05094]. Qualitative examples such as bowling collisions and first-person ball catching show that visual thoughts plus sparse tuning are necessary; text-only prompt augmentation or interpolation without adaptation is insufficient [2510.05094].

In image reasoning, UV-CoT reports that visual CoT learned from preference optimization can match or surpass fully supervised visual CoT with fewer data. On six main datasets, UV-CoT with 0% labels reaches an average of about 0.473, near the 0.482 of fully supervised Visual-CoT-7B, while UV-CoT with 10% labels reaches about 0.494 and becomes best overall [2504.18397]. Zero-shot and high-resolution evaluations further show advantages in OCR-heavy and spatial tasks [2504.18397]. VisReason fine-tuning raises Qwen2.5-VL-7B from 0.770 to 0.791 overall on the Visual-CoT benchmark, and further to 0.807 with VisReason-Pro; gains are especially large in fine-grained recognition and spatial relation reasoning, while IoU@0.5 and IoU@0.75 on RoI localization also improve [2511.17731].

In multimodal reasoning benchmarks, TVI-CoT improves its Qwen3-VL-8B backbone by +6.1% on MMMU, +3.8% on MathVerse, +3.4% on MathVista, and +3.4% on ScienceQA, with especially large gains in geometry, figure QA, and vision-dominant categories [2606.08464]. SSV-CoT raises Qwen2-VL-7B from 60.9 to 72.2 on MathVista and from 16.3 to 23.5 on MathVision, supporting the claim that sequential visual access matters most when visual evidence must be queried in multiple stages [2603.26737]. DaP-ICoT simultaneously improves reasoning performance and cuts token consumption: on M\(^3\)CoT with Qwen2-VL-7B, total tokens drop from 1,146 for ICoT to 314 for DaP-ICoT, a 72.6% reduction, while accuracy rises from the best baseline 48.7% to 57.2% [2603.21754].

In medical reasoning, expert-annotated visual CoT matters more than synthetic CoT. On S-Chain, ExGra-Med improves from 49.4/46.9 accuracy/F1 in the Q4-only setting to 60.4/59.6 with SV-CoT, and to 64.8/62.6 when combined with MedRAG. ROI localization quality also rises sharply: for ExGra-Med, mIoU increases from 4.3 with GPT-synthetic CoT to 25.3 with expert SV-CoT [2510.22728]. The component-wise ablation that injects gold intermediate steps shows that if grounded intermediate reasoning is correct, the final diagnostic mapping becomes nearly trivial [2510.22728].

In generation tasks beyond video, Shape-of-Thought reports 88.4% on component numeracy and 84.8% on structural topology, substantially above direct and text-only CoT baselines, while trace stability reaches 91.3 compared with 32.7 for text-only CoT [2601.21081]. FlowVLA shows that forcing a model to “think in motion” by predicting optical flow first leads to better downstream robotics control: on LIBERO, FlowVLA reaches an average success rate of 88.1 compared with 84.0 for the UniVLA re-implementation and 81.1 for CoT-VLA [2508.18269]. In image editing, Bagel-GVCoT achieves SC\(_g\)=8.87, PQ\(_g\)=8.76, and O\(_g\)=8.53 on SREdit-Bench, outperforming text-CoT and tool-based visual CoT baselines in spatially sophisticated scenes [2603.01893].

Not all evidence favors longer or more elaborate visual chains. A controlled maze study comparing Language CoT, Grounding CoT, and Visual CoT finds that long or visual CoT mainly accelerates convergence but does not raise the final performance ceiling; concise CoT containing only essential grounding steps generalizes best across maze sizes, a result described as a “short is long” effect [2511.22586]. This suggests that visual chain-of-thought is not universally synonymous with longer reasoning traces; in some settings, minimal grounded traces may be the most generalizable.

## 6. Limitations, controversies, and future directions

The literature identifies several recurring limitations. One is data and annotation cost. VisReason and S-Chain both rely on extensive annotation pipelines; S-Chain in particular estimates about 600 hours of physician time and about 1500 hours of professional translation for its multilingual dataset [2510.22728]. UV-CoT avoids box annotation but depends on evaluator quality and still requires large-scale preference data generation [2504.18397]. GVCoT addresses edit-region supervision through a 1.8M-sample dataset, but its masks and boxes are derived through a multi-stage pipeline involving Qwen2.5-VL, SAM2, and BiRefNet, so supervision quality still depends on upstream models [2603.01893].

A second limitation is cost and reproducibility of visual-thought generation. VChain relies on proprietary GPT-4o and gpt-image-1, with roughly two GPT calls per keyframe and increasing API cost as keyframe count grows; its keyframe editing process can accumulate oversaturation and oversmoothing artifacts [2510.05094]. Thinking with Comics depends on Gemini-3 Pro Image for high-quality multi-panel generation and is limited to mostly short, linear narratives [2602.02453]. More generally, “Visual Thoughts” observes that image-form thoughts can be powerful but often require many image tokens or costly generation steps [2505.15510].

A third limitation is faithfulness. The original “Description then Decision” work explicitly notes that prompt-induced CoT in vision-language models has no formal guarantee of faithfulness [2311.09193]. S-Chain is motivated by the same concern in medicine: synthetic CoTs may be plausible but unfaithful to visual evidence, which is particularly problematic in high-stakes domains [2510.22728]. The broader “Visual Thoughts” study reinforces this point by showing that clarity and conciseness of the representation, not mere fidelity to the original image, determine whether visual thoughts actually help reasoning [2505.15510].

A fourth limitation is scaling to long, complex, or dynamic settings. VChain notes that long videos or complex multi-agent scenes may require more keyframes, making reasoning harder and more expensive [2510.05094]. DaP-ICoT depends on segmentation quality and uses a heuristic confidence threshold \(\tau\), leaving room for learned gating policies [2603.21754]. SSV-CoT is currently image-only and depends on saliency quality; extending structured sequential cognition to video would require temporal region structures and temporal selection policies [2603.26737]. TVI-CoT still incurs additional compute with each \(\langle\text{Look}\rangle\) and remains limited by the precision of region-level grounding [2606.08464].

The main future directions are correspondingly clear. Several papers point toward richer structured states beyond pixels: explicit object-centric or physical state representations in VChain [2510.05094], more accurate box generation or detector integration in UV-CoT [2504.18397], true 3D or depth-aware visual CoT beyond pseudo-depth in VisReason-Pro [2511.17731], and multi-view or 3D asset generation from visual traces in Shape-of-Thought [2601.21081]. Others emphasize multimodal planning loops: interactive agents, robotics, AR/VR, and tool use are explicit targets in VChain, FlowVLA, and VisReason [2510.05094; 2508.18269; 2511.17731]. TVI-CoT and SSV-CoT suggest a more unified long-term direction: reasoning systems in which “thinking” and “looking” are both explicit, adaptive, and repeatedly interleaved rather than separated into a single visual encoding phase and a purely textual deliberation phase [2606.08464; 2603.26737].

A plausible synthesis is that visual chain-of-thought is becoming less a single technique than a design principle. The shared principle is to externalize, query, or optimize intermediate representations in the target visual modality—or in a representation that faithfully compresses that modality—so that reasoning can proceed through inspectable visual states rather than through text alone. Across current work, the main open question is no longer whether visual CoT can help, but which visual intermediates are most faithful, concise, efficient, and generalizable for a given class of tasks [2505.15510].

Source: https://www.emergentmind.com/topics/visual-chain-of-thought