Visual Chain-of-Thought Insights

• Visual Chain-of-Thought is a multimodal reasoning paradigm that combines text with explicit visual states like keyframes, bounding boxes, and region selections.
• It employs diverse visual representations—from region-grounded outputs to latent prompt embeddings—to guide reasoning and enhance downstream applications.
• Empirical studies reveal that incorporating visual intermediates significantly boosts performance in tasks such as video generation, image reasoning, and robotics control.

Visual Chain-of-Thought refers to a family of multimodal reasoning paradigms in which intermediate reasoning is represented not only as text but also as visually grounded states, regions, images, trajectories, or other visual artifacts that can be inspected, reused, and sometimes generated by the model itself. Across recent work, the common departure from standard text-only Chain-of-Thought is that the model does not reason only over linguistic descriptions of an image or video; instead, it externalizes or conditions on intermediate visual evidence such as bounding boxes, keyframes, optical flow, comics, object-centric crops, edited images, or grounded scene descriptions, and uses those intermediate states to support downstream reasoning or generation (Cheng et al., 21 May 2025).

1. Conceptual foundations and definitions

The central idea appears in several distinct but convergent formulations. In video generation, a chain-of-visual-thought is defined as a sparse sequence of keyframes,

$$chain_{\text{vis}} = [\mathbf{img}_0,\mathbf{img}_1,\dots,\mathbf{img}_{N-1}],$$

paired with textual thoughts,

$$chain_{\text{txt}} = [\mathbf{txt}_0,\mathbf{txt}_1,\dots,\mathbf{txt}_{N-1}],$$

where each pair is a critical intermediate visual state of a scenario (Huang et al., 6 Oct 2025). In image reasoning, visual CoT is defined as a sequence of region selections plus text reasoning, where each step focuses on a specific image region that is causally important for the final answer (Zhao et al., 25 Apr 2025). In dataset-centric formulations such as VisReason, the model outputs a sequence of actions $Y=(a_0,a_1,\dots,a_T)$, with each action containing a textual reasoning snippet and a bounding box predicting where to look next, so that the model literally “thinks by zooming” (Li et al., 21 Nov 2025). In TVI-CoT, the chain is explicitly interleaved by control tokens such as $\langle\text{Think}\rangle$, $\langle\text{Look}\rangle$, and $\langle\text{Answer}\rangle$, allowing repeated switching between textual reasoning and visual grounding (Hu et al., 7 Jun 2026).

Several papers define the contrast with standard textual CoT in closely related terms. Textual CoT is a step-by-step natural-language rationale; visual CoT instead makes the steps themselves visually grounded operations, visual states, or visual outputs (Zhao et al., 25 Apr 2025). In the VChain formulation, classical CoT “never commits to concrete visual states or images,” whereas the multimodal model is required to reason about consequences and instantiate them as images at discrete time points (Huang et al., 6 Oct 2025). In TVI-CoT, conventional multimodal CoT is described as “vision-blind reasoning,” because the image is encoded once and subsequent reasoning proceeds in text space only (Hu et al., 7 Jun 2026). In the earlier “Description then Decision” paradigm, the intermediate chain is still textual, but it is explicitly a linguistic chain-of-thought about visual content: the model first describes image information relevant to the question, then reasons over that description to decide among alternatives (Wu et al., 2023).

A broader theoretical synthesis is given by “Visual Thoughts,” which argues that both textual-MCoT and interleaved-MCoT work because they inject visual thoughts into the reasoning process, and that the improvements depend on the clarity and conciseness of those visual thoughts rather than on output modality alone (Cheng et al., 21 May 2025). This suggests that “visual chain-of-thought” is best understood not as one single output format, but as a general mechanism for carrying task-relevant visual information into deeper reasoning.

2. Forms of visual thought representation

The literature now spans several distinct representational forms. A first class is region-grounded reasoning. UV-CoT models visual CoT as a chain over bounding boxes and region-conditioned responses, where the model samples multiple candidate regions, evaluates them, and constructs preference pairs over the resulting reasoning trajectories (Zhao et al., 25 Apr 2025). VisReason and VisReason-Pro represent each reasoning step by a scene description, a normalized RoI bounding box $[x_1,y_1,x_2,y_2]\in[0,1]^4$, and a rationale explaining why that region is relevant or sufficient (Li et al., 21 Nov 2025). S-Chain formalizes grounded medical reasoning as a structured four-stage sequence $Y=(Y_1,Y_2,Y_3,Y_4)$, where $Y_1$ is ROI localization, $Y_2$ lesion description, $$chain_{\text{txt}} = [\mathbf{txt}_0,\mathbf{txt}_1,\dots,\mathbf{txt}_{N-1}],$$0 lesion grading, and $$chain_{\text{txt}} = [\mathbf{txt}_0,\mathbf{txt}_1,\dots,\mathbf{txt}_{N-1}],$$1 the final diagnosis (Le-Duc et al., 26 Oct 2025).

A second class is explicit visual-state generation. VChain uses sparse keyframes as “snapshots” of a scenario’s evolution, generated by GPT-4o and gpt-image-1, then uses those keyframes to guide sparse inference-time tuning of a video generator (Huang et al., 6 Oct 2025). GVCoT for image editing generates a visual thought map $$chain_{\text{txt}} = [\mathbf{txt}_0,\mathbf{txt}_1,\dots,\mathbf{txt}_{N-1}],$$2 that highlights where editing should occur and then conditions final editing on that thought map, so the intermediate reasoning is itself an image (Yin et al., 2 Mar 2026). Shape-of-Thought generates interleaved textual plans and rendered intermediate object states, so the chain is a visible progressive assembly of an object rather than a hidden latent rationale (Huo et al., 28 Jan 2026).

A third class uses higher-level visual sequences. Thinking with Comics externalizes intermediate reasoning into a sequence of comic panels with temporal and causal structure, so each panel is an intermediate reasoning state and the ordered layout becomes a visual chain of thought (Chen et al., 2 Feb 2026). FlowVLA treats optical flow as an intermediate motion representation, forcing the model to first predict $$chain_{\text{txt}} = [\mathbf{txt}_0,\mathbf{txt}_1,\dots,\mathbf{txt}_{N-1}],$$3 and then predict $$chain_{\text{txt}} = [\mathbf{txt}_0,\mathbf{txt}_1,\dots,\mathbf{txt}_{N-1}],$$4, yielding a causal chain $$chain_{\text{txt}} = [\mathbf{txt}_0,\mathbf{txt}_1,\dots,\mathbf{txt}_{N-1}],$$5 (Zhong et al., 25 Aug 2025). In SSV-CoT, the image is decomposed into question-relevant regions and reasoning unfolds as a sequence of region-selection actions over a structured visual bank $$chain_{\text{txt}} = [\mathbf{txt}_0,\mathbf{txt}_1,\dots,\mathbf{txt}_{N-1}],$$6 (Guo et al., 21 Mar 2026).

A fourth class is latent or prompt-space visual CoT. “Chain of Thought Prompt Tuning in Vision LLMs” does not produce human-readable intermediate images or boxes, but creates a sequence of prompt embeddings $$chain_{\text{txt}} = [\mathbf{txt}_0,\mathbf{txt}_1,\dots,\mathbf{txt}_{N-1}],$$7 for each class, combined recursively with dynamic weights $$chain_{\text{txt}} = [\mathbf{txt}_0,\mathbf{txt}_1,\dots,\mathbf{txt}_{N-1}],$$8, so that reasoning is distributed across prompt steps in latent space (Ge et al., 2023). End-to-End Visual Chain-of-Thought for chart summarization similarly treats visual CoT as an internal latent reasoning process induced by instruction fine-tuning rather than as an explicit output sequence (Choi et al., 24 Feb 2025).

The “Visual Thoughts” framework places these formats under a unified taxonomy: Natural Language, Structured Language, Edited Image, and Generative Image (Cheng et al., 21 May 2025). That taxonomy is useful because it makes explicit that visual chain-of-thought may be expressed as text that caches visual content, as structured symbolic descriptions, or as image-form reasoning artifacts.

3. Mechanisms and architectures

The architectural mechanisms vary, but several recurring patterns are visible. One pattern is decomposition into reasoning and execution. VChain is a three-stage inference-time framework: Visual Thought Reasoning, Sparse Inference-Time Tuning, and Video Sampling. A large multimodal model reasons about consequences and key moments; those keyframes are then used to fine-tune Wan via LoRA; finally, the tuned generator samples the full video conditioned on concatenated textual thoughts (Huang et al., 6 Oct 2025). GVCoT also factorizes the task: $$chain_{\text{txt}} = [\mathbf{txt}_0,\mathbf{txt}_1,\dots,\mathbf{txt}_{N-1}],$$9 so the model first generates a spatial cue and then performs the edit (Yin et al., 2 Mar 2026). FlowVLA adopts an analogous decomposition for world modeling: $Y=(a_0,a_1,\dots,a_T)$0 with optical flow as the explicit intermediate motion plan (Zhong et al., 25 Aug 2025).

A second pattern is iterative, step-conditioned visual access. TVI-CoT encodes the image once into visual tokens $Y=(a_0,a_1,\dots,a_T)$1, but when $Y=(a_0,a_1,\dots,a_T)$2 is emitted, it computes reasoning-conditioned attention over those tokens: $Y=(a_0,a_1,\dots,a_T)$3 then injects the attended visual representation back into the context (Hu et al., 7 Jun 2026). SSV-CoT similarly builds a structured set of salient regions and learns a policy over them, with an explicit STOP action, so visual access becomes sequential and goal-driven rather than static (Guo et al., 21 Mar 2026). DaP-ICoT addresses the same issue from a token-efficiency perspective: Dynamic Visual Thought Integration inserts visual thoughts only when confidence is low, and Precise Visual Thought Guidance uses SAM2 and cross-modal similarity to select object-level, semantically coherent visual thoughts instead of broken patch snippets (Liu et al., 23 Mar 2026).

A third pattern is explicit grounding through generated or selected visual entities. UV-CoT treats region selection as part of the CoT trajectory and optimizes it with Score-DPO so that the model prefers chains whose regions lead to better current and next-step answers (Zhao et al., 25 Apr 2025). VisReason frames visual CoT as a sequence of actions $Y=(a_0,a_1,\dots,a_T)$4, with RoIs serialized into tokens so the model predicts both reasoning and coordinates in one autoregressive stream (Li et al., 21 Nov 2025). S-Chain couples ROI localization and clinical reasoning so that the chain is not merely explanatory but causally structured: localization precedes description, grading, and diagnosis (Le-Duc et al., 26 Oct 2025).

A fourth pattern is explicit use of visual intermediate states as working memory. Shape-of-Thought uses previously generated rendered states as context for future textual plans and future rendered states, creating a multimodal autoregressive trace $Y=(a_0,a_1,\dots,a_T)$5 (Huo et al., 28 Jan 2026). The paper shows that removing visual history sharply lowers trace stability, which supports the claim that visual intermediate states act as a working memory rather than as decoration (Huo et al., 28 Jan 2026). A related idea appears in VChain, where previously generated keyframes are stitched together and shown to GPT-4o for next-frame reasoning, so consequence prediction is conditioned on the whole prior visual trace (Huang et al., 6 Oct 2025).

4. Training and supervision paradigms

Visual chain-of-thought has been learned under several supervision regimes. One regime is supervised fine-tuning on explicit visual reasoning traces. VisReason provides 489K examples and VisReason-Pro 165K examples, each with multi-round visual CoT and explicit RoIs; Qwen2.5-VL is fine-tuned with standard cross-entropy over the full output sequence of rationales, bounding boxes, and answers (Li et al., 21 Nov 2025). S-Chain provides 12,000 expert-annotated medical images and over 700k multilingual VQA pairs; its structured visual CoT supervision is also cast as autoregressive sequence prediction with cross-entropy over localization, lesion description, grading, and diagnosis (Le-Duc et al., 26 Oct 2025). Shape-of-Thought supervises both text and images in one sequence and optimizes a sum of next-token cross-entropy and rectified-flow MSE over image latents (Huo et al., 28 Jan 2026).

A second regime uses weak or unsupervised signals. UV-CoT eliminates human bounding-box annotations by generating candidate boxes, using an evaluator model to rank the resulting answers, and then optimizing a score-dependent DPO objective over preferred and dispreferred chains (Zhao et al., 25 Apr 2025). SSV-CoT learns sequential visual access without region-level annotations; it first bootstraps a pseudo-trajectory from question-conditioned saliency and then refines the visual policy with GRPO using answer correctness, formatting, reasoning length, and visual-budget penalties (Guo et al., 21 Mar 2026). The “Description then Decision” strategy requires no explicit rationale supervision at all; the description phase is induced purely through prompting of pretrained vision-LLMs (Wu et al., 2023).

A third regime is inference-time adaptation. VChain fine-tunes only LoRA adapters of Wan 2.1 T2V per prompt using 3–6 keyframes, each treated as a 1-frame video, and reuses Wan’s original flow-matching objective: $Y=(a_0,a_1,\dots,a_T)$6 with Visual Thoughts as the sparse supervision set (Huang et al., 6 Oct 2025). DaP-ICoT also operates entirely at inference time: a confidence score

$Y=(a_0,a_1,\dots,a_T)$7

controls whether a new visual thought is inserted (Liu et al., 23 Mar 2026).

A fourth regime combines supervised fine-tuning and reinforcement learning. GVCoT first uses two SFT stages to teach mask-form or zoom-in-form visual thoughts, then uses Flow-GRPO with rewards for format, IoU, CoT-edit consistency, and image quality to jointly improve reasoning traces and edits (Yin et al., 2 Mar 2026). TVI-CoT uses supervised fine-tuning on interleaved traces plus an auxiliary grounding loss,

$Y=(a_0,a_1,\dots,a_T)$8

so the model learns not only to emit $Y=(a_0,a_1,\dots,a_T)$9 tokens but also to attend to the correct regions when it does so (Hu et al., 7 Jun 2026). S-Chain adds two alignment losses beyond SFT: a margin-based ROI anchoring loss that pushes CoT embeddings toward ROI tokens and away from non-ROI tokens, and a supervised contrastive loss that separates CoT embeddings by disease class (Le-Duc et al., 26 Oct 2025).

A fifth regime treats visual CoT as latent rather than explicit. The chart summarization work on V-CoT induces a stepwise visual analysis procedure through carefully designed instructions such as “first identifying the chart type, then describing the axes and scales, and finally highlighting the major trends and data points,” but trains only on final summaries with a standard negative log-likelihood objective (Choi et al., 24 Feb 2025). Prompt-tuning work similarly builds a chain across prompt embeddings and visual biases, but the intermediate reasoning remains latent in the prompt sequence rather than explicit in the output (Ge et al., 2023).

5. Empirical evidence across tasks

Empirical findings are strong but heterogeneous, and they reveal that visual CoT is most beneficial when the task requires explicit visual state tracking, localized evidence, or repeated visual verification. In video generation, VChain reports large gains on reasoning-related human dimensions: Physics, Commonsense, and Causal Reasoning rise from roughly 32–33% for the T2V baseline and 38–42% for prompt augmentation to 58–62% for the full method, while perceptual quality metrics remain comparable (Huang et al., 6 Oct 2025). Qualitative examples such as bowling collisions and first-person ball catching show that visual thoughts plus sparse tuning are necessary; text-only prompt augmentation or interpolation without adaptation is insufficient (Huang et al., 6 Oct 2025).

In image reasoning, UV-CoT reports that visual CoT learned from preference optimization can match or surpass fully supervised visual CoT with fewer data. On six main datasets, UV-CoT with 0% labels reaches an average of about 0.473, near the 0.482 of fully supervised Visual-CoT-7B, while UV-CoT with 10% labels reaches about 0.494 and becomes best overall (Zhao et al., 25 Apr 2025). Zero-shot and high-resolution evaluations further show advantages in OCR-heavy and spatial tasks (Zhao et al., 25 Apr 2025). VisReason fine-tuning raises Qwen2.5-VL-7B from 0.770 to 0.791 overall on the Visual-CoT benchmark, and further to 0.807 with VisReason-Pro; gains are especially large in fine-grained recognition and spatial relation reasoning, while [email protected] and [email protected] on RoI localization also improve (Li et al., 21 Nov 2025).

In multimodal reasoning benchmarks, TVI-CoT improves its Qwen3-VL-8B backbone by +6.1% on MMMU, +3.8% on MathVerse, +3.4% on MathVista, and +3.4% on ScienceQA, with especially large gains in geometry, figure QA, and vision-dominant categories (Hu et al., 7 Jun 2026). SSV-CoT raises Qwen2-VL-7B from 60.9 to 72.2 on MathVista and from 16.3 to 23.5 on MathVision, supporting the claim that sequential visual access matters most when visual evidence must be queried in multiple stages (Guo et al., 21 Mar 2026). DaP-ICoT simultaneously improves reasoning performance and cuts token consumption: on M$\langle\text{Think}\rangle$0CoT with Qwen2-VL-7B, total tokens drop from 1,146 for ICoT to 314 for DaP-ICoT, a 72.6% reduction, while accuracy rises from the best baseline 48.7% to 57.2% (Liu et al., 23 Mar 2026).

In medical reasoning, expert-annotated visual CoT matters more than synthetic CoT. On S-Chain, ExGra-Med improves from 49.4/46.9 accuracy/F1 in the Q4-only setting to 60.4/59.6 with SV-CoT, and to 64.8/62.6 when combined with MedRAG. ROI localization quality also rises sharply: for ExGra-Med, mIoU increases from 4.3 with GPT-synthetic CoT to 25.3 with expert SV-CoT (Le-Duc et al., 26 Oct 2025). The component-wise ablation that injects gold intermediate steps shows that if grounded intermediate reasoning is correct, the final diagnostic mapping becomes nearly trivial (Le-Duc et al., 26 Oct 2025).

In generation tasks beyond video, Shape-of-Thought reports 88.4% on component numeracy and 84.8% on structural topology, substantially above direct and text-only CoT baselines, while trace stability reaches 91.3 compared with 32.7 for text-only CoT (Huo et al., 28 Jan 2026). FlowVLA shows that forcing a model to “think in motion” by predicting optical flow first leads to better downstream robotics control: on LIBERO, FlowVLA reaches an average success rate of 88.1 compared with 84.0 for the UniVLA re-implementation and 81.1 for CoT-VLA (Zhong et al., 25 Aug 2025). In image editing, Bagel-GVCoT achieves SC$\langle\text{Think}\rangle$1=8.87, PQ$\langle\text{Think}\rangle$2=8.76, and O$\langle\text{Think}\rangle$3=8.53 on SREdit-Bench, outperforming text-CoT and tool-based visual CoT baselines in spatially sophisticated scenes (Yin et al., 2 Mar 2026).

Not all evidence favors longer or more elaborate visual chains. A controlled maze study comparing Language CoT, Grounding CoT, and Visual CoT finds that long or visual CoT mainly accelerates convergence but does not raise the final performance ceiling; concise CoT containing only essential grounding steps generalizes best across maze sizes, a result described as a “short is long” effect (Du et al., 27 Nov 2025). This suggests that visual chain-of-thought is not universally synonymous with longer reasoning traces; in some settings, minimal grounded traces may be the most generalizable.

6. Limitations, controversies, and future directions

The literature identifies several recurring limitations. One is data and annotation cost. VisReason and S-Chain both rely on extensive annotation pipelines; S-Chain in particular estimates about 600 hours of physician time and about 1500 hours of professional translation for its multilingual dataset (Le-Duc et al., 26 Oct 2025). UV-CoT avoids box annotation but depends on evaluator quality and still requires large-scale preference data generation (Zhao et al., 25 Apr 2025). GVCoT addresses edit-region supervision through a 1.8M-sample dataset, but its masks and boxes are derived through a multi-stage pipeline involving Qwen2.5-VL, SAM2, and BiRefNet, so supervision quality still depends on upstream models (Yin et al., 2 Mar 2026).

A second limitation is cost and reproducibility of visual-thought generation. VChain relies on proprietary GPT-4o and gpt-image-1, with roughly two GPT calls per keyframe and increasing API cost as keyframe count grows; its keyframe editing process can accumulate oversaturation and oversmoothing artifacts (Huang et al., 6 Oct 2025). Thinking with Comics depends on Gemini-3 Pro Image for high-quality multi-panel generation and is limited to mostly short, linear narratives (Chen et al., 2 Feb 2026). More generally, “Visual Thoughts” observes that image-form thoughts can be powerful but often require many image tokens or costly generation steps (Cheng et al., 21 May 2025).

A third limitation is faithfulness. The original “Description then Decision” work explicitly notes that prompt-induced CoT in vision-LLMs has no formal guarantee of faithfulness (Wu et al., 2023). S-Chain is motivated by the same concern in medicine: synthetic CoTs may be plausible but unfaithful to visual evidence, which is particularly problematic in high-stakes domains (Le-Duc et al., 26 Oct 2025). The broader “Visual Thoughts” study reinforces this point by showing that clarity and conciseness of the representation, not mere fidelity to the original image, determine whether visual thoughts actually help reasoning (Cheng et al., 21 May 2025).

A fourth limitation is scaling to long, complex, or dynamic settings. VChain notes that long videos or complex multi-agent scenes may require more keyframes, making reasoning harder and more expensive (Huang et al., 6 Oct 2025). DaP-ICoT depends on segmentation quality and uses a heuristic confidence threshold $\langle\text{Think}\rangle$4, leaving room for learned gating policies (Liu et al., 23 Mar 2026). SSV-CoT is currently image-only and depends on saliency quality; extending structured sequential cognition to video would require temporal region structures and temporal selection policies (Guo et al., 21 Mar 2026). TVI-CoT still incurs additional compute with each $\langle\text{Think}\rangle$5 and remains limited by the precision of region-level grounding (Hu et al., 7 Jun 2026).

The main future directions are correspondingly clear. Several papers point toward richer structured states beyond pixels: explicit object-centric or physical state representations in VChain (Huang et al., 6 Oct 2025), more accurate box generation or detector integration in UV-CoT (Zhao et al., 25 Apr 2025), true 3D or depth-aware visual CoT beyond pseudo-depth in VisReason-Pro (Li et al., 21 Nov 2025), and multi-view or 3D asset generation from visual traces in Shape-of-Thought (Huo et al., 28 Jan 2026). Others emphasize multimodal planning loops: interactive agents, robotics, AR/VR, and tool use are explicit targets in VChain, FlowVLA, and VisReason (Huang et al., 6 Oct 2025, Zhong et al., 25 Aug 2025, Li et al., 21 Nov 2025). TVI-CoT and SSV-CoT suggest a more unified long-term direction: reasoning systems in which “thinking” and “looking” are both explicit, adaptive, and repeatedly interleaved rather than separated into a single visual encoding phase and a purely textual deliberation phase (Hu et al., 7 Jun 2026, Guo et al., 21 Mar 2026).

A plausible synthesis is that visual chain-of-thought is becoming less a single technique than a design principle. The shared principle is to externalize, query, or optimize intermediate representations in the target visual modality—or in a representation that faithfully compresses that modality—so that reasoning can proceed through inspectable visual states rather than through text alone. Across current work, the main open question is no longer whether visual CoT can help, but which visual intermediates are most faithful, concise, efficient, and generalizable for a given class of tasks (Cheng et al., 21 May 2025).