Visual-CoG: Chain of Visual Guidance

• Visual-CoG is a design principle that integrates structured visual intermediates into multimodal chain-of-thought reasoning, enabling explicit guidance and reduced logical gaps.
• Key motifs include decomposing complex tasks into visual subproblems, employing selective visual access, and using iterative corrections to boost interpretability and efficiency.
• Empirical studies demonstrate significant improvements in accuracy and control across tasks, with optimized token consumption and reinforcement learning strategies enhancing performance.

Searching arXiv for papers on Visual-CoG and closely related formulations to ground the article in the cited literature. Visual-Chain of Guidance (Visual-CoG) denotes a family of multimodal methods that insert explicit visual intermediates into a chain-of-thought-like process, so that reasoning, generation, or control is guided not only by text but also by structured visual evidence, visual access decisions, or visually grounded latent states. In the literature, the term covers several non-identical but related formulations: visual-textual infillings for sequential data, chains of anchors for 3D grounding, “visual thoughts” as intermediaries between image tokens and deeper reasoning layers, saliency-ranked region access in multimodal LLMs, stage-aware rewards for text-to-image generation, keyframe chains for video generation, spatial priors for embodied control, and geometric reasoning trajectories for pointing localization (Rose et al., 2023, Abdelrahman et al., 2023, Cheng et al., 21 May 2025, Guo et al., 21 Mar 2026, Ling et al., 13 May 2026, Li et al., 23 Jun 2026). This suggests that Visual-CoG is best understood as a general design principle for structured visual guidance rather than a single standardized architecture.

1. Conceptual emergence and terminological scope

An early explicit formulation appeared in "Visual Chain of Thought: Bridging Logical Gaps with Multimodal Infillings" (Rose et al., 2023), which extended chain-of-thought prompting beyond unimodal text by generating visual-textual intermediate steps between sparse key frames or instructions. In that setting, Visual-CoG was used to create synthetic multimodal infillings that were selected for consistency with surrounding context and intended to reduce logical gaps in sequential data.

A second strand emerged in "CoT3DRef: Chain-of-Thoughts Data-Efficient 3D Visual Grounding" (Abdelrahman et al., 2023), where visual grounding was reformulated as a sequence-to-sequence task that first predicts a chain of anchors and then the final referent. Here the central claim was not image generation but interpretability and data efficiency: the model exposes intermediate object selections, allowing failures to be traced to specific anchor errors.

A broader theoretical consolidation appears in "Visual Thoughts: A Unified Perspective of Understanding Multimodal Chain-of-Thought" (Cheng et al., 21 May 2025). That work argues that multimodal chain-of-thought improves large vision-LLMs by incorporating visual thoughts, defined as intermediate reasoning steps that distill instruction-relevant visual content into a cache-like representation. It further distinguishes Textual-MCoT and Interleaved-MCoT while proposing four expression forms for visual thoughts: Natural-Language Visual Thoughts, Structured-Language Visual Thoughts, Edited-Image Visual Thoughts, and Generative-Image Visual Thoughts. Within this perspective, Visual-CoG functions as the bridge from raw image content to later linguistic reasoning.

Because later papers reuse the term for markedly different mechanisms—stage-aware reinforcement learning in text-to-image generation, explicit spatial priors in robot control, or bidirectional latent guidance in diffusion—there is a recurring nomenclature issue. A common source of confusion is to treat all Visual-CoG systems as interleaved image-text decoding schemes. The published record does not support that reduction: some systems emit explicit visual artifacts, some query or retrieve visual regions, some operate only in latent space, and some use visual guidance only during training or sparse inference-time tuning (Li et al., 25 Aug 2025, Huang et al., 6 Oct 2025, Ma et al., 25 Nov 2025, Ye et al., 22 Dec 2025).

2. Recurrent design motifs

Across domains, Visual-CoG methods repeatedly decompose a difficult multimodal task into interpretable visual subproblems. In CoT3DRef, these are the anchor sequence and final target; in VChain they are sparse causally important keyframes paired with textual thoughts; in PointVG-R they are hand detection, fingertip keypoint extraction, ray construction, and target alignment; in GTA-VLA they are structured reasoning tokens $$C = [C_{\text{task}}; C_{\text{vision}}; C_{\text{robot}}]$$ (Abdelrahman et al., 2023, Huang et al., 6 Oct 2025, Ling et al., 13 May 2026, Li et al., 23 Jun 2026). The shared idea is that the intermediate structure is not merely explanatory output but a computational scaffold that constrains subsequent prediction.

A second motif is selective or adaptive visual access. "Beyond Static Visual Tokens: Structured Sequential Visual Chain-of-Thought Reasoning" (Guo et al., 21 Mar 2026) replaces static visual prefixes with a question-relevant saliency map, connected-component region extraction, a ranked structured region bank, and a learned step-wise policy over regions plus a STOP token. "Let's Think with Images Efficiently!" (Liu et al., 23 Mar 2026) uses Dynamic Visual Thought Integration to decide whether visual insertion is needed based on a confidence threshold, and Precise Visual Thought Guidance to select semantically coherent object-level sub-images from SAM2 outputs. "Chain-of-Cooking" (Xu et al., 29 Jul 2025) uses a Dynamic Patch Selection Module to retrieve previously generated image patches that are most related to the current textual contents. In GTA-VLA, optional human spatial priors—affordance points, boxes, and traces—are serialized into the backbone’s coordinate-token space and directly condition reasoning (Ling et al., 13 May 2026). These variants differ operationally, but each rejects the assumption that one static visual encoding suffices for multi-step reasoning.

A third motif is iterative, bidirectional, or stage-wise correction. In "Chain-of-Cooking: Cooking Process Visualization via Bidirectional Chain-of-Thought Guidance" (Xu et al., 29 Jul 2025), forward CoT uses the merged latent $z_t^p$ to guide denoising while backward CoT uses $z_{t+1}^p$ as a reconstruction target to refine the current latent prompt. In "Visual-Aware CoT: Achieving High-Fidelity Visual Consistency in Unified Models" (Ye et al., 22 Dec 2025), Adaptive Visual Planning first produces a checklist of visual consistency requirements and Iterative Visual Correction repeatedly evaluates and edits the current output until checks pass or the iteration limit is reached. In "Visual-CoG: Stage-Aware Reinforcement Learning with Chain of Guidance for Text-to-Image Generation" (Li et al., 25 Aug 2025), the generation pipeline is decomposed into semantic reasoning, process refining, and outcome evaluation, each with its own reward signal. The common principle is that visual guidance is distributed over the trajectory rather than concentrated at the final output.

3. Representative realizations across tasks

Domain	Visual-CoG mechanism	Representative paper
Sequential storytelling and summarization	Recursive multimodal infillings between neighboring steps	(Rose et al., 2023)
3D visual grounding	Seq2Seq chain of anchors followed by final target	(Abdelrahman et al., 2023)
Cooking process visualization	DPS, Semantic Evolution, Bidirectional CoT Guidance	(Xu et al., 29 Jul 2025)
Text-to-image generation	Semantic reasoning, process refining, outcome evaluation with stage-aware rewards	(Li et al., 25 Aug 2025)
Video generation	Sparse chain-of-visual-thought keyframes with LoRA tuning	(Huang et al., 6 Oct 2025)
Unified multimodal generation	Adaptive Visual Planning and Iterative Visual Correction	(Ye et al., 22 Dec 2025)
Multimodal reasoning in MLLMs	Structured saliency-ranked region bank and sequential region selection	(Guo et al., 21 Mar 2026)
Embodied VLA control	Optional spatial priors plus structured spatial-visual CoT	(Ling et al., 13 May 2026)
Pointing-based grounding	Hand, keypoint, ray, and target reasoning trajectory	(Li et al., 23 Jun 2026)

These systems differ in where the visual chain resides. In VCoT, the chain is explicit synthetic data augmentation; in CoT3DRef and PointVG-R, it is a supervised reasoning trajectory; in SSV-CoT and DaP-ICoT, it is a learned visual-access policy inside MLLM decoding; in Chain-of-Cooking and VITA, it is partly latent; in VACoT and Visual-CoG RL, it is a planning-and-correction or reward-decomposition mechanism for generation (Rose et al., 2023, Abdelrahman et al., 2023, Ma et al., 25 Nov 2025, Liu et al., 23 Mar 2026).

This variation also clarifies a technical misconception: Visual-CoG is not restricted to explicit image outputs inserted into the reasoning transcript. Several influential formulations operate through latent prompts, discrete codebooks, visual retrieval, or spatial prior tokens rather than visible intermediate images (Xu et al., 29 Jul 2025, Ma et al., 25 Nov 2025, Ling et al., 13 May 2026).

4. Architectures, objectives, and optimization strategies

The mathematical structure of Visual-CoG methods typically factorizes a difficult output distribution through intermediate guided states. CoT3DRef makes this explicit by modeling a chain of anchors $A$ and final target $t$ as

$$P(A,t\mid P,U)=\prod_{i=1}^{T}P(a_i\mid P,U,a_{<i}),$$

with anchor supervision implemented through sequential box losses in addition to parallel referring and distractor losses (Abdelrahman et al., 2023). PointVG-R uses an analogous factorization over reasoning stages and final box,

$$P(b,R\mid I;\theta)=P(R\mid I;\theta)\times P(b\mid I,R;\theta),$$

and then refines policy learning with cold-start SFT plus GRPO and an adaptive importance-weighting strategy based on group variance (Li et al., 23 Jun 2026).

Diffusion-based and unified generation variants couple visual guidance to denoising or iterative editing. Chain-of-Cooking defines a standard denoising loss and a semantic consistency loss over latent prompts,

$$\mathcal L=\mathcal L_{CE}+\eta\,\mathcal L_{CS},\quad \eta=0.1,$$

while classifier-free guidance at inference uses $w=7.5$ (Xu et al., 29 Jul 2025). VACoT trains first with supervised fine-tuning on planning and correction traces and then with flow-GRPO under a composite reward

$$R_{\mathrm{total}}(Y,V,T)=R_{\mathrm{visual}}(Y,Z_{\mathrm{plan}},V)+R_{\mathrm{text}}(Y,T),$$

where $z_t^p$0 decomposes over checklist items with type-specific scorers such as DINO similarity for identity and CSD-Score for style (Ye et al., 22 Dec 2025).

Reinforcement learning formulations are especially prominent in recent text-to-image and reasoning systems. Visual-CoG RL for autoregressive text-to-image generation defines three rewards—semantic reasoning $z_t^p$1, process refining $z_t^p$2, and outcome evaluation $z_t^p$3—and simply sums them:

$z_t^p$4

Optimization then uses a PPO-style clipped surrogate objective over sampled total rewards (Li et al., 25 Aug 2025). SSV-CoT uses a two-stage procedure: heuristic supervised fine-tuning with a curriculum coefficient and behavior cloning loss, followed by GRPO reinforcement learning with reward terms for task accuracy, format, length, and visual budget, plus a KL regularizer to remain close to the SFT policy (Guo et al., 21 Mar 2026).

A different optimization logic appears in VITA, where visual and action streams are forced into a shared discrete codebook $z_t^p$5 and trained with self-supervised reconstruction losses for future frames and action trajectories, combined on synchronized data as

$z_t^p$6

The resulting discrete latent sequence is then decoded in parallel into predicted frames and robot actions, yielding what the paper terms an implicit visual CoT for motion planning (Ma et al., 25 Nov 2025).

5. Empirical record

In grounding and localization, Visual-CoG methods report substantial gains. CoT3DRef improves multiple backbones on Nr3D and Sr3D under full-data training: for MVT, Nr3D rises from 55.1% to 60.4% and Sr3D from 64.5% to 73.2%; for SAT, Nr3D rises to 58.1% and Sr3D to 69.1%; for LAR, Nr3D reaches 58.0% and Sr3D 69.0% (Abdelrahman et al., 2023). Under 10% data on Sr3D, SAT rises from approximately 48.8% to 66.4% and MVT from approximately 48.8% to 65.2%. PointVG-R reports mIoU increasing from approximately 0.455 for zero-shot Qwen2.5-VL-7B to approximately 0.598 for standard SFT, 0.645 for V-CoT SFT, and 0.757 for the full method, described as a 15.86-point improvement over the baseline (Li et al., 23 Jun 2026).

In multimodal reasoning, sequential visual access also yields measurable gains. SSV-CoT, using a Qwen2-VL-7B backbone, improves M3CoT from 43.6 to 44.9, ScienceQA from 56.3 to 57.3, and LLaVA-W ROUGE-L from 32.7 to 35.7; on mathematical benchmarks it raises MathVista from 60.9 to 72.2 and MathVision from 16.3 to 23.5, with average performance increasing from 38.6 to 47.9 (Guo et al., 21 Mar 2026). DaP-ICoT reports larger gains relative to static ICoT baselines: on Qwen2-VL-7B, M3CoT reaches 57.2% in 0-shot and 58.7% in 1-shot versus 38.0% and 44.8%; ScienceQA reaches 75.9% and 78.5% versus 54.2% and 67.0%; MME Perception+Cognition reaches 2012.2 and 2076.0 versus 1587.3 and 1709.3 (Liu et al., 23 Mar 2026).

In image and video generation, the reported benefits center on semantic consistency, coherence, and reasoning-sensitive evaluation. Chain-of-Cooking introduces the CookViz dataset with 40 K intermediate image-text pairs and reports approximately 4–6 point FID reduction versus fine-tuned Stable Diffusion, together with +0.006 CLIP-T, +0.010 CLIP-I, and human Consistency/Coherency gains of approximately 0.3–0.5 points (Xu et al., 29 Jul 2025). Visual-CoG RL raises Show-o on GenEval from 68.29 to 83.86 and on T2I-CompBench from 34.67 to 36.84; on VisCog-Bench, baseline Show-o scores 58.07, the variant without semantic reasoning reward scores 65.34, and full Visual-CoG reaches 77.50, with especially large gains on Unusual Position and Unusual Color (Li et al., 25 Aug 2025). VACoT reports an OmniContext average of 8.26 compared with 7.89 for Uni-CoT and a GenEval overall score of 0.84 compared with 0.83 for Uni-CoT, while stating that most images satisfy checklist items within 2–3 iterations (Ye et al., 22 Dec 2025). VChain reports VBench 78.49%, Frame 71.67%, Temporal 65.82%, Align 67.77%, Physics 58.01%, Commonsense 60.16%, and Causal 62.12%, outperforming baselines and ablations on complex multi-step scenarios (Huang et al., 6 Oct 2025).

In embodied systems, Visual-CoG has been used to improve robustness and controllability rather than only reasoning accuracy. GTA-VLA attains 98.6% average success on LIBERO and 81.2% average success on SimplerEnv, and under SimplerEnv-Plus OOD shifts reaches 61.4% overall versus 52.3% for X-VLA, 7.3% for $z_t^p$7, and 3.7% for OpenVLA (Ling et al., 13 May 2026). In ambiguous settings, unseen-object ambiguity improves from 27.8% unguided to 40.9% with affordance-point guidance and 56.9% with box guidance. VITA reports improvements of 14.5%, 9.6%, and 12.1% over existing baselines on CALVIN, LIBERO, and SimplerEnv, together with an average success rate of 80.5% across six real-world tasks (Ma et al., 25 Nov 2025).

6. Interpretability, efficiency, and reported limitations

Interpretability is a central justification for Visual-CoG. CoT3DRef explicitly states that by visualizing attention at each decoding step one can see why a particular anchor was chosen, and that if the chain breaks at anchor $z_t^p$8 the error source is immediately known (Abdelrahman et al., 2023). VCoT frames its multimodal infillings as interpretable reasoning chains for storytelling and summarization (Rose et al., 2023). "Visual Thoughts" (Cheng et al., 21 May 2025) sharpens this point by arguing that the improvement from multimodal chain-of-thought depends on clarity and conciseness rather than pixel-perfect fidelity: correlation analysis reports no significant link between fidelity and accuracy, but very strong links for clarity and conciseness. It also describes the dominant information path as image $z_t^p$9 VT $z_{t+1}^p$0 reasoning rather than image $z_{t+1}^p$1 reasoning directly, with visual thought tokens serving as the “hot lane” into deeper layers.

Efficiency, however, is a recurrent concern. DaP-ICoT reports a 72.6% decrease in token consumption on M³CoT with Qwen2-VL-7B, reducing average tokens from 1146 to 314 (Liu et al., 23 Mar 2026). GTA-VLA decouples slow reasoning from fast control, updating its VLM branch at approximately 2 Hz while the action head runs at approximately 10 Hz (Ling et al., 13 May 2026). VITA reports 60 Hz action-step throughput, contrasting this with 5–10 Hz in prior predict-then-act CoT models (Ma et al., 25 Nov 2025). VChain estimates approximately 3 minutes for visual thought reasoning, 5–7 minutes for sparse inference-time tuning on a single A100 GPU for 5–6 keyframes, and 3 to 15 minutes for video sampling, with overall overhead reported as no more than approximately 20% of full video sampling time (Huang et al., 6 Oct 2025). These results indicate that Visual-CoG can either reduce multimodal overhead by making visual access selective or increase inference cost by adding structured guidance loops.

The limitations reported across papers are heterogeneous but technically consistent. Chain-of-Cooking states that the current model struggles with highly dynamic backgrounds, multi-person collaborations, and ultra-fine-grained manipulations such as multiple kneading steps (Xu et al., 29 Jul 2025). SSV-CoT notes additional compute cost from sequential visual access and reliance on the quality of the question-conditioned saliency map (Guo et al., 21 Mar 2026). Visual-CoG RL acknowledges heavy reliance on external detectors and validators such as GroundingDINO, CLIP, and HPS, as well as expensive and potentially unstable RL finetuning (Li et al., 25 Aug 2025). GTA-VLA states that current guidance and CoT live in 2D image space, making extension to full 3D priors an important future direction (Ling et al., 13 May 2026). DaP-ICoT notes that SAM2 segmentation may fail on cluttered scenes and that the confidence threshold is global rather than learned (Liu et al., 23 Mar 2026). VChain identifies keyframe quality drift, API cost, and the trade-off between over-fitting to static keyframes and under-fitting sparse guidance (Huang et al., 6 Oct 2025).

Taken together, these results define Visual-CoG as a broad research program for making visual reasoning trajectories explicit, actionable, and optimizable. Its shared hypothesis is that multimodal systems improve when visual evidence is not merely encoded once at the input boundary but organized into a chain—retrieved, generated, ranked, edited, or supervised—whose intermediate states remain available to later reasoning, generation, or control.