Gen-VCoT: Generative Visual Chain-of-Thought

• The paper presents Gen-VCoT, which uses RGB visual intermediates to provide interpretable multimodal reasoning for spatial, geometric, and semantic tasks.
• It employs a three-stage pipeline involving SAM for visual grounding, Marigold for geometric reasoning, and Qwen2-VL for semantic integration.
• Empirical results highlight task-dependent benefits, showing improvement in spatial and depth reasoning while revealing challenges for simple factual queries.

Gen-VCoT, short for Generative Visual Chain-of-Thought, is a multimodal reasoning framework that externalizes intermediate reasoning as human-interpretable RGB images rather than relying exclusively on text-based chain-of-thought. In the formulation introduced in "Gen-VCoT: Generative Visual Chain-of-Thought Reasoning via Diffusion-Based RGB Intermediate Representations" (Zhou et al., 15 Jun 2026), the system decomposes reasoning into visual grounding (Where), geometric reasoning (How), and semantic reasoning (What), using SAM segmentation, Marigold depth maps, and Qwen2-VL integration, together with an adaptive router that selects reasoning depth. The framework is motivated by the claim that text-centric CoT is a poor fit for tasks whose difficulty lies in grounding, spatial layout, depth ordering, and object separation, while the reported results also emphasize that the optimal reasoning representation is task-dependent rather than universally visual (Zhou et al., 15 Jun 2026).

1. Definition and conceptual basis

Gen-VCoT is defined as a pipeline that takes an image $I$ and question $Q$, generates one or more intermediate visual products, and then asks an MLLM to answer using both the original image and the generated evidence. Its central thesis is that one can use generated RGB visual intermediates—specifically segmentation visualizations and pseudo-colored depth maps—as a visual analogue of chain-of-thought. The paper presents these intermediates as simultaneously dense, directly viewable, and naturally consumable by existing vision-LLMs without bespoke decoders (Zhou et al., 15 Jun 2026).

The framework is positioned against two common alternatives. First, standard multimodal reasoning remains largely text-centric, even when the underlying task is visual. Second, prior visual-CoT mechanisms often employ either opaque latent tokens or sparse tool outputs. Gen-VCoT instead treats intermediate reasoning states as RGB images. In the paper’s own framing, this is meant to satisfy four desiderata at once: interpretability, end-to-end processing, dense representation, and decodability (Zhou et al., 15 Jun 2026).

Conceptually, the method assumes that some reasoning problems are more naturally expressed in vision than in language. The paper’s motivating examples are tasks involving object separation, relative position, and near-versus-far structure. The framework therefore treats segmentation as an explicit answer to where relevant entities are, depth as an explicit answer to how they are arranged geometrically, and the final MLLM stage as the place where those visual products are integrated into a semantic answer (Zhou et al., 15 Jun 2026).

A common misconception is that Gen-VCoT is presented as a universal replacement for text CoT. The paper does not support that interpretation. Its CLEVR-style results are used to argue the opposite: visual intermediates may help on spatial and depth questions but may hurt on simple factual or attribute queries, and the best reasoning representation depends on the task (Zhou et al., 15 Jun 2026).

2. Three-stage reasoning pipeline

The pipeline has three sequential stages: visual grounding, geometric reasoning, and semantic reasoning. The first stage generates an RGB instance segmentation map; the second generates an RGB pseudo-colored depth map; the third feeds the original image together with these intermediates into a multimodal LLM for answer production (Zhou et al., 15 Jun 2026).

In the visual grounding stage, Gen-VCoT uses the Segment Anything Model (SAM) to build an instance segmentation map. The paper does not describe language-conditioned segmentation that directly grounds the question text into regions. Instead, it applies a dense grid prompt strategy over the image and later relies on the MLLM to interpret the resulting masks in light of the question. With $N=8$, the system uses $8 \times 8 = 64$ grid points. For each point, SAM returns candidate masks, the mask with the highest predicted IoU score is selected, and masks below the area threshold $\tau_{\text{area}} = 100$ pixels are removed. Remaining instances are rendered into an RGB segmentation image on a black background, each with a distinct random color sampled from $[50,255]^3$ (Zhou et al., 15 Jun 2026).

In the geometric reasoning stage, the system uses Marigold, specifically the LCM (Latent Consistency Model) variant, to generate a dense monocular depth map. The paper emphasizes that this is the stage where diffusion-based generation is most explicit. Depth is predicted with 4 inference steps, 5-member ensemble averaging, fp16 precision, and image size $512 \times 512$. The scalar depth field is then normalized and converted into a rainbow pseudo-color map, where red denotes near regions and violet denotes far regions (Zhou et al., 15 Jun 2026).

In the semantic reasoning stage, Qwen2-VL-7B-Instruct receives a multi-image prompt containing three images: the original image, the segmentation map, and the depth map. The exact prompt template provided by the paper instructs the model that the second image is “a segmentation map with objects color-coded by instance” and the third is “a depth map using rainbow colormap (red=near, violet=far),” then asks it to answer concisely. Baseline evaluation uses only the original image with a shorter prompt (Zhou et al., 15 Jun 2026).

The following table summarizes the fixed expert modules and their reported operating details.

Component	Instantiation	Key details
Visual grounding	SAM ViT-Large	$8 \times 8$ grid prompts, 64 points, $\tau_{\text{area}} = 100$ pixels
Geometric reasoning	Marigold LCM	fp16, 4 inference steps, 5-member ensemble, $512 \times 512$
Semantic reasoning	Qwen2-VL-7B-Instruct	Multi-image prompt over original, segmentation, and depth
Routing	BERT-base	110M parameters, predicts path_1, path_1+2, or full

The significance of this design is not simply that the system adds auxiliary images. The paper’s claim is that the intermediate outputs constitute the actual visual chain-of-thought, because they externalize grounding and geometry before final answering, and because they remain inspectable by humans throughout the pipeline (Zhou et al., 15 Jun 2026).

3. Router, prompting, and systems behavior

A key systems component is the adaptive reasoning router, introduced because not all questions benefit from all intermediates. The router is a BERT-base classifier with 110M parameters that takes the question text only and predicts one of three paths: path_1 for segmentation only, path_1+2 for segmentation plus depth, and full for all three stages. The router is trained with a combined loss $Q$0, with $Q$1 to penalize more expensive paths (Zhou et al., 15 Jun 2026).

The appendix-level training details reported for the router are modest. It uses 100 question-path pairs annotated by the authors, AdamW, learning rate $Q$2, batch size 16, and 3 epochs, taking less than 1 minute on an RTX 3090. Validation accuracy is reported as 85% on a held-out set of 50 questions. The paper notes that most router errors arise on ambiguous questions that could reasonably use either path_1+2 or full (Zhou et al., 15 Jun 2026).

The router is also where an important presentation inconsistency appears. The explicit router label set lists only the three intermediate-bearing paths, but the pseudocode includes a baseline fallback, and later discussion argues that simple questions should bypass intermediate steps with a path_baseline. The paper does not fully resolve this inconsistency. In practice, the article’s own discussion treats baseline bypassing as desirable for simple factual questions, even though the method section enumerates only the three non-baseline labels (Zhou et al., 15 Jun 2026).

The system is explicitly not trained end-to-end. Gen-VCoT is assembled from frozen expert modules—SAM ViT-Large, Marigold LCM, and Qwen2-VL-7B-Instruct—and the router is the only newly trained component. There is no reported fine-tuning of the segmentation model, depth estimator, or reasoning MLLM, and there is no joint supervision aligning intermediate images to answer correctness (Zhou et al., 15 Jun 2026).

The implementation profile reflects this modular design. On a single NVIDIA RTX 3090 with 24 GB VRAM, the models are loaded sequentially. The reported breakdown is: SAM loading 1.1 s, SAM inference 1.1 s; Marigold loading 2.1 s, Marigold inference 2.5 s; Qwen2-VL loading 11.9 s; Gen-VCoT reasoning 0.8 s/question; and baseline reasoning 0.3 s/question. The paper further states that preprocessing cost can be amortized when many questions are asked about the same image, though its appendix and main-text amortized timing estimates are themselves inconsistent (Zhou et al., 15 Jun 2026).

4. Empirical profile and task-dependent performance

The paper reports two evaluation regimes. The first is an initial evaluation on a complex synthetic scene with 19 questions. In that setting, Gen-VCoT reaches 15/19 = 78.9%, versus 13/19 = 68.4% for the baseline, a +10.5% absolute gain. Category-wise, the paper reports +25% on spatial questions, +50% on depth questions, no change on recognition and counting, −50% on attribute questions, and +20% on complex reasoning questions (Zhou et al., 15 Jun 2026).

The second regime includes a CLEVR-style evaluation over 10 synthetic scenes and 80 questions. Here the picture changes substantially. One table reports Gen-VCoT 56/80 = 70.0% versus baseline 68/80 = 85.0%. A separate three-way comparison reports baseline 68/80 = 85.0%, Gen-VCoT 50/80 = 62.5%, and Text CoT 73/80 = 91.2%. The paper does not explain the discrepancy between 70.0% and 62.5% for Gen-VCoT on CLEVR, so this remains an unresolved reporting inconsistency (Zhou et al., 15 Jun 2026).

Despite that inconsistency, the trend emphasized by the authors is stable: Text CoT is best on simple factual CLEVR-style questions, baseline image-only reasoning is next, and visual intermediates are worst. The paper interprets this as evidence that structured text is a more precise channel for existence, count, and attribute queries, whereas RGB intermediates can inject noise. The method is therefore not framed as a universal accuracy booster, but as a representation that is specifically advantageous when geometric and spatial cues dominate (Zhou et al., 15 Jun 2026).

The stage ablations support this reading. Across 3 scenes × 8 questions × 4 configurations = 96 total evaluations, the paper reports that the Full (Seg+Depth) pipeline is best on spatial and depth tasks; No Depth (Seg only) is good on spatial and poor on depth; No Seg (Depth only) is good on both spatial and depth; and the Baseline is poor on both. The stated conclusion is that segmentation contributes object identity and counting, while depth contributes distance and ordering, making the two stages complementary rather than redundant (Zhou et al., 15 Jun 2026).

These results directly motivate one of the framework’s central claims: the optimal reasoning representation is task-dependent. The paper explicitly summarizes this as follows: simple factual and attribute queries favor Text CoT, while spatial reasoning, depth perception, and complex reasoning favor Visual representations (Zhou et al., 15 Jun 2026).

5. Relation to adjacent visual CoT frameworks

Gen-VCoT belongs to a broader family of multimodal CoT methods, but its defining feature is the use of RGB intermediate images rather than text-only rationales, latent visual tokens, or implicit latent planning. Related work illustrates several distinct design points.

VoCoT externalizes multimodal reasoning as a structured autoregressive sequence built around object descriptions, coordinate tokens, and dynamically inserted object visual features. Its core ideas are object-centric reasoning paths and a visually grounded representation of object concepts, and its concrete model instantiation, VolCano, uses OpenAI CLIP ViT-L/14, Mistral-7B, and a two-layer MLP to support grounded multi-step reasoning in LMMs (Li et al., 2024).

VG-CoT shifts emphasis from generation modality to trustworthiness. It constructs a dataset and benchmark in which each reasoning step is explicitly aligned to object regions and OCR evidence through a fully automated pipeline using YOLO, PaddleOCR, GPT-4o, and Grounding DINO. Its benchmark evaluates Rationale Quality, Answer Accuracy, and Reasoning-Answer Alignment, making grounded step-level evidence the central object of evaluation (Lim et al., 23 Apr 2026).

For video reasoning, VTI-CoT and ViTCoT both argue that text-only CoT is insufficient because actual inference should revisit visual evidence. VTI-CoT pairs textual reasoning steps with video intervals and representative frames, then renders the entire multimodal trace into a single canvas for efficient supervision (Zhang et al., 4 Jun 2026). ViTCoT instead inserts a selected key-video into a two-stage reasoning process and reports average gains over vanilla video CoT across several MLLMs and reasoning styles (Zhang et al., 14 Jul 2025).

Generation-oriented variants define still other forms of visual CoT. IV-CoT keeps the chain implicit by decomposing text-to-image conditioning into a structural-to-semantic query cascade supervised with training-only sketches, so the chain remains latent rather than decoded (Li et al., 23 Jun 2026). GVCoT for image editing generates an explicit image-space visual thought to localize where to edit before producing the final edit, and optimizes reasoning and editing jointly (Yin et al., 2 Mar 2026). Visual-Aware CoT introduces Adaptive Visual Planning and Iterative Visual Correction for reference-conditioned generation, using structured visual checklists to preserve identity, style, and attribute consistency (Ye et al., 22 Dec 2025). VCoT-Grasp applies the same principle to robotic manipulation by predicting an intermediate bounding box before crop-conditioned grasp generation (Zhang et al., 7 Oct 2025).

Taken together, these works suggest a broader design space. Some systems externalize reasoning through coordinates and object features (Li et al., 2024), some through region-grounded textual rationales (Lim et al., 23 Apr 2026), some through retrieved frames or rendered canvases (Zhang et al., 4 Jun 2026), some through implicit latent plans (Li et al., 23 Jun 2026), and Gen-VCoT through RGB segmentation and depth images (Zhou et al., 15 Jun 2026). A plausible implication is that “visual CoT” is not a single format but a family of intermediate representations with different trade-offs in interpretability, bandwidth, and task fit.

6. Limitations, inconsistencies, and significance

The paper states several explicit limitations. First, Gen-VCoT uses fixed expert models without task-specific fine-tuning, which may restrict performance on domain-specific or unusual images. Second, the three-stage sequential design adds latency, with about 8 s preprocessing per image for segmentation and depth. Third, evaluation is limited mostly to synthetic scenes; although the paper mentions real-world benchmark evaluation on GQA and CLEVR as ongoing, the reported experiments are primarily synthetic or synthetic CLEVR-like (Zhou et al., 15 Jun 2026).

A further limitation is error propagation. The framework depends strongly on expert-module quality, so segmentation failures and depth estimation errors can propagate into the answering stage. The paper’s own CLEVR discussion underscores domain sensitivity: grid-based SAM can produce irrelevant masks in simple scenes, and Marigold depth on synthetic CLEVR objects may lack meaningful gradients. The router is introduced as a mitigation strategy, but the current formulation does not demonstrate a full routing regime among baseline, text CoT, visual CoT, and combined modes (Zhou et al., 15 Jun 2026).

Methodologically, the work also contains unresolved reporting issues. Gen-VCoT CLEVR accuracy is reported as both 70.0% and 62.5%, and amortized per-question efficiency appears as both 5.2 s and approximately 1.8 s. The router presentation is likewise inconsistent about whether a baseline path is an explicit label or only an implied fallback. These inconsistencies do not negate the framework’s conceptual contribution, but they constrain strict quantitative interpretation (Zhou et al., 15 Jun 2026).

The framework’s lasting significance lies in a narrower claim. Gen-VCoT does not establish that visual intermediates are always superior. Rather, it argues that multimodal reasoning should not be tied to a single universal intermediate format. In the paper’s own synthesis, sometimes text CoT is best, sometimes visual CoT is best, and adaptive systems should learn to select the right reasoning representation for the task (Zhou et al., 15 Jun 2026).

In that sense, Gen-VCoT marks a specific transition in the visual-CoT literature: from hidden or purely verbal intermediate reasoning toward explicit RGB visual chain-of-thought. Its segmentation images answer where, its depth images answer how, and its MLLM stage answers what follows from those visible intermediates. The broader implication is not that reasoning has become fully solved by RGB externalization, but that interpretable visual intermediate states can function as a practical reasoning substrate for some classes of multimodal problems (Zhou et al., 15 Jun 2026).