Gen-VCoT: Generative Visual Chain-of-Thought Reasoning via Diffusion-Based RGB Intermediate Representations

Abstract: Multimodal LLMs (MLLMs) excel at visual reasoning but rely on text-based chain-of-thought (CoT), lacking interpretable visual intermediates. Existing methods use opaque tokens or external tools, missing key properties. We propose Gen-VCoT, a framework using expert vision models to generate RGB images as reasoning intermediates. It has three stages: visual grounding (SAM segmentation), geometric reasoning (Marigold depth maps), and semantic reasoning (Qwen2-VL integration). An adaptive router selects reasoning depth. Evaluations show Gen-VCoT improves spatial (25% better) and depth (50% better) questions, but may hurt simple factual queries. Text CoT outperforms visual intermediates on CLEVR (91.2% vs 62.5%), showing task-dependent optimal representations. Gen-VCoT establishes a new paradigm for interpretable multimodal reasoning.

• The paper's main contribution is introducing an adaptive, three-stage visual reasoning pipeline that leverages interpretable RGB intermediates.
• The methodology integrates expert segmentation and diffusion-based depth estimation to enhance spatial, geometric, and object-centric processing, achieving up to 50% improvement on depth queries.
• Experimental results show that explicit visual intermediates reduce hallucination and improve reasoning accuracy, with dynamic routing optimizing performance per task.

Gen-VCoT: Generative Visual Chain-of-Thought Reasoning via Diffusion-Based RGB Intermediate Representations

Motivation and Background

The paper presents Gen-VCoT, a framework addressing significant limitations in current Multimodal LLMs (MLLMs) for visual reasoning. Existing approaches predominantly rely on textual chain-of-thought (CoT) to decompose tasks but are fundamentally restricted when the reasoning requires spatial, geometric, or object-centric processing that is inherently visual. While recent work such as COVT leverages continuous visual tokens, these intermediates lack transparency, inhibiting interpretability and decodability. Further, methods like Visual Sketchpad depend on sparse geometric primitives or external tools, sacrificing the density and resolution of visual representation. Gen-VCoT leverages expert vision models to generate dense, interpretable RGB visual intermediates, systematically decomposing the reasoning process into explicit stages aligned with the underlying visual task structure.

Methodology

Three-Stage Visual Reasoning Pipeline

Gen-VCoT decomposes visual question answering into three sequential stages:

1. Visual Grounding (Where): The Segment Anything Model (SAM) generates instance segmentation maps via a grid-of-points strategy, producing color-coded regions marking distinct objects relevant to the query.
2. Geometric Reasoning (How): Marigold, a diffusion-based depth estimator, produces pseudo-colored depth maps using rainbow colormapping, establishing visual cues for spatial ordering (red = near, violet = far).
3. Semantic Reasoning (What): An MLLM, specifically Qwen2-VL, receives the original image, segmentation, and depth as joint input, integrating structured visual evidence to form the final answer.

   Figure 1: Gen-VCoT pipeline architecture. Each intermediate product (segmentation and depth) is generated via expert models and passed to Qwen2-VL for answer synthesis.

This pipeline is dynamically modulated by an adaptive router trained via BERT-base, which selects among several paths (segmentation only, segmentation + depth, full) in accordance with question complexity.

Dense Intermediate Visual Products

Gen-VCoT produces intermediates that are inherently interpretable and decodable. The segmentation and depth maps produced can be visually inspected and serve as dense pixel-level evidence for both the model and human analysts.

Figure 2: Gen-VCoT intermediate outputs. (a)~Input synthetic scene; (b)~instance-level segmentation; (c)~pseudo-colored depth map.

Experimental Evaluation

Synthetic and CLEVR-style Scenes

The framework is evaluated on synthetic scenes including indoor, street, and park domains. Twenty-four queries spanning recognition, spatial, depth, counting, attribute, and reasoning categories are benchmarked across multiple pipeline configurations.

Figure 3: Example synthetic evaluation scenes with their segmentation and depth intermediates.

Key findings:

• Spatial Relationship Answers: Gen-VCoT outperforms direct MLLM inference by +25% for spatial queries; depth reasoning shows a +50% improvement.
• Complex Reasoning: The presence of explicit visual intermediates substantially reduces hallucination and enhances object ordering accuracy.

  Figure 4: Qualitative comparison demonstrates improved spatial reasoning and object ordering from Gen-VCoT versus baseline.

Ablation Studies

Ablation experiments isolate the contribution of segmentation and depth stages, demonstrating that each step provides orthogonal improvements. Without depth, spatial ordering is less reliable; without segmentation, counting and object identification suffer.

Figure 5: Ablation study for spatial ordering in street scene. The full pipeline yields correct leftmost object, while partial configurations degrade reliability.

CLEVR-Style Analysis and Modality Trade-offs

CLEVR-style factual queries highlight an important contradiction: indiscriminate introduction of dense visual intermediates can degrade performance on simple tasks compared to both direct image inputs and text CoT. In CLEVR scenarios, text CoT yields 91.2% accuracy, considerably outperforming Gen-VCoT (62.5%) and the baseline (85%). This underscores that the optimal intermediate modality is task-dependent and motivates the adaptive router.

Efficiency and Practical Considerations

The pipeline achieves practical batching efficiency (5.2s/question) when model loading costs are amortized. While there is an inherent latency in preprocessing steps, this is offset in scenarios requiring multiple queries per image. All model components are publicly available and integrated through HuggingFace and Diffusers libraries.

Theoretical Implications and Future Directions

Gen-VCoT anchors the “generation as understanding” paradigm, extending prior work on repurposing diffusion models for dense vision tasks. Its explicit chain-of-reasoning at the visual level not only advances interpretability but also enables downstream compositional reasoning. The adaptive router’s empirical motivation further solidifies that modality selection should be data-driven and question-conditioned.

Potential future directions include:

• Scaling the pipeline to GQA, CLEVR, and real-world datasets
• Reinforcement-based router training to optimize answer correctness and efficiency
• Integration of video diffusion models for temporal chain-of-thought reasoning
• Co-training or fine-tuning MLLMs with explicit visual intermediates for enhanced synergy

Limitations

The primary limitations include reliance on fixed expert models, latency from sequential pipeline stages, and degraded performance in domains where visual intermediates introduce noise. There is also a need for broader benchmark coverage and domain adaptation.

Conclusion

Gen-VCoT introduces a systematic, generative, visual chain-of-thought reasoning framework integrating segmentation and depth maps as explicit, interpretable intermediates. The approach demonstrates robust improvements on spatial and depth reasoning tasks but highlights task-dependence of intermediate representation modality, as factual queries are better served by textual intermediates. The adaptive routing mechanism is empirically justified and essential for maximizing accuracy and efficiency. These findings contribute to advancing interpretable and generalizable multimodal reasoning.