---
title: Visualization-of-Thought Prompting
url: https://www.emergentmind.com/topics/visualization-of-thought-vot-prompting
type: topic
---

# Visualization-of-Thought Prompting

Visualization-of-Thought (VoT) prompting refers to a family of techniques that augment or restructure the internal reasoning processes of large language models (LLMs), including multimodal models (MLLMs), by explicitly externalizing intermediate reasoning steps as visual, structural, or graphical representations. VoT prompting replaces purely textual or latent chains of thought with explicit, stepwise, human-interpretable artifacts—such as diagrams, tables, sketches, trees, or code-generated images—that scaffold inference, promote reliability, elucidate model processes, and enhance complex reasoning, particularly in domains where spatial, set-theoretic, or multimodal synthesis is key [2406.05369][2508.04842][2406.14562][2405.13872][2501.07542][2404.03622].

## 1. Foundational Principles and Taxonomy

Visualization-of-Thought prompting is distinguished by requiring the model to “draw out” its intermediate analysis or reasoning trace, rather than solely relying on chain-of-thought (CoT) verbalizations. A VoT prompt explicitly directs the LLM to construct one or more visual or structured artifacts reflecting its understanding at each inference hop. These can take diverse forms:

- **Set/Diagram-Based**: Venn Diagram Prompting (VD) asks the model to partition document facts into overlaps, uniques, and irrelevants with respect to the query, mirroring Venn diagram regions [2406.05369].
- **Tabular/Analytic**: Charts-of-Thought requires the model to extract data points from a figure, build a structured table, verify correctness, and only then perform the ultimate analytic (e.g., VLAT chart reasoning) [2508.04842].
- **External Sketch/Whiteboard**: Whiteboard-of-Thought has the LLM emit code (Matplotlib/Turtle) to generate literal images for each reasoning step, then feeds these images back to itself for further analysis [2406.14562].
- **Mental Image/ASCII-Grid**: Classic VoT in textual models elicits model-rendered ASCII-art or emoji-grids after each reasoning step, emulating the “mind’s eye” [2404.03622].
- **Explicit Visual Rationale Extraction**: Image-of-Thought (IoT) prompting decomposes a problem into subgoals, interleaving sub-image (crop, segmentation, annotation) and textual rationales in MLLMs [2405.13872].
- **Multimodal Interleaving**: Multimodal VoT (MVoT) prompts models to alternate text and genuine image-token sequence blocks, yielding a visual sketchpad within the generation process [2501.07542].
- **Graphical Process Transparency**: Systems like PrompTHis and iToT expose the model’s or user’s prompt-editing or reasoning-tree history as explicit, navigable visualizations [2403.09615][2409.00413].

VoT prompting serves both functional and interpretability objectives: it scaffolds complex reasoning for difficult tasks and provides external artifacts for humans to scrutinize or manipulate.

## 2. Formal Mechanisms of Visualization-of-Thought

VoT prompting generalizes the chain-of-thought paradigm—notationally, for a prompt input $x$, the model alternates between emitting a textual step $z_i$ and a visual artifact $v_i$:

$$
z_i \sim p_\theta(z_i | x, z_{1:i-1}, v_{1:i-1})
$$
$$
v_i \sim p_\theta(v_i | x, z_{1:i}, v_{1:i-1})
$$

For more complex VoT variants:

- **Whiteboard-of-Thought**: At each step $i$, the model generates drawing code $\mathcal{C}_{i}$ (e.g., Matplotlib/Turtle), which is executed to yield $W_i$, then updates its chain-of-thought given $(S, W_{1:i})$ [2406.14562].
- **Image-of-Thought (IoT)**: For an image/question pair $(I,Q)$, the MLLM decomposes $Q$ into subgoals $\{SG_i\}$, for each invokes a visual operation (detection, segmentation, etc.) on $I$ yielding $VR_i$, and explains $VR_i$ in text $TR_i$. The multimodal rationale series $\langle SG_i, VR_i, TR_i \rangle_{1..n}$ supports the final answer [2405.13872].
- **MVoT**: Autoregressive MLLMs emit alternately blocks of text and image tokens, with explicit loss objectives matching visually coherent generation (e.g., token discrepancy loss for alignment) [2501.07542].

Algorithmically, VoT prompting may involve deterministic pipelines (structured prompt layouts), alternating text/image generation, or backend tool invocation (e.g., external code execution and re-feeding of results).

## 3. Representative VoT Instantiations

**Venn Diagram Prompting (VD):** VD prompting treats the query as a universal set $\xi$ and each document as a set $A,B,C,...$; the model is instructed to partition fact regions—$A\cap\xi$, $A\cap B\cap\xi$, $A\cap\xi^c$—and only use content inside the answer-supporting region (e.g., $\xi' = (D_2\cap D_3) \cup (D_1 \cap D_2 \cap D_3)$) for synthesis [2406.05369].

**Charts-of-Thought:** The prompt stages require models to extract (tabulate) raw chart data, verify via table-image alignment, then execute specified analysis, yielding substantial improvements on VLAT benchmarks and surpassing human performance for several chart types (e.g., Claude-3.7-sonnet: VLAT score 50.17 vs. human 28.82) [2508.04842].

**Whiteboard-of-Thought:** The model generates and consumes its own visualizations (code to image to context), demonstrating up to +92% accuracy gain on tasks (BIG-Bench ASCII, spatial navigation) where chain-of-thought fails ($\Delta\mathrm{Acc}=+92\%$) [2406.14562].

**Mind's Eye VoT:** For multi-hop 2D spatial navigation and visual tiling, interleaved verbal reasoning and model-drawn ASCII grids enable higher accuracy—e.g., GPT-4 VoT route planning success rate 14.72% (vs. 9.48% CoT), next-step accuracy 54.68% (vs. 47.18% CoT) [2404.03622].

**Image-of-Thought:** IoT prompting produces stepwise, tightly coupled visual (image crop, bounding box, segmentation) and textual rationales, increasing MMBench accuracy (GPT-4o: 87.6% with IoT vs. 86.2% CoT) and supporting broad improvements in spatial and knowledge-intensive categories [2405.13872].

**MVoT:** In MLLMs trained for interleaved text and image token sequences, MVoT provides end-to-end visualized reasoning traces and exhibits robust accuracy, particularly on spatially challenging domains (e.g., FrozenLake: CoT 61.48%, MVoT 85.60%) [2501.07542].

## 4. Empirical Benchmarks and Impact

VoT prompting consistently improves performance on tasks requiring spatial reasoning, set operations, or multifaceted data synthesis, particularly in prolonged or knowledge-intensive scenarios.

**Document Synthesis:** VD prompting increased answer correctness on PubMedQA from 0.384 to 0.535 and Long-Context QA from 0.752 to 0.802, with explicit overlap/unique partitioning yielding more comprehensive and position-invariant synthesis [2406.05369].

**Chart Reasoning:** Charts-of-Thought boosted GPT-4.5 VLAT scores by +21.8% and Claude-3.7-sonnet by +13.5% over baseline, with human-level or better accuracy on all major chart types [2508.04842].

**Visual/Spatial Tasks:** Mind’s Eye VoT and MVoT unlocked spatial competence—63.94% accuracy on visual tiling (vs. 54.15% CoT) [2404.03622], and MVoT outperformed text-only approaches by 24 percentage points on FrozenLake [2501.07542].

**Multimodal Rationale:** IoT prompting increased MMBench, MME, and MMVet results across categories, especially those requiring cross-modal alignment or detailed visual inference; ablative removal of visual rationales resulted in 4–7% degradation in knowledge and spatial categories [2405.13872].

**User-Facing Systems:** PrompTHis and iToT leveraged VoT to improve transparency, user intervention, and co-creative processes, facilitating both prompt understanding in generative art and intervention in multi-branch symbolic/logical reasoning [2403.09615][2409.00413].

## 5. Theoretical Analysis and Interpretability

VoT prompts introduce a scaffolding effect—a division of a monolithic inference task into tractable, stepwise sub-tasks—easing working memory load on LLMs and aligning with cognitive science accounts of human problem decomposition [2406.05369][2508.04842]. Position bias, empirically observed in LLMs where early-context tokens disproportionately affect output, is mitigated in VoT frameworks such as VD; set-membership, not linear order, determines relevance and inclusion [2406.05369].

Visualization and explicit structural partitioning also aid traceability and hallucination reduction. In Charts-of-Thought, the verified extraction phase reduces accidental hallucination or misreading of visual data, while in Whiteboard-of-Thought, error analysis localizes failure to image understanding, not upstream generation, suggesting high interpretability [2406.14562][2508.04842].

## 6. Best Practices, Design Patterns, and Generalization

The construction of effective VoT prompts typically involves:

- **Explicit Interleaving:** Alternate text and visual/structural stubs for each reasoning step.
- **Precise Instructions:** Clearly demarcate tasks for extraction, verification, partitioning.
- **Stepwise Verification:** In chart/graph reasoning, verification (as in Charts-of-Thought) is critical to performance.
- **Citation and Provenance:** In document synthesis, tag facts with document identifiers for traceability.
- **Template Adaptation:** Extend VoT scaffolds to any domain with structured rationale—e.g., graphs (adjacency matrices), tables (markdown), audio (spectrograms), spatial navigation (ASCII or image sketches), or tree-of-thought reasoning (node-link diagrams).

Generalization across modalities is evident: VoT concepts have been used to scaffold audio reasoning (spectrograms), prompt-edit history graphs (PrompTHis), and even to propose extensions into robotics (camera-action traces), design (floor plan evolution), and algorithm tracing (memory diagrams) [2403.09615][2405.13872][2501.07542].

## 7. Limitations and Future Research Directions

Despite robust empirical improvements, current limitations of VoT techniques include:

- **Modal Bottlenecks:** Where visualization is text-based (ASCII), representational power is limited to grids/shapes [2404.03622].
- **Visual Perception Reliance:** In Whiteboard-of-Thought, main failure modes relate to MLLM's visual understanding, not the reasoning process per se [2406.14562].
- **Computational Overhead:** Each reasoning hop may require substantial additional computation (e.g., generating and processing 100–300 image tokens per step in MVoT) [2501.07542].
- **Prompt Fragility:** VoT pattern performance can be sensitive to instruction wording (e.g., omission of “reasoning” in Mind’s Eye VoT drops accuracy by 5–10%) [2404.03622].
- **Interactive/Adaptive Design:** Optimal integration of VoT into real-time, user-facing workflows and the design of fully differentiable or learnable VoT submodules remain open challenges [2409.00413][2405.13872].

Prospective avenues include more sophisticated multimodal rendering (video/3D), tighter integration of external tools, learnable rationale selection, and deployment within embodied or interactive systems for fully transparent, adaptive reasoning chains [2501.07542][2405.13872].

Source: https://www.emergentmind.com/topics/visualization-of-thought-vot-prompting