---
title: 'Intermediate Visual Thoughts: IVT Paradigm'
url: https://www.emergentmind.com/topics/intermediate-visual-thoughts
type: topic
---

# Intermediate Visual Thoughts: IVT Paradigm

Intermediate visual thoughts (IVTs) are explicit, manipulable visual representations generated or maintained by intelligent models during the multi-step reasoning process. Unlike static image encodings used as a one-pass context, IVTs function as provisional “sketches,” “diagrams,” “abstracts,” or “intermediate images” in the model’s cognitive workspace, anchoring and guiding subsequent reasoning steps. IVTs may be pixel-space images, symbolic diagrams, sketches, feature maps, latent embeddings, or other structured visualizations. They serve as dynamic intermediaries between language-like planning and high-dimensional visual generation, enabling stepwise spatial reasoning, constraint satisfaction, and the correction of hallucinations. IVTs are now central to a new paradigm—often called thinking with images—in which visual artifacts are not only model outputs but core mediators of intelligent deliberation.

## 1. Foundational Principles and Definitions

Intermediate visual thoughts (IVTs) are formally defined as model-generated visual artifacts that are created, manipulated, or updated at intermediate steps within a composite reasoning process. In contrast to traditional vision–language pipelines, which encode an input image \(I\) to a static vector \(v = \Phi_V(I)\) and perform all reasoning within the language domain \(x_t \sim P(x_t \mid x_{<t}, v, Q)\) [2506.23918], the IVT paradigm interleaves visual artifacts—\(z_t \in \mathcal{T}_{\text{text}} \cup \mathcal{I}_{\text{vis}}\)—such that
\[
z_t \sim P(z_t \mid S_t, I, Q)
\]
where \(S_t\) is the multimodal reasoning state. IVTs may include provisional pixel blueprints from diffusion models [2602.01608], explicit 2D or 3D diagrams [2503.11790, 2601.14602], sketches and abstracts [2505.20164, 2406.09403], or feature-level latent states [2505.15510, 2507.15652]. Their essential characteristics are:

- **Explicitness:** IVTs are directly generated and used by the model, not hidden internal features.
- **Intermediate Position:** IVTs occur at non-terminal steps within a chained or iterative pipeline, not merely as final outputs.
- **Manipulability:** IVTs can be critiqued, refined, or composed—often closing a feedback loop with planning modules.
- **Functionality:** The presence of IVTs enables grounding of logical constraints, supports self-correction, and enhances fidelity in spatial or compositional reasoning.

## 2. Architectures and Generation Methodologies

IVT-centric systems span a rich spectrum of architectural design, which can be organized following the three-stage cognitive autonomy schema [2506.23918]:

| Stage                   | Mechanism                                   | Representative Methods            |
|-------------------------|---------------------------------------------|------------------------------------|
| Tool-driven exploration | Calls to external vision tools, code APIs   | Sketchpad [2406.09403], VisuoThink [2504.09130] |
| Programmatic manipulation | Generation of code to draw or edit images   | Visual Sketchpad [2406.09403], 3D Scratchpad [2601.14602] |
| Intrinsic imagination   | Native interleaved vision-language generation| MVoT [2501.07542], DeepSketcher [2509.25866], SoT [2601.21081], Thinking with Generated Images [2505.22525] |

- **Collaborative Loops:** Autoregressive (language) and diffusion (visual) models can operate in a closed, simulate–criticize–refine cycle, with an LLM generating constraints, a diffusion simulator instantiating an IVT \(R_t\), and a Critic (often a vision–language model) verifying satisfaction against spatial or physical requirements [2602.01608].
- **Explicit Rendering Chains:** Systems such as Visual Sketchpad expose APIs for direct drawing: lines, boxes, masks, markers, invoking specialist detectors or segmenters as needed [2406.09403].
- **Visual Abstracts and Sketches:** Visual Abstract Thinking prompts models using sketches or edge maps that prune irrelevant detail, augmenting expressive yet compact intermediate visual representations [2505.20164].
- **3D Spatial Scratchpads:** 3D workspaces allow models or agentic reasoning pipelines to plan placement, orientation, and camera view, with per-step renders providing visual feedback at each reasoning turn [2601.14602].
- **Latent Embedding Editors:** Internal manipulation of visual-state embeddings—without requiring external rendering—yields highly aligned, compositional visual thoughts and supports differentiable end-to-end training [2509.25866].

## 3. Mathematical and Algorithmic Formalizations

Many IVT frameworks encode the reasoning/feedback cycle via a recurrent state update:
\[
\mathcal{S}_t = \{P_t, R_t, F_t\}
\]
where \(P_t\) is the planning prompt, \(R_t\) is the generated IVT (image), and \(F_t\) is the critic's (vision-language model's) feedback [2602.01608]. The interaction unfolds as:
1. **Planner:** \(P_t = \mathcal{M}_{\text{plan}}(Q, F_{t-1}, H_{t-1})\)
2. **Simulator:** \(R_t \sim p_{\text{world}}(x \mid P_t, \mathcal{C}_t)\)
3. **Critic:** \((v_t, F_t) = \mathcal{M}_{\text{critic}}(R_t, Q)\)

In programmatic systems, code is generated (e.g., Matplotlib snippets), executed, and the resulting image is ingested as the next-step context along with textual state and the action chain [2503.11790].

For autoregressive transformers supporting joint text-vision token streams [2505.22525, 2501.07542], the model's prediction at each step alternates between
\[
\hat{z}_i = P_{\theta}(z_i | x, \hat{z}_1, \hat{v}_1, ..., \hat{z}_{i-1}, \hat{v}_{i-1})
\]
\[
\hat{v}_i = P_{\theta}(v_i | x, \hat{z}_1, \hat{v}_1, ..., \hat{z}_i, \hat{v}_{i-1})
\]
mirroring dual-coding theory.

Visualization-of-Thought (VoT) for text-only LLMs employs prompt engineering to directly elicit text-based ASCII or emoji grid depictions (“mental imagery”), which function as IVTs for spatial tasks [2404.03622].

## 4. Functional Role: Verification, Grounding, and Self-Reflection

IVTs serve several core algorithmic purposes:

- **Grounding Symbolic Constraints:** IVTs embed symbolic constraints into pixel space, enabling structured planners to “see” subgoal outcomes (e.g., geometric constructions, piece arrangements) [2602.01608, 2601.21081].
- **Incremental Error Correction:** Feedback from a Critic (scored \(v_t \in [0,1]\), with accompanying rationale \(F_t\)) guides planner refinement, effectively localizing error propagation before it accumulates [2602.01608].
- **Compositional Integration:** Chain-of-Images (CoI) and Visual Chain-of-Thought (Visual-CoT) pipelines insert IVTs at each reasoning step, transforming complex symbolic operation sequences into concrete pattern recognition and verification [2311.09241, 2511.02779].
- **Mitigation of Hallucinations:** Extraction of “visual factual knowledge” at intermediate transformer layers, as in EVA, allows re-injection of image-derived evidence, counteracting language-prior hallucinations [2507.15652].
- **Planning and Search:** Tree-structured and graph-of-thought frameworks represent each candidate world state as a pair (textual description, rendered diagram), enabling visual self-checks, path-pruning, and backtracking [2504.09130, 2503.11790].
- **Internal Manipulation:** DeepSketcher demonstrates continuous-state IVT modification through an “embedding editor,” moving visual tokens according to natural-language edit actions [2509.25866].

## 5. Empirical Impact and Benchmarking

A consistent empirical finding is that IVTs, even when simulated with gold sketches or diagrams, yield substantial accuracy gains in complex spatial, mathematical, and planning tasks relative to text-only chain-of-thought baselines. Key quantitative results:

- Collaborative Thoughts achieves 100% correct region counts versus <60% for both AR-Only and Diffusion-Only on cutting tasks, and reduces angle task token cost by four orders of magnitude [2602.01608].
- 3D Scratchpad improves text alignment on GenAI-Bench by 32% (0.83 vs 0.63), with pronounced benefits in spatial and compositional attributes [2601.14602].
- MIRA benchmarks report a mean relative accuracy gain of 33.7% when gold IVTs are provided to models; text-only CoT offers only minor or negative improvement on spatial tasks [2511.02779].
- DeepSketcher, with tool-free embedding editing, improves geometry and logic benchmarks by 3-8 points over the state of the art, and attention maps from IVT updates closely track programmatic intent [2509.25866].
- On geometry, graph, and chess benchmarks, Visual Sketchpad yields mean accuracy improvements of 12.7% (math) and 8.6% (vision) [2406.09403].
- In programmatic planning domains, use of conceptual diagrams triples or quadruples plan success rates compared to textual-only or naive search baselines [2503.11790].
- Visual Abstract Thinking (sketches as prompts) delivers +15.4% over no-prompt and outperforms textual CoT and tool-based approaches in visual reasoning [2505.20164].

## 6. Analysis of Limitations and Challenges

Despite the strong foundational case for IVTs, several limitations have emerged:

- **Compounding Errors:** Unified multimodal models often accumulate generation errors in their IVT sequences, resulting in hallucinations, loss of alignment, or semantically inconsistent visuals. Benchmarking with Mentis Oculi demonstrates that compounded IVT errors frequently degrade overall multi-step performance; even when correct intermediate visuals are injected, models may fail to leverage them in decision making [2602.02465].
- **Interpretation vs. Generation Gap:** Models often fail not in generating plausible visuals but in utilizing them for downstream action selection. This reflects architectural deficits in deep visual–textual grounding [2602.02465].
- **Computational Overhead:** Each visual thought imposes non-trivial costs in both token budget and runtime. See [2506.23918] for explicit discussion of token explosion and the need for efficient visual representation schemes.
- **Generalization Rigor:** Many empirical wins arise when models are provided—or simulate—ground-truth or high-fidelity IVTs. Full autonomy in generating necessary, optimal visual intermediates remains a challenge [2511.02779, 2601.21081].

Selected ablation studies show that omitting IVTs or their schemas typically results in steep accuracy declines (from 90.2% to 58% on Blocksworld) [2503.11790]. Qualitative analyses confirm that performance correlates with the clarity and conciseness of the IVT chosen [2505.15510].

## 7. Future Directions and Open Problems

Contemporary research identifies several next steps and unresolved issues:

- **Dynamic Representation:** The search for optimal IVT forms continues: explicit diagrams, sketches, latent maps, 3D scenes, semantic layouts, and hybrid approaches are all active research frontiers [2505.15510, 2601.14602].
- **Interleaved Planning and Perception:** Closing the loop on visual planning at every chain step, especially via joint optimization or end-to-end differentiable architectures, is prioritized [2602.01608, 2509.25866, 2503.11790].
- **Benchmarks and Evaluation:** New benchmarks require the autonomous generation and use of IVTs for compositional reasoning, process-level reward models, and step-fidelity evaluation [2511.02779, 2606.23918].
- **Rigorous Error Analysis:** Understanding, quantifying, and mitigating compounding IVT errors—both in generation and in utilization—remains essential for progress toward robust, human-aligned multimodal intelligence [2602.02465].
- **Cross-modal Causality and Simulation:** Extension of IVTs beyond isolated reasoning into domains requiring visual causal modeling (e.g., dynamics, physics, real-world interaction) is underexplored [2606.23918, 2505.22525].
- **Unified Architectures:** There is an urgent need for native, closed-loop, vision-language frameworks—beyond modular tool invocation—that support efficient, coherent visual–textual chains of thought at scale [2506.23918, 2509.25866, 2601.21081, 2602.01608].

## Summary Table: Major Paradigms and Representative Methods

| Paradigm                 | Mechanism / Key Idea               | Example Paper / Framework               |
|--------------------------|------------------------------------|-----------------------------------------|
| Tool-augmented (Stage 1) | Externally drawn, interpreted IVTs | Visual Sketchpad [2406.09403], VisuoThink [2504.09130] |
| Programmatic (Stage 2)   | Code-based diagram generation      | VisualScratchpad [2406.09403], 3D Scratchpad [2601.14602], Visualizing Thought [2503.11790] |
| Intrinsic imagination    | End-to-end chain with visual tokens| MVoT [2501.07542], DeepSketcher [2509.25866], SoT [2601.21081], Thinking with Generated Images [2505.22525] |

IVTs have fundamentally expanded the computational horizon of multimodal AI, providing both a target for evaluation and a vehicle for compositional, grounded intelligence. Nonetheless, unifying robust, efficient, and fully-autonomous IVT-centric reasoning remains an open frontier.

Source: https://www.emergentmind.com/topics/intermediate-visual-thoughts