---
title: 'DeepSketcher: Multimodal Visual Reasoning'
url: https://www.emergentmind.com/topics/deepsketcher
type: topic
---

# DeepSketcher: Multimodal Visual Reasoning

DeepSketcher is a multimodal reasoning system situated in the “thinking with images” paradigm, defined as a comprehensive suite comprising both an image-text interleaved dataset and a self-contained model. Its central claim is that a Vision Language Model can alternate between natural-language reasoning, free-form visual manipulation instructions, and updated visual context without invoking external tools or repeatedly re-encoding generated images. In DeepSketcher, visual manipulation is internalized in the model’s own embedding space, so intermediate “visual thoughts” become part of the reasoning process rather than an auxiliary execution stage [2509.25866].

## 1. Conceptual framing

DeepSketcher addresses a shift from text-dominant chain-of-thought to image-interactive reasoning. The underlying premise is that a model may improve multimodal reasoning by iteratively attending to fine-grained regions through intermediate visual operations. The paper formalizes the interaction pattern as
\[
(E_{v,0}, Q)\to (R_0,\,\mathtt{Act}_0)\to E_{v,1}\to (R_1,\,\mathtt{Act}_1)\dots\to A,
\]
where \(E_{v,t}\in\mathbb{R}^{K\times D}\) denotes the visual token sequence after \(t\) edits, \(R_t\) is a natural-language reasoning step, and \(\mathtt{Act}_t\) is a free-form edit instruction encoded into an action embedding \(E_{\mathrm{act}}\in\mathbb{R}^{32\times D}\) [2509.25866].

This formulation distinguishes DeepSketcher from conventional tool-calling VLMs. Prior external-tool workflows typically require generating an instruction, executing that instruction outside the model, rendering or editing an image, and then re-encoding the resulting pixels. DeepSketcher instead performs “thinking with images” without external tools: all visual edits are internalized in an embedding editor module, and inference does not require external code execution or pixel re-encoding [2509.25866].

A plausible implication is that DeepSketcher is best understood not as a generic sketch renderer or image editor, but as a reasoning architecture in which visual state transitions are first-class internal variables.

## 2. Image–text interleaved dataset

The DeepSketcher dataset contains 31 K full chain-of-thought trajectories, each alternating between a natural-language reasoning step \(R_t\), a free-form “tool call” edit instruction \(\mathtt{Act}_t\), and a code-rendered image \(I_{t+1}\) obtained by editing the original rendering code. In total, it includes over 59 K visual manipulations spanning broad operation types [2509.25866].

The annotation pipeline has two rounds. In the seed stage, two proprietary LLM experts, Solver and Code Editor, collaborate on 6 K examples from CoSyn-400K to generate gold-standard traces. In the expansion stage, a fine-tuned intermediate model bootstraps a further 25 K trajectories via the same agentic loop at far lower API cost. Code-based editing ensures 100% repeatability and eliminates spatial-grounding noise. Quality control uses multi-model verification with GPT-4.1, Claude 3.7, and Qwen2.5-VL-72B to filter and confirm answer correctness, and the construction pipeline also includes syntax/render checks, LLM self-critique, back-off edits, and rejection sampling to drop “easy” examples solved by the base model [2509.25866].

The dataset covers geometry, chart and diagram reasoning, color and annotation tasks, counting and spatial reasoning, and logic puzzles and combinatorial tasks. The paper reports at least nine broad manipulation categories with the following proportions [2509.25866]:

| Category | Share |
|---|---:|
| Labeling/Annotation | 20.9% |
| Highlighting | 17.7% |
| Color Operations | 12.5% |
| Circle Drawing | 11.8% |
| Line Drawing | 11.7% |
| Point Marking | 6.6% |
| Area/Region Operations | 4.5% |
| Shape Drawing | 4.3% |
| Others | 4.3% |

The training corpus totals 31 K trajectories, consisting of 6 K seed and 25 K expanded examples. A held-out in-house “Indicator-500” test set of 500 code-rendered VQA examples is used for editor ablations. Inputs are standardized to sequences of text segments \(\mathcal{S}_t\) interleaved with image tokens \(E_{v,t}\) [2509.25866].

## 3. Model architecture and embedding editor

DeepSketcher uses Qwen2.5-VL-7B as the backbone visual-language transformer. Its visual encoder is a frozen image transformer that produces \(K\times D\) tokens, while a text encoder and autoregressive LLM head generate the reasoning tokens \(R_t\) and edit instructions \(\mathtt{Act}_t\). The architectural novelty lies in the Embedding Editor, a Q-Former–style cross-attention module that feeds action embeddings into visual token updates [2509.25866].

The editor operator is written as
\[
E_v^{(t+1)} \;=\; E_v^{(t)} \;+\; f_{\theta}\bigl(E_v^{(t)},\,E_{\mathrm{act}^{(t)}}\bigr).
\]
In practice, \(f_{\theta}\) is implemented via one or more cross-attention plus FFN layers. Repeated stacking yields \(E_V^{\mathrm{pred}}\), a manipulated embedding that becomes the next visual context [2509.25866].

This design makes “visual thoughts” latent rather than pixelized. The model does not need to render an image after every intermediate action, even though the training data are derived from code-rendered edits. A plausible implication is that DeepSketcher converts tool-mediated visual operations into a learned state-transition mechanism over visual tokens, thereby compressing tool use into the backbone’s own internal dynamics.

The paper provides a geometric VQA example in which the model first requests “draw tangent line at point A,” updates the visual embedding, and then continues reasoning from the manipulated visual state. The resulting trace follows the pattern
\[
R_0:\text{“I see circle \(C\)…”}\to \mathtt{<tool\_call>\ draw\ tangent\ at\ A}\to E_{v,1}\to R_1:\text{“Now I notice the two tangents….”}
\]
rather than a purely textual chain-of-thought [2509.25866].

## 4. Objectives and training regimen

DeepSketcher is trained in three stages on 8 \(\times\) A100. Stage 1 warms up the LLM on the 6 K seed set for 5 epochs with learning rate \(5\mathrm{e}{-6}\), while freezing the editor and visual encoder. Stage 2 trains the embedding editor on the full 31 K dataset for 10 epochs with learning rate \(1\mathrm{e}{-4}\), while freezing the LLM. Stage 3 jointly fine-tunes the LLM and editor for 2 epochs with learning rate \(5\mathrm{e}{-6}\). The paper also notes data augmentation via the img2code pipeline to enrich diversity [2509.25866].

The optimization objective is split across phases. Phase 1 uses an LM warm-up loss over the interleaved sequence:
\[
\mathcal{L}^{\mathrm{phase\text{-}1}}_{\mathrm{LM}}
= -\frac{1}{\sum_i|\mathcal S^{(i)}|}
\sum_{i=1}^N
\sum_{t=0}^{T^{(i)}-1}
\sum_{\tau\in\mathcal{S}^{(i)}_t}
\log P_\theta\!\bigl(x_\tau^{(i)}\mid x_{<\tau}^{(i)},\,E_{v,\le t}^{(i)}\bigr).
\]

Phase 2 combines editor supervision with language modeling:
\[
\mathcal{L}^{\mathrm{phase\text{-}2}}(\theta)
= \|\,E_V^{\mathrm{pred}} - E_V^{\mathrm{gt}}\,\|_1
+\mathcal{L}^{\mathrm{phase\text{-}2}}_{\mathrm{LM}}(\theta),
\]
where the LM term has the same form as the Phase 1 objective, except that the LM now conditions on editor-produced embeddings [2509.25866].

The training strategy separates linguistic adaptation, editor learning, and joint alignment. This suggests that DeepSketcher treats visual-edit internalization as a specialized capability that benefits from being learned after a text-and-vision backbone has already been stabilized on seed trajectories.

## 5. Empirical behavior

Evaluation is reported on five public benchmarks—MathVerse, MathVision (mini), MathVista (mini), LogicVista, and WeMath—using overall accuracy, together with the in-house Indicator-500 set using pass@1 accuracy [2509.25866].

Against the Qwen2.5-VL-7B baseline, DeepSketcher-7B improves average performance by +3.9 points across the five public benchmarks. The gains are especially pronounced on geometry and counting tasks in MathVision, where the paper reports +5.3, and on logical reasoning in LogicVista, where it reports +8.3. In comparisons to tool-calling VLMs such as VILASR and DeepEyes and to “inner-visual-thought” models such as Zebra-CoT and Mirage, DeepSketcher is described as being within 1–2 points of GPT-4.1 on several tasks and as outperforming all prior open synthesizers [2509.25866].

The ablations are central to the paper’s argument. On a CoSyn-400K subset, agentic curation improves pass@8 for Qwen2.5-VL-72B from 0.67 to 0.72 when paired with Code Editor, and for GPT-4.1 from 0.72 to 0.80. On Indicator-500 and the public sets, the Stage 3 editor improves average performance by +3.8 points over a text-only model and closes the gap to the “agentic oracle” [2509.25866].

The failure analysis is correspondingly specific. The paper reports approximate attention deviations, where the editor sometimes attends to a wrong region when instructions are ambiguous. It also notes that open-world images, exemplified by MathVista cases with faces, can partially generalize but still degrade [2509.25866]. These observations indicate that the model’s internal visual editing is effective in code-rendered domains but remains sensitive to grounding ambiguity and domain shift.

## 6. Relation to adjacent sketch and visual-manipulation research

DeepSketcher belongs to a broader research landscape in which sketches, strokes, or intermediate visual states are used to guide generation, editing, or reasoning, but its problem definition is distinct from several adjacent systems.

FaceShop is an interactive system for sketch-based face image editing. It uses a web-based interface with geometry strokes, color strokes, a mask tool, and an iris tool; its workflow recomputes a forward pass in a conditional completion network on each stroke event, at approximately 60 ms on GPU, and displays the updated composite. It combines image completion and image translation in a unified framework for interactive image editing at \(512\times512\) resolution [1804.08972]. DeepSketcher differs in target task and representation: it is not an interactive pixel editor for face regions, but a VLM that performs interleaved image-text reasoning through latent visual-token updates [2509.25866].

DiffSketching addresses sketch-to-image synthesis with diffusion models. It fine-tunes a class-conditional diffusion U-Net under a sketch perceptual loss \(L_p\) and an image identity loss \(L_i\), uses a frozen photo-sketching network and pretrained feature extractors, and reports Sketchy test-set results including FID \(=6.46\) and IS \(=89.91\) [2305.18812]. Painter takes a different route by teaching auto-regressive language models to generate virtual brush strokes directly as XML-like textual snippets, using OPT-125M and OPT-1.3B backbones with residual cross-attention to frozen ResNet-50 features [2308.08520]. Self-Supervised Sketch-to-Image Synthesis studies exemplar-based sketch-to-image synthesis through synthetic sketch generation, a two-encoder auto-encoder for content/style disentanglement, and a refinement GAN, reporting CelebA-HQ \(1024^2\) FID \(13.6\) for AE+GAN [2012.09290]. Deep Sketch-Based Modeling, by contrast, focuses on single-sketch 3D reconstruction and identifies style variance, imprecise perspective, and sparsity as the principal obstacles for adapting image-based methods to sketch input [2011.06133].

Taken together, these systems show several non-identical roles for sketches in modern vision research: as user controls for localized image editing, as sparse conditions for image synthesis, as sequential stroke programs, and as inputs to 3D reconstruction. DeepSketcher extends the motif into multimodal reasoning by treating visual manipulation itself as an internal operation of the VLM rather than as a preprocessing, rendering, or postprocessing step [2509.25866].

## 7. Limitations, extensions, and broader significance

The paper’s proposed extensions are aligned with its current constraints. One direction is to move beyond code-rendered domains to mixed real-world imagery via learned “img2edit” modules. Another is to expose intermediate “visual thoughts” as images, for example through latent-to-pixel decoders, in order to improve interpretability [2509.25866].

The broader applications named in the paper include educational tools that interactively sketch proofs or diagrams, as well as extensions to 3D scene understanding, chart editing, and robotic manipulation instructions. The central open question is how to learn reliably from noisier, human-authored data while maintaining the controllability afforded by code space [2509.25866].

A plausible implication is that DeepSketcher reframes multimodal reasoning around a learned edit algebra over visual tokens. Its significance lies less in producing final images than in showing that intermediate visual manipulations can be trained as internal computational steps, with measurable gains on geometry, counting, and logic benchmarks [2509.25866].

Source: https://www.emergentmind.com/topics/deepsketcher