DeepSketcher: Multimodal Visual Reasoning

• DeepSketcher is a multimodal system that integrates natural language and latent visual edits, redefining reasoning through internal state transitions.
• It employs an innovative Embedding Editor to update visual token embeddings, eliminating the need for external image rendering or tool calls.
• The model’s training strategy and dataset curation yield measurable gains in geometry, counting, and logical reasoning tasks across benchmarks.

DeepSketcher is a multimodal reasoning system situated in the “thinking with images” paradigm, defined as a comprehensive suite comprising both an image-text interleaved dataset and a self-contained model. Its central claim is that a Vision LLM can alternate between natural-language reasoning, free-form visual manipulation instructions, and updated visual context without invoking external tools or repeatedly re-encoding generated images. In DeepSketcher, visual manipulation is internalized in the model’s own embedding space, so intermediate “visual thoughts” become part of the reasoning process rather than an auxiliary execution stage (Zhang et al., 30 Sep 2025).

1. Conceptual framing

DeepSketcher addresses a shift from text-dominant chain-of-thought to image-interactive reasoning. The underlying premise is that a model may improve multimodal reasoning by iteratively attending to fine-grained regions through intermediate visual operations. The paper formalizes the interaction pattern as

$$(E_{v,0}, Q)\to (R_0,\,\mathtt{Act}_0)\to E_{v,1}\to (R_1,\,\mathtt{Act}_1)\dots\to A,$$

where $E_{v,t}\in\mathbb{R}^{K\times D}$ denotes the visual token sequence after $t$ edits, $R_t$ is a natural-language reasoning step, and $\mathtt{Act}_t$ is a free-form edit instruction encoded into an action embedding $E_{\mathrm{act}}\in\mathbb{R}^{32\times D}$ (Zhang et al., 30 Sep 2025).

This formulation distinguishes DeepSketcher from conventional tool-calling VLMs. Prior external-tool workflows typically require generating an instruction, executing that instruction outside the model, rendering or editing an image, and then re-encoding the resulting pixels. DeepSketcher instead performs “thinking with images” without external tools: all visual edits are internalized in an embedding editor module, and inference does not require external code execution or pixel re-encoding (Zhang et al., 30 Sep 2025).

A plausible implication is that DeepSketcher is best understood not as a generic sketch renderer or image editor, but as a reasoning architecture in which visual state transitions are first-class internal variables.

2. Image–text interleaved dataset

The DeepSketcher dataset contains 31 K full chain-of-thought trajectories, each alternating between a natural-language reasoning step $R_t$, a free-form “tool call” edit instruction $\mathtt{Act}_t$, and a code-rendered image $I_{t+1}$ obtained by editing the original rendering code. In total, it includes over 59 K visual manipulations spanning broad operation types (Zhang et al., 30 Sep 2025).

The annotation pipeline has two rounds. In the seed stage, two proprietary LLM experts, Solver and Code Editor, collaborate on 6 K examples from CoSyn-400K to generate gold-standard traces. In the expansion stage, a fine-tuned intermediate model bootstraps a further 25 K trajectories via the same agentic loop at far lower API cost. Code-based editing ensures 100% repeatability and eliminates spatial-grounding noise. Quality control uses multi-model verification with GPT-4.1, Claude 3.7, and Qwen2.5-VL-72B to filter and confirm answer correctness, and the construction pipeline also includes syntax/render checks, LLM self-critique, back-off edits, and rejection sampling to drop “easy” examples solved by the base model (Zhang et al., 30 Sep 2025).

The dataset covers geometry, chart and diagram reasoning, color and annotation tasks, counting and spatial reasoning, and logic puzzles and combinatorial tasks. The paper reports at least nine broad manipulation categories with the following proportions (Zhang et al., 30 Sep 2025):

Category	Share
Labeling/Annotation	20.9%
Highlighting	17.7%
Color Operations	12.5%
Circle Drawing	11.8%
Line Drawing	11.7%
Point Marking	6.6%
Area/Region Operations	4.5%
Shape Drawing	4.3%
Others	4.3%

The training corpus totals 31 K trajectories, consisting of 6 K seed and 25 K expanded examples. A held-out in-house “Indicator-500” test set of 500 code-rendered VQA examples is used for editor ablations. Inputs are standardized to sequences of text segments $\mathcal{S}_t$ interleaved with image tokens $E_{v,t}\in\mathbb{R}^{K\times D}$0 (Zhang et al., 30 Sep 2025).

3. Model architecture and embedding editor

DeepSketcher uses Qwen2.5-VL-7B as the backbone visual-language transformer. Its visual encoder is a frozen image transformer that produces $E_{v,t}\in\mathbb{R}^{K\times D}$1 tokens, while a text encoder and autoregressive LLM head generate the reasoning tokens $E_{v,t}\in\mathbb{R}^{K\times D}$2 and edit instructions $E_{v,t}\in\mathbb{R}^{K\times D}$3. The architectural novelty lies in the Embedding Editor, a Q-Former–style cross-attention module that feeds action embeddings into visual token updates (Zhang et al., 30 Sep 2025).

The editor operator is written as

$E_{v,t}\in\mathbb{R}^{K\times D}$4

In practice, $E_{v,t}\in\mathbb{R}^{K\times D}$5 is implemented via one or more cross-attention plus FFN layers. Repeated stacking yields $E_{v,t}\in\mathbb{R}^{K\times D}$6, a manipulated embedding that becomes the next visual context (Zhang et al., 30 Sep 2025).

This design makes “visual thoughts” latent rather than pixelized. The model does not need to render an image after every intermediate action, even though the training data are derived from code-rendered edits. A plausible implication is that DeepSketcher converts tool-mediated visual operations into a learned state-transition mechanism over visual tokens, thereby compressing tool use into the backbone’s own internal dynamics.

The paper provides a geometric VQA example in which the model first requests “draw tangent line at point A,” updates the visual embedding, and then continues reasoning from the manipulated visual state. The resulting trace follows the pattern

$E_{v,t}\in\mathbb{R}^{K\times D}$7

rather than a purely textual chain-of-thought (Zhang et al., 30 Sep 2025).

4. Objectives and training regimen

DeepSketcher is trained in three stages on 8 $E_{v,t}\in\mathbb{R}^{K\times D}$8 A100. Stage 1 warms up the LLM on the 6 K seed set for 5 epochs with learning rate $E_{v,t}\in\mathbb{R}^{K\times D}$9, while freezing the editor and visual encoder. Stage 2 trains the embedding editor on the full 31 K dataset for 10 epochs with learning rate $t$0, while freezing the LLM. Stage 3 jointly fine-tunes the LLM and editor for 2 epochs with learning rate $t$1. The paper also notes data augmentation via the img2code pipeline to enrich diversity (Zhang et al., 30 Sep 2025).

The optimization objective is split across phases. Phase 1 uses an LM warm-up loss over the interleaved sequence: $t$2

Phase 2 combines editor supervision with language modeling: $t$3 where the LM term has the same form as the Phase 1 objective, except that the LM now conditions on editor-produced embeddings (Zhang et al., 30 Sep 2025).

The training strategy separates linguistic adaptation, editor learning, and joint alignment. This suggests that DeepSketcher treats visual-edit internalization as a specialized capability that benefits from being learned after a text-and-vision backbone has already been stabilized on seed trajectories.

5. Empirical behavior

Evaluation is reported on five public benchmarks—MathVerse, MathVision (mini), MathVista (mini), LogicVista, and WeMath—using overall accuracy, together with the in-house Indicator-500 set using pass@1 accuracy (Zhang et al., 30 Sep 2025).

Against the Qwen2.5-VL-7B baseline, DeepSketcher-7B improves average performance by +3.9 points across the five public benchmarks. The gains are especially pronounced on geometry and counting tasks in MathVision, where the paper reports +5.3, and on logical reasoning in LogicVista, where it reports +8.3. In comparisons to tool-calling VLMs such as VILASR and DeepEyes and to “inner-visual-thought” models such as Zebra-CoT and Mirage, DeepSketcher is described as being within 1–2 points of GPT-4.1 on several tasks and as outperforming all prior open synthesizers (Zhang et al., 30 Sep 2025).

The ablations are central to the paper’s argument. On a CoSyn-400K subset, agentic curation improves pass@8 for Qwen2.5-VL-72B from 0.67 to 0.72 when paired with Code Editor, and for GPT-4.1 from 0.72 to 0.80. On Indicator-500 and the public sets, the Stage 3 editor improves average performance by +3.8 points over a text-only model and closes the gap to the “agentic oracle” (Zhang et al., 30 Sep 2025).

The failure analysis is correspondingly specific. The paper reports approximate attention deviations, where the editor sometimes attends to a wrong region when instructions are ambiguous. It also notes that open-world images, exemplified by MathVista cases with faces, can partially generalize but still degrade (Zhang et al., 30 Sep 2025). These observations indicate that the model’s internal visual editing is effective in code-rendered domains but remains sensitive to grounding ambiguity and domain shift.

6. Relation to adjacent sketch and visual-manipulation research

DeepSketcher belongs to a broader research landscape in which sketches, strokes, or intermediate visual states are used to guide generation, editing, or reasoning, but its problem definition is distinct from several adjacent systems.

FaceShop is an interactive system for sketch-based face image editing. It uses a web-based interface with geometry strokes, color strokes, a mask tool, and an iris tool; its workflow recomputes a forward pass in a conditional completion network on each stroke event, at approximately 60 ms on GPU, and displays the updated composite. It combines image completion and image translation in a unified framework for interactive image editing at $t$4 resolution (Portenier et al., 2018). DeepSketcher differs in target task and representation: it is not an interactive pixel editor for face regions, but a VLM that performs interleaved image-text reasoning through latent visual-token updates (Zhang et al., 30 Sep 2025).

DiffSketching addresses sketch-to-image synthesis with diffusion models. It fine-tunes a class-conditional diffusion U-Net under a sketch perceptual loss $t$5 and an image identity loss $t$6, uses a frozen photo-sketching network and pretrained feature extractors, and reports Sketchy test-set results including FID $t$7 and IS $t$8 (Wang et al., 2023). Painter takes a different route by teaching auto-regressive LLMs to generate virtual brush strokes directly as XML-like textual snippets, using OPT-125M and OPT-1.3B backbones with residual cross-attention to frozen ResNet-50 features (Pourreza et al., 2023). Self-Supervised Sketch-to-Image Synthesis studies exemplar-based sketch-to-image synthesis through synthetic sketch generation, a two-encoder auto-encoder for content/style disentanglement, and a refinement GAN, reporting CelebA-HQ $t$9 FID $R_t$0 for AE+GAN (Liu et al., 2020). Deep Sketch-Based Modeling, by contrast, focuses on single-sketch 3D reconstruction and identifies style variance, imprecise perspective, and sparsity as the principal obstacles for adapting image-based methods to sketch input (Zhong et al., 2020).

Taken together, these systems show several non-identical roles for sketches in modern vision research: as user controls for localized image editing, as sparse conditions for image synthesis, as sequential stroke programs, and as inputs to 3D reconstruction. DeepSketcher extends the motif into multimodal reasoning by treating visual manipulation itself as an internal operation of the VLM rather than as a preprocessing, rendering, or postprocessing step (Zhang et al., 30 Sep 2025).

7. Limitations, extensions, and broader significance

The paper’s proposed extensions are aligned with its current constraints. One direction is to move beyond code-rendered domains to mixed real-world imagery via learned “img2edit” modules. Another is to expose intermediate “visual thoughts” as images, for example through latent-to-pixel decoders, in order to improve interpretability (Zhang et al., 30 Sep 2025).

The broader applications named in the paper include educational tools that interactively sketch proofs or diagrams, as well as extensions to 3D scene understanding, chart editing, and robotic manipulation instructions. The central open question is how to learn reliably from noisier, human-authored data while maintaining the controllability afforded by code space (Zhang et al., 30 Sep 2025).

A plausible implication is that DeepSketcher reframes multimodal reasoning around a learned edit algebra over visual tokens. Its significance lies less in producing final images than in showing that intermediate visual manipulations can be trained as internal computational steps, with measurable gains on geometry, counting, and logic benchmarks (Zhang et al., 30 Sep 2025).