Beyond NL2Code: A Structured Survey of Multimodal Code Intelligence

Abstract: While LLMs have substantially advanced text-to-code synthesis, many real programming tasks specify intent through visual artifacts such as screenshots, charts, documents, vector drawings, videos, and interactive states. These tasks require models to connect visual perception to executable programs, because correctness depends not only on syntax but also on layout, geometry, data semantics, editability, interaction behavior, and domain-specific constraints that apply after execution. This survey examines Multimodal Code Intelligence, covering systems that generate, edit, refine, execute, or reason with code under visually grounded inputs and outputs. We first formulate the field by the role that code plays in each task, distinguishing code as a rendered artifact, an editable symbolic structure, a scientific representation, an intermediate reasoning trace, or an executable policy or tool interface. We then organize benchmarks and methods into four domains: Graphical User Interface, Scientific Visualization, Structured Graphics, and Frontier Tasks and Frameworks. This taxonomy connects mature artifact-generation problems to emerging agentic and unified settings and allows us to compare how different tasks treat evidence of correctness. Looking ahead, we argue that future research may benefit from four verification-centered directions. Multi-signal validation can combine complementary evidence of correctness, multi-state verification can test behavior across execution trajectories, cross-task transfer testing can probe reusable visual-code skills, and verifiable agent traces can reveal whether agent actions are grounded in visual evidence. Together, these directions may move multimodal code generation from single-output imitation toward evidence-grounded executable systems.

• The paper introduces a structured taxonomy of Multimodal Code Intelligence spanning direct generation, code editing, refinement, programmatic tool use, and executable policies across four major domains.
• The paper shows that visual similarity is an incomplete correctness measure, with IWR-Bench models reaching 64.25% visual fidelity but only 24.39% interactive-function accuracy.
• The paper proposes multi-signal and multi-state validation, cross-task transfer tests, and verifiable agent traces to assess semantic fidelity, structure, editability, execution, and interaction reliability.

Scope and motivation

This survey, authored by a consortium spanning Meituan and several academic institutions (2606.15932), addresses the gap between text-driven code generation (NL2Code) and the growing class of programming tasks whose intent is specified through visual artifacts: screenshots, charts, documents, vector drawings, videos, and interactive states. The authors term this area Multimodal Code Intelligence (MCI), covering systems that generate, edit, refine, execute, or reason with code under visually grounded inputs and outputs. The literature snapshot extends through January 2026, with emphasis on 2022–2026 work; the survey is maintained alongside a continuously updated repository of papers and benchmarks.

The survey's central organizing claim is that correctness in these tasks cannot be reduced to visual similarity. A generated webpage can match a reference screenshot while omitting event handlers; a chart can render plausibly while encoding wrong data; an SVG can look correct while being non-editable; a CAD program can compile while violating design constraints. This observation motivates both the taxonomy and the verification-centered agenda that structures the paper.

Task formulation

The authors formalize MCI by extending the NL2Code mapping $\mathcal{C} = \operatorname{LLM}(\mathcal{T})$ to settings where visual context $\mathcal{I}$, rendered feedback, or visually specified intent is central. Three synthesis sub-tasks are distinguished:

• Multimodal direct generation: $$\mathcal{C}_{\text{gen}} = \operatorname{MLLM}(\mathcal{I}, \mathcal{T}_{\text{desc}})$$, where the bottleneck is visual fidelity after execution.
• Instruction-driven code editing: modification of visual content via textual instructions or visual prompts (bounding boxes, encircled regions), optionally source-agnostic.
• Reference-based code refinement: correction of a flawed draft $\mathcal{C}_{\text{draft}}$ against a visual reference, using the draft as a structural prior.

Beyond synthesis, two reasoning paradigms are formalized. In programmatic tool-use, code serves as an intermediate trace—either solved directly via execution or used to transform the image into an augmented view for subsequent inference. In executable policy, code maps observations and goals to environment actions, as in embodied control and GUI navigation.

The survey also introduces a role-based classification of code: rendered artifact, editable symbolic structure, scientific representation, intermediate reasoning trace, or executable policy/tool interface. This role determines which evidence signals are meaningful for validation—a framing that recurs throughout the domain reviews.

Graphical user interface

The GUI domain covers web and mobile application generation. Web-to-code benefits from a closed loop—HTML/CSS/JavaScript, browser rendering, WebDriver interaction—that makes evaluation practical but biases it toward appearance-level correctness when only rendering is observed. Benchmarks are grouped by correctness signal: static benchmarks (WebSight, Design2Code, WebCode2M) evaluate structural and visual alignment; dynamic benchmarks (Interaction2Code, MRWeb, Web-Bench, IWR-Bench, WebGen-Bench) test functionality through execution; specialized arenas assess human preference or agent-task performance.

The strongest quantitative evidence for the survey's thesis comes from this domain: on IWR-Bench, models achieve 64.25% visual fidelity but only 24.39% interactive function. The implication is direct—rendered similarity and interaction correctness measure different capabilities, and optimizing visual overlap alone rewards implementations that fail under user actions. Methodologically, the field has progressed from SFT on synthetic pairs (WebSight, Web2Code), through multi-agent decomposition (ScreenCoder, Frontend Diffusion, TDDev), to feedback-driven optimization (WebGen-Agent, ReLook, UI2Code$^N$) and agent-centric evaluation (Coder-CUA), where a Computer-Use Agent verifies functional correctness by executing tasks rather than comparing screenshots.

Mobile generation lacks this execution substrate: native apps are compiled binaries without crawlable source or shared runtimes. Current benchmarks therefore rely on proxies—APPUI uses mockup reconstruction (1.1k image-code pairs across 12 categories), CANVAS targets Figma tool workflows (598 tasks from 3.3k designs), UICrit provides critique annotations, and UIClip supplies learned screenshot-quality scores. None directly verifies native runtime behavior, a structural constraint the authors state plainly rather than treat as solvable within current evaluation practice.

Scientific visualization

This domain spans statistical charts, structured documents, academic presentations, and scientific demonstrations, unified by the requirement that generated code preserve scientific semantics—not merely rendered appearance.

Statistical charts split into asymmetric formulations. NL-to-Chart is under-specified synthesis evaluated against analytic intent (nvBench, MatPlotBench, VisEval, PlotCraft); Chart-to-Code is constrained reconstruction evaluated against data semantics and visual encodings (ChartX, Plot2Code, ChartMimic). Methods have moved from SFT on chart-code pairs (ChartCoder with its Snippet-of-Thought strategy on Chart2Code-160k) toward preference-driven optimization against executed outcomes: DualDPO applies DPO with synthesized preferences, while MSRL and ChartMaster use GRPO with multimodal reward feedback. The authors flag a persistent caveat: rewards based mainly on visual similarity can prefer plausible renderings containing wrong recovered data, incorrect grouping, or misleading axes.

Structured documents pose a multi-grammar recovery problem—reading order, table grids, formula trees, and cross-region references must survive serialization into Markdown, HTML, or LaTeX. Benchmarks progress from BLEU-based evaluation (IM2LaTeX-100K) through structural edit distance (TEDS in PubTabNet) to render-based verification (Tab2LaTeX, Table2LaTeX-RL), reflecting recognition that textual similarity does not guarantee executable correctness. Methods span pipeline OCR, end-to-end VLMs (dots.ocr, DeepSeek-OCR), and RL with verifiable rewards (olmOCR 2's binary unit tests, FD-RL's format-decoupled objectives).

Academic presentations shift evaluation from source reconstruction to communication design. Slide benchmarks (SlidesBench, Zenodo10K with PPTEval scoring, TSBench) and poster benchmarks (Paper2Poster with its Paper Quiz reader-simulation metric, P2PEval) provide partial evidence of rhetorical sequencing and communicative effectiveness, but the authors note that whether a poster foregrounds the intended argument remains open.

Scientific demonstration is identified as the most validation-sensitive subdomain: a molecule can render cleanly while violating chemistry, and an animation can look intuitive while proving the wrong theorem. InteractScience combines programmatic functional testing with visually-grounded qualitative testing; TheoremExplainBench evaluates Manim-generated theorem animations.

Structured graphics

Structured graphics—SVG, diagrams, CAD—instantiate what the authors call the domain's central failure mode: structural non-equivalence, where outputs render plausibly while losing object hierarchy, relation graphs, or parametric construction history.

For SVG, methods divide into differentiable-rasterizer optimization (DiffVG, LIVE, VectorFusion), neural sequence models (DeepSVG, IconShop), and LLM-era systems adding data scale, reasoning traces, and rendering feedback (StarVector, Chat2SVG, RLRF's rendering-aware RL). The dominant path-based representation is characterized as low-level and weakly aligned with semantic objects, explaining why no single method family resolves the fidelity–editability tradeoff.

For diagrams, the task is logic compilation rather than drawing: reversed arrows, missing branch conditions, or changed relation types can hide behind high pixel similarity. Flow2Code evaluates flowchart-to-code translation via Pass@k over 1.6k samples in 15 languages; StarFlow tests sketch-to-workflow JSON; MMVG extends to Verilog from block diagrams. The authors caution that synthetic leakage—models learning renderer templates without preserving relation graphs—must be controlled.

For CAD, the target is parametric correctness: recovering construction logic, constraints, and feature dependencies, not surface geometry. Methods progress from command-sequence reconstruction (DeepCAD, ~8k test samples) through executable CadQuery scripts (Query2CAD, CAD-Coder with GRPO, CAD-Recode from point clouds) to feedback-aware pipelines (CADFusion's SFT+DPO, CAD-Judge's Compiler-as-a-Judge, ReCAD's SFT-RL). Compilation success is explicitly positioned as necessary but weaker than verified geometric validity, constraint satisfaction, and editability.

Frontier tasks and frameworks

Five frontier settings stress process reliability rather than artifact quality.

Programmatic visual manipulation treats code as an executable interface for inspecting visual evidence ("Thinking with Image"). Evaluation signals—final-answer accuracy, trace executability, process rewards—are all indirect: none proves a crop, mask, or sketch is causally responsible for the answer. The authors propose operation replay, region grounding, evidence ablation, and counterfactual-image tests as stronger protocols.

Video code generation splits into Code-to-Video (authoring scaffold; MMMC, PresentEval) and Video-to-Code (policy recovery; Video2Code with 115k triplets). Both directions share a discrete–continuous mismatch: code specifies sparse anchors while interpolation, pacing, and motion dynamics remain delegated to renderers or discarded during abstraction. A manipulation policy can succeed at a task while discarding velocity, contact timing, and recovery behavior from the demonstration.

Embodied control instantiates the executable-policy formulation. Code as Policies, ProgPrompt, RoboScript, and EmbodiedCoder emit inspectable action scripts; VLM-CaR generates dense reward functions; Octopus applies RL with environmental feedback. The central bottleneck is physical grounding, and the key open question is where to place the boundary between symbolic planning and continuous control—binary simulator rewards can hide unsafe intermediate actions and simulator-specific affordances.

Visually grounded programming exposes a compression bottleneck: visual context is often converted into textual surrogates before code synthesis, risking loss of spatial relations, graph topology, and UI state. HumanEval-V makes images indispensable (253 tasks); SWE-bench MM compiles 617 JavaScript repair tasks with visual scenarios; CodeV filters 133 repository tasks requiring visual cues. Methods either convert visuals to structured intermediates (Code-Vision's Mermaid conversion) or use visual feedback inside repair agents (GUIRepair's Image2Code/Code2Image loop). Pass rates remain weak grounding evidence if repository context permits text-only shortcuts—an ablation concern the authors emphasize.

Unified multimodal code generation confronts a generalization paradox: broader task coverage does not demonstrate transferable visual-code abstractions. Systems such as JanusCoder (JanusCode-800K), VisCoder2 (VisCode-Multi-679K), and VisCodex (MCD-598k) unify via data mixtures and model merging; VinciCoder adds coarse-to-fine visual-similarity rewards. The unresolved assumption, stated explicitly, is that training on many code-image-instruction tuples induces reusable notions of axes, panels, nodes, layout hierarchy, and state change—current reports validate broad task acceptance more often than controlled primitive-level transfer.

Verification-centered future directions

The survey consolidates its per-domain findings into four research directions:

1. Multi-signal validation: a diagnostic profile separating visual similarity, execution success, semantic fidelity, structural validity, editability, and interaction correctness, rather than a single scalar reward. The authors catalog proxy judges (visual, text/code, preference, agent replay, trace) with their characteristic failure modes—for instance, visual judges over-score surface match, and preference judges are prompt-sensitive and weakly reproducible.
2. Multi-state verification: evaluating execution episodes—initial states, actions, transitions, validator outputs, recovery cases—rather than isolated renderings, with substrate-specific checks (DOM assertions, emulator gestures, synchronization checks, controller diagnostics).
3. Cross-task transfer testing: minimal protocols comparing base mixtures, source-augmented mixtures, and matched-size controls on provenance-filtered targets, reporting positive and negative transfer so unification becomes falsifiable.
4. Verifiable agent traces: evidence logs recording cited visual regions, changed code regions, expected validators, replay results, and rollback decisions, enabling failure attribution across perception, synthesis, execution, and action selection.

Limitations

The authors acknowledge several constraints. Coverage is bounded by publicly available artifacts; benchmark-proposing works are likely overrepresented relative to deployed systems lacking public releases. Cross-method comparison is limited because benchmarks observe different slices of correctness, so the survey deliberately avoids universal rankings in favor of within-domain comparison and failure-mode analysis. Closed-source reports, private evaluation sets, and rapidly changing arXiv releases limit reproducibility. The cross-task transfer discussion is explicitly agenda-setting: current evaluations rarely isolate causal transfer, and deployment-facing concerns remain underexplored.

Conclusion

This survey organizes Multimodal Code Intelligence into four domains—Graphical User Interface, Scientific Visualization, Structured Graphics, and Frontier Tasks and Frameworks—unified by a role-based formulation of code and a consistent evaluation lens. Its principal analytical contribution is the demonstration, supported by concrete evidence such as the 64.25%-versus-24.39% fidelity-function gap on IWR-Bench, that visual similarity is a useful lower bound but an unreliable proxy for correctness across interaction behavior, data semantics, symbolic structure, and parametric constraints. The proposed agenda—multi-signal validation, multi-state verification, cross-task transfer testing, and verifiable agent traces—positions the field to move from single-output imitation toward evidence-grounded executable systems, with the accompanying repository providing a maintained resource for tracking this evolving landscape.