VisCodex: Unified Multimodal Code Generation

• VisCodex is a unified multimodal code generation framework that integrates visual inputs and text through a task-vector merging methodology.
• It employs a ViT-based visual encoder, a cross-modal projector, and a Transformer decoder fine-tuned on a large, diverse multimodal coding dataset.
• Key applications include image-to-code UI generation, chart code replication, and symbolic SVG reasoning for downstream robotics and documentation tasks.

VisCodex is a unified multimodal code generation framework that enables open-source multimodal LLMs (MLLMs) to synthesize code from both visual and textual inputs. Designed to bridge deficiencies in vision-LLMs’ (VLMs) code generation and to extend coding LLMs with visual reasoning, VisCodex merges advanced visual comprehension and program synthesis by leveraging an arithmetic, task-vector–based parameter merging methodology. The resulting architecture is capable of interpreting complex visual stimuli—including UI mockups, rendered charts, and code-augmented question–answer pairs—and generating syntactically and functionally correct source code, including HTML, Python, and algorithmic solutions (Jiang et al., 13 Aug 2025). The approach is grounded in large, curated datasets and evaluated through rigorous multimodal benchmarks.

1. Architecture and Model Merging

VisCodex’s architecture integrates three core components: a ViT-based visual encoder with 2D Rotary Position Embedding (RoPE), a cross-modal projector that embeds image features in the LLM’s input space, and a Transformer-based LLM decoder. Rather than full end-to-end multimodal pre-training, VisCodex employs an “arithmetical model merging” strategy:

• The model starts from a base LLM (Qwen2.5).
• Two fine-tuned descendants are selected: a vision–LLM (Qwen2.5-VL) for visual-semantic grounding, and a code LLM (OpenCodeReasoning-Nemotron) specialized in program synthesis.
• The parameter differences relative to the base (“task vectors”) are

$$\tau_{\mathrm{vlm}} = \theta_{\mathrm{vlm}} - \theta_\mathrm{base}, \quad \tau_{\mathrm{code}} = \theta_{\mathrm{code}} - \theta_\mathrm{base}.$$

• The merged backbone parameters for VisCodex are initialized as a convex combination:

$$\theta_{\mathrm{VisCodex}} = \theta_{\mathrm{base}} + \lambda\,\tau_{\mathrm{vlm}} + (1 - \lambda)\,\tau_{\mathrm{code}}, \quad \lambda \in [0, 1].$$

• Vision encoder and projector parameters are frozen; only the language backbone is merged and further instruction-tuned.

This ensures that VisCodex inherits both visual understanding from the VLM and advanced coding skills from the code LLM (Jiang et al., 13 Aug 2025).

2. Training Datasets and Instruction Tuning

After merging, VisCodex undergoes supervised instruction tuning solely on the language backbone, using the Multimodal Coding Dataset (McD). McD comprises 598,000 instruction pairs distributed across four domains:

Domain	# Samples	Typical Task	Avg Length (tokens)
HTML Code	200k	Webpage screenshots → HTML+CSS	632 ± 144
Chart Code Pairs	210k	Images of charts → matplotlib code	551 ± 190
Code QA	59k	StackOverflow images + accepted code answers	1022 ± 776
Algorithmic Code	129k	LeetCode, Codeforces, contest problems	969 ± 321

Sources include high-quality HTML+CSS generated by GPT-4o, real and synthetic chart code pairs, refined StackOverflow QAs with images, and curated algorithmic problems. For all domains, images are essential, and code is filtered for correctness via automated execution or render (Jiang et al., 13 Aug 2025).

Supervised fine-tuning employs standard sequence-to-sequence cross-entropy loss:

$$\mathcal{L} = - \sum_{t=1}^T \log P_\theta(y_t \mid y_{<t}, x_\mathrm{text}, x_\mathrm{image}).$$

All vision modules remain frozen; only the LLM backbone is updated.

3. Benchmarking and Evaluation

VisCodex is evaluated primarily on four multimodal code benchmarks: Design2Code, ChartMimic, MMCode, and the newly proposed InfiBench-V. InfiBench-V is a 322-question, human-curated benchmark that requires both textual and visual reasoning, with categories covering front-end, back-end, data science, IT operations, and mobile/desktop programming. Each test sample demands image-grounded code comprehension or synthesis.

Evaluation metrics are tailored to output modality:

• Keyword Matching (rule-based, weighted): matches essential components in specification and output.
• Unit Testing: programmatically executes generated code and verifies output.
• GPT-4o Judge: adjudicates free-form answers for correctness.

Performance is reported as the mean of these metrics across all samples (Jiang et al., 13 Aug 2025).

4. Empirical Performance and Ablations

Experimental results demonstrate state-of-the-art performance among open-source MLLMs, with VisCodex-8B and VisCodex-33B closely tracking or matching proprietary models. InfiBench-V and other benchmark scores highlight the efficacy of the task-vector merging approach:

Model	Size	Design2Code	ChartMimic	MMCode	InfiBench-V	Average
VisCodex-8B	8B	90.1 / 90.9	74.8 / 74.1	11.0	72.1	68.8
VisCodex-33B	33B	90.5 / 91.1	79.3 / 78.5	15.6	78.6	72.3
GPT-4o	—	90.2 / 90.4	79.0 / 83.5	17.0	79.9	73.3

Without merging, average performance for VisCodex-8B drops from 68.8 to 66.3 (MMCode pass@1 declines from 11.0 to 6.8), emphasizing the necessity of combining vision and code specializations. Substituting alternative code task vectors (e.g., OpenThinker2, Qwen2.5-Coder) consistently yields improvements over general-purpose LLMs. Two-stage projector training and model replacement alone underperform the task-vector merging strategy (Jiang et al., 13 Aug 2025).

Qualitative assessment shows close matches to ground-truth HTML/chart code; however, minor off-by-one errors in layout, occasional HTML syntax misnesting, and limited robustness to rare domain-specific libraries are observed failure cases.

5. Symbolic Representation and Downstream Reasoning

The VisCodex paradigm is closely informed by results from the VCode benchmark and its agentic VCoder framework (Lin et al., 4 Nov 2025). SVG is advanced as a symbolic, declarative, and interpretable visual representation which preserves high-value semantic information from images—not merely as an output format, but as an actionable interface for downstream agents. The SVG abstraction encodes object instances, spatial relations, type, and textual content in a format that is compact, executable, and readily auditable.

Benchmarking in VCode motivates system-level chains where diagrams or natural images are translated into SVG whose rendering retains the symbolic facts of the original. CodeVQA, an evaluation protocol in which a policy model answers questions over rendered SVGs (“Render→VQA”), reliably measures symbolic fidelity and reveals that even state-of-the-art VLMs benefit from guided refinement and external perception modules (e.g., detectors, mask generators, OCR).

A plausible implication is that VisCodex can leverage this modality—producing SVG for downstream robotic reasoning, programmatic question answering, simulation, or interactive editing. This extends code synthesis beyond text into rich, symbolic visual abstraction (Lin et al., 4 Nov 2025).

6. Practical Applications and Research Prospects

VisCodex supports key application scenarios:

• Image-to-Code UI Generation: Translates mockups or screenshots into HTML/CSS or other code representations for rapid prototyping.
• Chart and Data Visualization Replication: Generates matplotlib or HTML chart code from graphical input, supporting data science workflows.
• Automated Documentation and QA: Processes annotated diagrams and StackOverflow pairs, bridging visual documentation and code.
• Symbolic Reasoning and Robotics: Exports interpretable SVG from photographs or scenes, suitable for agentic planning and manipulation.

Future research may explore joint end-to-end training on large paired vision-to-SVG or vision-to-code corpora, moving beyond test-time augmentation with external visual tools. Potential directions include extending SVG chains for parametric animations, integrating feedback loops for user-refined SVG, and bridging to 3D via mappings from 2D primitives to 3D meshes or scene graphs (Lin et al., 4 Nov 2025).

7. Limitations and Current Challenges

Despite its robust performance, VisCodex inherits key limitations from its components and training regimes:

• Professional and knowledge-intensive code generation remains challenging; models may mislabel historical or technical terms, especially in fine-grained or specialized domains.
• Reasoning involving 3D spatial relationships and depth is only partially addressed, with a persistent gap compared to direct image-to-answer models.
• Occasional layout errors in HTML/UI and occasional syntax errors in complex code outputs.
• Open-domain resilience remains lower than that of proprietary models on heavily domain-shifted input (Jiang et al., 13 Aug 2025, Lin et al., 4 Nov 2025).

These gaps suggest that ongoing improvement in both task-vector composition and multimodal dataset curation are requisite for further progress.

---

In summary, VisCodex exemplifies state-of-the-art open-source multimodal code generation by integrating visual perception and code synthesis through task-vector–based model merging, large-scale instruction tuning, and evaluation on code- and vision-intensive benchmarks. It reflects a broader trend towards symbolic visual abstraction and actionable code generation from multimodal data, informed by benchmark-driven developments in both code and vision–language modeling (Jiang et al., 13 Aug 2025, Lin et al., 4 Nov 2025).