UniCoder: Unified Visual-to-Code Generation via Symbolic Rewards and Reference-Guided Code Optimization

Abstract: Visual-to-Code generation, which transforms scientific plots, vector graphics, and webpages into executable scripts, demands a level of pixel-precise alignment that standard Multimodal LLMs (MLLMs) fail to achieve through Supervised Fine-Tuning (SFT) alone. While Reinforcement Learning (RL) offers a theoretical pathway to bridge this gap, its application is hindered by two fundamental obstacles: (1) \textit{Reward Coarseness}, where semantic metrics like CLIP scores fail to penalize fine-grained element deviations, and (2) \textit{Exploration Stagnation}, where the sparse, heterogeneous code search space prevents the policy from bootstrapping valid trajectories. To overcome these limitations, we introduce UniCoder, a unified RL framework that integrates two novel mechanisms. First, we propose \textbf{Symbolic Attribute Alignment}, which employs a lightweight auxiliary LLM to parse generated code into discrete visual attributes (e.g., hex colors, coordinate limits), enabling dense, element-wise reward computation. Second, to escape local optima, we devise \textbf{Reference-Guided Code Optimization}, a strategy that dynamically injects ground-truth trajectories into low-performing rollout groups, transforming blind exploration into guided policy improvement. Extensive experiments on ChartMimic, UniSVG, Design2Code and ScreenBench benchmarks demonstrate that our 8B-parameter model not only surpasses all open-source baselines but also achieves state-of-the-art performance comparable to proprietary models, establishing a new paradigm for generalized visual-to-code synthesis.

• The paper introduces a unified RL framework that leverages fine-grained symbolic attribute alignment for enhanced pixel-wise and semantic code precision.
• It integrates a dynamic reference-guided code optimization strategy to stabilize training and overcome exploration stagnation in sparse reward environments.
• Empirical results demonstrate significant gains on benchmarks like ChartMimic and Design2Code, outperforming both similar-scale open-source and specialized commercial models.

UniCoder: Unified Visual-to-Code Generation via Symbolic Rewards and Reference-Guided Code Optimization

Introduction

The synthesis of executable code from visual data—across domains such as scientific charts, vector graphics, and web design—poses stringent requirements for token-level semantic fidelity and precise pixel-wise alignment. Contemporary Multimodal LLMs (MLLMs), while powerful under supervised fine-tuning (SFT), demonstrate significant deficiencies when high-fidelity compositional or symbolic correspondence is essential. This limitation stems primarily from coarse reward structures and difficulties with exploration in reinforcement learning (RL) frameworks. "UniCoder: Unified Visual-to-Code Generation via Symbolic Rewards and Reference-Guided Code Optimization" (2606.31732) addresses these challenges by proposing a unified RL-based framework that couples symbolic attribute alignment with a dynamic reference-guided code optimization regime, achieving state-of-the-art alignment across a variety of code generation benchmarks.

Motivations and Problem Analysis

MLLM-driven visual-to-code synthesis is presently stymied by two canonical bottlenecks:

1. Reward Coarseness: Principal approaches use visual-semantic metrics (e.g., CLIP score), delivering high global similarity but failing to penalize fine-grained semantic errors or element-level misalignments. This leads to scenarios where outputs with critical semantic errors nonetheless achieve high rewards due to superficial style matching.
2. Exploration Stagnation: The sparsity of valid solutions and the brittle nature of code renders the RL reward landscape extremely sparse. Standard exploration mechanisms often become stuck in local minima, especially in multicoding-paradigm environments (Python, SVG, HTML).

   Figure 1: CLIP-based rewards offer high similarity for structurally erroneous outputs and exhibit training stagnation under vanilla RL paradigms.

Framework Overview

UniCoder introduces two key innovations for RL-based visual-to-code modeling:

• Symbolic Attribute Alignment: A task-specialized, lightweight LLM parses code outputs to extract domain-specific symbolic attributes (e.g., color hex codes, axis limits, DOM tags), providing fine-grained, dense rewards that disentangle structure from rendering artifacts.
• Reference-Guided Code Optimization: During RL rollouts, if sampled outputs perform poorly, the framework dynamically injects ground-truth reference code, ensuring presence of at least one optimal trajectory per rollout group, stabilizing advantage calculation and gradient propagation.

  Figure 2: UniCoder’s reward architecture combines attribute, execution, and visual consistency signals; policy updates leverage reference-guided group computation.

UniCoder utilizes a composite reward:

$$R_{total} = w_1 R_{exec} + w_2 R_{vis} + w_3 R_{attr}$$

with $R_{exec}$ as executability, $R_{vis}$ as CLIP visual similarity, and $R_{attr}$ as symbolic attribute alignment via soft recall.

Empirical Results

Experiments span authoritative multimodal code generation datasets: ChartMimic (Python plots), UniSVG (vector graphics), Design2Code and ScreenBench (real-world web pages). Benchmarks are evaluated using metrics covering execution rate, global visual similarity, and elementwise attribute precision.

UniCoder (8B) consistently outperforms all open-source baselines of similar scale across all domains. Notably:

• On ChartMimic, UniCoder attains 86.4% execution rate, compared to 77.3% for Qwen3-VL-8B and 60.2% for LLaVA-v1.6-7B.
• Design2Code fine-grained block, position, and color metrics show clear improvements with UniCoder, accentuating the benefits of reward densification and reference injection.

The method even approaches or surpasses proprietary closed-source models on specific alignment and layout reconstruction metrics, indicating that RL-based symbolic optimization can substantially reduce the performance gap with commercial foundation models.

Figure 3: All reward components—execution rate, attribute recall, CLIP similarity, and overall reward—exhibit consistent improvement throughout Reference-Guided GRPO training.

Qualitative analysis reveals that baseline models often manifest color hallucinations, loss of elements, and incorrect structure, whereas UniCoder yields high-fidelity outputs with negligible structural errors.

Figure 4: Baselines exhibit color palette hallucinations and structural omissions; UniCoder achieves both accurate color codes and structural integrity.

Further cross-task qualitative comparisons underscore the improved generalization of UniCoder over strong baseline models (Figures 5–9).

Ablation Analysis

Component-wise ablation reveals:

• Attribute Reward Ablation: Loss of elementwise supervision leads to pronounced decline in position and color alignment—demonstrating that global CLIP rewards are insufficient for local structure fidelity.
• Reference Injection Ablation: Omission destabilizes group advantage computation and increases frequency of code execution failures, corroborating the necessity of guided exploration in sparse reward settings.

Practical and Theoretical Implications

Practical Impact: UniCoder dramatically elevates the accessibility of high-accuracy program synthesis from visual data for downstream scientific, design, and front-end engineering applications. The proposed framework is computationally viable—leveraging an 8B policy with a 3B attribute extractor—and robust across code paradigms.

Theoretical Implications: The integration of symbolic attribute extraction as a dense RL reward augments the standard RLHF pipeline with explicit compositional supervision, bridging the gap between pixel-level and compositional reasoning. Reference-guided rollouts generalize the idea of contrastive RL with gold anchors, providing a template for RL in sparse, structured domains.

Future Directions: Generalizing symbolic reward schemas to interactive graphical tasks, animations, or dynamic software artifacts remains open. Further, progress in automatic attribute schema extraction and adaptive reward tuning will broaden model applicability beyond static reconstructions.

Conclusion

UniCoder achieves state-of-the-art multimodal code synthesis by introducing a fine-grained symbolic reward and guided exploration mechanism into RL-based code generation, consistently outperforming both open-source and (in specialized settings) commercial systems (2606.31732). It demonstrates that highly-structured reward modeling and intelligent reference utilization are critical for bridging the compositional fidelity gap in visual-to-code generation. These methods are likely to become foundational for future visual programming and intelligent design assistants as interfaces and assets grow in visual-coding complexity.