OmniDiagram: Advancing Unified Diagram Code Generation via Visual Interrogation Reward

Abstract: The paradigm of programmable diagram generation is evolving rapidly, playing a crucial role in structured visualization. However, most existing studies are confined to a narrow range of task formulations and language support, constraining their applicability to diverse diagram types. In this work, we propose OmniDiagram, a unified framework that incorporates diverse diagram code languages and task definitions. To address the challenge of aligning code logic with visual fidelity in Reinforcement Learning (RL), we introduce a novel visual feedback strategy named Visual Interrogation Verifies All (\textsc{Viva}). Unlike brittle syntax-based rules or pixel-level matching, \textsc{Viva} rewards the visual structure of rendered diagrams through a generative approach. Specifically, \textsc{Viva} actively generates targeted visual inquiries to scrutinize diagram visual fidelity and provides fine-grained feedback for optimization. This mechanism facilitates a self-evolving training process, effectively obviating the need for manually annotated ground truth code. Furthermore, we construct M3$^2$Diagram, the first large-scale diagram code generation dataset, containing over 196k high-quality instances. Experimental results confirm that the combination of SFT and our \textsc{Viva}-based RL allows OmniDiagram to establish a new state-of-the-art (SOTA) across diagram code generation benchmarks.

• The paper introduces a unified framework that integrates SFT and Viva-guided RL to enhance multimodal diagram code generation.
• The methodology leverages a large-scale M3²Diagram dataset and GRPO optimization, achieving state-of-the-art execution rates and visual fidelity.
• Empirical results demonstrate robust performance across Diagram-to-Code, Diagram Editing, and Text-to-Code tasks in multiple diagram languages.

OmniDiagram: A Unified paradigm for Multimodal Diagram Code Generation via Visual Interrogation Reward

Motivation and Context

OmniDiagram presents a unified approach to diagram code generation, moving beyond earlier methods limited by narrow task formulation or restricted language coverage. Prior models typically address a single modality (e.g., Image-to-Code), support only a small set of diagrammatic languages, or are confined to specific tasks such as Text-to-Code or Editing without true unification. The new framework is designed to address these shortcomings, harmonizing the demands of handling diverse diagram types, supporting multiple code syntaxes (LaTeX, Mermaid, PlantUML), and cross-task requirements for both construction and modification of diagram code.

A central technical challenge is aligning the generated code with both execution robustness and visual fidelity, especially in a reinforcement learning setting where fully aligned reward signals are difficult to obtain. OmniDiagram approaches this problem through a hybrid SFT+RL training pipeline, culminating in a novel, generative reward mechanism—the Visual Interrogation Verifies All (Viva)—which acts as an adaptive, question-driven visual supervisor.

Figure 1: Overcoming the barriers of single-modality: The comprehensive landscape of OmniDiagram.

Methodology

Dataset Construction and Coverage

The M3²Diagram dataset is introduced to substantiate the omni-task, omni-language paradigm. It spans 196k high-quality samples, including 31k curated from open-source corpora and 165k synthesized via a multi-step, top-down generation pipeline engineered for broad semantic and topological diversity. The dataset is balanced across LaTeX, Mermaid, and PlantUML, and covers three major task axes: Diagram-to-Code, Diagram Editing, and Text-to-Code, culminating in a structured $3 \times 3$ task-language matrix.

A reasoning-enriched subset (77k samples) is distilled via trajectories expounded from Gemini-2.5-Flash, facilitating analysis on the impact of explicit reasoning against pure direct synthesis. The dataset is partitioned (8:1) for SFT and RL, and carefully decontaminated using clustering based on perceptual hashing to maintain a uniform spread across complexity and diagram type.

Figure 2: Breakdown of the 196k-sample M3²Diagram dataset, supplemented by 77k reasoning-enriched samples, categorized across languages, tasks, and data sources.

Model Training Pipeline

Training proceeds in two main stages:

• Supervised Fine-Tuning (SFT): Qwen2.5-VL-3B/7B backbones are adapted to the M3²Diagram corpus, bootstrapping the models with syntactic and task compliance.
• Viva-Guided RL (GRPO): RL with a VQA-based reward signal optimizes for high-fidelity and executable code. For each RL rollout, properly rendered code output is visually interrogated with a battery of fine-grained, instance-specific visual questions (Vivas). An answer set adjudicated by a reward model (Qwen3-VL-32B) yields a smoothed, partially creditable reward signal, which is aggregated with a hard format adherence check. Candidates failing compilation are penalized. Policy updates employ Group Relative Policy Optimization (GRPO), normalizing the advantage by group-level statistics.

  Figure 3: Overview of the OmniDiagram methodology. The framework illustrates the end-to-end flow from scalable data synthesis to model training, highlighting the two-stage optimization strategy combining an SFT foundation with the proposed Viva-guided RL mechanism.

Visual Interrogation Verifies All (Viva) Strategy

The Viva mechanism extends beyond brittle template- or global-similarity based visual feedback. For every task instance, ten targeted visual questions are generated offline (e.g., “Is there a red diamond-shaped node labeled ‘Condition’?”). At training time, the RL evaluator answers these based on the rendered image, producing a per-question soft score in $[0,1]$, allowing for nuanced assessment even on partially correct outputs. This not only stabilizes learning by reducing reward variance (theoretically analyzed and empirically validated), but also delivers reward signals that are robust to the multiplicity and ambiguity inherent in diagram-to-code mappings.

Empirical Results

OmniDiagram establishes new SOTA results for open-source multimodal diagram code generation across Diagram-to-Code, Diagram Editing, and Text-to-Code on both the M3²Bench and external benchmarks (CoSyn, VisPlotBench). The RL-enhanced variants (3B, 7B) outperform strong open-source baselines, including Qwen2.5-VL-72B and the InternVL3.5 series, and in specific cases approach or match proprietary models such as Gemini-3-Flash and GPT-5-mini particularly in execution rate and visual scores.

Key quantitative highlights:

• On M3²Bench Diagram-to-Code, OmniDia-7B (RL) achieves 94.3% execution rate and a visual score of 75.5, outperforming Qwen2.5-VL-72B by 9.7 and 20.5, respectively.
• On VisPlotBench-Mermaid (Text-to-Code), OmniDia-7B (RL) achieves 86.3% execution, a 29.0 point improvement over Qwen2.5-VL-7B.
• Ablation studies demonstrate that direct output reasoning (Chain-of-Thought) impairs code renderability despite higher visual detail, indicating a trade-off—the so-called “Syntax Tax”—between semantic richness and syntactic rigidity. RL aligns the model towards execution-correct outputs, further boosting robustness and visual metrics.

  Figure 4: Progression of the overall reward during the RL training process for OmniDiagram.

Qualitative Showcase and Visual Verification

Diversity and task unification are visually demonstrated across all benchmarks:

Figure 5: Qualitative showcase of our model across three modalities (LaTeX, Mermaid, PlantUML) and three tasks. Left: Diagram-to-Code reconstruction. Middle: Diagram Editing. Right: Text-to-Code from natural language.

Additionally, the Viva mechanism’s question-based approach is elucidated by presenting task-relevant visual checks—for instance, verifying node count, shape style, or specific color assignments (Figures 7, 8, 9).

Figure 6: Qualitative example of visual verification questions for the Text-to-Code task.

Figure 7: Qualitative example of visual verification questions for the Diagram-to-Code task.

Figure 8: Qualitative example of visual verification questions for the Diagram Editing task.

Limitations

While empirically robust and compositionally scalable, OmniDiagram maintains a fixed reward weighting ($\alpha$) between visual and syntax rewards. The framework relies on GRPO; further benchmarking versus other RL paradigms (PPO, DPO) under the Viva regime is left for future work. Training and reward evaluation are resource-intensive, motivating future research on leaner, more efficient pipelines for data synthesis, reward collection, and RL supervision.

Implications and Future Directions

OmniDiagram demonstrates that unified multimodal code generation is possible at scale, even under heterogeneous task formulations, provided the reward infrastructure is sufficiently granular and visually grounded. The M3²Diagram dataset both raises the evaluation bar and provides a critical resource for the field. Viva represents a paradigm shift toward human-analogous, fine-grained visual interrogation in RL training, encouraging further exploration of reward models tightly coupled to cognitive verification rather than holistic similarity or simple execution success.

The practical implications are broad: scalable diagrammatic knowledge automation, code-driven system diagramming, graph-based data analytics, and, crucially, the possibility of closed-loop design-edit pipelines where generation, editing, and validation are all handled in an integrated multimodal system.

Conclusion

OmniDiagram advances the state of art for unified diagrammatic code generation by systematically integrating large-scale omni-multimodal data, SFT initialization, and visual question-answering-based reward optimization. The combination of extensive coverage (M3²Diagram), a robust training pipeline, and the Viva reward mechanism yields strong empirical performance and establishes a flexible foundation for subsequent progress in multimodal generative AI. The framework’s generalizability and extensibility signal a new direction for RL-driven alignment in structured code synthesis and beyond.