VFIG: Vectorizing Complex Figures in SVG with Vision-Language Models

Abstract: Scalable Vector Graphics (SVG) are an essential format for technical illustration and digital design, offering precise resolution independence and flexible semantic editability. In practice, however, original vector source files are frequently lost or inaccessible, leaving only "flat" rasterized versions (e.g., PNG or JPEG) that are difficult to modify or scale. Manually reconstructing these figures is a prohibitively labor-intensive process, requiring specialized expertise to recover the original geometric intent. To bridge this gap, we propose VFIG, a family of Vision-LLMs trained for complex and high-fidelity figure-to-SVG conversion. While this task is inherently data-driven, existing datasets are typically small-scale and lack the complexity of professional diagrams. We address this by introducing VFIG-DATA, a large-scale dataset of 66K high-quality figure-SVG pairs, curated from a diverse mix of real-world paper figures and procedurally generated diagrams. Recognizing that SVGs are composed of recurring primitives and hierarchical local structures, we introduce a coarse-to-fine training curriculum that begins with supervised fine-tuning (SFT) to learn atomic primitives and transitions to reinforcement learning (RL) refinement to optimize global diagram fidelity, layout consistency, and topological edge cases. Finally, we introduce VFIG-BENCH, a comprehensive evaluation suite with novel metrics designed to measure the structural integrity of complex figures. VFIG achieves state-of-the-art performance among open-source models and performs on par with GPT-5.2, achieving a VLM-Judge score of 0.829 on VFIG-BENCH.

• The paper introduces VFIG, a novel approach that uses vision-language models and reinforcement learning to automate the conversion of complex raster figures into editable SVG code.
• The methodology employs a two-stage curriculum with supervised fine-tuning followed by reinforcement learning, yielding high structural fidelity and semantic correctness.
• Empirical evaluations demonstrate that VFIG significantly outperforms existing models in visual similarity, editability, and human preference metrics.

VFig: Advancing SVG Vectorization for Complex Figures with Vision-LLMs

Motivation and Problem Statement

The paper "VFIG: Vectorizing Complex Figures in SVG with Vision-LLMs" (2603.24575) targets a salient gap in technical illustration: the automated reconstruction of complex rasterized figures (PNG/JPG) into editable, high-fidelity SVG code. While SVG provides resolution independence, semantic editability, and standardized support across platforms, original vector sources are routinely lost in scientific and technical workflows. Manual SVG reconstruction is prohibitively labor-intensive, especially for figures exhibiting hierarchical layouts, dense annotations, and intricate connectivity. The task demands cross-modal reasoning over both visual structure and syntactic SVG code generation, posing formidable challenges in compositional fidelity and semantic correctness.

Figure 1: Overview of VFig. Given complex raster images (top row) as input, VFig generates editable, high-fidelity SVG code (pink box). Rendering the generated SVG (bottom row) produces outputs nearly indistinguishable from the inputs.

Dataset Construction: VFig-Data

Recognizing the crucial role of training data in enabling robust figure-to-SVG conversion, the authors introduce VFig-Data—a 66K-pair dataset prioritizing structural complexity and editability. Unlike prior SVG datasets focusing on icons or decorative graphics, VFig-Data encompasses diagram-centric scientific figures, both extracted from real-world papers and synthesized programmatically.

Figure 2: Examples of VFig-Data and academic data from three sources: simple diagrams, complex layouts, and a curated set of shapes/arrows to enable structured SVG generation.

Data generation proceeds via two complementary pipelines: (i) VLM-driven describe-and-generate approaches for real-world figures—leveraging Gemini models for improved structured description and subsequent SVG code synthesis, and (ii) procedurally generated diagrams with precise control over geometric primitives, visual attributes, and connectivity. Rigorous filtering is applied to remove figures unsuited for vectorization (natural images, equations) and to ensure SVG code predominantly consists of semantic primitives rather than unwieldy paths, optimizing for editability and efficient modeling.

Figure 3: Data generation and filtering pipelines for academic figures, VLM-based diagram generation from crawled images, and shape/arrow templates with randomized elements.

Model Architecture and Training Paradigm

The proposed VFig model follows a two-stage curriculum: (1) supervised fine-tuning (SFT) on structurally simpler diagrams to establish primitive and basic layout generation, and (2) transfer to complex scientific figures for compositional learning and fidelity. SFT is formalized over the conditional distribution $p_\theta(y|x)$, optimizing next-token likelihood via standard autoregressive objectives.

To address the visual-textual mismatch and remaining errors in rendered outputs, reinforcement learning (RL) refinements are introduced post-SFT. The RL protocol employs Group Relative Policy Optimization (GRPO) [deepseekmath], leveraging rubric-based rewards extracted via VLM-based judges (Gemini-3-Flash) from rendered SVG comparisons. Rewards decompose into presence, layout, connectivity, and details (styling/text accuracy), whose average forms the overall scalar reward. Strong correlation is observed between VLM-judge outputs and human evaluation metrics ($r = 0.89$ overall).

Evaluation Methodology and Benchmarking

Comprehensive evaluation is performed via VFig-Bench—a test suite emphasizing structural complexity and editability, with multi-granular metrics: SSIM, LPIPS, VisualSim (embedding cosine similarity), VLM-Judge, SVG cleanliness, and render rate.

VFig is benchmarked against classical raster-to-vector algorithms (Potrace, VTracer), learning-based SVG methods (DeepSVG, DiffVG, Im2Vec), open-source VLMs (OmniSVG (Yang et al., 8 Apr 2025), StarVector (Rodriguez et al., 2023), Qwen3-VL (Bai et al., 26 Nov 2025)), and proprietary baselines (Gemini-3-Pro [google_gemini3_2025], GPT-5.2 [openai_gpt5_2_2025]).

Figure 4: Qualitative comparison across models. VFig's rendered SVG outputs more faithfully preserve figure structure and semantic attributes, as reflected in higher Gemini judge scores.

Empirical Findings and Ablation Insights

VFig outperforms existing open-source models (including StarVector, OmniSVG, Qwen3-VL-4B) on VFig-Bench, SVG-Diagrams, and Molmo2-Diagram in both visual and structural metrics. With SFT+RL, VFig achieves a VLM-Judge score of 0.829, competitive with GPT-5.2 (0.858) despite a much smaller backbone.

Key empirical results:

• Curriculum SFT significantly improves compositional stability: Two-stage training yields higher render success rate and more consistent structure compared to single-stage SFT.
• RL with structure-aware rewards further enhances fidelity: Full rubric optimization achieves highest judge-based scores and code cleanliness; pixel-level objectives (e.g., LPIPS) improve perceptual similarity but reduce structural correctness.
• Qwen3-VL-8B backbone achieves best overall performance on visual metrics, but Qwen3-VL-4B remains competitive considering computational efficiency.
• Human preference evaluation aligns closely with VLM-Judge metrics: VFig decisively outperforms Qwen3-VL-4B in "good rate" and pairwise wins; pairwise fractions highlight its competitiveness against proprietary systems.

  Figure 5: Annotated issue distribution in human evaluation, identifying predominant failure modes in arrows and layout overlap.

  

  Figure 6: Pairwise win fractions in human evaluation, visualizing model preferences and VFig's competitive standing.

Rule-Based Structural Evaluation

Fine-grained rule-based evaluation on programmatically generated diagrams allows rigorous assessment of semantic fidelity in shapes and arrows.

Figure 7: Rule-based evaluation for shapes and arrows on sample 000024, quantifying visual and structural correctness.

VFig's RL-enhanced outputs consistently achieve higher shape and arrow scores compared to SFT-only and baseline models.

Failure Modes and Limitations

Typical failure cases include inaccurate rendering of thin lines, arrowheads, small text elements, and 3D shapes, as well as color/style deviations. Model generalization is weakest on out-of-distribution graphics (logos, icons) not emphasized during training. RL reward design (heavy reliance on Gemini judge) and data scope contribute to these limitations.

Implications and Future Directions

This study demonstrates that targeted data curation, hierarchical SFT curriculum, RL reward engineering, and structure-focused evaluations are crucial for bridging the gap between open-source and proprietary vision-language SVG generation. The approach illustrates how compositional reasoning and semantic editability for complex diagrams can be reliably automated within the constraints of current VLMs.

Practically, VFig enables scalable, editable conversion of scientific and technical figures, accelerating revision, reuse, and visualization workflows. Theoretically, it advances the study of code generation conditioned on visual inputs at high structural complexity.

Future directions include (i) broader coverage of non-diagram vector graphics, (ii) exploration of alternative RL rewards combining perceptual and structural signals, (iii) improved generalization to dense and stylized figures, and (iv) deeper integration with multimodal reasoning/span localization for diagram interpretation.

Conclusion

VFig introduces a principled pipeline for high-fidelity SVG code generation from complex raster figures utilizing vision-LLMs, contributing new data resources, training protocols, and benchmarks for the field. Empirical results establish VFig as the state-of-the-art among open-source models and competitive with leading proprietary systems. Public release of the models, data, and evaluation suite positions VFig as a foundation for subsequent research on editable scientific figure synthesis and multimodal symbolic coding.