---
title: 'VFIG: Vision-Language Figure-to-SVG Conversion'
url: https://www.emergentmind.com/topics/vfig
type: topic
---
# VFIG: Vision-Language Figure-to-SVG Conversion
Searching arXiv for the specific term and closely related papers to ground the article in current literature.
VFIG is a family of vision-language models for figure-to-SVG conversion: given a rasterized figure such as a PNG or JPEG, it generates editable Scalable Vector Graphics code intended to reconstruct the figure while preserving structure, layout, text, and connector semantics. The system is designed for complex scientific and technical figures rather than generic raster tracing, and is introduced together with VFIG-DATA, a dataset of **66K** curated figure–SVG pairs, and VFIG-BENCH, an evaluation suite for structural fidelity. Its training recipe combines a coarse-to-fine supervised curriculum with reinforcement learning refinement, and the reported VFIG-4B system reaches a **VLM-Judge score of 0.829 on VFIG-BENCH**, state-of-the-art among open-source models and close to **GPT-5.2** on the same benchmark [2603.24575].
## 1. Task definition and representational scope
VFIG addresses the problem of recovering structured vector graphics from flattened raster figures. The paper motivates this problem by noting that SVG is resolution-independent, semantically editable, text-based, and widely supported, whereas original vector source files are often unavailable and only raster exports remain [2603.24575]. The intended figures include scientific diagrams, architecture diagrams, flowcharts, process diagrams, and multi-panel technical illustrations.
The task is formulated as conditional generation. Given an input raster figure image $x \in \mathcal{X}$, the model generates an SVG program $y \in \mathcal{Y}$, using paired training data
$$
\mathcal{D}=\{(x_i,y_i)\}_{i=1}^{N}.
$$
The output is treated as structured code and generated autoregressively. The model is prompted with instructions such as “Generate the SVG for this figure” or “Convert this figure into valid SVG code.” In deployment, the generated program is expected to contain a valid `` block.
A central design choice is that VFIG is not aimed at path-heavy raster tracing. The paper emphasizes semantic primitives and structured editability. In both curation and evaluation, SVG elements are grouped into categories including basic primitives (`rect`, `circle`, `ellipse`), connectors (`line`, `polyline`), complex shapes (`path`, `polygon`), and text. This places the task closer to structured program synthesis than to contour approximation alone.
## 2. VFIG-DATA and data curation
VFIG-DATA combines real-world scientific figures with procedurally generated diagrams, reflecting the paper’s claim that the task is inherently data-driven but poorly served by prior small-scale datasets [2603.24575]. The dataset contains **66K** curated figure–SVG pairs, and the overall training mixture additionally includes **78K** filtered examples from existing academic SVG datasets.
The real-world subset is built from Paper2Fig figures and a large-scale crawl of recent arXiv papers. The appendix states that **259,073 arXiv documents** after January 2025 were queried, with figures extracted from LaTeX `\includegraphics` references and embedded PDFs rasterized using PyMuPDF. After image filtering, **45K high-quality figures** were retained. Because these figures are often only available as raster assets, the paper uses a two-stage VLM pipeline for SVG creation: a description stage that produces a detailed structured account of shapes, colors, positions, text, styles, and relationships, followed by an SVG generation stage conditioned on both the original image and that description. The selected pipeline is described as **Gemini-3-Pro $\rightarrow$ Gemini-3-Pro**, and in an internal human evaluation over **332 tasks**, **1088 responses**, and **5 annotators**, it was preferred in **88.7\%** of decisive comparisons over **Gemini-3-Pro $\rightarrow$ GPT-5.1**.
The procedural subset is intended to supply precise supervision for attributes that are difficult to recover from raster figures alone. It uses **19 layout templates**; with probability **0.3**, two templates are combined by splitting the canvas and adding cross-connections. The shape vocabulary contains **18 total** shapes: **12 flat shapes** and **6 pseudo-3D shapes**. Styling includes **7 fill styles**, several border styles, randomized stroke widths, rounded corners with probability **0.6**, and **1–2 font families** sampled from a set of **8**. Directed connections are sampled by
$$
c \sim \mathcal{U}\!\left(\lfloor n\, r_\ell \rfloor,\; \lfloor n\, r_h \rfloor\right), \qquad r_\ell \sim \mathcal{U}(0.4, 0.6), \quad r_h \sim \mathcal{U}(0.6, 0.8).
$$
Arrows are straight **60\%** of the time and quadratic Bézier **40\%**, and are attached to true shape boundaries via geometric intersection or raycasting.
Filtering is used to suppress unsuitable figures and non-semantic SVG. Image filtering classifies figures into `KEEP`, `IMAGE`, `MATH`, and `PLOT`, retaining only `KEEP`. Code filtering defines
$$
B = \#(\text{basic primitives}), \quad K = \#(\text{connectors}), \quad C = \#(\text{complex shapes}), \quad N=B+K+C,
$$
and keeps only SVGs satisfying
$$
\frac{B+K}{N} \ge 0.4, \qquad C \le 50.
$$
The associated structural descriptors are
$$
\mathrm{EC} = \log(1+N+T),
$$
$$
\mathrm{Clean} = \frac{B+K}{N} = 1-\frac{C}{N},
$$
$$
\mathrm{PD} = \frac{C}{N} = 1-\mathrm{Clean},
$$
where $T$ is the number of text elements. These quantities are later reused in evaluation.
## 3. Model family and SVG generation pipeline
VFIG is presented as a family of fine-tuned VLMs rather than as a new architecture built from scratch [2603.24575]. The evaluated backbones include **Qwen3-VL-4B**, **Qwen3-VL-8B**, **Qwen2.5-VL-3B**, and **InternVL3.5-4B**. The main open-source system is built on **Qwen3-VL-4B-Instruct**.
The adaptation strategy is deliberately narrow: the paper freezes the vision encoder and multimodal projector, and fine-tunes only the language model parameters using LoRA. For the main Qwen3-VL-4B configuration, the reported SFT settings include LoRA rank **64**, maximum sequence length **8192**, image maximum pixels **262,144**, learning rate **2e-5**, per-device batch size **1**, gradient accumulation **16**, **3** epochs, cosine scheduling, warmup ratio **0.1**, and **bf16** precision.
Inference is autoregressive code generation. The maximum generation length is **8192 tokens**, and greedy decoding (`do_sample=False`) is used by default because SVG is a structured program-synthesis task in which minor token errors can break syntax or renderability. After generation, the `` content is extracted. The paper does not define a custom SVG grammar or tokenizer; a plausible implication is that task adaptation is achieved primarily through dataset design and objective shaping rather than through a specialized decoder formalism.
## 4. Coarse-to-fine supervised learning and RL refinement
The supervised stage optimizes the conditional likelihood
$$
\mathcal{L}_{\text{SFT}} = -\mathbb{E}_{(x,y)\sim\mathcal{D}} \left[\log p_\theta(y \mid x)\right].
$$
The distinctive aspect is the curriculum. Training begins on simpler diagrams and primitive-heavy data, then transitions to complex real scientific figures. The paper characterizes this as first learning “atomic primitives” and then refining global diagram composition. It explicitly states that direct training on complex figures from the start often causes unstable convergence, degenerate outputs, and difficulty in learning local primitives and global structure jointly [2603.24575].
Reinforcement learning is then applied using **Group Relative Policy Optimization (GRPO)**. The objective is
$$
\mathcal{L}_{\text{GRPO}} = \mathbb{E}_{x \sim \mathcal{D},\, \hat{y} \sim \pi_\theta(\cdot|x)} \left[ R(x,\hat{y}) - \beta \,\mathrm{KL}\!\left( \pi_\theta(\cdot|x)\,\|\,\pi_{\text{ref}}(\cdot|x) \right) \right].
$$
Here $\pi_\theta$ is the current policy, $\pi_{\text{ref}}$ is the frozen SFT policy, and the KL coefficient is **0.01**. RL rollouts render sampled SVG candidates with CairoSVG and score them using **Gemini-3-Flash** as a judge. The reward is the unweighted average of four rubric components,
$$
\mathcal{R} = \frac{1}{4}\left(r_{\text{pres}} + r_{\text{layout}} + r_{\text{conn}} + r_{\text{det}}\right),
$$
where the terms correspond to presence, layout, connectivity, and details. Malformed or unrenderable SVG receives zero reward.
The reported RL configuration uses **8** samples per prompt, learning rate **9e-6**, entropy bonus **0.001**, train batch size **64**, validation batch size **64**, optimization minibatch size **16**, micro-batch per GPU **1**, maximum prompt length **9000**, maximum response length **8500**, maximum model length **17,500**, **bf16** precision, **4× L40S** hardware, and approximately **30 hours** of training. On **100 annotated examples**, the judge’s overall Pearson correlation with human judgments is reported as **0.89**, with category-level correlations of **0.79** for presence, **0.63** for layout, **0.83** for connectivity, and **0.87** for details. This suggests that RL is being driven by a reward that is structurally aligned with the target task rather than by pixel similarity alone.
## 5. VFIG-BENCH and empirical performance
VFIG-BENCH is introduced because existing SVG benchmarks are described as being dominated by icons, emojis, and simple graphics rather than structurally complex scientific figures [2603.24575]. The benchmark contains **392 realistic scientific figures** held out from VFIG-DATA, and the evaluation protocol is complemented by **500** held-out samples from Molmo2-Diagram and **474** official test examples from SVG-Diagrams.
Evaluation is explicitly multi-granular. Rasterized predictions are scored with **SSIM** and **LPIPS**. Image-level feature similarity is summarized as **VisualSim**, defined as the average cosine similarity of DINO, CLIP, and SigLIP image embeddings. Holistic structural quality is measured by **VLM-Judge**, which is the mean score of Gemini and GPT judges over presence, layout, connectivity, and details. SVG code quality is measured using **Clean** and **Render**, the latter being the successful rendering rate.
The main VFIG-BENCH results are summarized below.
| Model | VLM-Judge | Note |
|---|---:|---|
| OmniSVG-4B | 0.039 | Open-source baseline |
| Starvector-8B | 0.548 | Open-source baseline |
| Qwen3-VL-4B | 0.466 | Backbone without VFIG training |
| VFIG-4B (SFT) | 0.781 | Two-stage supervised model |
| VFIG-4B (SFT+RL) | 0.829 | Best open-source result |
| GPT-5.2 | 0.858 | Closed-source comparison |
For **VFIG-4B (SFT+RL)** on VFIG-BENCH, the paper reports **SSIM 0.778**, **LPIPS 0.212**, **VisualSim 0.957**, **VLM-Judge 0.829**, **Clean 0.853**, and **Render 0.960**. On **Molmo2-Diagram**, the same model achieves **VLM-Judge 0.834**; on **SVG-Diagrams**, **0.705**. A larger **Qwen3-VL-8B** variant with two-stage SFT and RL reaches **VLM-Judge 0.845** on VFIG-BENCH, with **VisualSim 0.960**.
The ablations attribute a substantial share of the gain to curriculum and RL. For **Qwen3-VL-4B**, one-stage SFT yields **VLM-Judge 0.712**, while two-stage SFT reaches **0.737**, and render rate improves from **0.749** to **0.933**. On VFIG-BENCH, **VFIG-4B (SFT)** scores **0.781**, and **VFIG-4B (SFT+RL)** rises to **0.829**. Reward ablations show that removing any of the four rubric components harms performance; among the reported variants, omitting layout or details produces some of the largest degradations.
The benchmark also highlights a methodological contrast with classical tracing. **VTracer** attains **SSIM 0.950** and **VLM-Judge 0.838** on VFIG-BENCH, but **Clean 0.000**, because it generates path-heavy non-semantic SVG. This illustrates the paper’s central distinction between visual resemblance and semantic editability.
## 6. Limitations, scope, and terminological ambiguities
VFIG is most directly targeted at diagram-centric figure reconstruction. The paper identifies scientific diagrams, architecture figures, flowcharts, pipeline diagrams, process illustrations, and technical block diagrams as strong application domains [2603.24575]. It also states that the training and benchmark distributions are biased toward structured scientific diagrams rather than toward arbitrary vector graphics.
The reported weaknesses are concentrated in fine-grained fidelity. The paper explicitly notes difficulties with thin lines, arrowheads, small text, exact fonts, subtle styling, local geometric details, and 3D-like or perspective-like shapes. Font fidelity is described as consistently weak, and generalization to icons, logos, sketches, object drawings, and sparse geometric art is limited. The reward design is also acknowledged to be imperfect: equal weighting of presence, layout, connectivity, and details may not optimally reflect human preference, and the exploration of hybrid structural-plus-pixel rewards remains limited.
A further source of confusion is nomenclature. In current arXiv literature, **VFIG** refers to figure-to-SVG conversion with vision-language models [2603.24575]. It is distinct from **GVIF**, the “Generative Visual Information Fidelity” metric for generative semantic communications [2505.10405]; from **VFG**, the “Variational Flow Graphical Model” [2207.02722]; from **VFI** systems such as “TLB-VFI” for video frame interpolation [2507.04984]; and from **FIGS**, the “Faint Infrared Grism Survey” in astronomy [1706.02669]. This suggests that, despite superficial acronym overlap, VFIG occupies a separate research line centered on structured visual program generation and editable vector reconstruction.
Source: https://www.emergentmind.com/topics/vfig