---
title: 'FigEdit: A Benchmark for Chart Editing'
url: https://www.emergentmind.com/topics/figedit
type: topic
---

# FigEdit: A Benchmark for Chart Editing

to=arxiv_search  微信天天中彩票  大发时时彩怎么 code
to=arxiv_search.search code
FigEdit is a large-scale benchmark for scientific figure editing introduced to evaluate a specific claim: a chart is not merely an arrangement of pixels but a visual representation of structured data governed by a graphical grammar. In this formulation, a chart image is the rendering of an underlying specification rather than a free-form bitmap, so editing a chart is treated as a structured transformation problem rather than a pixel-manipulation task. FigEdit comprises over 30,836 samples, covers 10 chart types, and is organized into five task families—single edits, multi edits, conversational edits, visual-guidance-based edits, and style transfer—using executable Vega/Vega-Lite specifications, rendered before–after pairs, and natural-language instructions [2512.00752].

## 1. Formal model and scope

The benchmark formalizes a natural image as a mapping
$$
I : \mathbb{R}^2 \rightarrow \mathbb{R}^3
$$
from coordinates to RGB values, but formalizes a chart as
$$
I = R(\sigma),
$$
where the specification $\sigma$ lives in a structured space $\Sigma$ and is decomposed as
$$
\sigma = (C, S).
$$
Here, content $C$ includes the dataset $D$, chart type $\tau$, and encodings, while style $S$ includes palettes, fonts, strokes and fills, gridlines, legend layout, spacing, and margins [2512.00752].

On this view, an atomic edit is a function on specifications,
$$
f_e : \Sigma \rightarrow \Sigma,
$$
so the edited output is produced as
$$
I^\star = R(\sigma^\star), \quad \sigma^\star = f_e(\sigma).
$$
Examples stated for this framework include adding a bar for a category with a specified value, or increasing axis-label font size while leaving data and geometry unchanged. The formulation emphasizes invariants that are specific to charts: data–encoding alignment, axis coherence, legend integrity, and text correctness [2512.00752].

A common misconception is that chart editing can be handled as ordinary image editing with stronger prompting. FigEdit is built precisely to challenge that assumption. The benchmark argues that state-of-the-art editors trained on natural images are optimized for perceptual realism and prompt alignment, not for preserving executable chart semantics such as correct bar heights, consistent legends, or stable axis ranges [2512.00752].

## 2. Benchmark composition and construction

FigEdit covers 10 chart types: area, line, bar, stacked-bar, pie, donut, box, violin, scatter, and dot. The benchmark uses curated real-world datasets from sources including Kaggle, UCI, World Bank WDI, OECD PISA, US Census, BLS, BEA, WHO, FRED, and NOAA, with chart-class preference lists used to encourage coherent dataset–chart pairings [2512.00752].

Base figures are produced through LLM-guided Vega spec proposal. The model is instructed to output a JSON object
$$
o = \{ \text{vega\_spec} = \sigma, \ \text{dataset} = d \}, \quad d \in \mathcal{A},
$$
with a valid Vega v6 schema, inline `"values"` data, and field structures appropriate to the chart type. Invalid outputs are rejected and resampled [2512.00752].

Construction includes several explicit validation layers. Schema and field-shape validation enforce chart-type requirements. Duplicate and near-duplicate control uses exact and structural signatures, including
$$
h_{\mathrm{exact}}(\sigma), \quad h_{\mathrm{multi}}(\sigma), \quad s_{\mathrm{val}}(\sigma), \quad s_{\mathrm{struct}}(\sigma),
$$
to reject repeated or trivially similar specifications. Semantic diversity is further enforced by token-overlap control over categorical terms using a sliding window and Jaccard-style overlap thresholding [2512.00752].

The underlying edit generation process uses Vega/Vega-Lite as the editable representation. For each base chart, the system samples feasible operations, generates natural-language instructions with OP tags, applies deterministic edit functions to the specification, validates the edited spec, and renders the ground-truth image pair. This yields paired images together with executable before–after specifications rather than only image-space annotations [2512.00752].

## 3. Task families and edit vocabulary

FigEdit defines five task families of increasing difficulty. In the single-edit task, the input is an original chart image and a natural-language instruction describing exactly one atomic edit. In the multi-edit task, the instruction describes two or more edits composed in a canonical order. In the conversational task, the system must apply sequential edits across turns while preserving prior modifications. In the style-transfer task, a target chart must preserve its content while matching a style attribute taken from a reference chart. In the visual-guidance task, a separate guidance image marks the target region with a thin red circle, and the edit must apply only there [2512.00752].

The benchmark’s atomic operation set is
$$
\mathcal{O} = \{
\text{change\_datapoint\_color},
\text{increase\_text\_size},
\text{decrease\_text\_size},
\text{change\_background\_color},
\text{increase\_category\_spacing},
\text{decrease\_category\_spacing},
\text{add\_title},
\text{add\_datapoint},
\text{remove\_datapoint}
\}.
$$
These are grouped in the paper as data-centric operations, text operations, style editing, layout or spacing adjustment, and font operations [2512.00752].

The benchmark also reports the operation distribution. In the single-edit split, add element has 1,941 images, remove element 1,892, add title 1,942, change background color 1,944, change data color 1,729, adjust category spacing 1,729, and font adjustment 2,943, for a total of 14,105 single edits. Multi edit includes 3,370 dual-operation instances and 2,660 triple-operation instances. Visual guidance includes 1,666 change-data-color cases and 1,819 remove-element cases. Style transfer includes 1,511 transfer-style examples, 1,728 change-data-color examples, and 387 category-spacing examples [2512.00752].

Instructions are natural-language sentences followed by canonical OP tags. The parameterization includes, for example, `new_bg=<css-or-#hex>` for background edits, `axis_label_size=<int 6..30>` for font changes, and structured arguments such as `target_category`, `target_series`, `position`, `category`, and `value` for datapoint addition, recoloring, or removal. This pairing of free-form language with deterministic operation metadata is one of the benchmark’s defining features [2512.00752].

## 4. Evaluation protocol and the metric-gap problem

FigEdit evaluates outputs in image space but treats semantic correctness as central. The benchmark uses SSIM, PSNR, LPIPS, CLIP similarity, OCR similarity, and an LLM-based instruction score. OCR similarity is defined as
$$
\mathrm{Sim}_{\text{OCR}} = 1 - \frac{\mathrm{EditDist}(s_{\widehat{I}}, s_{I^\star})}{\max(|s_{\widehat{I}}|, |s_{I^\star}|)},
$$
where OCR strings are extracted from the generated image and the ground truth [2512.00752].

The LLM-based judge uses GPT-4o and produces `score_instruction`, `score_preservation`, `score_quality`, and an overall `score`, each on a 1–5 scale, together with JSON reasoning. The paper’s point is not merely that charts need more metrics, but that chart edits can be visually close to the source while remaining semantically invalid. High SSIM or PSNR can therefore coexist with failure to execute the requested edit [2512.00752].

This metric-gap is a central contribution of the benchmark. The analysis explicitly states that classic metrics overestimate performance in chart editing because only small regions may change while the semantic meaning changes severely. A model can ignore the requested modification, misplace a bar, distort a legend, or alter an axis range and still achieve strong pixel-level scores. FigEdit therefore uses OCR and LLM-as-judge evaluation to expose failures that image-reconstruction metrics miss [2512.00752].

A second misconception addressed by the benchmark is that preservation can be judged solely by closeness to the input. In chart editing, preservation is constrained by chart grammar. Unedited components should remain intact, but edited components must also remain semantically valid with respect to the underlying data and encodings. This makes “preservation” stricter than ordinary visual similarity [2512.00752].

## 5. Baseline models and empirical findings

The benchmark evaluates four representative image-editing systems: GPT-Image, Imagen 4, OmniGen2, and InstructPix2Pix. For single and multi edits, the models receive the original chart and the instruction. For conversational edits, multi-turn prompting is used where supported. For visual guidance, the original chart and the red-circled guidance image are provided. For style transfer, a target chart and a reference chart are supplied in a two-panel prompt [2512.00752].

The results support the benchmark’s thesis. On single edits, Imagen 4 attains the highest SSIM at 0.7726 and PSNR at 13.04, but has low semantic scores: `Instr 1.58`, `Preserv 1.51`, and `Qual 2.05`. GPT-Image has lower SSIM and PSNR, at 0.7295 and 10.32, but much stronger LLM scores: `Instr 3.47`, `Preserv 1.71`, and `Qual 2.45`. InstructPix2Pix and OmniGen2 are intermediate, with OmniGen2 reaching `Instr 3.35`, `Preserv 2.55`, and `Qual 2.85` on single edits [2512.00752].

The multi-edit setting is harder for all models. Imagen 4 falls to `Instr 1.26` and `Preserv 1.32`, while GPT-Image reaches `Instr 2.51` and `Preserv 1.63`; InstructPix2Pix and OmniGen2 remain weak as well. In conversational editing, GPT-Image is notably stronger on instruction following, with `Instr 4.59`, but preservation remains only moderate at `2.51`. In visual guidance, GPT-Image obtains the best OCR score at 0.4665 and strong quality at `3.95`, yet even there the paper reports frequent failures to limit the edit to the circled region. In style transfer, Imagen 4 again has the highest SSIM at 0.8500 and PSNR at 14.00, while GPT-Image leads on semantic measures with OCR 0.5092 and LLM scores `Instr 3.06`, `Preserv 3.57`, and `Qual 4.16` [2512.00752].

The per-instruction breakdown identifies data-centric edits as especially difficult. “Add a new data point” and “Remove an existing data point” are the hardest categories across systems. Reported failures include misplacing bars or points, changing the wrong element, distorting axes, or neglecting the requested removal entirely. This supports the broader conclusion that current editors treat charts as generic images rather than as structured visualizations [2512.00752].

## 6. Position in the broader editing literature

FigEdit occupies a distinct place within the rapidly expanding literature on image editing. General benchmarks and editors such as ImgEdit introduce 1.2 million instruction-based edit pairs, a VLM-based editor, and a benchmark spanning basic, hard single-turn, and multi-turn settings, but are designed for open-domain image editing rather than chart-specific graphical grammar [2505.20275]. Fine-grained editors such as PartEdit target object parts by learning special textual tokens that produce localization masks and then blending source and edited features during denoising [2502.04050]. Bounding-box-guided editors such as FineEdit use multi-level bounding box injection, a 1.2 million-pair dataset, and a dedicated region-editing benchmark to improve localization and background preservation [2604.10954].

Latency-optimized localized editors address a different axis. FlashEdit explicitly decouples speed, structure, and semantics through One-Step Inversion-and-Editing, Background Shield, and Sparsified Spatial Cross-Attention, reporting under 0.2 seconds per edit and a 150.84$\times$ speedup over DDIM+Prompt-to-Prompt while emphasizing background consistency and localized changes [2509.22244]. FireRed-Image-Edit further scales instruction-based editing through a 1.6B-sample corpus, balanced generation and editing data, preference optimization, online RL, and layout-aware OCR rewards for text editing [2602.13344]. Earlier optimization-based work such as Forgedit focused on per-image fine-tuning, vector projection in text-embedding space, and UNet forgetting strategies to control identity similarity and editing strength [2309.10556].

FigEdit’s contribution is not that these methods are irrelevant to figures, but that their success criteria are still largely image-space criteria. The benchmark argues that scientific charts require structure-aware models that can reason over executable specifications, data tables, encodings, legends, and typography jointly. This suggests that future figure editing systems will need to combine multimodal editing with chart parsing, specification editing, or other grammar-aware representations rather than relying on pixel-level manipulation alone [2512.00752].

## 7. Research implications and future directions

FigEdit is intended as a benchmark, but it also functions as a problem statement for structure-aware figure editing. The paper points toward models that parse a chart into data, marks, encodings, and style, apply edits at the specification level, and re-render the result. It also identifies open problems: robust parsing of arbitrary scientific charts into executable specifications, faithful edit application when the original specification is unavailable, evaluation protocols that compare specifications or data when possible, and extension to more complex multi-panel figures and additional chart types [2512.00752].

This position has direct implications for “FigEdit”-style systems in the broader sense used by adjacent editing work. FlashEdit, FineEdit, PartEdit, and ImgEdit all emphasize localized edits, preservation outside the edit region, and interactive workflows, which are properties that remain relevant for plots, diagrams, and document figures [2509.22244][2604.10954][2502.04050][2505.20275]. FigEdit sharpens that agenda by showing that localization and preservation, while necessary, are not sufficient for scientific charts unless the model also respects the underlying structured transformation defined by the graphical grammar [2512.00752].

In that sense, FigEdit marks a shift in how figure editing is conceptualized. It treats scientific charts as rendered programs rather than as ordinary images, supplies a common benchmark grounded in real data and executable specifications, and demonstrates that contemporary image editors remain weak on the semantic constraints that matter in scientific workflows. Its central legacy is therefore methodological: chart editing should be evaluated, and likely modeled, as specification-aware transformation rather than as perceptually plausible image synthesis [2512.00752].

Source: https://www.emergentmind.com/topics/figedit