---
title: 'ArtEdit-Bench: Art Editing Benchmark'
url: https://www.emergentmind.com/topics/artedit-bench
type: topic
---

# ArtEdit-Bench: Art Editing Benchmark

ArtEdit-Bench denotes an art-focused benchmark concept for instruction-based image editing: a benchmark intended to measure how reliably a model can apply artistic, stylistic, semantic, and localized edits to an input image while preserving non-target content and respecting compositional constraints. In the current literature represented here, ArtEdit-Bench is not introduced as a single formal release; rather, it is repeatedly treated as a natural extension of recent benchmarks for complex instruction following, referring-expression editing, localization, unified edit evaluation, and art-domain knowledge assessment. Taken together, these works define the main design axes of such a benchmark: compositional instruction structure, localized control, multi-turn interaction, mask-aware consistency measurement, and art-specific evaluation criteria [2506.12830] [2602.05577] [2506.03448] [2505.20275] [2607.02007].

## 1. Conceptual scope and benchmark lineage

ArtEdit-Bench belongs at the intersection of instruction-based image editing and art-oriented evaluation. The relevant precursor benchmarks do not share a single objective. "ComplexBench-Edit" focuses on complex, multi-instruction, and chain-dependent editing; "RefEdit-Bench" isolates failures on referring expressions in cluttered scenes; "ImgEdit-Bench" organizes broad editing evaluation around instruction adherence, editing quality, and detail preservation, including multi-turn interaction; "LocateEdit-Bench" addresses the forensic question of where an edit occurred; and "EduArt" shows that art-domain evaluation requires format diversity and item-level measurement rather than multiple-choice saturation alone [2506.12830] [2506.03448] [2505.20275] [2602.05577] [2607.02007].

A useful way to situate ArtEdit-Bench is to treat it as an art-domain analogue of these benchmarks rather than as a mere style-transfer testbed. That framing matters because the cited works consistently show that current models can appear strong on simple or global edits while failing on compositional dependencies, entity disambiguation, localization, or iterative editing.

| Source benchmark | Primary emphasis | Transferable element for ArtEdit-Bench |
|---|---|---|
| ComplexBench-Edit | Complex multi-instruction editing | Parallel and chain-dependent edit structure |
| RefEdit-Bench | Referring-expression editing | Multi-entity disambiguation |
| ImgEdit-Bench | Unified edit evaluation | Three-dimensional scoring and multi-turn tasks |
| LocateEdit-Bench | Edit localization | Pixel-level edited-region masks |
| EduArt | Art-domain benchmark design | Format diversity and psychometric analysis |

This benchmark lineage also guards against a common misconception: an art-editing benchmark is not exhausted by measuring global style change. The precursor literature points instead toward a broader target capability set, including localized style application, region-specific preservation, sequential edits, reference resolution, and evaluation protocols that do not over-reward visually plausible but instruction-inaccurate outputs.

## 2. Task structure: compositionality, reference resolution, and interaction

The most explicit structural template comes from "ComplexBench-Edit," which organizes each sample around three instructions and three levels of complexity: **Parallel**, **Two-chain**, and **Three-chain**. Parallel instructions are logically independent; Two-chain has a dependent pair plus one independent instruction, formally \(T_1 \rightarrow T_2,\; T_3\) independent; Three-chain uses \(T_1 \rightarrow T_2 \rightarrow T_3\). The benchmark also defines a hierarchical editing taxonomy spanning object-level, attribute-level, and global-level operations, with categories such as Add Object, Change Object, Delete Object, Change Color, Change Pose, Change Material, Change Content, Change Text, Change Background, and Change Style [2506.12830].

For an art-oriented benchmark, the source text explicitly proposes adapting this structure to artistic dependencies. Examples given include independent edits such as changing brush style, altering the palette to warm colors, and darkening the background; a two-step chain such as converting to watercolor and then overlaying ink outlines; and a longer chain such as changing style to cubism, then adding a surreal floating object, then blurring the background while keeping the figure sharp. This suggests that ArtEdit-Bench would be most informative when it evaluates not just whether a model can stylize an image, but whether it can maintain an internal notion of scene state across dependent artistic operations [2506.12830].

"RefEdit-Bench" adds a second structural requirement: the ability to resolve referring expressions in complex multi-entity scenes. It is built on RefCOCO, contains 200 images divided into Easy and Hard subsets, and uses five edit categories—change color, change object, add object, remove object, and change texture. The Hard subset is defined by multiple similar instances and overlapping or nearby objects, so that the editing system must identify the correct referent from contextual language such as relative position, clothing, or nearby objects [2506.03448]. In an ArtEdit-Bench setting, this directly motivates prompts of the form “turn only the foreground figure into monochrome” or “stylize the leftmost building as ukiyo-e while keeping the rest realistic,” although that exact formulation would be an inference.

"ImgEdit-Bench" broadens the task definition further by including a **Basic-Edit Suite**, a challenging **Understanding–Grounding–Editing (UGE) Suite**, and a **Multi-Turn Suite**. The multi-turn suite is divided into **Content Memory**, **Content Understanding**, and **Version Backtracking**, thereby treating persistent constraints, pronoun-based or implicit references, and undo-like operations as first-class editing competencies [2505.20275]. A plausible implication is that a mature ArtEdit-Bench would need not only single-turn artistic transformations but also iterative creative workflows, since artistic editing often proceeds through refinement, preservation of prior constraints, and selective reversal of earlier changes.

## 3. Data construction and annotation pipelines

The precursor benchmarks supply several compatible construction pipelines, each emphasizing a different aspect of supervision quality. "ComplexBench-Edit" begins from MSCOCO images and applies a Vision Content Filter with two explicit constraints: an **Intra-class Frequency Constraint**, which discards any image where some category appears more than twice, and a **Category Diversity Constraint**, which discards any image with fewer than three distinct categories. Formally, retained images satisfy \(\forall c \in C(I),\; N_c(I)\leq 2 \land |C(I)|\geq 3\). Instructions are then generated by an MLLM using the image, an object list from MSCOCO, and a chosen edit-type combination; chain instructions are generated sequentially with prior instructions as context. Automatic validation is performed by a second MLLM for object-scene compatibility, rationality of references and attributes, and conflicts between instructions, after which two PhD students review a subset for edit-type consistency and post-edit reasonableness [2506.12830].

The same source explicitly proposes how this pipeline could be transferred to an art setting: replace MSCOCO with an art dataset, adapt the type taxonomy to artistic attributes such as composition, style, and brushwork, and substitute realism-oriented plausibility checks with artistic coherence checks. Because that proposal appears as a transfer suggestion rather than a reported implementation, it is best understood as a benchmark design pattern rather than a released ArtEdit-Bench protocol [2506.12830].

"RefEdit" contributes a different pipeline centered on targeted synthetic supervision. Its training data, RefEdit-Data, contains more than 20,000 editing triplets generated through GPT-4o text-side scene and instruction generation, FLUX image generation at \(1024 \times 1024\), Grounded SAM mask extraction from expanded referring expressions, and edit realization with FlowChef or Inpaint Anything depending on edit type. The authors emphasize that each image is constructed to contain multiple similar entities with distinguishing features, and that this targeted dataset corrects a failure mode not addressed by generic large-scale editing corpora [2506.03448]. For ArtEdit-Bench, the transferable principle is straightforward: targeted synthetic data for localized artistic edits may be more valuable than much larger but less specific style-edit collections.

"ImgEdit" provides a large-scale curation template. Its 1.2 million edit pairs are derived from high-resolution LAION-Aesthetics images, filtered by resolution and aesthetic score, grounded with YOLO-World and SAM2, filtered further by CLIPScore and area ratio, and converted into task-specific edits through FLUX, SDXL, IP-Adapter, ControlNet, Canny/Depth LoRA, and post-hoc GPT-4o quality filtering. It also records multi-turn interaction sequences and, for many tasks, masks or boxes for edited objects [2505.20275]. This suggests that an art benchmark could likewise combine dense metadata, task-specific generation pipelines, and high-resolution source material rather than relying on low-resolution style-transfer pairs alone.

"LocateEdit-Bench" contributes a mask-generation pipeline absent from most editing benchmarks. Starting from OmniEdit-derived instructions, it filters to object-centric separable edits, regenerates outputs with four modern instruction-based editors, extracts the target object via ChatGPT, and obtains a pixel-level mask using SAM3, retaining only masks with confidence \(\geq 0.5\). The final dataset contains 231,093 edited images with pixel-level masks [2602.05577]. For ArtEdit-Bench, that pipeline is especially significant if localization or region-specific preservation is to be evaluated directly rather than inferred.

## 4. Evaluation frameworks and metrics

The cited literature converges on the view that no single metric is sufficient. "ComplexBench-Edit" evaluates instruction adherence with an MLLM-based 5-point score \(S_i \in \{0,1,2,3,4,5\}\) for each instruction, and defines a multiplicative chain score for dependent edits:
\[
S_{\text{chain}} = S_1 \times S_2 \times \dots \times S_N.
\]
This formulation zeroes the chain score if any required step fails. The same work argues that standard whole-image \(L1/L2\) measures are flawed for consistency analysis, and therefore defines potentially edited regions from the union of original and edited bounding boxes, computes \(R_{\text{consistent}} = \text{Image Area} \setminus R_{\text{edit}}\), and evaluates \(L1/L2\) only over \(R_{\text{consistent}}\) [2506.12830]. The source text further proposes replacing boxes with segmentation masks, or using \(\text{LPIPS}\) or \(\mathrm{SSIM}\) over \(R_{\text{consistent}}\), in art domains.

"ImgEdit-Bench" formalizes a broader three-dimensional scoring model: **instruction adherence**, **image-editing quality**, and **detail preservation**. A key design decision is that editing quality and preservation are capped by instruction adherence:
\[
\tilde{s}_{\text{edit}}=\min(s_{\text{edit}}, s_{\text{inst}}), \qquad
\tilde{s}_{\text{pres}}=\min(s_{\text{pres}}, s_{\text{inst}}).
\]
This prevents a model from scoring highly on technical cleanliness or preservation when it has not actually executed the requested edit [2505.20275]. For an art benchmark, this is a principled way to avoid rewarding visually attractive but semantically wrong stylizations.

"RefEdit-Bench" evaluates **Semantic Consistency (SC)**, **Perceptual Quality (PQ)**, and an overall score
\[
O = \sqrt{SC \times PQ}.
\]
It also introduces **Modified VIEScore**, in which ground-truth masks crop the target region before GPT-4o judges correctness, thereby making localized evaluation more sensitive to whether the correct entity was edited [2506.03448]. That mechanism is immediately relevant to art tasks requiring style changes restricted to one object or region.

"LocateEdit-Bench" adds explicit localization metrics. It evaluates edited-region prediction with Accuracy, Precision, Recall, F1, AUC, Dice, and IoU, and does so under two protocols: full-set evaluation across all four editors and cross-editor generalization from one editor to all others [2602.05577]. This shifts evaluation from “did the image look correct?” to “can the edited region itself be segmented?”, a distinction that matters for controllability, auditability, and non-target preservation in artistic editing systems.

"EduArt" contributes an orthogonal but important lesson: benchmark quality depends on format diversity and psychometric characterization. It measures item difficulty and item discrimination and uses logistic regression to isolate the effects of format, image presence, language, and model identity [2607.02007]. A plausible implication is that ArtEdit-Bench should not report only aggregate edit scores; it should also analyze which task formats discriminate among strong models and which ones saturate.

## 5. Empirical lessons from adjacent benchmarks

The empirical record across these benchmarks shows that apparently strong editing models remain brittle once evaluation moves beyond simple single-turn prompts. On "ComplexBench-Edit," Gemini-CoT achieves the best overall average editing score, **40.47**, compared with **36.70** for Gemini, but scores drop sharply with complexity: in **Three-chain – Obj-At**, Gemini scores **15.10** and Gemini-CoT **17.54**. Performance is also edit-type dependent: Gemini-CoT is best on Add Object (**48.23**), Change Object (**44.78**), Delete Object (**66.38**), Change Color (**41.98**), Change Material (**32.50**), Change Text (**28.68**), and Change Background (**38.98**), but it is weaker on Change Style (**30.69**) than Step1X-Edit (**45.54**) and OmniGen (**44.75**). On background preservation, OmniGen has the best consistency with average \(L1=0.0377\) and \(L2=0.0068\), whereas Gemini-CoT reaches \(L1=0.0846\) and \(L2=0.0281\), indicating a trade-off between aggressive editing and non-target consistency. Human preference data further show Gemini-CoT preferred over Gemini in **67%** of pairs, over OmniGen in **76%**, and over Step1X-Edit in **63%** [2506.12830].

These findings are directly relevant to ArtEdit-Bench because artistic editing often combines global transformation with local preservation. The precursor evidence indicates that stronger instruction following can coincide with worse background preservation, and that specialized models may excel on global style changes even when they lag on compositional object-attribute edits. That pattern cautions against treating “style” as a proxy for overall editing competence.

"RefEdit" shows a related failure mode: models trained on millions of samples still struggle when the prompt must identify the correct entity in a complex scene. The benchmark uses 200 RefCOCO-derived images, evenly split between Easy and Hard. The authors report that RefEdit, trained on only 20,000 editing triplets, outperforms Flux/SD3-model-based baselines trained on millions of data and improves both RefEdit-Bench and PIE-Bench performance [2506.03448]. The key lesson for ArtEdit-Bench is that data specificity can dominate raw scale when the task requires localized artistic control over ambiguous compositions.

"ImgEdit-Bench" arrives at a similar conclusion from another angle. It contains **811** total test samples across **14** sub-tasks and shows that GPT-4o-Image is strongest overall, while ImgEdit-E1 and Step1X-Edit are the strongest open-source systems. Yet even frontier proprietary models remain weak on multi-turn **content memory** and **content understanding**, despite being better at **version backtracking** [2505.20275]. This suggests that a benchmark evaluating only single-turn art stylization would miss an important part of realistic creative workflows.

"LocateEdit-Bench" adds evidence from the forensic side. With **231,093** edited images, it finds that semantic-heavy segmenters outperform low-level anomaly detectors on instruction-based edits; SegFormer reaches average F1 of about **84.8%**, Dice about **82.6**, and mIoU about **77.1**, while Mesorch reaches average F1 about **86.1%**, Dice about **82.1**, and mIoU about **76.8**. Cross-editor generalization degrades markedly, and BAGEL-edited images are consistently the hardest to localize [2602.05577]. For ArtEdit-Bench, a plausible implication is that localization-aware evaluation should expect strong editor-specific effects rather than assuming one universal notion of edit traceability.

Finally, "EduArt" shows that evaluation format can change conclusions even in the same domain. Six models exceed **90%** on `multiple_choice_radio`, but the abstract reports that models above **94 percent** on multiple choice can fall to **23.9 percent** on open completion and **6.2 percent** on error identification. The benchmark also reports strong psychometric properties, with mean discrimination **0.514** and **82.3 percent** good discriminators [2607.02007]. Although EduArt evaluates multimodal LLMs rather than image editors, its central finding generalizes: benchmark formats that saturate on recognition tasks can hide major capability deficits in production, correction, or transformation tasks.

## 6. Limitations, misconceptions, and likely development paths

A first limitation is domain mismatch. The precursor editing benchmarks are largely grounded in realistic image corpora—MSCOCO, RefCOCO, LAION, OpenImages, and OmniEdit-derived natural imagery—rather than paintings, illustrations, or stylized compositions [2506.12830] [2506.03448] [2505.20275] [2602.05577]. Consequently, any direct transfer to ArtEdit-Bench remains partial. The cited texts explicitly acknowledge this for ComplexBench-Edit and RefEdit, both of which present their art-domain relevance as an extension rather than a completed benchmark implementation.

A second limitation concerns region definition. ComplexBench-Edit uses bounding boxes to estimate edited and consistent regions, which the source text itself describes as coarse; LocateEdit-Bench improves this by generating pixel-level masks through SAM3, but its edit space is restricted to **Object Addition**, **Object Swap**, and **Attribute Modification** [2506.12830] [2602.05577]. An art benchmark would likely require finer masks for stylized, overlapping, or texture-level edits, especially when the intended transformation affects brushwork, color fields, or layered compositional regions rather than discrete object instances. This is an inference, but it follows directly from the limitations described.

A third issue is evaluator reliability. ComplexBench-Edit and RefEdit rely on MLLM-based judgment; ImgEdit-Bench uses GPT-4o and also trains ImgEdit-Judge; EduArt demonstrates that task format itself strongly modulates accuracy and discrimination [2506.12830] [2506.03448] [2505.20275] [2607.02007]. Together, these results undermine the misconception that a single VLM judge or a single score dimension can faithfully summarize artistic editing performance. In an art benchmark, evaluator calibration, format diversity, and expert review would therefore be central design concerns.

The most plausible development path for ArtEdit-Bench is a hybridization of the benchmark families summarized above. Such a benchmark would combine: compositional multi-instruction structure and chain-aware scoring from ComplexBench-Edit; referring-expression and multi-entity difficulty from RefEdit; unified edit-quality and multi-turn evaluation from ImgEdit-Bench; pixel-level localization from LocateEdit-Bench; and psychometric analysis plus art-domain task diversity from EduArt [2506.12830] [2506.03448] [2505.20275] [2602.05577] [2607.02007]. The cited texts also point toward specific extensions: mask-based rather than box-based consistency evaluation, style-aware preservation metrics, structured plan consumption instead of plain prompt concatenation, more diverse instruction types, and expert human studies for domains where aesthetic judgment is irreducible.

In that synthesized sense, ArtEdit-Bench is best understood not as a single fixed dataset already standardized in the literature, but as an emerging benchmark design program for rigorous evaluation of art-oriented instruction-based image editing. Its defining challenge is to measure artistic controllability without collapsing the problem into either global style transfer alone or generic image-edit similarity metrics.

Source: https://www.emergentmind.com/topics/artedit-bench