ArtEdit-Bench: Art Editing Benchmark

• ArtEdit-Bench is an art-oriented benchmark that defines evaluation axes such as compositional instruction structure, localized control, and multi-turn interactions.
• It integrates methodologies from ComplexBench-Edit, RefEdit-Bench, ImgEdit-Bench, LocateEdit-Bench, and EduArt to adapt art-domain pipelines and psychometric analysis.
• The benchmark emphasizes precise evaluation of localized style changes, sequential edit consistency, and art-specific quality metrics in image editing.

ArtEdit-Bench denotes an art-focused benchmark concept for instruction-based image editing: a benchmark intended to measure how reliably a model can apply artistic, stylistic, semantic, and localized edits to an input image while preserving non-target content and respecting compositional constraints. In the current literature represented here, ArtEdit-Bench is not introduced as a single formal release; rather, it is repeatedly treated as a natural extension of recent benchmarks for complex instruction following, referring-expression editing, localization, unified edit evaluation, and art-domain knowledge assessment. Taken together, these works define the main design axes of such a benchmark: compositional instruction structure, localized control, multi-turn interaction, mask-aware consistency measurement, and art-specific evaluation criteria (Wang et al., 15 Jun 2025, Wu et al., 5 Feb 2026, Pathiraja et al., 3 Jun 2025, Ye et al., 26 May 2025, Spinaci et al., 2 Jul 2026).

1. Conceptual scope and benchmark lineage

ArtEdit-Bench belongs at the intersection of instruction-based image editing and art-oriented evaluation. The relevant precursor benchmarks do not share a single objective. "ComplexBench-Edit" focuses on complex, multi-instruction, and chain-dependent editing; "RefEdit-Bench" isolates failures on referring expressions in cluttered scenes; "ImgEdit-Bench" organizes broad editing evaluation around instruction adherence, editing quality, and detail preservation, including multi-turn interaction; "LocateEdit-Bench" addresses the forensic question of where an edit occurred; and "EduArt" shows that art-domain evaluation requires format diversity and item-level measurement rather than multiple-choice saturation alone (Wang et al., 15 Jun 2025, Pathiraja et al., 3 Jun 2025, Ye et al., 26 May 2025, Wu et al., 5 Feb 2026, Spinaci et al., 2 Jul 2026).

A useful way to situate ArtEdit-Bench is to treat it as an art-domain analogue of these benchmarks rather than as a mere style-transfer testbed. That framing matters because the cited works consistently show that current models can appear strong on simple or global edits while failing on compositional dependencies, entity disambiguation, localization, or iterative editing.

Source benchmark	Primary emphasis	Transferable element for ArtEdit-Bench
ComplexBench-Edit	Complex multi-instruction editing	Parallel and chain-dependent edit structure
RefEdit-Bench	Referring-expression editing	Multi-entity disambiguation
ImgEdit-Bench	Unified edit evaluation	Three-dimensional scoring and multi-turn tasks
LocateEdit-Bench	Edit localization	Pixel-level edited-region masks
EduArt	Art-domain benchmark design	Format diversity and psychometric analysis

This benchmark lineage also guards against a common misconception: an art-editing benchmark is not exhausted by measuring global style change. The precursor literature points instead toward a broader target capability set, including localized style application, region-specific preservation, sequential edits, reference resolution, and evaluation protocols that do not over-reward visually plausible but instruction-inaccurate outputs.

2. Task structure: compositionality, reference resolution, and interaction

The most explicit structural template comes from "ComplexBench-Edit," which organizes each sample around three instructions and three levels of complexity: Parallel, Two-chain, and Three-chain. Parallel instructions are logically independent; Two-chain has a dependent pair plus one independent instruction, formally $T_1 \rightarrow T_2,\; T_3$ independent; Three-chain uses $T_1 \rightarrow T_2 \rightarrow T_3$. The benchmark also defines a hierarchical editing taxonomy spanning object-level, attribute-level, and global-level operations, with categories such as Add Object, Change Object, Delete Object, Change Color, Change Pose, Change Material, Change Content, Change Text, Change Background, and Change Style (Wang et al., 15 Jun 2025).

For an art-oriented benchmark, the source text explicitly proposes adapting this structure to artistic dependencies. Examples given include independent edits such as changing brush style, altering the palette to warm colors, and darkening the background; a two-step chain such as converting to watercolor and then overlaying ink outlines; and a longer chain such as changing style to cubism, then adding a surreal floating object, then blurring the background while keeping the figure sharp. This suggests that ArtEdit-Bench would be most informative when it evaluates not just whether a model can stylize an image, but whether it can maintain an internal notion of scene state across dependent artistic operations (Wang et al., 15 Jun 2025).

"RefEdit-Bench" adds a second structural requirement: the ability to resolve referring expressions in complex multi-entity scenes. It is built on RefCOCO, contains 200 images divided into Easy and Hard subsets, and uses five edit categories—change color, change object, add object, remove object, and change texture. The Hard subset is defined by multiple similar instances and overlapping or nearby objects, so that the editing system must identify the correct referent from contextual language such as relative position, clothing, or nearby objects (Pathiraja et al., 3 Jun 2025). In an ArtEdit-Bench setting, this directly motivates prompts of the form “turn only the foreground figure into monochrome” or “stylize the leftmost building as ukiyo-e while keeping the rest realistic,” although that exact formulation would be an inference.

"ImgEdit-Bench" broadens the task definition further by including a Basic-Edit Suite, a challenging Understanding–Grounding–Editing (UGE) Suite, and a Multi-Turn Suite. The multi-turn suite is divided into Content Memory, Content Understanding, and Version Backtracking, thereby treating persistent constraints, pronoun-based or implicit references, and undo-like operations as first-class editing competencies (Ye et al., 26 May 2025). A plausible implication is that a mature ArtEdit-Bench would need not only single-turn artistic transformations but also iterative creative workflows, since artistic editing often proceeds through refinement, preservation of prior constraints, and selective reversal of earlier changes.

3. Data construction and annotation pipelines

The precursor benchmarks supply several compatible construction pipelines, each emphasizing a different aspect of supervision quality. "ComplexBench-Edit" begins from MSCOCO images and applies a Vision Content Filter with two explicit constraints: an Intra-class Frequency Constraint, which discards any image where some category appears more than twice, and a Category Diversity Constraint, which discards any image with fewer than three distinct categories. Formally, retained images satisfy $$\forall c \in C(I),\; N_c(I)\leq 2 \land |C(I)|\geq 3$$. Instructions are then generated by an MLLM using the image, an object list from MSCOCO, and a chosen edit-type combination; chain instructions are generated sequentially with prior instructions as context. Automatic validation is performed by a second MLLM for object-scene compatibility, rationality of references and attributes, and conflicts between instructions, after which two PhD students review a subset for edit-type consistency and post-edit reasonableness (Wang et al., 15 Jun 2025).

The same source explicitly proposes how this pipeline could be transferred to an art setting: replace MSCOCO with an art dataset, adapt the type taxonomy to artistic attributes such as composition, style, and brushwork, and substitute realism-oriented plausibility checks with artistic coherence checks. Because that proposal appears as a transfer suggestion rather than a reported implementation, it is best understood as a benchmark design pattern rather than a released ArtEdit-Bench protocol (Wang et al., 15 Jun 2025).

"RefEdit" contributes a different pipeline centered on targeted synthetic supervision. Its training data, RefEdit-Data, contains more than 20,000 editing triplets generated through GPT-4o text-side scene and instruction generation, FLUX image generation at $1024 \times 1024$, Grounded SAM mask extraction from expanded referring expressions, and edit realization with FlowChef or Inpaint Anything depending on edit type. The authors emphasize that each image is constructed to contain multiple similar entities with distinguishing features, and that this targeted dataset corrects a failure mode not addressed by generic large-scale editing corpora (Pathiraja et al., 3 Jun 2025). For ArtEdit-Bench, the transferable principle is straightforward: targeted synthetic data for localized artistic edits may be more valuable than much larger but less specific style-edit collections.

"ImgEdit" provides a large-scale curation template. Its 1.2 million edit pairs are derived from high-resolution LAION-Aesthetics images, filtered by resolution and aesthetic score, grounded with YOLO-World and SAM2, filtered further by CLIPScore and area ratio, and converted into task-specific edits through FLUX, SDXL, IP-Adapter, ControlNet, Canny/Depth LoRA, and post-hoc GPT-4o quality filtering. It also records multi-turn interaction sequences and, for many tasks, masks or boxes for edited objects (Ye et al., 26 May 2025). This suggests that an art benchmark could likewise combine dense metadata, task-specific generation pipelines, and high-resolution source material rather than relying on low-resolution style-transfer pairs alone.

"LocateEdit-Bench" contributes a mask-generation pipeline absent from most editing benchmarks. Starting from OmniEdit-derived instructions, it filters to object-centric separable edits, regenerates outputs with four modern instruction-based editors, extracts the target object via ChatGPT, and obtains a pixel-level mask using SAM3, retaining only masks with confidence $\geq 0.5$. The final dataset contains 231,093 edited images with pixel-level masks (Wu et al., 5 Feb 2026). For ArtEdit-Bench, that pipeline is especially significant if localization or region-specific preservation is to be evaluated directly rather than inferred.

4. Evaluation frameworks and metrics

The cited literature converges on the view that no single metric is sufficient. "ComplexBench-Edit" evaluates instruction adherence with an MLLM-based 5-point score $S_i \in \{0,1,2,3,4,5\}$ for each instruction, and defines a multiplicative chain score for dependent edits: $$S_{\text{chain}} = S_1 \times S_2 \times \dots \times S_N.$$ This formulation zeroes the chain score if any required step fails. The same work argues that standard whole-image $L1/L2$ measures are flawed for consistency analysis, and therefore defines potentially edited regions from the union of original and edited bounding boxes, computes $$R_{\text{consistent}} = \text{Image Area} \setminus R_{\text{edit}}$$, and evaluates $L1/L2$ only over $T_1 \rightarrow T_2 \rightarrow T_3$0 (Wang et al., 15 Jun 2025). The source text further proposes replacing boxes with segmentation masks, or using $T_1 \rightarrow T_2 \rightarrow T_3$1 or $T_1 \rightarrow T_2 \rightarrow T_3$2 over $T_1 \rightarrow T_2 \rightarrow T_3$3, in art domains.

"ImgEdit-Bench" formalizes a broader three-dimensional scoring model: instruction adherence, image-editing quality, and detail preservation. A key design decision is that editing quality and preservation are capped by instruction adherence: $T_1 \rightarrow T_2 \rightarrow T_3$4 This prevents a model from scoring highly on technical cleanliness or preservation when it has not actually executed the requested edit (Ye et al., 26 May 2025). For an art benchmark, this is a principled way to avoid rewarding visually attractive but semantically wrong stylizations.

"RefEdit-Bench" evaluates Semantic Consistency (SC), Perceptual Quality (PQ), and an overall score

$T_1 \rightarrow T_2 \rightarrow T_3$5

It also introduces Modified VIEScore, in which ground-truth masks crop the target region before GPT-4o judges correctness, thereby making localized evaluation more sensitive to whether the correct entity was edited (Pathiraja et al., 3 Jun 2025). That mechanism is immediately relevant to art tasks requiring style changes restricted to one object or region.

"LocateEdit-Bench" adds explicit localization metrics. It evaluates edited-region prediction with Accuracy, Precision, Recall, F1, AUC, Dice, and IoU, and does so under two protocols: full-set evaluation across all four editors and cross-editor generalization from one editor to all others (Wu et al., 5 Feb 2026). This shifts evaluation from “did the image look correct?” to “can the edited region itself be segmented?”, a distinction that matters for controllability, auditability, and non-target preservation in artistic editing systems.

"EduArt" contributes an orthogonal but important lesson: benchmark quality depends on format diversity and psychometric characterization. It measures item difficulty and item discrimination and uses logistic regression to isolate the effects of format, image presence, language, and model identity (Spinaci et al., 2 Jul 2026). A plausible implication is that ArtEdit-Bench should not report only aggregate edit scores; it should also analyze which task formats discriminate among strong models and which ones saturate.

5. Empirical lessons from adjacent benchmarks

The empirical record across these benchmarks shows that apparently strong editing models remain brittle once evaluation moves beyond simple single-turn prompts. On "ComplexBench-Edit," Gemini-CoT achieves the best overall average editing score, 40.47, compared with 36.70 for Gemini, but scores drop sharply with complexity: in Three-chain – Obj-At, Gemini scores 15.10 and Gemini-CoT 17.54. Performance is also edit-type dependent: Gemini-CoT is best on Add Object (48.23), Change Object (44.78), Delete Object (66.38), Change Color (41.98), Change Material (32.50), Change Text (28.68), and Change Background (38.98), but it is weaker on Change Style (30.69) than Step1X-Edit (45.54) and OmniGen (44.75). On background preservation, OmniGen has the best consistency with average $T_1 \rightarrow T_2 \rightarrow T_3$6 and $T_1 \rightarrow T_2 \rightarrow T_3$7, whereas Gemini-CoT reaches $T_1 \rightarrow T_2 \rightarrow T_3$8 and $T_1 \rightarrow T_2 \rightarrow T_3$9, indicating a trade-off between aggressive editing and non-target consistency. Human preference data further show Gemini-CoT preferred over Gemini in 67% of pairs, over OmniGen in 76%, and over Step1X-Edit in 63% (Wang et al., 15 Jun 2025).

These findings are directly relevant to ArtEdit-Bench because artistic editing often combines global transformation with local preservation. The precursor evidence indicates that stronger instruction following can coincide with worse background preservation, and that specialized models may excel on global style changes even when they lag on compositional object-attribute edits. That pattern cautions against treating “style” as a proxy for overall editing competence.

"RefEdit" shows a related failure mode: models trained on millions of samples still struggle when the prompt must identify the correct entity in a complex scene. The benchmark uses 200 RefCOCO-derived images, evenly split between Easy and Hard. The authors report that RefEdit, trained on only 20,000 editing triplets, outperforms Flux/SD3-model-based baselines trained on millions of data and improves both RefEdit-Bench and PIE-Bench performance (Pathiraja et al., 3 Jun 2025). The key lesson for ArtEdit-Bench is that data specificity can dominate raw scale when the task requires localized artistic control over ambiguous compositions.

"ImgEdit-Bench" arrives at a similar conclusion from another angle. It contains 811 total test samples across 14 sub-tasks and shows that GPT-4o-Image is strongest overall, while ImgEdit-E1 and Step1X-Edit are the strongest open-source systems. Yet even frontier proprietary models remain weak on multi-turn content memory and content understanding, despite being better at version backtracking (Ye et al., 26 May 2025). This suggests that a benchmark evaluating only single-turn art stylization would miss an important part of realistic creative workflows.

"LocateEdit-Bench" adds evidence from the forensic side. With 231,093 edited images, it finds that semantic-heavy segmenters outperform low-level anomaly detectors on instruction-based edits; SegFormer reaches average F1 of about 84.8%, Dice about 82.6, and mIoU about 77.1, while Mesorch reaches average F1 about 86.1%, Dice about 82.1, and mIoU about 76.8. Cross-editor generalization degrades markedly, and BAGEL-edited images are consistently the hardest to localize (Wu et al., 5 Feb 2026). For ArtEdit-Bench, a plausible implication is that localization-aware evaluation should expect strong editor-specific effects rather than assuming one universal notion of edit traceability.

Finally, "EduArt" shows that evaluation format can change conclusions even in the same domain. Six models exceed 90% on multiple_choice_radio, but the abstract reports that models above 94 percent on multiple choice can fall to 23.9 percent on open completion and 6.2 percent on error identification. The benchmark also reports strong psychometric properties, with mean discrimination 0.514 and 82.3 percent good discriminators (Spinaci et al., 2 Jul 2026). Although EduArt evaluates multimodal LLMs rather than image editors, its central finding generalizes: benchmark formats that saturate on recognition tasks can hide major capability deficits in production, correction, or transformation tasks.

6. Limitations, misconceptions, and likely development paths

A first limitation is domain mismatch. The precursor editing benchmarks are largely grounded in realistic image corpora—MSCOCO, RefCOCO, LAION, OpenImages, and OmniEdit-derived natural imagery—rather than paintings, illustrations, or stylized compositions (Wang et al., 15 Jun 2025, Pathiraja et al., 3 Jun 2025, Ye et al., 26 May 2025, Wu et al., 5 Feb 2026). Consequently, any direct transfer to ArtEdit-Bench remains partial. The cited texts explicitly acknowledge this for ComplexBench-Edit and RefEdit, both of which present their art-domain relevance as an extension rather than a completed benchmark implementation.

A second limitation concerns region definition. ComplexBench-Edit uses bounding boxes to estimate edited and consistent regions, which the source text itself describes as coarse; LocateEdit-Bench improves this by generating pixel-level masks through SAM3, but its edit space is restricted to Object Addition, Object Swap, and Attribute Modification (Wang et al., 15 Jun 2025, Wu et al., 5 Feb 2026). An art benchmark would likely require finer masks for stylized, overlapping, or texture-level edits, especially when the intended transformation affects brushwork, color fields, or layered compositional regions rather than discrete object instances. This is an inference, but it follows directly from the limitations described.

A third issue is evaluator reliability. ComplexBench-Edit and RefEdit rely on MLLM-based judgment; ImgEdit-Bench uses GPT-4o and also trains ImgEdit-Judge; EduArt demonstrates that task format itself strongly modulates accuracy and discrimination (Wang et al., 15 Jun 2025, Pathiraja et al., 3 Jun 2025, Ye et al., 26 May 2025, Spinaci et al., 2 Jul 2026). Together, these results undermine the misconception that a single VLM judge or a single score dimension can faithfully summarize artistic editing performance. In an art benchmark, evaluator calibration, format diversity, and expert review would therefore be central design concerns.

The most plausible development path for ArtEdit-Bench is a hybridization of the benchmark families summarized above. Such a benchmark would combine: compositional multi-instruction structure and chain-aware scoring from ComplexBench-Edit; referring-expression and multi-entity difficulty from RefEdit; unified edit-quality and multi-turn evaluation from ImgEdit-Bench; pixel-level localization from LocateEdit-Bench; and psychometric analysis plus art-domain task diversity from EduArt (Wang et al., 15 Jun 2025, Pathiraja et al., 3 Jun 2025, Ye et al., 26 May 2025, Wu et al., 5 Feb 2026, Spinaci et al., 2 Jul 2026). The cited texts also point toward specific extensions: mask-based rather than box-based consistency evaluation, style-aware preservation metrics, structured plan consumption instead of plain prompt concatenation, more diverse instruction types, and expert human studies for domains where aesthetic judgment is irreducible.

In that synthesized sense, ArtEdit-Bench is best understood not as a single fixed dataset already standardized in the literature, but as an emerging benchmark design program for rigorous evaluation of art-oriented instruction-based image editing. Its defining challenge is to measure artistic controllability without collapsing the problem into either global style transfer alone or generic image-edit similarity metrics.