---
title: 'I2EBench2.0: Image Editing Benchmark'
url: https://www.emergentmind.com/topics/i2ebench2-0
type: topic
---

# I2EBench2.0: Image Editing Benchmark

Searching arXiv for the benchmark paper and its predecessor to ground the article in the latest published versions.
I2EBench2.0 is a comprehensive benchmark for instruction-based image editing (IIE), defined as the task in which a model takes an input image \(V_I\) and a natural-language instruction \(T\) and produces an edited image that follows the instruction while preserving irrelevant content and maintaining visual quality. The benchmark extends the original NeurIPS’24 I2EBench by introducing joint evaluation of single-round and multi-round editing, broadening the evaluation space to 16 single-round dimensions and 7 multi-round dimensions, and grounding its automatic scoring procedures in human studies and cross-judge analysis [2606.15570]. Within the benchmark’s framing, the central difficulty of IIE evaluation is that instructions are heterogeneous, the edits range from high-level semantic operations to low-level restoration, a single scalar metric cannot capture all relevant properties, and interactive editing requires robustness across sequences of edits rather than only one-shot transformations [2606.15570][2408.14180].

## 1. Origins, scope, and motivation

I2EBench2.0 was created to address a specific evaluation gap in instruction-based image editing. Prior work had notable limitations: small datasets, limited edit types, focus on mask-guided or otherwise narrow protocols, very limited or no multi-round evaluation, and dependence on generic metrics such as CLIP, PSNR, SSIM, and LPIPS applied uniformly across disparate editing tasks [2606.15570][2408.14180]. In contrast, I2EBench2.0 is explicitly designed as a benchmark for mask-free instruction-based editing that evaluates both semantic correctness and low-level image quality using dimension-specific automatic metrics rather than a single universal score [2606.15570].

The benchmark inherits the core idea of the original "I2EBench: A Comprehensive Benchmark for Instruction-based Image Editing" [2408.14180] and expands it into a framework that simultaneously evaluates single-round and multi-round editing [2606.15570]. A round is one editing step: single-round editing applies one instruction to the original image, whereas multi-round editing applies a sequence of 2–5 instructions, each operating on the previous output [2606.15570]. This distinction matters because practical editing workflows are iterative, and failure in any intermediate step can propagate forward. The benchmark therefore treats multi-round editing not as repeated single-round testing, but as a separate regime with its own success criterion and its own dimensions [2606.15570].

The benchmark is intended both as an evaluation tool and as what may be called an *“analysis benchmark”* (*Editor’s term*): it measures current performance and also exposes systematic weaknesses of existing IIE models, especially under multi-round interaction [2606.15570]. The paper explicitly characterizes it as both a benchmark and a research probe, using comparative evaluation across strong baseline systems to derive guidance for future model design [2606.15570].

## 2. Dataset construction and benchmark composition

Each evaluation sample consists of an input image \(V_I\), one or more natural-language instructions \(T\), and the output of an IIE model, after which I2EBench2.0 computes dimension-specific scores for instruction following, preservation of non-target regions, low-level quality, and multi-round robustness [2606.15570]. The dataset contains over 2,000 images and over 6,700 editing instructions [2606.15570]. For the 16 single-round dimensions, the paper reports approximately 140 images per dimension, yielding roughly \(16 \times 140 \approx 2{,}240\) image-dimension instances [2606.15570][2408.14180].

Images are drawn from public datasets covering both general scenes and restoration settings. The benchmark explicitly mentions MS COCO for natural scenes and objects, as well as task-specific sources for deblurring, dehazing, snow, rain, shadow, low-light, and watermark removal [2606.15570]. The predecessor benchmark also lists Berkeley segmentation, GoPro, HIDE, Dense-Haze, Haze4K, LOL, LHP-Rain, CSD, SRD, and CLWD / WDNet among the sources used for the 16-dimensional framework [2408.14180]. The resulting coverage spans animals, objects, plants, humans, scenery, and global edits [2606.15570][2408.14180].

For each image-dimension pair, human annotators first write an original instruction. To increase linguistic diversity, the benchmark then uses ChatGPT to produce diverse paraphrases that preserve intent while varying syntax and vocabulary [2606.15570][2408.14180]. Each instruction is additionally assigned one of six content categories: Animal, Object, Scenery, Plant, Human, or Global [2606.15570]. This category labeling supports downstream analysis of category-dependent performance differences.

Multi-round annotations are provided for each applicable high-level dimension except Region Accuracy, with 2–5 instructions per image [2606.15570]. The benchmark’s design choice to use atomic instructions in single-round settings and composition through multi-round sequences is deliberate. This suggests that metric reliability and interpretability were prioritized over one-shot compositional complexity, while still approximating realistic editing workflows via sequential interaction [2606.15570].

## 3. Evaluation dimensions and metrics

The benchmark is organized into 8 high-level single-round dimensions, 8 low-level single-round dimensions, and 7 multi-round high-level dimensions [2606.15570]. The high-level dimensions focus on semantic correctness and localization, while the low-level dimensions capture restoration and enhancement quality. Region Accuracy occupies a bridge position because it evaluates localization and preservation rather than direct semantic correctness [2606.15570][2408.14180].

### Single-round dimensions

| Group | Dimensions | Metric basis |
|---|---|---|
| High-level | Counting; Direction Perception; Object Removal; Object Replacement; Background Replacement; Color Alteration; Style Alteration; Region Accuracy | GPT‑4V for six semantic dimensions, CLIP for Style Alteration, SSIM for Region Accuracy |
| Low-level | Deblurring; Haze Removal; Lowlight Enhancement; Noise Removal; Rain Removal; Shadow Removal; Snow Removal; Watermark Removal | SSIM against ground-truth clean images |

For Counting, Direction Perception, Object Removal, Object Replacement, Background Replacement, and Color Alteration, the benchmark uses GPT‑4V as an automatic judge with human-designed question–answer templates [2606.15570]. Counting asks questions such as “How many cats are on the shoe rack?” and scores the percentage of correct matches to annotated answers [2606.15570]. Direction Perception uses yes/no or relational questions about object position, while Object Removal asks whether a target object remains present after editing [2606.15570]. Object Replacement checks whether the new target object is present and the original object is absent, Background Replacement checks whether the background matches the requested description, and Color Alteration asks GPT‑4V to identify the edited object’s color [2606.15570].

Style Alteration is evaluated with CLIP image–text similarity using prompts of the form
\[
\text{“an image with } \cdots \text{ style”},
\]
where the placeholder is filled by the requested style [2606.15570][2408.14180]. Region Accuracy is computed by masking the intended edit region, whitening that region in both original and edited images, and then measuring SSIM on the remaining pixels so that a high score indicates preservation of non-target regions [2606.15570][2408.14180].

For the eight low-level dimensions, the metric is SSIM against a ground-truth clean image:
\[
\text{SSIM}(I_{\text{edit}, I_{\text{gt}) \in [0,1].
\]
These scores are scaled to 0–100 when reported in the benchmark tables [2606.15570]. The low-level tasks are Deblurring, Haze Removal, Lowlight Enhancement, Noise Removal, Rain Removal, Shadow Removal, Snow Removal, and Watermark Removal [2606.15570].

### Multi-round dimensions

The multi-round benchmark evaluates seven high-level dimensions: Counting, Direction Perception, Object Removal, Object Replacement, Background Replacement, Color Alteration, and Style Alteration [2606.15570]. Region Accuracy is excluded because multi-round region masks would have to be recomputed for each model’s intermediate outputs, which is incompatible with a fixed automated benchmark [2606.15570].

The multi-round scoring rule is intentionally strict. For a sequence of \(K\) instructions, an example is successful only if all \(K\) rounds succeed [2606.15570]. For GPT‑4V-based dimensions, the benchmark evaluates each round separately and assigns a sample score of 1 only if every round is correct; otherwise the sample is a failure. The reported dimension score is the multi-round success rate:
\[
\text{Score} = \frac{\text{\# multi-round successes}{\text{\# samples} \times 100.
\]
For Style Alteration, the benchmark again uses CLIP-based style similarity across rounds while retaining the same all-rounds-must-succeed interpretation [2606.15570]. This scoring rule makes multi-round performance substantially more demanding than single-round performance because small intermediate errors accumulate.

## 4. Human alignment and evaluation protocol

A central design principle of I2EBench2.0 is alignment with human judgment [2606.15570]. To test this, the benchmark conducts a user study for each of the 16 single-round and 7 multi-round dimensions. Human raters are shown the instruction \(T\), the original image \(V_I\), and edited outputs \(\{V_1, \ldots, V_M\}\) from \(M\) models, and are asked to rank all outputs from best to worst according to overall judgment, combining instruction faithfulness and visual quality [2606.15570]. Ranks are converted into human scores by mapping rank 1 to score \(M\), rank \(M\) to score 1, and rank \(k\) to score \(M-k+1\) [2606.15570][2408.14180]. The per-dimension human score for a model is then averaged over the sampled examples:
\[
\text{HumanScore}_{\text{dim} = \frac{1}{N} \sum_{i=1}^N s_{i,\text{model}.
\]

The benchmark then compares these human-derived rankings with rank-based scores produced by its automatic metrics [2606.15570]. Figure-based analysis in the paper reports strong positive correlations, described in terms of Spearman’s \(\rho\), between automatic ranks and human evaluation across all dimensions [2606.15570]. The predecessor benchmark similarly reports significant positive correlations between I2EBench rank scores and human rank scores across its 16 dimensions [2408.14180]. Although the provided summary does not list exact coefficient values, the stated conclusion is that better benchmark scores reliably correspond to better human preference rankings [2606.15570].

I2EBench2.0 further evaluates judge dependence by replacing GPT‑4V with Qwen3VL-8B and LLaVA-1.5-7B in cross-judge experiments [2606.15570]. It computes average ranks across judges to verify consistency, and the paper states that rankings are very consistent across different judges, with small average-rank variations [2606.15570]. This is important because high-level evaluation relies on powerful MLLMs, and the benchmark’s claim is not that judge choice is irrelevant, but that the overall conclusions remain stable across multiple evaluators [2606.15570].

Operationally, the benchmark’s single-round protocol feeds each \((V_I, T)\) pair into the model once, obtains one edited image using official inference settings, and applies the dimension-specific evaluator: GPT‑4V-based Q/A, CLIP similarity, Region Accuracy SSIM, or low-level SSIM [2606.15570]. The multi-round protocol initializes \(V^{(0)} = V_I\), applies instructions \(T_1, T_2, \ldots, T_K\) sequentially to produce \(V^{(1)}, V^{(2)}, \ldots, V^{(K)}\), evaluates each round, and counts the sample as successful only if all rounds pass [2606.15570].

## 5. Baseline models and empirical findings

The benchmark evaluates a wide set of IIE models using official code and checkpoints, with one edited image generated per instruction [2606.15570]. The reported model set includes HIVE, InstructDiffusion, InstructPix2Pix, MagicBrush, MGIE, InstructEdit, InstructAny2Pix, HQ-Edit, FLUX Kontext, and Qwen-image [2606.15570]. The predecessor benchmark had evaluated eight models, while the extended version’s tables show at least ten [2408.14180][2606.15570].

On single-round low-level dimensions, the benchmark reports that low-level tasks are mostly moderately solved, but with substantial diversity across methods [2606.15570]. InstructDiffusion often leads in Deblurring, Haze Removal, Rain Removal, and Watermark Removal, while Qwen-image achieves very high scores in Lowlight Enhancement, including 73.7% in one table and 72.8% in another [2606.15570]. FLUX Kontext scores very high in Haze Removal and Rain Removal / Watermark Removal, including 79.7% on rain and 92.2% on watermark removal in one reported table [2606.15570]. MGIE and HQ-Edit are competitive on some low-level edits but are not consistently best [2606.15570].

On single-round high-level dimensions, Counting and Direction Perception are strongly discriminative [2606.15570]. Qwen-image reaches roughly 65% on original instructions and 59% on diverse instructions for Counting, and exceeds 80% on Direction Perception [2606.15570]. FLUX Kontext is also strong, with approximately 49–45% on Counting and 76–78% on Direction Perception [2606.15570]. Earlier models such as InstructPix2Pix, InstructDiffusion, and InstructEdit often remain below 20% on counting [2606.15570]. For object-centric operations, Qwen-image is reported as best or near-best in Object Removal, Object Replacement, and Background Replacement, with approximately 78–75%, 88–97%, and 96–93%, respectively [2606.15570]. MagicBrush and MGIE also perform relatively well in Object Replacement and Background Replacement, typically in the 60–80% range [2606.15570]. Color Alteration is comparatively strong for Qwen-image and FLUX Kontext, both above roughly 80%, whereas Style Alteration remains difficult for all models, with maximum scores only around the 27–28% CLIP similarity range [2606.15570].

Multi-round evaluation reveals much larger weaknesses [2606.15570]. Counting becomes extremely difficult: all models have very low multi-round counting accuracy, often below 5%, with Qwen-image at about 11% [2606.15570]. Direction Perception degrades but remains feasible for the strongest models, with Qwen-image around 57% on original and 53% on diverse instructions, and FLUX Kontext around 45–38% [2606.15570]. Object Removal and Object Replacement also degrade substantially, though Qwen-image still reaches around 39–43% on removal and 64–68% on replacement [2606.15570]. Background Replacement and Color Alteration degrade less severely; Qwen-image retains about 82% on multi-round background replacement versus around 96% in single-round evaluation, and about 66–68% on multi-round Color Alteration [2606.15570]. Style Alteration remains low and similar across models in the multi-round setting, around 23–27% [2606.15570].

The benchmark also reports trade-offs. Some models are strong at low-level restoration but weaker at nuanced instruction semantics; others are relatively strong in high-level semantics but less stable on low-level metrics [2606.15570]. Qwen-image and FLUX Kontext are singled out as the most balanced across high-level and low-level as well as single-round and multi-round settings [2606.15570]. Qualitative examples further identify failure modes such as over-editing non-target regions, hallucinated objects, incorrect counting despite plausible images, and progressive drift across rounds [2606.15570].

## 6. Research implications, comparative position, and limitations

The paper distills several research insights from the benchmark results [2606.15570]. First, multi-round robustness is a major unsolved problem. The gap between single-round and multi-round performance is clear across dimensions, and early sequence errors propagate, especially for object-centric tasks such as counting, removal, and replacement [2606.15570]. This suggests that future models may need explicit mechanisms for edit history, state tracking, and error correction, though such architectural prescriptions are presented as research guidance rather than as part of the benchmark itself.

Second, the benchmark states that aesthetic quality degrades across rounds, using an aesthetic predictor score (AP) similar to CLIP-MLP from LAION-5B [2606.15570]. AP tends to drop in multi-round editing, indicating accumulating artifacts and quality loss [2606.15570]. Third, no single model dominates all dimensions: InstructDiffusion is strong on low-level tasks, MagicBrush and MGIE on some high-level edits, and Qwen-image and FLUX on complex semantics and balanced performance [2606.15570]. Fourth, instruction wording matters. The benchmark reports that some models perform notably worse under diverse paraphrased instructions, especially in object removal, whereas models leveraging strong LLM or MLLM front-ends are more robust to instruction variation [2606.15570]. The earlier benchmark formalized this sensitivity through an instruction-robustness change rate,
\[
S^{i} = \frac{|S_o^{i}-S_d^{i}|}{\text{MIN}(S_o^{i}, S_d^{i})},
\]
where \(S_o^i\) and \(S_d^i\) are scores under original and diverse instructions, respectively [2408.14180].

The benchmark also reports category-dependent performance [2606.15570][2408.14180]. Scenery and Global categories consistently obtain higher scores because they are easier and often correspond to global edits that do not require precise localization [2606.15570]. Animal and Human categories are harder because they demand fine-grained semantics and geometry [2606.15570]. This category pattern situates I2EBench2.0 not merely as a collection of tasks, but as a controlled testbed for differentiating which kinds of semantic precision remain difficult for current editors.

Relative to prior benchmarks, I2EBench2.0 distinguishes itself by combining 8 high-level semantic dimensions, 8 low-level editing dimensions, and 7 multi-round dimensions in one framework [2606.15570]. Compared with TedBench and TedBench++, it offers broader coverage and explicit multi-round evaluation; compared with EditBench, it is suitable for mask-free instruction-based models; compared with EditVal, it covers more than geometric aspects and relies less on manual scoring; and compared with MagicBrush, Emu Edit, and SmartEdit test sets, it goes beyond single-round evaluation and generic CLIP- or DINO-based scoring [2606.15570]. Its use of multimodal LLM judges with human-designed Q/A templates, human-alignment studies, and cross-judge rank analysis is presented as one of its defining methodological novelties [2606.15570].

The benchmark’s limitations are also explicit. Domain coverage is broader than prior work but still focused on typical natural images and classical low-level conditions; artistic or domain-specific areas such as medical, satellite, or anime are not explicitly included [2606.15570]. Instructions are in English only, and cross-lingual robustness is not evaluated [2606.15570]. The benchmark does not yet measure diversity of valid outputs under underspecified instructions, nor does it test ambiguous or infeasible instructions [2606.15570]. High-level evaluation depends on powerful MLLMs, which introduces cost and latency [2606.15570]. Complex compositional one-shot instructions are not directly benchmarked; instead, they are approximated through multi-round sequences [2606.15570]. Region Accuracy is absent from multi-round evaluation for practical reasons of annotation and automation [2606.15570].

The released resources include the dataset, human-annotated instructions and paraphrases, multi-round sequences, category labels, evaluation scripts, MLLM integrations, SSIM and CLIP implementations, and edited images from all evaluated models, all hosted in the public repository associated with the benchmark [2606.15570][2408.14180]. In that sense, I2EBench2.0 functions as both a standardized benchmark and a reproducible infrastructure for evaluating new IIE models under a common protocol [2606.15570].

Source: https://www.emergentmind.com/topics/i2ebench2-0