EdiVal-Bench: Multi-turn Image Editing

• EdiVal-Bench is a multi-turn benchmark for instruction-based image editing that uses an object-centric approach to assess sequential editing tasks.
• It integrates object detection with vision-language models to evaluate instruction following, content consistency, and visual quality across editing episodes.
• Covering 572 images and 1,716 instructions over 9 edit types, it exposes paradigm-specific failure modes and promotes scalable, automated evaluation.

Searching arXiv for the specified benchmark and closely related prior work. EdiVal-Bench is a multi-turn benchmark for instruction-based image editing, introduced as an instantiation of the EdiVal-Agent evaluation framework. It is designed for automated, scalable, and fine-grained evaluation from an object-centric perspective, and covers 572 images from GEdit-Bench with 1,716 diverse editing instructions over 9 instruction types. The benchmark targets a setting in which each editing episode applies a sequence of editing instructions, with a default length of 3 turns, and it evaluates 11 state-of-the-art editing models spanning autoregressive, flow-matching, and diffusion paradigms (Chen et al., 16 Sep 2025).

1. Origin, motivation, and relation to prior benchmarks

EdiVal-Bench was proposed in response to a persistent evaluation bottleneck in instruction-based image editing. The motivating critique is twofold. First, reference-based metrics such as L1/L2 distances and semantic similarity via CLIP or DINO depend on hand-curated or model-generated references, thereby covering only a small subset of valid edits and inheriting biases from prior generative models. Second, VLM-based evaluation can exhibit poor spatial reasoning, hallucinated object properties, insensitivity to subtle or local changes, and miscalibration for synthetic artifacts, which weakens objective, fine-grained assessment (Chen et al., 16 Sep 2025).

The benchmark is directly connected to the earlier EditVal benchmark, which standardized quantitative evaluation for diffusion-based, text-guided image editing. EditVal consisted of a curated dataset of 92 images from MS-COCO, 648 unique edit operations, 13 fine-grained edit types, and an automated evaluation pipeline based on pre-trained vision-LLMs, with complementary large-scale human studies on Amazon Mechanical Turk (Basu et al., 2023). EditVal focused on single-turn editing and emphasized the lack of a unified benchmark across diverse edit types, especially spatial manipulations. EdiVal-Bench extends that line of work into multi-turn instruction-based editing and broadens model coverage beyond diffusion-only systems to include autoregressive and flow-matching models (Chen et al., 16 Sep 2025).

A common misconception is to treat EdiVal-Bench as a direct renaming of EditVal. The available evidence indicates otherwise. EditVal and EdiVal-Bench are related but distinct resources: the former benchmarks diffusion-based text-guided image editing with 13 edit types in a single-turn setting, whereas the latter is a multi-turn benchmark instantiated from EdiVal-Agent, covers 9 instruction types over 572 images, and evaluates 11 models across three model paradigms (Basu et al., 2023).

2. Benchmark composition and task taxonomy

EdiVal-Bench is structured around editing episodes rather than isolated edits. Each episode applies a sequence of instructions to a real-world image, with the default protocol using 3 turns. The dataset contains 572 images from GEdit-Bench and 1,716 diverse editing instructions (Chen et al., 16 Sep 2025).

The instruction space is grouped into 6 semantic categories and 9 concrete instruction types:

Semantic category	Instruction types
Subject-centric	subject_add, subject_remove, subject_replace
Attribute-centric	color_alter, material_alter
Text-related	text_change
Relational	position_change
Counting	count_change
Global	background_change

The typed instructions are generated contextually per image; one example given is “Change the color of metal white airplane to blue.” This organization reflects the benchmark’s emphasis on grounded, context-aware editing rather than generic prompt templates (Chen et al., 16 Sep 2025).

EdiVal-Bench evaluates 11 state-of-the-art editing models. The benchmarked autoregressive models are GPT-Image-1, Nano Banana (Gemini 2.5 Flash), and Gemini 2.0 Flash. The flow-matching models are Qwen-Image-Edit, Step1X-Edit, FLUX.1-Kontext-dev, and OmniGen. The diffusion-based models are InstructPix2Pix, MagicBrush, UltraEdit, and AnyEdit. Both open- and closed-source models are included (Chen et al., 16 Sep 2025).

This model coverage matters because it enables comparisons across editing paradigms rather than within a single generative family. A plausible implication is that EdiVal-Bench was designed not only to score existing systems but also to expose paradigm-specific failure modes in multi-turn editing.

3. Object-centric generation of multi-turn editing episodes

EdiVal-Bench is built through the EdiVal-Agent pipeline, whose first stage is decomposition. The goal is to extract a full, structured list of visible, semantically meaningful objects in each image, together with attributes such as color, material, text, count, and foregroundness. A VLM such as GPT-4o is used to parse object lists from the image and return JSON, after which Grounding-DINO filters for objects that can actually be visually grounded; ambiguous and low-confidence objects are dropped. The system maintains three dynamic object pools: all ever present, currently unedited, and currently available for editing, denoted $\mathcal{P}^{\text{all}_t}$, $\mathcal{P}^{\text{unch}_t}$, and $\mathcal{P}^{\text{avail}_t}$ (Chen et al., 16 Sep 2025).

The second stage is instruction generation. At each turn, a new instruction type is sampled, avoiding repetition when possible, with subject_add as fallback. GPT-4o then generates a natural-language instruction conditioned on the current scene state and the dynamic object pools. After each instruction, the pools are updated because objects may be added, removed, or tagged as edited. Background changes receive special handling: after such an edit, background consistency is no longer required or computed (Chen et al., 16 Sep 2025).

This object-centric construction is central to the benchmark’s semantics. It ensures that instructions are grounded in the actual object state of the image and can therefore be evaluated by symbolic checks, detector outputs, or targeted VLM queries. Compared with single-turn benchmarks, this protocol captures compositional and state-dependent editing behavior that emerges only when edits accumulate over turns. This suggests that EdiVal-Bench is intended to test not merely instruction compliance, but state tracking and edit-history robustness.

4. Evaluation pipeline and scoring dimensions

EdiVal-Bench uses a fully automated, multi-axis evaluation protocol implemented through EdiVal-Agent. The evaluation separates instruction following, content consistency, and visual quality rather than collapsing them into a single undifferentiated score (Chen et al., 16 Sep 2025).

Instruction following is handled differently depending on the type of edit. Symbolically verifiable types—add, remove, replace, position, and count—are evaluated solely by Grounding-DINO. The detector checks existence or nonexistence, correct spatial relation based on center positions, and correct instance count using detection box counts. Semantic types—color, material, text, and background change—use Qwen2.5-VL, with detector-guided cropping so that the VLM attends only to the relevant object or location (Chen et al., 16 Sep 2025).

Content consistency measures preservation of unchanged content. For each unchanged object, corresponding regions are extracted from the base image $I^0$ and the edited image $I^t$. Two metrics are then computed. The first is DINOv3 cosine similarity,

$$s^{\mathrm{obj}} = \cos[\phi(I^0_{\mathrm{obj}}), \phi(I^t_{\mathrm{obj}})]$$

where $\phi(\cdot)$ is the DINOv3 embedding of the object crop. The second is a normalized pixel L1 metric,

$$q^{\mathrm{obj}} = 1 - \frac{\|I^0_{\mathrm{obj}} - I^t_{\mathrm{obj}}\|_1}{Z}$$

with $Z = C \cdot |\Omega| \cdot \Delta$ for channel count, region size, and dynamic range. For background, all detected objects are masked and the same metrics are computed on the background region (Chen et al., 16 Sep 2025).

Visual quality is measured using Human Preference Score v3 (HPSv3), which the paper describes as more sensitive to artifacts than VLMs. The framework also tracks low-level measures such as the 99.9% luminance quantile to capture overexposure and drift across turns. Luminance is defined using the Rec. 709 formula

$Y = 0.2126 R + 0.7152 G + 0.0722 B .$

Aggregation proceeds along two main axes: instruction following $\mathcal{P}^{\text{unch}_t}$0, defined as the fraction of images where all edits up to turn $\mathcal{P}^{\text{unch}_t}$1 are successful, and content consistency $\mathcal{P}^{\text{unch}_t}$2, defined as the mean of object and background DINOv3 similarities at turn $\mathcal{P}^{\text{unch}_t}$3. The overall combined score is the geometric mean

$\mathcal{P}^{\text{unch}_t}$4

while visual quality is tracked separately from the main score (Chen et al., 16 Sep 2025).

This decomposition is notable because it prevents high visual appeal from obscuring failures in instruction following or preservation. A plausible implication is that the benchmark is explicitly designed to diagnose trade-offs rather than to reward a single optimization target.

5. Reliability, agreement with human judgment, and interpretability

A central claim of EdiVal-Bench is that its evaluation framework improves agreement with human judgment. In a human agreement study on instruction following, EdiVal-Agent achieved 81.3% agreement with human raters, compared with 75.2% for VLM-only evaluation using Qwen-2.5-VL and 65.4% for CLIP-dir. Inter-annotator agreement, described as the ceiling, was 85.5% (Chen et al., 16 Sep 2025).

The benchmark attributes this improvement to the integration of object detection with VLMs. That integration is especially important for tasks involving spatial understanding and object presence, including add, remove, replace, position, and count. For color, material, text, and background changes, detector-guided cropping before the VLM query improves reliability by constraining the evaluation to the intended region (Chen et al., 16 Sep 2025).

The paper also argues that HPSv3 better correlates with human judgments of visual quality than VLM-based quality assessment. VLMs are reported to miss serious artifacts and to overrate images affected by overexposure or streaking artifacts. More broadly, the automated pipeline is described as matching or exceeding prior zero-shot VLM and reference-based evaluation methods in both precision and interpretability (Chen et al., 16 Sep 2025).

These findings can be situated against the earlier EditVal results. EditVal already showed that an automated pipeline based on a stronger object-centric VLM, OwL-ViT, was more reliable than CLIP for certain fine-grained and spatial edits, and that its automated scores were strongly correlated with human preferences for the edit types considered (Basu et al., 2023). EdiVal-Bench generalizes that object-centric logic to a broader multi-turn setting and makes the evaluator itself modular.

6. Empirical findings and exposed failure modes

The benchmark reports several model-level findings. Nano Banana leads in overall balance between instruction following and consistency and also has low latency. GPT-Image-1 is best at instruction following but has poor content consistency and high latency, often regenerating or restyling images in ways that harm object and background stability. Autoregressive models outperform non-autoregressive models—specifically diffusion and flow-matching systems—in multi-turn editing, maintaining context coherence more effectively. Qwen-Image-Edit is described as strong in single-turn settings but subject to rapid degradation over multiple turns (Chen et al., 16 Sep 2025).

The benchmark also exposes specific failure modes. One is exposure bias and cumulative drift: non-autoregressive models trained for single-turn or short-history editing accumulate errors in multi-turn settings. Qwen-Image-Edit is given as an example, showing increased overexposure and artifacts across turns, with performance dropping sharply by the third turn. Another is subtype weakness: all models struggle on tasks such as count_change, where even the best models achieve under 25% success. Spatial reasoning tasks such as position_change and numerical edits are identified as difficult for both models and evaluation tools. A further pattern is aesthetic drift: some autoregressive models, including GPT-Image-1, improve perceived image beauty but at the cost of fidelity to the base image’s style (Chen et al., 16 Sep 2025).

These findings resonate with earlier observations from EditVal. That benchmark found no universal winner across edit types, identified Instruct-Pix2Pix, Null-Text, and SINE as top-performing on average, and concluded that most methods fail on spatial operations such as changing the position of an object (Basu et al., 2023). EdiVal-Bench preserves the emphasis on spatial and object-centric diagnostic evaluation, but it adds the temporal dimension of multi-turn editing, where degradation can compound rather than remain localized to a single edit.

A plausible implication is that EdiVal-Bench shifts the field’s attention from isolated edit success toward trajectory stability across editing episodes.

7. Extensibility, released resources, and research significance

EdiVal-Bench is presented as automated and parallelizable end to end: decomposition, instruction generation, and evaluation do not require hand annotation or paired reference images. The framework is explicitly modular, allowing new object detectors, semantic models, improved VLMs, or human-aligned reward models to be integrated without major system change. This design is intended to support future upgrades in evaluation accuracy over time (Chen et al., 16 Sep 2025).

The benchmark’s significance lies in its combination of fine-grained object-centric reasoning, multi-turn evaluation, and multi-dimensional scoring. It decomposes results by instruction type, turn, and model; combines symbolic reasoning with learned semantic features; and separates instruction following, preservation, and aesthetics. The associated project page is listed as https://tianyucodings.github.io/EdiVAL-page/ (Chen et al., 16 Sep 2025).

In the broader history of image-editing benchmarks, EditVal provided a standardized benchmark for quantitatively evaluating diffusion-based text-guided image editing, with a curated dataset, 13 edit types, and both automated and human protocols (Basu et al., 2023). EdiVal-Bench can be understood as the multi-turn, object-centric continuation of that agenda. This suggests a broader methodological shift: benchmarking is moving away from reference-image dependence and coarse text-image alignment toward structured scene decomposition, symbolic verifiability, and evaluator modularity.