EdiVal-Agent: Object-Centric Editing Evaluation

• EdiVal-Agent is an object-centric framework for multi-turn, instruction-based image editing that decomposes images into semantically meaningful objects.
• It synthesizes context-aware edit instructions and evaluates results using a hybrid approach combining vision-language models, detectors, and quality scorers.
• The framework enhances interpretability by localizing failures to specific objects and offers a scalable benchmark across diverse editor families.

EdiVal-Agent is an automated, scalable, object-centric framework for generating and evaluating multi-turn instruction-based image editing. It was introduced to address a specific evaluation bottleneck: existing protocols either depend on paired reference images, which limit coverage and inherit biases from prior generative models, or rely on zero-shot vision-LLMs (VLMs), whose prompt-based judgments of instruction following, content consistency, and visual quality are often imprecise. In its canonical formulation, EdiVal-Agent decomposes an image into grounded objects, synthesizes diverse context-aware edit instructions, and evaluates the resulting edits along three axes—instruction following, content consistency, and visual quality—by combining VLMs, open-vocabulary detectors, semantic and pixel-level feature extractors, and learned human-preference scorers (Chen et al., 16 Sep 2025).

1. Problem formulation and design rationale

EdiVal-Agent is designed for multi-turn instruction-based image editing, where a model must execute a sequence of edits while preserving all content not targeted by the current instruction. The framework is motivated by three stated challenges: generating a broad set of realistic editing scenarios without hand-crafted reference pairs; verifying, in a fine-grained and interpretable manner, that each edit both follows the instruction and preserves all other content; and measuring perceptual realism in a way that correlates with human preference while remaining sensitive to common generative artifacts (Chen et al., 16 Sep 2025).

The framework’s defining commitment is object-centricity. Rather than scoring an edited image only at the whole-image level, it represents the image as a set of semantically meaningful foreground objects with associated attributes and grounded boxes. This decomposition supports symbolic verification for edits that can be checked geometrically or by object presence, while also enabling localized semantic and pixel-level comparisons for content preservation. The intended consequence is interpretability: failures can be localized to specific objects, attributes, or background regions instead of being collapsed into a single global score.

A recurrent simplification in prior evaluation practice is the assumption that prompt-only VLM scoring is sufficient. EdiVal-Agent was introduced precisely against that assumption. Its hybrid design combines detector-based symbolic checks with VLM-based semantic judgments and separate quality models, reflecting the position that no single component adequately captures the full structure of multi-turn editing performance.

2. Object-centric decomposition and context-aware instruction synthesis

The workflow begins with object-centric decomposition. A VLM, specifically GPT-4o, is prompted to list all clearly visible foreground objects in JSON form. Each entry uses the key "{material} {color} {object}", omitting unknown fields, and records the attributes {object, color, material, text, count, foreground}. Grounding-DINO is then used to filter hallucinated or low-confidence entities, retaining only boxes with score $\ge 0.35$ and normalized area $\le 0.4$. The output is a grounded object pool $\mathcal{P}_0$ mapping object names to attributes, bounding boxes, and detection scores (Chen et al., 16 Sep 2025).

Instruction synthesis is then organized around three dynamic pools maintained at turn $t$:

• $\mathcal{P}^{all}_t$: all objects ever mentioned
• $\mathcal{P}^{avail}_t$: objects currently editable
• $\mathcal{P}^{unch}_t$: objects never edited so far

With $\text{MAX\_TURNS}=3$, the framework samples an unused instruction type at each turn, falling back to subject_add if the sampled type is infeasible; selects object(s) from $\mathcal{P}^{avail}_t$; prompts GPT-4o with a task-specific template to generate a natural-language instruction; and updates all three pools according to the semantic effect of the chosen edit. The supported edit types are subject_add, subject_remove, subject_replace, color_alter, material_alter, text_change, position_change, count_change, and background_change (Chen et al., 16 Sep 2025).

The handling of background_change is notable. Once such an edit occurs, subsequent background consistency checks are disabled, and the generated instruction is augmented with the clause “make {foreground objects} unchanged.” This preserves comparability across turns by explicitly stating the preservation requirement even as the background semantics are modified.

This instruction-synthesis stage is not merely data generation. It defines the benchmark’s causal structure: each turn creates new preservation obligations, and the object pools track those obligations explicitly. That structure is what makes later evaluation interpretable on a per-turn basis.

3. Multi-stage evaluation and formal metrics

EdiVal-Agent evaluates each turn along three axes: instruction following, content consistency, and visual quality. These are computed by a multi-stage stack rather than a single end-to-end judge (Chen et al., 16 Sep 2025).

For instruction following, the framework distinguishes between symbolically verifiable and semantically verifiable edit types. Symbolically verifiable types—subject_add, subject_remove, subject_replace, position_change, and count_change—are evaluated entirely via Grounding-DINO detections. For subject_replace(old \rightarrow new), success is defined by

$$\max_{b \in B_{\text{old}},\, e \in E_{\text{new}}} \operatorname{IoU}(b,e) > 0,$$

where

$\le 0.4$0

Semantically verifiable types—color_alter, material_alter, text_change, and background_change—are evaluated on detector-guided crops from the edited image using Qwen2.5-VL with prompts such as “Is the {object} now {new_color}? Yes/No.”

The turn-level instruction-following score is

$\le 0.4$1

For content consistency, unchanged objects in $\le 0.4$2 are isolated using their original bounding boxes. Two measures are computed for each unchanged object $\le 0.4$3:

$\le 0.4$4

where $\le 0.4$5 is the DINOv3 embedding, and

$\le 0.4$6

Background regions $\le 0.4$7 are treated analogously after masking out all object boxes; if a background_change has occurred at or before turn $\le 0.4$8, background consistency is omitted for that and subsequent turns.

The turn-level content-consistency score is

$\le 0.4$9

optionally including $\mathcal{P}_0$0 when no background_change has occurred.

For visual quality, EdiVal-Agent uses HPSv3 to produce a scalar score $\mathcal{P}_0$1. It reports both the absolute score and the drift

$\mathcal{P}_0$2

so that “beautified” outputs with large stylistic drift can be distinguished from faithful edits with low drift. To detect over-exposure artifacts, the framework also converts RGB to luminance via

$\mathcal{P}_0$3

and tracks the per-image $\mathcal{P}_0$4 luminance quantile.

The overall performance at turn $\mathcal{P}_0$5 is defined as

$\mathcal{P}_0$6

A further positional check is defined for instructions such as “left of”: if $\mathcal{P}_0$7 and $\mathcal{P}_0$8 are box centers, the condition is $\mathcal{P}_0$9. This makes spatial verification explicit rather than inferential.

4. Tooling, benchmark construction, and model coverage

The pipeline is modular at every stage. Decomposition uses GPT-4o and Grounding-DINO with SwinT-OGC weights. Instruction generation uses task-specialized GPT-4o prompts. Instruction-following evaluation combines Grounding-DINO with Qwen2.5-VL (7B). Content consistency uses DINOv3 (ViT-B/16) plus normalized $t$0. Visual quality uses HPSv3, while low-level exposure drift is captured through luminance quantiles (Chen et al., 16 Sep 2025).

Instantiated on 572 real-world photos filtered from GEdit-Bench’s 606 images by removing 34 privacy-sensitive examples, the framework automatically builds EdiVal-Bench. The benchmark uses $t$1, yielding 1,716 total instructions across the nine supported edit types. This construction is significant because it replaces manual benchmark authoring with a procedural pipeline grounded in explicit object pools and edit semantics.

EdiVal-Bench evaluates 11 state-of-the-art editors spanning three model families:

Family	Editors
Autoregressive	Nano Banana (Gemini 2.5 Flash), GPT-Image-1, Gemini 2.0 Flash
Flow-matching	Qwen-Image-Edit, Step1X-Edit, FLUX.1-Kontext-dev, OmniGen
Diffusion	AnyEdit, UltraEdit, MagicBrush, InstructPix2Pix (IP2P)

This cross-paradigm coverage is central to the benchmark’s role. It is not restricted to a single architectural lineage, and several of its reported findings concern systematic differences between autoregressive, flow-matching, and diffusion editors under repeated editing.

5. Empirical findings and diagnosed failure modes

On instruction-following evaluation, EdiVal-Agent reports 81.3% agreement with human judgments over 4,576 annotations, compared with 75.2% for Qwen2.5-VL alone and 65.4% for CLIP directional scoring with a tuned threshold; the reported inter-annotator ceiling is 85.5%, and the improvement from the hybrid approach over VLM-only and CLIP baselines is stated to be statistically significant at $t$2 (Chen et al., 16 Sep 2025).

In multi-turn editing, autoregressive editors achieve the strongest balance between instruction following and content preservation. Nano Banana reports

$t$3

with latency approximately $t$4 s/img. GPT-Image-1 reports

$t$5

with latency approximately $t$6 s/img.

Flow-matching editors degrade more sharply across turns. Qwen-Image-Edit reports

$t$7

while its over-exposure drift is characterized by the $t$8 luminance statistic increasing from $t$9 and $\mathcal{P}^{all}_t$0 dropping from $\mathcal{P}^{all}_t$1 over three turns. FLUX.1-Kontext-dev is reported as having the best consistency among flow models, with

$\mathcal{P}^{all}_t$2

Diffusion editors are reported to struggle on instruction following, with values below 52% at $\mathcal{P}^{all}_t$3, while preserving moderate consistency.

Several failure modes are explicitly identified. count_change is the hardest task: even GPT-Image-1 achieves less than 25% at $\mathcal{P}^{all}_t$4, and most non-autoregressive models are near zero. position_change and subject_remove expose VLM hallucinations and spatial-reasoning failures when VLM-only evaluation is used. Over-exposure artifacts accumulate in flow-matching editors; notably, Qwen2.5-VL quality scores remain high in such cases, while HPSv3 and luminance quantiles detect the degradation. The paper attributes sharp multi-turn degradation in non-autoregressive models to exposure bias: models trained for single-turn editing drift when repeatedly editing their own outputs, whereas autoregressive models with full-history context are more stable.

A further comparative result concerns multi-turn editing versus a single complex prompt containing three instructions. For autoregressive models, sequential multi-turn pipelines outperform complex single-shot prompting; Nano Banana’s $\mathcal{P}^{all}_t$5 instruction-following score is 35.35% in multi-turn mode versus 28.14% with a single complex prompt. For flow-matching models, the reverse tendency is reported in at least one case: Qwen-Image-Edit scores 22.55% in multi-turn mode versus 27.62% with a concatenated prompt, consistent with the claim that avoiding repeated self-editing can reduce compounding exposure bias.

6. Relation to JarvisEvo and the broader EdiVal-Agent paradigm

A later integrated description accompanying JarvisEvo presents the EdiVal-Agent paradigm in a broader form: not only as an evaluation framework, but as a tightly coupled editing-and-evaluation agent. In that realization, a vision encoder ingests the source image $\mathcal{P}^{all}_t$6, a LLM with 8B-token capacity ingests the query $\mathcal{P}^{all}_t$7, and both modalities are fused in a shared Transformer stack. Over 200 parameterized Adobe Lightroom tools are exposed through an A2L wrapper. The same model weights are reused as an evaluator conditioned on $\mathcal{P}^{all}_t$8, where $\mathcal{P}^{all}_t$9 is the full editing trajectory, to generate a self-evaluation score and rationale $\mathcal{P}^{avail}_t$0 (Lin et al., 28 Nov 2025).

The core inference mechanism is the interleaved multimodal chain-of-thought (iMCoT), which alternates textual reasoning, tool selection, external tool execution, and state update:

1. $\mathcal{P}^{avail}_t$1 with > tags > > 2. $\mathcal{P}^{avail}_t$2 with <tool_call> tags > > 3. $\mathcal{P}^{avail}_t$3 > > 4. $\mathcal{P}^{avail}_t$4 > > The resulting editing trajectory is > > $\mathcal{P}^{avail}_t$5 > > This closes the perception–action loop by forcing each reasoning step to be grounded in the actual edited image rather than in text-only internal planning. > > JarvisEvo further introduces synergistic editor-evaluator policy optimization (SEPO), comprising co-evolving editor and evaluator loops. For the editor, the reward decomposes as > > $\mathcal{P}^{avail}_t$6 > > where $\mathcal{P}^{avail}_t$7 is a format reward, $\mathcal{P}^{avail}_t$8 is a tool-accuracy reward, and $\mathcal{P}^{avail}_t$9 is a pairwise-preference reward, > > $\mathcal{P}^{unch}_t$0 > > Optimization uses a Group Relative Policy Optimization (GRPO) objective with advantage > > $\mathcal{P}^{unch}_t$1 > > A selective loss masking mechanism masks self-evaluation tokens from the editor’s policy gradient. For the evaluator, the reward is > > $\mathcal{P}^{unch}_t$2 > > with > > $\mathcal{P}^{unch}_t$3 > > where $\mathcal{P}^{unch}_t$4. > > The training pipeline has three stages: supervised fine-tuning on 150K editing/evaluation samples; SEPO with dual RL loops on 20K samples; and reflection fine-tuning on 5K on-policy trajectories. The cold-start stage uses 110K editing samples with gold iMCoT traces and 40K evaluation samples; the SEPO stage uses 10K editing pairs and 10K evaluation pairs with $\mathcal{P}^{unch}_t$5 rollouts per query; the reflection set is produced on-policy when one trajectory wins over another. The stated purpose of this design is to mitigate reward hacking by continually recalibrating the evaluator to human-annotated scores while the editor trains against the evaluator’s current judgments. > > On ArtEdit-Bench-Lr, JarvisEvo is reported to outperform Nano-Banana on the English set with $\mathcal{P}^{unch}_t$6 versus $\mathcal{P}^{unch}_t$7, $\mathcal{P}^{unch}_t$8 versus $\mathcal{P}^{unch}_t$9, $\text{MAX\_TURNS}=3$0 versus $\text{MAX\_TURNS}=3$1, $\text{MAX\_TURNS}=3$2 versus $\text{MAX\_TURNS}=3$3, and overall score $\text{MAX\_TURNS}=3$4 versus $\text{MAX\_TURNS}=3$5, corresponding to an average relative improvement of 18.95% and a 44.96% improvement in pixel-level fidelity. On ArtEdit-Bench-Eval, it reports $\text{MAX\_TURNS}=3$6 and $\text{MAX\_TURNS}=3$7. This suggests a broadened interpretation of the EdiVal-Agent idea: evaluation is no longer only an external benchmarking apparatus, but part of a co-adaptive editing system in which editing and self-assessment are jointly optimized.