EditReward-Bench: Image Editing Reward Evaluation

• EditReward-Bench is a family of benchmarks for instruction-guided image editing that uses both pairwise and multi-candidate evaluations.
• The EditScore and EditReward versions differ in dataset construction, annotation protocols, and metric design, influencing evaluation outcomes.
• Recent research employs EditReward-Bench to stress-test reward models, revealing challenges in multi-way ranking and model consistency.

EditReward-Bench denotes reward-model evaluation benchmarks for instruction-guided image editing. In the recent literature, the name has been used for at least two related but non-identical benchmark constructions: the benchmark introduced with EditScore, which contains 3,072 pairwise preference tuples, and the benchmark introduced with EditReward, which is reported as 500 Groups (≈1.5K units) and supports $K=2/3/4$ candidate ranking. Both are designed to evaluate whether a model can judge edited images relative to a source image and an editing instruction, but they differ in dataset construction, annotation protocol, dimensional rubric, and metric design. Later work explicitly distinguishes these prior EditReward-Bench datasets from EditReward-Compass, which is introduced as a new benchmark rather than a synonym for EditReward-Bench (Luo et al., 28 Sep 2025, Wu et al., 30 Sep 2025, Bai et al., 13 May 2026).

1. Nomenclature and scope

In the cited literature, “EditReward-Bench” is not a single frozen artifact. The term refers to prior reward-model benchmarks released by other works—specifically EditScore and EditReward. This naming overlap matters because later papers sometimes report results on “EditReward-Bench” while adopting different task structures, dimensional rubrics, or evaluation protocols (Bai et al., 13 May 2026).

The EditScore benchmark is introduced as a “comprehensive benchmark to systematically evaluate reward models on editing quality.” Its design centers on pairwise preferences over edited outputs drawn from a heterogeneous model pool, with three evaluation axes specialized for instruction-guided editing: Prompt Following, Consistency, and Overall Quality (Luo et al., 28 Sep 2025).

The EditReward benchmark is introduced as “a stricter and more human-grounded evaluation for instruction-guided image editing reward models.” Its distinctive features are multi-dimensional labeling—Instruction Following and Visual Quality on a 4-point Likert scale—and multi-way preference tasks with $K=2$, $K=3$, and $K=4$ candidates (Wu et al., 30 Sep 2025).

A concise comparison is given below.

Benchmark usage	Composition	Evaluation emphasis
EditScore’s EditReward-Bench	3,072 pairwise preference tuples; 13 subtasks; 11 editing models	PF, C, O pairwise accuracy
EditReward’s EditReward-Bench	500 Groups (≈1.5K units); $K=2/3/4$; held-out test split	Instruction Following and Visual Quality; multi-way accuracy
EditReward-Compass	2,251 preference pairs; 36 tasks; cross-/intra-model sampling	IA, VC, VQ under realistic RL-style comparisons

2. Dataset constructions

The EditScore version of EditReward-Bench is organized around four conceptual categories with 13 concrete subtasks: Subject, Appearance, Scene, and Advanced. The subtasks are subject addition, subject removal, subject replace; color alteration, material modification, style transfer, tone transformation; background change, extract; and portrait beautification, text modification, motion change, and hybrid edit. To reflect real-world variability, the candidate pool of edited outputs is sourced from 11 heterogeneous editing models: Step1X-Edit, Step1X-Edit v1.1, Qwen-Image-Edit, OmniGen2, FLUX-Kontext-dev, FLUX-Kontext-pro, Bagel, MagicBrush, Omnigen, gpt-image-1, and Gemini-2.5-image-preview (Luo et al., 28 Sep 2025).

The EditReward version of EditReward-Bench is curated from a held-out test split of EditReward-Data. EditReward-Data aggregates 9,557 unique instruction–image pairs from six vetted editing benchmarks: GEdit-Bench, ImgEdit-Bench, MagicBrush, AnyEdit, EmuEdit, plus an internal set. For each instruction, 12 candidates are generated by six editing models—Step1X-Edit, FLUX.1 Kontext, Qwen-Image-Edit, BAGEL, Ovis-U1, OmniGen2—with multiple seeds, and seven candidates are randomly sampled for human scoring. The resulting benchmark comprises 500 high-quality groups sampled from the candidate pool and spanning seven distinct editing categories (Wu et al., 30 Sep 2025).

The two benchmarks therefore target overlapping but non-identical notions of reward evaluation. One emphasizes cross-model, pairwise discrimination across 13 subtasks; the other emphasizes stricter multi-candidate ordering and ranking consistency on a held-out test split. This suggests that benchmark scores reported under the shared label “EditReward-Bench” are not automatically interchangeable.

3. Annotation protocols and metrics

The EditScore benchmark employs a rigorous expert annotation protocol. For each input consisting of an instruction and source image, annotators are shown five candidate edited outputs randomly sampled from the model pool. Experts perform tiered ranking under each dimension—Prompt Following, Consistency, and Overall Quality—allowing ties such as “3|12|45.” Two independent expert raters annotate each example, and only cases with complete agreement in tiered rankings are retained. The tiered rankings are then converted into pairwise preference tuples by including only cross-tier comparisons and excluding intra-tier comparisons (Luo et al., 28 Sep 2025).

Its principal metric is pairwise preference accuracy, defined as

$$A_{\text{pair}} = \frac{1}{|P|} \sum_{(i,j)\in P} \mathbf{1}\{S(i) > S(j)\},$$

where $P$ is the set of human preference pairs and $S(i)$ is the scalar score assigned by the reward model to output $i$ under the specified dimension. In line with VIEScore, the framework decomposes evaluation into Semantic Consistency and Perceptual Quality, and the composite overall score is computed as the geometric mean $$S_{\text{final}} = \sqrt{S_{\text{SC}} \cdot S_{\text{PQ}}}$$ (Luo et al., 28 Sep 2025).

The EditReward benchmark uses a different annotation structure. Each group is annotated independently by three trained experts using the two-dimensional rubric of Instruction Following and Visual Quality, with scores on a 4-point Likert scale. Cross-checking and majority agreement are used to finalize labels. Evaluation supports pairwise and pointwise settings and constructs $K=2$0-, $K=2$1-, and $K=2$2-way tuples for multi-way preference tasks (Wu et al., 30 Sep 2025).

Later work using the EditReward benchmark formulates the task as K-way ranking: for a source image $K=2$3, editing instruction $K=2$4, and $K=2$5 candidates $K=2$6, the evaluator produces discrete scores on the same 1–5 rubric and a ranking $K=2$7. The ranking-accuracy metric counts exact agreement, including tie structure, between predicted and human rankings:

$K=2$8

For $K=2$9, this reduces to pairwise preference accuracy; for $K=3$0, the metric is stricter than top-1 because the entire ranking must match (Zhang et al., 9 May 2026).

4. Baseline findings

On the EditScore benchmark, proprietary VLMs form the upper tier. GPT-4.1 reaches $K=3$1, $K=3$2, $K=3$3; GPT-5 reaches $K=3$4, $K=3$5, $K=3$6; and Gemini-2.5-Pro reaches $K=3$7, $K=3$8, $K=3$9. Open-source VLM baselines underperform: Qwen2.5-VL-72B reaches $K=4$0, $K=4$1, $K=4$2. The specialized EditScore reward models close this gap, and EditScore-72B Avg@4 reaches $K=4$3, $K=4$4, $K=4$5, surpassing GPT-5 on overall accuracy (Luo et al., 28 Sep 2025).

On the EditReward benchmark, the reported baselines also show the difficulty of the task, especially as $K=4$6 increases. GPT-5 records $K=4$7 at $K=4$8, $K=4$9 at $K=2/3/4$0, and $K=2/3/4$1 at $K=2/3/4$2, with $K=2/3/4$3 overall. Gemini-2.5-Flash records $K=2/3/4$4, $K=2/3/4$5, $K=2/3/4$6, and $K=2/3/4$7. The best overall result in the table is EditReward with a MiMo-VL-7B backbone: $K=2/3/4$8 at $K=2/3/4$9, $$A_{\text{pair}} = \frac{1}{|P|} \sum_{(i,j)\in P} \mathbf{1}\{S(i) > S(j)\},$$0 at $$A_{\text{pair}} = \frac{1}{|P|} \sum_{(i,j)\in P} \mathbf{1}\{S(i) > S(j)\},$$1, $$A_{\text{pair}} = \frac{1}{|P|} \sum_{(i,j)\in P} \mathbf{1}\{S(i) > S(j)\},$$2 at $$A_{\text{pair}} = \frac{1}{|P|} \sum_{(i,j)\in P} \mathbf{1}\{S(i) > S(j)\},$$3, and $$A_{\text{pair}} = \frac{1}{|P|} \sum_{(i,j)\in P} \mathbf{1}\{S(i) > S(j)\},$$4 overall (Wu et al., 30 Sep 2025).

A recurrent empirical pattern is multi-way difficulty. The EditReward results show large drops from $$A_{\text{pair}} = \frac{1}{|P|} \sum_{(i,j)\in P} \mathbf{1}\{S(i) > S(j)\},$$5 to $$A_{\text{pair}} = \frac{1}{|P|} \sum_{(i,j)\in P} \mathbf{1}\{S(i) > S(j)\},$$6 for all models, indicating that accurate pairwise discrimination does not guarantee consistent multi-candidate ordering. This is one of the main reasons the benchmark is described as stricter than pairwise-only suites (Wu et al., 30 Sep 2025).

5. Reuse in later reward-model research

Subsequent work has used EditReward-Bench as a stress test for new reward-modeling paradigms. RewardHarness treats the EditReward benchmark purely as an evaluator benchmark: given a source image, an editing instruction, and $$A_{\text{pair}} = \frac{1}{|P|} \sum_{(i,j)\in P} \mathbf{1}\{S(i) > S(j)\},$$7 edited candidates, the system must produce a ranking that agrees with human preference labels. Using only 100 preference demonstrations from the EditReward training split, RewardHarness with Gemini-2.0-Flash as Sub-Agent attains $$A_{\text{pair}} = \frac{1}{|P|} \sum_{(i,j)\in P} \mathbf{1}\{S(i) > S(j)\},$$8 at $$A_{\text{pair}} = \frac{1}{|P|} \sum_{(i,j)\in P} \mathbf{1}\{S(i) > S(j)\},$$9, $P$0 at $P$1, and $P$2 at $P$3, with $P$4 average accuracy across EditReward-Bench and GenAI-Bench (Zhang et al., 9 May 2026).

SpatialReward uses EditReward-Bench as the “general reward modeling” benchmark and reports three dimensions: Prompt Following, Source Consistency, and Overall. In that unified pointwise setting, SpatialReward (8B) achieves $P$5, $P$6, and $P$7. Its ablations on EditReward-Bench report $P$8 for an SFT baseline without grounding, $P$9 for SFT with Box Only, $S(i)$0 for SFT with Think-with-Box, and $S(i)$1 for RL with Think-with-Box (Long et al., 7 Feb 2026).

MMRB2 positions image editing as one subtask within a broader multimodal reward benchmark and reports an EditReward baseline of $S(i)$2 on the single-image editing subset, compared with $S(i)$3 for a Qwen2.5-VL-7B judge using the same backbone without preference training. In the same evaluation, GPT-5 reaches $S(i)$4, and Gemini 3 Pro reaches $S(i)$5 on image editing (Hu et al., 18 Dec 2025).

These later uses do not erase protocol differences across EditReward-Bench variants. Rather, they show that the benchmark family became a common substrate for studying preference-trained reward models, MLLM-as-judge systems, in-context evaluators, and RL-oriented reward signals.

6. Limitations, ambiguities, and the transition to EditReward-Compass

Several limitations recur across the EditReward-Bench literature. In the EditScore benchmark, the paper does not report explicit inter-annotator agreement coefficients, does not provide instruction-length statistics or defined train/validation/test splits, and focuses on images rather than video or temporal consistency (Luo et al., 28 Sep 2025). In the EditReward benchmark, exact category-level breakdowns are not reported, the multi-way metric is intentionally all-or-nothing, and the protocol emphasizes majority agreement after cross-checking rather than reporting explicit $S(i)$6 statistics (Wu et al., 30 Sep 2025).

A further issue is nomenclature. A recurrent source of confusion is that “EditReward-Bench” refers to prior benchmarks released by EditScore and EditReward, whereas “EditReward-Compass” is the new benchmark introduced in “Edit-Compass & EditReward-Compass.” EditReward-Compass is explicitly described as not a synonym for EditReward-Bench (Bai et al., 13 May 2026).

EditReward-Compass is introduced to address limitations of earlier datasets. It contains 2,251 preference pairs, spans 36 tasks, and uses both cross-model and same-model intra-model sampling with FlowGRPO-inspired stochasticity to mirror real RL optimization decisions. Its judging pipeline reuses the dimensions Instruction Awareness, Visual Consistency, and Visual Quality, and preferences are verified in two stages by eight image-editing experts, with a pair retained only if all five annotators in the second stage reach unanimous agreement (Bai et al., 13 May 2026).

This transition is significant because earlier EditReward-Bench datasets were largely centered on cross-model sampling or narrower task coverage. EditReward-Compass adds Algorithmic Visual Reasoning, Dynamic Manipulation, World Knowledge Reasoning, Multi-Image editing, and Complex Paint, and is presented as a more realistic RL-style reward-model benchmark rather than a rebranding of EditReward-Bench (Bai et al., 13 May 2026).

7. Significance for reward modeling in image editing

EditReward-Bench established reward evaluation as a first-class problem in instruction-guided image editing. The benchmarks replaced generic image metrics with source-conditioned, instruction-conditioned human preferences, and they made reward-model performance measurable in the same operational space required by downstream selection and RL. The EditScore work explicitly links benchmark accuracy to robust online RL with Flow-GRPO on OmniGen2, while the EditReward work links benchmark alignment to large-scale data curation and improved Step1X-Edit training (Luo et al., 28 Sep 2025, Wu et al., 30 Sep 2025).

The benchmark family also exposed recurring failure modes of general-purpose judges. Open-source VLMs lagged specialized reward models on both pairwise and multi-way settings; same-model comparisons proved harder than different-model comparisons; and later analyses argued that perception failures such as “Attention Collapse” or weak cross-image comparison degrade consistency judgments (Long et al., 7 Feb 2026, Hu et al., 18 Dec 2025).

Taken together, the literature presents EditReward-Bench as both a benchmark family and a methodological turning point. It provided a human-grounded testbed for reward models in image editing, revealed sharp differences between generic VLM-as-judge systems and domain-specialized evaluators, and set the stage for successor benchmarks such as EditReward-Compass that seek broader task coverage, more realistic RL-style sampling, and more structured human-aligned judging (Bai et al., 13 May 2026).