Develop more accurate evaluation metrics for image editing that capture subject consistency

Develop and validate objective evaluation metrics for image editing that accurately measure subject consistency (i.e., preservation of a subject’s identity and appearance across edits), to address deficiencies in current benchmark assessments of region-based editing tasks.

Background

In constructing the image editing dataset, the authors observed that training with GPT-Image-Edit-1.5M can improve benchmark scores but severely disrupts character (subject) consistency, leading them to exclude this dataset from training.

This highlighted a broader limitation in existing evaluation practices for image editing: current metrics insufficiently capture subject consistency, especially for region-based edits, motivating the need for more accurate metrics.

References

This observation also highlights the need for more accurate evaluation metrics in image editing (such as subject consistency), which is left for future work.

— EMMA: Efficient Multimodal Understanding, Generation, and Editing with a Unified Architecture  (2512.04810 - He et al., 4 Dec 2025) in Section 3, Image Editing Data (IT2I)

However, scalable and fully objective evaluation of fine-grained local guidance remains an open research problem.

— RefDiT: Local Attribute Guidance in Reference-Based Image Generation  (2609.04976 - Mishra et al., 4 Sep 2026) in Discussion, subsection “Evaluation limitations”