LMM4Edit Evaluation Framework

• The framework introduces a multi-dimensional, human-aligned metric using LMMs for fine-grained text-guided image editing.
• It integrates quality, alignment, and preservation scores derived from extensive MOS ratings on the EBench-18K benchmark.
• Its modular design allows extension to document- and code-level editing, ensuring high correlation with expert human assessments.

LMM4Edit is an evaluation framework and metric based on Large Multimodal Models (LMMs) for benchmarking text-guided image editing (TIE) and, by extension, broader AI model editing tasks. The core contribution is an all-in-one, human-aligned, and multi-dimensional assessment protocol, validated on the EBench-18K dataset, and extensible to other domains such as document-level or code-level editing. LMM4Edit is distinguished by high granularity, explicit metric design, and deep integration with state-of-the-art LMM architectures (Xu et al., 22 Jul 2025).

1. Motivation and Context

The domain of text-guided image editing requires models to modify source images according to natural-language instructions. Traditional image quality assessment (IQA) metrics (e.g., SSIM, LPIPS, BIQI, BRISQUE) primarily capture low-level, natural-distortion artifacts but fail to quantify semantic and compositional changes or to assess the alignment between an editing instruction and the resulting image. Vision–language metrics such as CLIPScore, ImageReward, and PickScore focus on global text–image alignment, mainly for text-to-image generation, neglecting the crucial comparison between source and target images in the editing setting.

Recent large multimodal models (LMMs), such as GPT-4V and InternVL2, provide improved zero-shot image–instruction understanding, but remain suboptimal for fine-grained assessment of editing alignment and attribute preservation, especially when compared to human mean opinion scores (MOS) (Xu et al., 22 Jul 2025).

LMM4Edit was developed to address these deficits with an evaluation protocol that (a) captures core dimensions of perceptual quality, editing alignment, and attribute preservation, (b) measures task-specific instruction satisfaction via QA, (c) is validated at scale against expert human preferences, and (d) generalizes to scenarios beyond image editing, including document and code model editing (Xu et al., 22 Jul 2025, Zeng et al., 26 May 2025, Li et al., 10 Mar 2025).

2. The EBench-18K Benchmark

EBench-18K underpins LMM4Edit’s image evaluation protocol, providing the first large-scale, multi-dimensional benchmark for TIE:

• Composition: 1,080 high-resolution source images (split among 21 editing tasks: 13 high-level semantic and 8 low-level enhancements), 3,240 editing prompts, and 18,360 edited images generated by 17 state-of-the-art TIE models.
• Annotations: Each image is annotated with three separate MOS ratings (Quality, Alignment, Preservation) from 15 professional annotators using a 5-point continuous scale; scores are z-normalized and mapped to $[0, 100]$. Each image–prompt pair is also annotated with a binary yes/no for instruction satisfaction across 275,400 responses in total.
• Scope: Combining more than 1 million individual annotations, EBench-18K supports robust, high-resolution ground truth for training and evaluating evaluators—not only TIE models but also LMMs serving as evaluators (Xu et al., 22 Jul 2025).

Component	Statistic	Description
Source Images	1,080	Photographic, task-diversified
Edited Images	18,360	17 SOTA models × 1,080 sources
Editing Tasks	21	13 semantic, 8 enhancement
Annotators/Image	15	Per dimension (Q/E/P), per QA
MOS Values	55,080	3 dimensions × 18,360 images
QA Responses	275,400	18,360 images × 15 annotators
Prompts	3,240	1,080 instruction, 2,160 description

EBench-18K’s multi-dimensional labels facilitate comprehensive benchmarking of TIE models as well as evaluation protocols and provide the basis for LMM4Edit’s supervised score regression (Xu et al., 22 Jul 2025).

3. LMM4Edit Architecture and Methodology

LMM4Edit realizes an LMM-based metric system that predicts fine-grained scores on multiple axes and answers task-specific QA for any TIE scenario:

• Inputs: Edited image $I_e$, source image $I_s$, editing instruction prompt $T_p$
• Outputs: Continuous scores for Perceptual Quality ($S_q$), Editing Alignment ($S_e$), Attribute Preservation ($S_p$), and a binary QA result ($A$)
• Pipeline:
  • Stage 1: Instruction tuning for text score and QA, trained with cross-entropy loss.
  • Stage 2: Score regression via an MLP decoder, trained with MSE loss on MOS.
• Fine-tuning:
  • Instruction Tuning: Explicit natural-language prompts for each evaluation dimension.
  • AdaLoRA: Low-rank adaptation with adaptive rank allocation for both visual and LLM layers; enables efficient parameter updates ($I_e$1 of original parameters updated).
  • Two-Stage Training: First cross-entropy (rough format), then MSE (precise scores).

The mathematical design introduces the LMM4Edit aggregate score:

$I_e$2

where each $I_e$3 is normalized to $I_e$4, emphasizing editing alignment for TIE ranking (Xu et al., 22 Jul 2025).

4. Evaluation Protocols, Results, and Comparative Analysis

LMM4Edit achieves high alignment with human assessments, with Spearman rank correlations (SRCC) of 0.9136 for Quality, 0.8830 for Alignment, and 0.9048 for Preservation on EBench-18K. This performance surpasses the strongest fine-tuned baselines (InternVL2.5: 0.8836/0.8207/0.8841), with average improvements of +3.3% (Quality), +6.6% (Alignment), and +2.3% (Preservation).

Zero-shot validation on the I2EBench QA benchmark demonstrates leading generalization, exhibiting the highest yes/no accuracy among SOTA LMM-based evaluators without dataset-specific tuning.

Ablation studies confirm the necessity of visual+LLM LoRA (joint adaptation, +6% SRCC), projector fine-tuning (+5% SRCC), AdaLoRA over fixed-rank (+1.8% SRCC), and superior backbones (Qwen2.5-VL outperforms InternVL2.5 and LLaVA-NeXT by 2–3% SRCC) (Xu et al., 22 Jul 2025).

5. Connections to Document- and Code-Level Model Editing

LMM4Edit’s modular protocol generalizes beyond images:

• Document-level Model Editing: DocMEdit establishes the paradigm for editing model outputs in the regime where inputs, outputs, and updates are at full-document scale, with metrics focusing on both effectiveness (Document-ROUGE, Document-Entity) and locality of the edit (ROUGE and Entity Side Effect). Existing editing approaches (FT, MEMIT, RAG, SKEME) underperform on long documents, often corrupting a large fraction of unchanged text. LMM4Edit’s explicit, multi-view evaluation and robust ground truth suggest potential application areas—if extended to textual modalities—by combining context retrieval, local parameter adaptation, fact orchestration, and side-effect regularization (Zeng et al., 26 May 2025).
• Code Editing: EditLord operationalizes code editing as a compositional, rule-based process, where edits are factored through meta-rules inferred by an inductive LM, specifications, and final code generation. This explicit symbolic backbone is compatible with LMM4Edit’s architecture, enabling multimodal evaluation (source code, AST, NL spec, test results). Integrating meta-rule libraries as model input provides both interpretability and robustness, suggesting that future LMM4Edit variants may jointly reason across visual, textual, and symbolic data (Li et al., 10 Mar 2025).

6. Practical Recommendations, Limitations, and Prospects

LMM4Edit’s multi-dimensional evaluation framework is recommended for:

• Automatic benchmarking throughout the TIE model development cycle.
• Hyperparameter search using the aggregate metric $I_e$5.
• Extension to additional editing tasks through collection and labeling of new MOS/QA data, leveraging the same architectural pipeline.
• Resource-efficient fine-tuning owing to AdaLoRA’s parameter reuse.

While LMM4Edit currently achieves high human correlation and generalization, extensions to document and code editing require new benchmarks and aggregation strategies—particularly for handling long-context reasoning, fact insertion planning, and minimal side-effects in editing. Open challenges include automating relevant region identification, effective multi-fact integration, scaling to specialized or multilingual scenarios, and ensuring negligible side effects under robust edit schedules (Xu et al., 22 Jul 2025, Zeng et al., 26 May 2025, Li et al., 10 Mar 2025).

7. Significance, Impact, and Future Directions

LMM4Edit establishes a new standard for TIE evaluation in terms of dataset scale, annotation granularity, and human alignment. It closes the gap between LMM-based and human evaluation, particularly in editing alignment and attribute preservation—areas where prior scores and models struggled. Its all-in-one metric and extensible methodology presage a shift toward LMM-evaluated and LMM-edited domains spanning vision, language, and symbolic data.

A plausible implication is that future LMM4Edit-like frameworks, with modular input encodings and compositional meta-rule integration, will become foundational for reliable, interpretable, and scalable evaluation of both image and model editing tasks. The open-source release of data and code further accelerates adoption and community-driven extension (Xu et al., 22 Jul 2025).