TECCI: Tricky Edits of Collected and Curated Images

Abstract: Despite tremendous recent progress, current text-guided image editing methods still struggle with many aspects of editing involving instruction following, minimally editing the source image, and ensuring high visual quality. These problems are especially apparent when the requested edit is challenging, such as those that involve position, motion, viewpoint, scale and creative edits. To systematically test generative image editors, we propose a novel image editing benchmark -- TECCI: Tricky Edits of Collected and Curated Images. TECCI consists of a completely new set of images we are releasing. The images in TECCI span 7 image categories. The images and these categories were curated intentionally to target weaknesses of existing methods. The edit instructions in TECCI are automatically generated by Gemini, covering 5 edit types per source image. We also curated a set of 530 images for which we created challenging manually written edit instructions. Overall, TECCI contains 7550 pairs of images and edit instructions. We conduct human evaluations of five leading image editing models on TECCI. Humans judge outputs along three dimensions: 1) instruction following, 2) minimality of the edits, and 3) visual quality. To scale-up the evaluation, we also build an auto-rater using Gemini that achieves 74.7% accuracy in matching human evaluations. Our evaluations reveal that: 1) none of the models exceed a 22% overall success rate, demonstrating the challenging nature of TECCI, 2) Nano Banana Pro is the best performing model overall, 3) models perform significantly better at instruction following compared to minimal edits and visual quality, 4) models struggle with editing architecture and nature images which require strong understanding of spatial layout and intricate visual details. 5) reasoning and creative edits are the most difficult, whereas color and appearance edits are the easiest.

• The paper presents TECCI, a novel benchmark featuring 1,934 carefully curated images and a diverse set of challenging edit instructions to evaluate text-guided image editing models.
• The paper employs a multi-dimensional evaluation protocol that assesses instruction following, image consistency, and visual quality through both human judgments and an MLLM-based autorater.
• The paper reveals that even state-of-the-art models achieve low overall success rates (<23%), underscoring significant challenges in handling complex spatial reasoning and creative edit tasks.

TECCI: A Comprehensive Benchmark for Challenging Text-Guided Image Editing

Dataset Design and Structure

The TECCI benchmark ("Tricky Edits of Collected and Curated Images" (2606.01213)) establishes a standardized, high-difficulty testbed for evaluating state-of-the-art (SOTA) text-guided image editing models. Unlike prior datasets, TECCI includes 1,934 manually collected natural images—deliberately excluded from all model pretraining sets—spanning seven challenging categories: Text, Clock, Vehicle, Architecture, Art, Animal, and Nature. This curation focuses on structural complexity, rare attributes, and typical failure modes in current models, such as text rendering, intricate spatial layouts, and creative edits.

The dataset is bifurcated into two subsets:

• TECCI-IRCS: 530 images paired with manually crafted, highly challenging edit instructions emphasizing difficult, creative, or compound edits across semantic dimensions.
• TECCI-GGIS: 1,404 images each linked to five LLM-generated edit instructions (using Gemini 3 Pro), operationalized to probe reasoning, style transfer, background manipulation, and inventive image modifications.

Edit instructions cover 28 fine-grained types, with balanced sampling and meta-categorical coverage to drive cross-context generalization. The linguistic complexity of IRCS instructions notably exceeds that of synthetic (GGIS) prompts, as shown by a higher Type-Token Ratio and greater prevalence of Hapax Legomena, which supports deeper analysis of image-language alignment.

Figure 1: TECCI-IRCS skews towards more challenging image types, while GGIS uses uniform sampling for broader coverage.

Figure 2: Edit instruction distributions demonstrate the high diversity and category balance of TECCI.

Figure 3: Co-occurrence heatmaps highlight the deliberate cross-coverage of edit types and image categories, essential for testing generalization.

Evaluation Protocols and Criteria

TECCI adopts a multi-criteria evaluation paradigm, decomposing model performance into three axes:

• Instruction Following (IF): Semantic fidelity to the natural language edit instruction.
• Image Consistency (IC): Minimal, localized editing with strict preservation of non-targeted regions and object identity.
• Visual Quality (VQ): Artifact-free, photorealistic, and high-resolution integration of edits.

Human evaluations are conducted at scale: 1,315 image-instruction pairs (1050 from GGIS, 265 from IRCS), with each edited output assessed independently by five trained raters on all three dimensions, yielding 32,875 human judgments. Scoring is rigorously thresholded (mean ≥ 4.5/5 for each criterion) to define "success" at a high quality bar.

To scale the evaluation beyond what is feasible with human raters, an MLLM-based autorater (Gemini 3 Flash) is introduced. This model, prompted to rate IF, IC, and VQ independently using Chain-of-Thought analysis and Z-score normalization, achieves 74.7% agreement with human aggregate overall judgments, exceeding many prior metric alignment baselines.

Figure 4: The distribution of human evaluation scores reveals a high bar for edits to be considered successful.

Experimental Results: Model Performance Analysis

Five SOTA proprietary models are benchmarked: Nano Banana Pro, Nano Banana 2, Grok Imagine Pro, GPT Image 1.5, and Seedream 5.0 Lite. Evaluations are based both on single-sided scoring and side-by-side Elo comparison.

Key findings:

• Low absolute attainment: No model exceeds a 22.3% overall success rate at the stringent threshold, with IRCS subset success rates notably lower than GGIS, underscoring the real-world challenge presented by TECCI.
• Dimension imbalance: Models demonstrate higher proficiency in IF (up to 47.7 for GGIS) than in IC or VQ, with the latter often lagging due to over-editing or inability to seamlessly integrate changes.
• Nano Banana Pro consistently outperforms other models in overall, IC, and VQ; Grok Imagine Pro achieves slightly higher IF, and GPT Image 1.5 displays anomalously high IF but poor IC, reflecting tendencies to over-apply edits or introduce artifacts.

  Figure 5: Models show variable performance across categories and edit types, with text-centric and animal categories generally yielding higher scores.

Text and animal images yield higher performance, attributed to their object-centric focus and reduced background complexity. Architecture and nature images pose the most substantial hurdles, primarily due to their demand for accurate spatial reasoning and intricate detail preservation. Among edit types, low-level changes (color, style) achieve the highest success; complex reasoning, high-level semantic transformation, and creative edits remain unresolved bottlenecks.

Figure 6: Qualitative samples illustrate the spectrum of edit complexity and model performance, with human scores annotated for each output.

Autorater Scalability and Human Alignment

The MLLM-based autorater enables comprehensive automated evaluation over the entire dataset. Its quantitative trends mirror the core findings of human judgments—no model surpasses a 19.8% overall rate post-normalization. The autorater is modestly more optimistic on IF and VQ, but the ranking of models and identification of performance cliffs on complex edits are robust. The strong cross-setup alignment validates the autorater as a viable, reproducible large-scale alternative to human annotation for benchmarking iterative model improvements.

Implications and Future Directions

TECCI surfaces essential open challenges for the image editing community. The persistent difficulty with spatial, reasoning-intensive, and creative edits emphasizes:

• The limitations of current decoder and diffusion architectures in structured manipulation and minimality,
• Insufficiency of CLIP-aligned or cross-attentional modules for high-fidelity, localized editing,
• The need for new data-driven methods (possibly exploiting multi-turn, context-rich setups) and loss functions more sensitive to edit granularity and non-target-area integrity.

Practically, TECCI's rigorous test splits and standardized evaluation afford a stable platform for model selection, ablation, and transfer learning studies. The explicit separation of instruction types, image categories, and comprehensive captions also invites research in controlled augmentation, compositionality, and cross-modal understanding.

Theoretically, further progress on benchmarks such as TECCI will likely require deeper integration of explicit scene representations, 3D awareness, and improved language-to-image grounding, beyond prompt tuning or scale alone. The dataset's agnostic design makes it extensible to future multi-turn and non-natural image regimes.

Conclusion

TECCI introduces a high-variance, challenging standard for text-guided image editing evaluation grounded in uniquely curated, model-unseen natural images, linguistically rich and diverse instructions, and robust, multi-dimensional human and automatic metrics. The results reveal a substantial gap between SOTA system capabilities and practical editing requirements—most acutely for edits demanding creative reasoning, visual consistency, and minimal edit locality. TECCI is positioned to stimulate advances in multimodal understanding, controlled image synthesis, and reliable evaluation protocols in generative vision research.