PICABench: How Far Are We from Physically Realistic Image Editing? (2510.17681v2)

Abstract: Image editing has achieved remarkable progress recently. Modern editing models could already follow complex instructions to manipulate the original content. However, beyond completing the editing instructions, the accompanying physical effects are the key to the generation realism. For example, removing an object should also remove its shadow, reflections, and interactions with nearby objects. Unfortunately, existing models and benchmarks mainly focus on instruction completion but overlook these physical effects. So, at this moment, how far are we from physically realistic image editing? To answer this, we introduce PICABench, which systematically evaluates physical realism across eight sub-dimension (spanning optics, mechanics, and state transitions) for most of the common editing operations (add, remove, attribute change, etc.). We further propose the PICAEval, a reliable evaluation protocol that uses VLM-as-a-judge with per-case, region-level human annotations and questions. Beyond benchmarking, we also explore effective solutions by learning physics from videos and construct a training dataset PICA-100K. After evaluating most of the mainstream models, we observe that physical realism remains a challenging problem with large rooms to explore. We hope that our benchmark and proposed solutions can serve as a foundation for future work moving from naive content editing toward physically consistent realism.

• The paper introduces PICABench to systematically evaluate physical realism in instruction-based image editing.
• It proposes a region-grounded QA evaluation protocol to assess physical effects such as shadows, reflections, and deformations.
• The use of synthetic video data (PICA-100K) enables targeted supervision, yielding measurable improvements in editing accuracy and consistency.

PICABench: A Benchmark for Physically Realistic Image Editing

Motivation and Problem Statement

Instruction-based image editing has advanced rapidly, with modern models capable of following complex natural language instructions to manipulate images in semantically coherent ways. However, the field has largely prioritized semantic fidelity and perceptual quality, neglecting the physical realism of edits. Physically realistic editing requires that models not only perform the requested operation (e.g., object addition/removal) but also update all associated physical effects—such as shadows, reflections, deformations, and state transitions—consistent with the laws of optics, mechanics, and material state changes. Existing benchmarks and evaluation protocols are insufficient, as they focus on instruction completion and overlook these critical physical effects, resulting in models that produce visually plausible but physically implausible outputs.

PICABench: Benchmark Design and Taxonomy

PICABench is introduced as a comprehensive diagnostic benchmark to systematically evaluate physical realism in image editing. The benchmark is structured around three principal dimensions, each further divided into sub-dimensions with concrete, checkable criteria:

• Optics:
  • Light propagation (shadow geometry, occlusion, softness)
  • Reflection (view-dependent highlights, mirror consistency)
  • Refraction (background distortion through transparent media)
  • Light-source effects (integration of new light sources with scene illumination)
• Mechanics:
  • Deformation (material-consistent shape changes)
  • Causality (physically plausible contacts, supports, and force redistribution)
• State Transition:
  • Global (scene-wide changes: time of day, weather, phase transitions)
  • Local (object- or region-specific changes: wetting, burning, melting, fracturing)

The benchmark comprises 900 editing samples, each paired with multi-level natural language instructions (superficial, intermediate, explicit) targeting specific physical phenomena. Images are selected to maximize diversity in materials, lighting, and physical cues, and instructions are crafted to induce edits that test adherence to physical laws.

PICAEval: Region-Grounded, QA-Based Evaluation Protocol

Evaluating physical realism is non-trivial due to the lack of reference images and the contextual nature of physical plausibility. PICAEval addresses this by introducing a region-grounded, question-answering (QA) protocol:

• For each edited image, human annotators mark key regions where physical evidence should manifest (e.g., shadow zones, reflective surfaces, contact points).
• For each region, a set of binary (yes/no) questions is generated (using GPT-5 and manual review) to probe both instruction completion and physical consistency.
• At evaluation, a VLM (e.g., GPT-5, Qwen2.5-VL-72B) is prompted with the edited image, instruction, region, and question, and must answer based solely on visible evidence.

This protocol reduces hallucination, increases interpretability, and achieves high alignment with human judgments (Pearson $r > 0.9$ with human Elo rankings). The metric is robust to subtle, localized errors and enables fine-grained diagnostic analysis across physical sub-dimensions.

PICA-100K: Synthetic Video-Based Dataset for Physics-Aware Supervision

To address the lack of large-scale, physics-aware training data, the authors introduce PICA-100K, a synthetic dataset of 105,085 instruction-based editing samples derived from video generation:

• Scene and subject prompts are curated and refined using GPT-5, rendered into images with a T2I model (FLUX.1-Krea-dev).
• Motion-based edit instructions are generated and applied using an I2V model (Wan2.2-14B), simulating physically plausible transformations.
• The first and last frames of each video are extracted as (source, edited) pairs, with instructions and QA annotations generated automatically.

This pipeline enables precise, controllable supervision signals for training models to perform physically realistic edits, without the cost and limitations of real-world video annotation.

Experimental Results and Analysis

Benchmarking State-of-the-Art Models

Eleven open- and closed-source models are evaluated on PICABench using PICAEval. Key findings:

• All open-source models score below 60% accuracy on physical realism; only closed-source models (GPT-Image-1, Seedream 4.0) slightly exceed this threshold.
• Unified multimodal models (e.g., Bagel, OmniGen2) underperform compared to dedicated image editing models, indicating that visual understanding alone does not guarantee physical realism.
• Model performance improves with prompt specificity: explicit prompts yield higher scores than superficial or intermediate prompts, but the gains are limited by the models' lack of internalized physics knowledge.

Effectiveness of Video-Based Supervision

Fine-tuning FLUX.1-Kontext on PICA-100K yields consistent improvements across all physical sub-dimensions:

• Overall accuracy improves by +1.71% (from 48.93% to 50.64%) and physical consistency (PSNR) increases from 24.57 dB to 25.23 dB.
• Gains are observed in optics (reflection, light propagation), mechanics (deformation), and local state transitions.
• Training on a real video-based dataset (MIRA400K) underperforms compared to PICA-100K, highlighting the importance of targeted, synthetic supervision.

Evaluation Protocol Validity

PICAEval demonstrates high correlation with human preference rankings, outperforming baseline VLM-as-a-judge protocols that lack region-level grounding. The QA-based approach is more sensitive to nuanced physical violations and less prone to hallucination.

Limitations and Future Directions

• The PICA-100K dataset, while effective, is limited in scale and diversity due to the constraints of the synthetic generation pipeline.
• The current training paradigm relies on supervised finetuning; reinforcement learning or more advanced post-training strategies may yield further gains.
• The framework is restricted to single-image inputs and does not support multi-image or multi-condition conditioning.
• Future work should explore larger-scale, more diverse synthetic data, RL-based optimization, and multi-modal conditioning to further close the gap in physical realism.

Implications and Outlook

PICABench, PICAEval, and PICA-100K collectively establish a new standard for evaluating and improving physical realism in image editing. The results highlight a significant gap between current model capabilities and the requirements for physically plausible image manipulation. The findings suggest that progress in semantic and perceptual quality does not automatically translate to physical consistency, and that targeted supervision and evaluation are necessary to drive advances in this area.

The benchmark and dataset provide a foundation for future research on physics-aware generative models, with implications for applications in visual effects, digital content creation, scientific visualization, and any domain where physical plausibility is critical. As generative models approach world-simulation capabilities, integrating explicit physical reasoning and supervision will be essential for robust, trustworthy image editing systems.

Conclusion

PICABench provides a rigorous, fine-grained framework for diagnosing and advancing physical realism in image editing. The introduction of region-grounded evaluation and synthetic video-based supervision demonstrates measurable improvements but also exposes the persistent challenges in this domain. The work sets a clear agenda for future research: bridging the gap between semantic understanding and physical plausibility in generative visual models.