Effective coupling of visual understanding and image generation for physical realism

Establish methods to effectively couple visual understanding with generative image editing within unified multimodal vision–language architectures so that improved scene understanding translates into physically realistic edits, addressing the observed gap where enhanced understanding alone does not yield better physical realism in instruction-based image editing.

Background

The paper benchmarks unified multimodal models against dedicated image-editing systems and finds that stronger visual understanding does not automatically produce physically realistic edits. Unified architectures underperform on PICABench’s physics-aware criteria, indicating a disconnect between understanding and generation.

The authors highlight that explicit, physics-informed prompts help but do not fully bridge the gap, pointing to the need for architectural or training advances that tightly integrate understanding outputs with generative mechanisms to enforce physical consistency across optics, mechanics, and state transitions.