CM-GAN: Image Inpainting with Cascaded Modulation GAN and Object-Aware Training (2203.11947v3)

Abstract: Recent image inpainting methods have made great progress but often struggle to generate plausible image structures when dealing with large holes in complex images. This is partially due to the lack of effective network structures that can capture both the long-range dependency and high-level semantics of an image. We propose cascaded modulation GAN (CM-GAN), a new network design consisting of an encoder with Fourier convolution blocks that extract multi-scale feature representations from the input image with holes and a dual-stream decoder with a novel cascaded global-spatial modulation block at each scale level. In each decoder block, global modulation is first applied to perform coarse and semantic-aware structure synthesis, followed by spatial modulation to further adjust the feature map in a spatially adaptive fashion. In addition, we design an object-aware training scheme to prevent the network from hallucinating new objects inside holes, fulfilling the needs of object removal tasks in real-world scenarios. Extensive experiments are conducted to show that our method significantly outperforms existing methods in both quantitative and qualitative evaluation. Please refer to the project page: \url{https://github.com/htzheng/CM-GAN-Inpainting}.

• The paper introduces CM-GAN, integrating cascaded modulation and object-aware training to enhance the realism and coherence of inpainted images.
• It employs a dual-stream decoder that first synthesizes coarse global structures then refines them with local spatial details for consistent image completion.
• Experimental results on datasets like Places2 show significant improvements over prior methods using metrics such as FID and LPIPS.

Image Inpainting with Cascaded Modulation GAN and Object-Aware Training

The paper, "Image Inpainting with Cascaded Modulation GAN and Object-Aware Training," introduces a novel approach to tackle the consistently challenging problem of image inpainting, which involves the completion of missing regions within images. Building upon the foundation of success realized through generative adversarial networks (GANs) in computer vision, the authors propose the Cascaded Modulation GAN (CM-GAN). This new architecture integrates innovative network design and training schemes aimed at improving the quality of visual results, especially in challenging scenarios such as large missing areas or object distraction removal.

Methodology

CM-GAN's architecture distinguishes itself by using an encoder equipped with Fourier convolution blocks and a dual-stream decoder that features cascaded global-spatial modulation blocks at varying scale levels. The encoder achieves multi-scale feature extraction from input images with missing regions, while the dual-stream decoder comprises global modulation to synthesize coarse structures followed by spatial modulation for refining these structures with local details. This design choice allows for better synthesis of holistic and coherent image structures, addressing the challenge of maintaining global-local consistency which is frequently problematic in large hole inpainting.

In addition, the paper introduces an object-aware training scheme aimed at preventing unwanted hallucination of new objects within inpainted regions, commonly a necessary aspect in tasks focusing on the removal of specific objects from scenes. Using instance-level panoptic segmentation, the proposed training scheme generates realistic masks, mimicking real-world use cases like object removal. The avoidance of training scenarios that could lead to visual artifacts such as inappropriate object-like shapes or color bleeding is central to improving real-world application efficacy.

Furthermore, the methodology introduces a masked $R_1$ regularization, a variant of standard gradient penalty applied in adversarial training, fine-tuned for inpainting tasks. This approach stabilizes training by confining the regularization focus strictly within masked regions, reducing unintended penalties on parts of the image that are already valid.

Experimental Results

The research provides extensive experimental validation, showing that CM-GAN leads to significant improvements over existing methods across multiple metrics—Fréchet Inception Distance (FID), Learned Perceptual Image Patch Similarity (LPIPS), and both paired and unpaired Inception Discriminative Scores (P-IDS and U-IDS), particularly on datasets like Places2. These results underscore the model's capability to synthesize more realistic completions compared to state-of-the-art techniques such as ProFill, LaMa, and CoModGAN.

Implications and Future Directions

The implications of this work are notable in domains where image integrity is critical following object removal or restoration from damage. The proposed methods for preserving context and preventing spurious artifact generation are integral to applications in fields such as photo editing and enhancement, content creation, and digitization of printed materials.

Theoretically, the model paves the pathway for enhancing GANs with more sophisticated modulation techniques, particularly the cascade of global and spatial modulation mechanism which may inspire further research into combining different forms of feature modulations to resolve the intricacies of image structure completion.

Moving forward, there are opportunities to explore the integration of CM-GAN with emerging neural architectures like transformers, potentially affording even richer feature representations and more accurate inpainting capabilities. Specific domain-centric models, such as for medical imaging or high-detail architectural restoration, could benefit from specialized adaptations of this technology.

In summary, this paper presents a significant step towards optimizing and refining image inpainting, laying groundwork for further exploration in both computer vision research and practical application.