Image Inpainting for Irregular Holes Using Partial Convolutions (1804.07723v2)

Abstract: Existing deep learning based image inpainting methods use a standard convolutional network over the corrupted image, using convolutional filter responses conditioned on both valid pixels as well as the substitute values in the masked holes (typically the mean value). This often leads to artifacts such as color discrepancy and blurriness. Post-processing is usually used to reduce such artifacts, but are expensive and may fail. We propose the use of partial convolutions, where the convolution is masked and renormalized to be conditioned on only valid pixels. We further include a mechanism to automatically generate an updated mask for the next layer as part of the forward pass. Our model outperforms other methods for irregular masks. We show qualitative and quantitative comparisons with other methods to validate our approach.

• The paper introduces a novel partial convolution method that updates only valid pixels, enabling effective inpainting of irregular holes.
• The proposed U-Net based architecture incorporates masked convolutions and automatic mask updates to ensure semantically coherent restorations.
• Experimental results across ImageNet, Places2, and CelebA-HQ demonstrate superior qualitative performance over traditional methods like PatchMatch and other deep learning models.

Image Inpainting for Irregular Holes Using Partial Convolutions

The paper "Image Inpainting for Irregular Holes Using Partial Convolutions" by Guilin Liu et al. explores new methodologies to address image inpainting challenges, particularly when dealing with irregular hole shapes. Traditional deep learning approaches often focus on filling central rectangular holes through convolutional networks but suffer from artifacts and require extensive post-processing. In contrast, the authors propose a new framework based on partial convolutions designed to handle irregularly shaped holes effectively.

Introduction and Background

Image inpainting is a well-documented problem with numerous applications, including content removal and restoration in images. Historically, it has been tackled through both non-learning and learning approaches. Non-learning methods, such as PatchMatch, use image statistics to fill in gaps but lack semantic understanding, often leading to incongruous imagery in the filled regions. Conversely, contemporary deep learning methods offer semantic awareness through convolutional networks but typically initialize hole regions with arbitrary pixel values like the mean of ImageNet, leading to issues such as texture deficits and color discrepancies.

Proposed Methodology

The authors introduce Partial Convolutional Layers (PConv), a novel approach that updates only the valid pixels during convolutions while ignoring undefined regions within the holes. This is achieved through masked and re-normalized convolutions, ensuring outputs are based solely on valid inputs. Significantly, the framework includes an automatic mask update mechanism to propagate valid pixels layer-by-layer, eventually filling the hole with semantically coherent content.

Network Design

The proposed network architecture is a modified U-Net with partial convolutional layers substituted for traditional ones. This not only enhances the semantic coherence of the inpainted regions but also facilitates the use of skip connections—a feature previously ineffective with standard convolutions due to propagation of noise from invalid pixels.

Experimental Setup

To validate their approach, the authors developed an extensive irregular mask dataset derived from occlusion/dis-occlusion mask estimation methods, categorizing the masks based on the hole-to-image area ratios and their proximity to image borders. The network was trained on three datasets: ImageNet, Places2, and CelebA-HQ, utilizing a combination of L1, perceptual, style, and total variation (TV) losses to optimize both low and high-level feature restoration.

Results and Comparisons

The model demonstrated superior qualitative and quantitative performance across various datasets and comparisons. Specifically, it outperformed non-learning approaches like PatchMatch and other deep learning frameworks such as the Iizuka et al. GL model and Yu et al.'s generative inpainting model. Additionally, a user paper corroborated the quantitative metrics, indicating a clear preference for the outputs of the proposed method in both unlimited and limited time scenarios.

Implications and Future Work

The introduction of partial convolutions opens avenues for more advanced inpainting systems that can handle complex mask shapes without catastrophic failure as holes increase in size. Moreover, the method's generalizability was showcased through its application in image super-resolution tasks by redefining low-resolution inputs into partial convolution-compatible formats.

Nevertheless, challenges remain, particularly for highly structured images or extremely large holes. Future research could explore adaptive models that further refine mask updating mechanisms or integrate multi-scale generative models to enhance fine-grained details.

Conclusion

This paper presents a significant step forward in image inpainting, showcasing the robustness and applicability of partial convolutions for irregular hole restoration. The efficacy of the proposed method is evidenced by its strong empirical performance and clear advantages over existing techniques, paving the way for more versatile and reliable image editing tools in computational imaging and related fields.