Region Normalization for Image Inpainting (1911.10375v2)

Abstract: Feature Normalization (FN) is an important technique to help neural network training, which typically normalizes features across spatial dimensions. Most previous image inpainting methods apply FN in their networks without considering the impact of the corrupted regions of the input image on normalization, e.g. mean and variance shifts. In this work, we show that the mean and variance shifts caused by full-spatial FN limit the image inpainting network training and we propose a spatial region-wise normalization named Region Normalization (RN) to overcome the limitation. RN divides spatial pixels into different regions according to the input mask, and computes the mean and variance in each region for normalization. We develop two kinds of RN for our image inpainting network: (1) Basic RN (RN-B), which normalizes pixels from the corrupted and uncorrupted regions separately based on the original inpainting mask to solve the mean and variance shift problem; (2) Learnable RN (RN-L), which automatically detects potentially corrupted and uncorrupted regions for separate normalization, and performs global affine transformation to enhance their fusion. We apply RN-B in the early layers and RN-L in the latter layers of the network respectively. Experiments show that our method outperforms current state-of-the-art methods quantitatively and qualitatively. We further generalize RN to other inpainting networks and achieve consistent performance improvements. Our code is available at https://github.com/geekyutao/RN.

• The paper introduces Region Normalization (RN), a novel technique that separately normalizes corrupted and uncorrupted regions using mask-guided statistics.
• It details two variants—RN-B for early layers and RN-L for later layers—that significantly enhance reconstruction quality with improved PSNR, SSIM, and l1 loss metrics.
• The study motivates further exploration of region-specific normalization in diverse computer vision tasks to boost training efficiency and model robustness.

An Examination of Region Normalization for Image Inpainting

The paper "Region Normalization for Image Inpainting" addresses a crucial challenge in the training of neural networks for image inpainting by proposing a novel normalization technique named Region Normalization (RN). Image inpainting involves reconstructing corrupted regions of an input image and has various applications in image editing tasks, including object removal and image restoration. Existing methods largely overlook the potential adverse effects of applying full-spatial Feature Normalization (FN) techniques, such as Batch Normalization (BN) and Instance Normalization (IN), to images containing corrupted regions. These traditional normalization techniques can cause mean and variance shifts, undermining model performance in image inpainting tasks.

Key Contributions

The authors introduce a spatial region-wise normalization method, RN, which aims to resolve the issues associated with mean and variance shifts during the normalization phase in neural networks applied to image inpainting. RN operates by dividing spatial pixels into distinct regions based on an input mask and calculating region-specific mean and variance for normalization.

1. Basic Region Normalization (RN-B): Designed for early layers of the inpainting network where input features have significant corrupted areas. RN-B separates and normalizes corrupt and uncorrupted regions independently, based on the inpainting mask.
2. Learnable Region Normalization (RN-L): Applied in later layers of the network where corrupted regions begin to blend. It autonomously detects potentially corrupted areas and performs normalization using a learned region mask, further refining the fusion of corrupted and uncorrupted regions through a global affine transformation.

Numerical Results

Empirical evaluations on Places2 and CelebA datasets demonstrate that networks utilizing RN outperform those using traditional normalization approaches significantly in terms of PSNR, SSIM, and $l_1$ loss metrics. The superiority of the proposed RN is pronounced as the mask area increases, showcasing the robustness of RN in scenarios with extensive corrupted regions.

The paper provides comprehensive quantitative comparisons against state-of-the-art inpainting methods such as Contextual Attention, Partial Convolution, Gated Convolution, and EdgeConnect. Results confirm that RN-equipped networks consistently deliver higher fidelity reconstructions.

Implications and Future Directions

The implications of this paper are twofold. Practically, adopting RN has shown clear enhancements in image inpainting tasks, indicating its potential for broader adoption in applications requiring image editing and restoration. Theoretically, RN stimulates further investigation into context-specific normalization, particularly for tasks involving inputs with heterogeneous spatial characteristics.

Future research could explore the application of RN in other computer vision domains where input data isn't spatially homogenous, such as object detection and classification tasks. Such exploration could lead to improvements in training efficiency and model performance, leveraging RN's adaptability in handling region-specific feature normalizations.

In conclusion, the introduction of RN is a notable advancement in the field of image inpainting, offering a robust solution to previously overlooked normalization challenges. The work paves the way for further developments in tailored normalization practices suited to domain-specific neural network training.