Towards Interactive Image Inpainting via Sketch Refinement (2306.00407v3)

Abstract: One tough problem of image inpainting is to restore complex structures in the corrupted regions. It motivates interactive image inpainting which leverages additional hints, e.g., sketches, to assist the inpainting process. Sketch is simple and intuitive to end users, but meanwhile has free forms with much randomness. Such randomness may confuse the inpainting models, and incur severe artifacts in completed images. To address this problem, we propose a two-stage image inpainting method termed SketchRefiner. In the first stage, we propose using a cross-correlation loss function to robustly calibrate and refine the user-provided sketches in a coarse-to-fine fashion. In the second stage, we learn to extract informative features from the abstracted sketches in the feature space and modulate the inpainting process. We also propose an algorithm to simulate real sketches automatically and build a test protocol with different applications. Experimental results on public datasets demonstrate that SketchRefiner effectively utilizes sketch information and eliminates the artifacts due to the free-form sketches. Our method consistently outperforms the state-of-the-art ones both qualitatively and quantitatively, meanwhile revealing great potential in real-world applications. Our code and dataset are available.

• The paper presents SketchRefiner, a two-stage framework that refines user sketches using a Registration Module and an Enhancement Module to accurately guide image reconstruction.
• The method employs a Partial Sketch Encoder and Sketch Feature Aggregation to integrate refined sketch features, significantly improving PSNR, SSIM, and FID scores.
• Evaluations on datasets such as ImageNet, CelebA-HQ, and Places confirm enhanced realism and user alignment, positioning SketchRefiner as a state-of-the-art inpainting solution.

Overview of "Towards Interactive Image Inpainting via Robust Sketch Refinement"

The paper "Towards Interactive Image Inpainting via Robust Sketch Refinement" introduces a novel approach to address challenges in interactive image inpainting, particularly focusing on the integration and refinement of user-provided sketches. Traditional image inpainting methods often struggle with restoring complex structures in corrupted image regions due to the inherent randomness and free-form nature of user sketches. This research proposes a two-stage method, aptly named SketchRefiner, to overcome these challenges and enhance the image inpainting process by effectively utilizing user sketches.

Methodology

SketchRefiner is structured into two primary stages: sketch refinement and sketch-modulated image inpainting.

1. Sketch Refinement:
   • The sketch refinement stage is designed to address the misalignment and incoherence typical of user-drawn sketches. This is achieved using a two-part sketch refinement network (SRN).
   • The first part, the Registration Module (RM), uses a cross-correlation loss function to align user sketches with the intended image structures.
   • The second part, the Enhancement Module (EM), refines the structural inconsistencies in the sketches.
   • The combined output of RM and EM is a refined sketch that accurately reflects the user's intentions while tolerating randomness.
2. Sketch-Modulated Image Inpainting:
   • For the image inpainting process, the refined sketches serve as a means to guide image reconstruction.
   • A Partial Sketch Encoder (PSE) extracts multidimensional features from sketches in the latent space, rather than pixel-level, ensuring better integration into the inpainting process.
   • These features are aggregated using Sketch Feature Aggregation (SFA) blocks, which modulate the inpainting network by fusing extracted features with image textures.

Results

The proposed SketchRefiner method was evaluated on several public datasets, including ImageNet, CelebA-HQ, and Places, alongside a newly developed sketch-based test protocol simulating real-world applications. The results indicate that SketchRefiner consistently outperforms existing state-of-the-art methods both qualitatively and quantitatively.

• The refined sketches eliminated artifacts present in previous methods that relied on less robust edge-based techniques.
• SketchRefiner's ability to tolerate free-form input and integrate abstract sketch information clearly demonstrated an improvement in the realism and user-alignability of inpainted images.
• Quantitatively, SketchRefiner yielded higher PSNR and SSIM scores and lower FID values, marking a significant advancement in image inpainting tasks.

Implications and Future Work

The implications of this research are twofold, encompassing both practical and theoretical dimensions. Practically, SketchRefiner enhances interactive image editing tools by providing a more intuitive and reliable means for users to guide complex image inpainting tasks through sketches. Theoretically, this methodology offers insights into addressing the unpredictability of human input within neural network frameworks.

Future research directions are poised to refine the model further, particularly in mitigating over-refinement in sketches with complex structures and exploring the utilization of SketchRefiner in colorized input scenarios. Expanding the model's understanding of diverse sketching styles and integrating more adaptive user interaction feedback mechanisms are also recommended for further exploration.

In conclusion, the paper establishes a robust framework for refining and leveraging user-drawn sketches in interactive image inpainting, marking a step forward in bridging the gap between user intention and AI interpretation in image restoration tasks.