GenWarp: Single Image to Novel Views with Semantic-Preserving Generative Warping

Abstract: Generating novel views from a single image remains a challenging task due to the complexity of 3D scenes and the limited diversity in the existing multi-view datasets to train a model on. Recent research combining large-scale text-to-image (T2I) models with monocular depth estimation (MDE) has shown promise in handling in-the-wild images. In these methods, an input view is geometrically warped to novel views with estimated depth maps, then the warped image is inpainted by T2I models. However, they struggle with noisy depth maps and loss of semantic details when warping an input view to novel viewpoints. In this paper, we propose a novel approach for single-shot novel view synthesis, a semantic-preserving generative warping framework that enables T2I generative models to learn where to warp and where to generate, through augmenting cross-view attention with self-attention. Our approach addresses the limitations of existing methods by conditioning the generative model on source view images and incorporating geometric warping signals. Qualitative and quantitative evaluations demonstrate that our model outperforms existing methods in both in-domain and out-of-domain scenarios. Project page is available at https://GenWarp-NVS.github.io/.

• The paper introduces GenWarp, a framework that integrates warping signals with generative diffusion to preserve semantic details during novel view synthesis.
• It employs a two-stream U-net architecture combining a semantic preserver and a diffusion model using warped coordinate embeddings as geometric priors.
• Experimental results on datasets like RealEstate10K and ScanNet show that GenWarp outperforms baseline methods, improving metrics such as FID and PSNR.

Overview of Semantic-Preserving Generative Warping for Single-Shot Novel View Synthesis

Generating novel views from a single image is a complex task that has seen significant advancements in recent years. The paper "Semantic-Preserving Generative Warping for Single-Shot Novel View Synthesis" introduces a novel framework named GenWarp, designed to address the challenges faced by previous approaches in this domain. GenWarp integrates geometric warping and generative modeling through advanced attention mechanisms to produce high-quality novel views with preserved semantic details.

Introduction

The generation of novel views from a single image is highly relevant for applications such as portrait design, cartoon creation, and movie production. Traditional Text-to-Image (T2I) models like Stable Diffusion exhibit limitations in multi-view generation due to their lack of inherent 3D scene awareness. Recent methods combining T2I models with Monocular Depth Estimation (MDE) offer a promising yet imperfect solution. These methods, which rely on warping input images using depth maps followed by inpainting to fill occluded regions, often struggle with noisy depth predictions and the loss of semantic coherence.

Methodology

GenWarp introduces a more sophisticated approach by explicitly integrating warping signals into the generative process rather than treating it as a separate inpainting task. The core innovation lies in augmenting self-attention mechanisms with cross-view attention, effectively allowing the model to learn where to warp and where to generate content. This integration occurs directly within the attention layers of a diffusion model fine-tuned for this purpose.

Two-Stream Architecture

The proposed architecture includes a semantic preserver network and a diffusion model, both based on the U-net architecture. The semantic preserver encodes the input view into a feature map, while the diffusion model generates the novel view by integrating features from both the input and the novel view. The key novelty is the use of warped coordinate embeddings, which act as geometric priors based on the input image's depth map and desired camera viewpoint. This conditioning facilitates a robust generative process that respects the geometric transformation between views.

Augmented Self-Attention

The model extends the self-attention mechanism by incorporating cross-view attention, enabling the fusion of input view features with the target view features. This hybrid attention mechanism allows the model to balance between generating new content and warping existing features accurately. By concatenating the self-attention map and the cross-view attention map, the model aligns semantic details from the input view with the generated novel view, preserving consistency and reducing artifacts.

Performance Evaluation

The efficacy of GenWarp is validated through extensive experiments on datasets such as RealEstate10K, ScanNet, and in-the-wild images. Qualitative results demonstrate GenWarp's superior capability in generating coherent and contextually consistent novel views, even in challenging scenarios involving large viewpoint changes. Quantitative metrics, including FID and PSNR, further corroborate these findings, showing that GenWarp outperforms baseline methods like GeoGPT and traditional warping-and-inpainting approaches.

Implications and Future Work

GenWarp addresses a crucial limitation in novel view synthesis by effectively combining depth-based warping with advanced generative modeling. This method not only enhances the quality of generated views but also ensures semantic coherence, making it highly applicable to various real-world scenarios. Future research could explore optimizing the attention mechanisms further, integrating additional sensor inputs, or expanding to more complex scene understanding tasks.

In conclusion, GenWarp represents a significant step forward in single-shot novel view synthesis, leveraging sophisticated attention mechanisms to achieve higher fidelity and more semantically accurate generative results. This work lays the groundwork for future innovations in combining geometric transformations with generative modeling, potentially leading to more advanced applications in AI-driven content creation.