MVDiffusion: Enabling Holistic Multi-view Image Generation with Correspondence-Aware Diffusion (2307.01097v7)

Abstract: This paper introduces MVDiffusion, a simple yet effective method for generating consistent multi-view images from text prompts given pixel-to-pixel correspondences (e.g., perspective crops from a panorama or multi-view images given depth maps and poses). Unlike prior methods that rely on iterative image warping and inpainting, MVDiffusion simultaneously generates all images with a global awareness, effectively addressing the prevalent error accumulation issue. At its core, MVDiffusion processes perspective images in parallel with a pre-trained text-to-image diffusion model, while integrating novel correspondence-aware attention layers to facilitate cross-view interactions. For panorama generation, while only trained with 10k panoramas, MVDiffusion is able to generate high-resolution photorealistic images for arbitrary texts or extrapolate one perspective image to a 360-degree view. For multi-view depth-to-image generation, MVDiffusion demonstrates state-of-the-art performance for texturing a scene mesh.

• The paper introduces a novel correspondence-aware attention mechanism within a latent diffusion framework to enforce multi-view consistency.
• The paper demonstrates state-of-the-art improvements in metrics like FID, IS, CS, and PSNR, indicating superior coherence over traditional methods.
• The paper’s approach offers significant implications for virtual reality and automated scene generation, paving the way for scalable immersive content creation.

Overview of MVDiffusion: Enabling Holistic Multi-view Image Generation with Correspondence-Aware Diffusion

The paper presents an innovative approach, MVDiffusion, aimed at generating consistent multi-view images from text prompts integrated with pixel-to-pixel correspondences. This methodology diverges from prior iterative warping and inpainting techniques by enabling simultaneous image generation via a text-to-image diffusion model, enhanced with correspondence-aware mechanisms. This strategy effectively mitigates the common error accumulation challenge in multi-view text-to-image synthesis.

The framework employs a stable diffusion model, integrating novel correspondence-aware attention (CAA) layers that facilitate cross-view interactions and ensure multi-view consistency. The synthesized images maintain coherence across perspectives, demonstrated through two primary tasks: panorama generation and multi-view depth-to-image generation. Despite being trained on a relatively modest dataset of 10,000 panoramas, the MVDiffusion framework can generate high-resolution and photorealistic panoramic images, as well as extrapolate 360-degree views from single perspective images.

Key Components and Methodology

MVDiffusion leverages the latent diffusion model (LDM) architecture that includes a variational autoencoder (VAE), a time-conditional UNet denoising network, and a condition encoder. The core innovation lies in the correspondence-aware attention (CAA) mechanism, which enforces multi-view consistency through learned pixel-to-pixel correspondences across synthesized image views. The training process freezes the stable diffusion model's weights, optimizing the CAA layers to ensure that the original generalization capabilities remain unaffected.

In the panorama generation task, images are generated and stitched to form a seamless panoramic view. The model demonstrates proficiency in handling arbitrary specification of texts per view for image synthesis, illustrating adaptability across diverse content types, including indoor, outdoor, and stylized scenes, even without explicit training data for each domain.

For the multi-view depth-to-image generation task, MVDiffusion achieves state-of-the-art results, enabling texture mapping onto scene meshes. This capability is crucial for applications in virtual and augmented reality, where 3D representations of environments enhance user experience.

Empirical Evaluation and Results

Through rigorous experimental evaluation, MVDiffusion attains state-of-the-art performance on established tasks. The paper provides robust quantitative metrics, where improvements in Fréchet Inception Distance (FID), Inception Score (IS), and CLIP Score (CS) are evident over baseline methods, highlighting the efficiency and quality of synthesized images. The PSNR-based evaluation further quantifies multi-view consistency, indicating superior coherence in MVDiffusion-generated images compared to existing methodologies.

Implications and Future Directions

The contributions of MVDiffusion extend beyond the immediate successes in the tasks presented. The innovative integration of CAA into stable diffusion processes sets a precedent for future advancements in broader generative models, including video synthesis and object generation with complex geometries. This approach could potentially be adapted and refined for scalability to even larger systems and more intricate, real-world applications.

Moreover, MVDiffusion exemplifies a pivotal step towards practical and versatile domain-specific content creation. The implications for virtual environment construction, automated scene generation for filmmaking, and other immersive experience technologies are substantial.

However, the paper also acknowledges limitations, notably the computational resources required for generation scalability. Addressing these challenges is a crucial aspect of future work, where more efficient resource utilization would enable even broader deployment.

Through meticulous design and empirical validation, MVDiffusion emerges as a compelling advancement in the domain of multi-view image generation, setting a robust foundation for continuing developments in this dynamic arena.