AuraFusion360: Augmented Unseen Region Alignment for Reference-based 360° Unbounded Scene Inpainting (2502.05176v3)

Abstract: Three-dimensional scene inpainting is crucial for applications from virtual reality to architectural visualization, yet existing methods struggle with view consistency and geometric accuracy in 360{\deg} unbounded scenes. We present AuraFusion360, a novel reference-based method that enables high-quality object removal and hole filling in 3D scenes represented by Gaussian Splatting. Our approach introduces (1) depth-aware unseen mask generation for accurate occlusion identification, (2) Adaptive Guided Depth Diffusion, a zero-shot method for accurate initial point placement without requiring additional training, and (3) SDEdit-based detail enhancement for multi-view coherence. We also introduce 360-USID, the first comprehensive dataset for 360{\deg} unbounded scene inpainting with ground truth. Extensive experiments demonstrate that AuraFusion360 significantly outperforms existing methods, achieving superior perceptual quality while maintaining geometric accuracy across dramatic viewpoint changes.

• The paper introduces AuraFusion360, a novel pipeline for 360° unbounded scene inpainting that enhances view consistency and geometric accuracy.
• Its methodology includes Adaptive Guided Depth Diffusion (AGDD) and depth-aware unseen mask generation to improve initial point placement and occlusion detection.
• The paper introduces the 360° Unbounded Scenes Inpainting Dataset (360-USID), the first comprehensive benchmark for evaluating 360° scene inpainting methods.

Overview of AuraFusion360: Augmented Unseen Region Alignment for Reference-based 360° Unbounded Scene Inpainting

The paper "AuraFusion360: Augmented Unseen Region Alignment for Reference-based 360° Unbounded Scene Inpainting" contributes to the field of 3D scene inpainting by addressing critical challenges associated with view consistency and geometric accuracy in 360° unbounded scenes. This research presents a novel approach, AuraFusion360, which employs advanced techniques to enhance object removal and hole filling in three-dimensional (3D) scenes that utilize Gaussian Splatting for representation. The methodology introduced in the paper promises significant improvements over existing models by maintaining coherence across different views even with substantial viewpoint changes.

Methodology

This paper focuses on three main components to achieve its objectives:

1. Depth-Aware Unseen Mask Generation:
   • The proposed method introduces a depth-aware mechanism to generate unseen masks. These masks accurately identify occluded regions requiring inpainting by leveraging multi-view geometric information. By pooling data from different perspectives, the tailored masks enable precise occlusion detection, ensuring that the reconstruction process pays special attention to unseen areas requiring reconstruction.
2. Adaptive Guided Depth Diffusion (AGDD):
   • To enhance initial point placement without additional training, the paper incorporates a zero-shot approach using AGDD. This technique enables the alignment of estimated depth with existing depth metrics, addressing challenges related to scale discrepancies in monocular depth estimation. AGDD achieves this by adapting depth diffusion processes for unseen mask regions, optimizing them for accuracy and structural coherence in 360° unbounded scenes.
3. SDEdit-Based Detail Enhancement:
   • The authors utilize SDEdit for enhancing fine details and ensuring coherence across multi-view inpainted scenes. Instead of applying random noise, the paper's method employs DDIM Inversion to preserve structural integrity during denoising. This innovation allows the model to reconstruct missing details while mitigating the risk of generating artifact-laden content, ensuring uniformity in RGB guidance.

Contributions

The paper makes several significant contributions to 3D scene inpainting:

• Introduction of a unified pipeline for 360° unbounded scene inpainting that combines object removal, depth-aware region identification, and multi-view consistency to ensure high-fidelity reconstructions.
• Development of a new dataset, the 360° Unbounded Scenes Inpainting Dataset (360-USID), which serves as the first comprehensive benchmark for evaluating 360° scene inpainting methods. This dataset provides a rich collection of scenes to assess perceptual quality, geometric accuracy, and consistency of inpainting tasks.

Experimental Results

Extensive experiments demonstrate AuraFusion360's capabilities, showcasing its superiority over existing methods such as Gaussian Grouping and reference-based approaches like GScream and Infusion. AuraFusion360 achieves higher perceptual quality and geometric accuracy, with notable improvements in object removal and artifact reduction. Across various scenes, the method produced more coherent inpainted regions, maintaining structural integrity even in complex geometric arrangements.

Implications and Future Work

AuraFusion360 paves the way for significant advancements in virtual reality (VR), augmented reality (AR), and architectural visualization by providing more realistic and geometrically accurate 3D scene reconstructions. The techniques developed, particularly the integration of multi-view information and improved depth alignment strategies, suggest broader applications in fields that rely on immersive and realistic computer-generated content.

Further research could explore the scalability of AuraFusion360's approach for dynamic scenes or extend its capabilities with linguistic inputs for more interactive inpainting, guiding scene reconstructions using semantic cues. The availability of the 360-USID dataset encourages future methodological developments that could be quantitatively evaluated against established benchmarks.

In summary, AuraFusion360 represents a significant stride in the domain of 3D scene inpainting, with robust methodologies that address long-standing challenges in view consistency and geometric precision, setting a new standard for future innovations within the field.