ComboVerse: Compositional 3D Assets Creation Using Spatially-Aware Diffusion Guidance (2403.12409v1)

Abstract: Generating high-quality 3D assets from a given image is highly desirable in various applications such as AR/VR. Recent advances in single-image 3D generation explore feed-forward models that learn to infer the 3D model of an object without optimization. Though promising results have been achieved in single object generation, these methods often struggle to model complex 3D assets that inherently contain multiple objects. In this work, we present ComboVerse, a 3D generation framework that produces high-quality 3D assets with complex compositions by learning to combine multiple models. 1) We first perform an in-depth analysis of this ``multi-object gap'' from both model and data perspectives. 2) Next, with reconstructed 3D models of different objects, we seek to adjust their sizes, rotation angles, and locations to create a 3D asset that matches the given image. 3) To automate this process, we apply spatially-aware score distillation sampling (SSDS) from pretrained diffusion models to guide the positioning of objects. Our proposed framework emphasizes spatial alignment of objects, compared with standard score distillation sampling, and thus achieves more accurate results. Extensive experiments validate ComboVerse achieves clear improvements over existing methods in generating compositional 3D assets.

• The paper introduces spatially-aware score distillation sampling (SSDS) to enhance spatial alignment in 3D asset creation.
• It decomposes scenes into individual objects for accurate single-object reconstruction and subsequent multi-object combination.
• Empirical evaluations using metrics like CLIP-Score and GPT-3DScore demonstrate substantial improvements in spatial coherence and positional accuracy.

Exploring ComboVerse: Advance in Compositional 3D Assets Creation through Spatially-Aware Diffusion Guidance

Introduction to ComboVerse

Recent developments in 3D content creation from 2D imageries have opened new avenues for application in AR/VR, gaming, and beyond. The inception of ComboVerse pivots around the challenge of generating 3D assets from single images, especially those containing multiple objects which represent a more substantial complexity due to their intricate compositions. This work meticulously discusses the identified "multi-object gap" in current models and introduces a sophisticated framework that leverages spatially-aware score distillation sampling (SSDS) to guide the amalgamation of objects, achieving notable improvements in the generation of compositional 3D assets.

Unpacking the "Multi-Object Gap"

The core observation that underpins ComboVerse is the identified deficiency in existing 3D generative models when dealing with scenarios baring more than a single object. A thorough analysis of this gap helped in understanding the shortcoming from two fronts: model and data.

• Model Perspective: Current feed-forward models are designed with a bias toward single-object generation, faltering when introduced to composite scenes.
• Data Perspective: The preponderance of single-object datasets like Objaverse means that models lack the necessary exposure to complex multi-object scenarios, leading to suboptimal performance when confronted with such tasks.

ComboVerse navigates these challenges by advocating for a compositional approach, reminiscent of the methodologies adopted by skilled human artists. This involves generating individual objects in isolation and then accurately combining them in accordance with their spatial relationships within the scene.

Spatially-Aware Score Distillation Sampling: A Novel Approach

The standout contribution of ComboVerse is its application of spatially-aware score distillation sampling (SSDS). This technique advances beyond the standard score distillation sampling by prioritizing spatial alignment and relationship among objects for their accurate arrangement. The novel SSDS loss introduced, focuses on enhancing objects' spatial context by reweighting attention maps, enabling a more refined guidance mechanism for positioning objects. Experiments validate the superiority of this method in attaining spatial congruity of objects, thus generating more realistic and spatially coherent 3D assets.

ComboVerse in Action

The workflow of ComboVerse is divided into two pivotal stages: single-object reconstruction and multi-object combination.

1. Single-Object Reconstruction: Decompartmentalizes the scene into individual objects for independent generation.
2. Multi-Object Combination: Leverages the pre-trained models along with the proposed SSDS to guide the spatial arrangement of the independently generated objects into a cohesive compositional 3D asset.

Empirical Evaluation

The benchmark for evaluating ComboVerse comprised 100 complex scenes, showing a broad range of scenarios. A comparative analysis with existing state-of-the-art techniques evidenced substantial improvements not just in object generation but crucially in their spatial combination. Through qualitative and quantitative assessments, including the use of novel metrics like CLIP-Score and GPT-3DScore, ComboVerse demonstrated its capacity to significantly outperform benchmarks, particularly in terms of handling occlusions, positional accuracy, and overall compositional integrity.

Conclusion and Future Directions

ComboVerse marks an advance in the field of generative AI and 3D content creation by addressing the nuanced challenge of generating compositional 3D assets from single images. By introducing a framework that incorporates a methodical analysis of the "multi-object gap" and employing spatially-aware score distillation sampling, this work sets a new benchmark for the generation of complex 3D scenes.

The implications of this research extend beyond academic interest, harboring the potential to revolutionize content creation across AR/VR, gaming, and digital entertainment. Looking forward, the methodologies and insights gleaned from ComboVerse could fuel further explorations into more effective 3D generation techniques, especially those capable of tackling intricate scenes comprising numerous objects with varying spatial and compositional requirements.