The generation of 3D scenes from user-specified conditions offers a promising avenue for alleviating the production burden in 3D applications. Previous studies required significant effort to realize the desired scene, owing to limited control conditions. We propose a method for controlling and generating 3D scenes under multimodal conditions using partial images, layout information represented in the top view, and text prompts. Combining these conditions to generate a 3D scene involves the following significant difficulties: (1) the creation of large datasets, (2) reflection on the interaction of multimodal conditions, and (3) domain dependence of the layout conditions. We decompose the process of 3D scene generation into 2D image generation from the given conditions and 3D scene generation from 2D images. 2D image generation is achieved by fine-tuning a pretrained text-to-image model with a small artificial dataset of partial images and layouts, and 3D scene generation is achieved by layout-conditioned depth estimation and neural radiance fields (NeRF), thereby avoiding the creation of large datasets. The use of a common representation of spatial information using 360-degree images allows for the consideration of multimodal condition interactions and reduces the domain dependence of the layout control. The experimental results qualitatively and quantitatively demonstrated that the proposed method can generate 3D scenes in diverse domains, from indoor to outdoor, according to multimodal conditions.
MaGRITTe is a novel method for generating 3D scenes from combinations of input conditions including partial images, layouts in top view format, and textual descriptions.
The method leverages pretrained text-to-image models and neural radiance fields (NeRF) to efficiently generate diverse and detailed 3D scenes without requiring extensive datasets.
Experiments demonstrate MaGRITTe’s ability to outperform benchmarks in generating realistic 3D scenes that align with multimodal input conditions for both indoor and outdoor setups.
MaGRITTe aims to reduce content creation burdens in 3D environments, offering future research directions in enhancing realism and interactivity for applications in virtual and augmented reality.
The paper introduces MaGRITTe, a novel method for generating and controlling 3D scenes from a combination of input conditions: partial images, layouts represented in top view format, and textual descriptions. This research addresses the challenge of efficiently generating detailed and diverse 3D scenes without requiring extensive datasets. By decomposing the process into generating 2D images from conditions and then constructing 3D scenes from these 2D images, MaGRITTe leverages the strengths of text-to-image models and neural radiance fields to facilitate the generation process. This method provides a significant step towards reducing the burden of content creation in 3D environments by allowing detailed scene control using multiple modalities.
MaGRITTe's approach encompasses converting input conditions to a common representation, generating 360-degree images, and subsequently deriving 3D scenes through depth estimation and neural radiance fields (NeRF). The method details include:
Experiments were conducted to validate the effectiveness of MaGRITTe across various metrics, with datasets created specifically for both indoor and outdoor scenes. These included comparisons with existing methods and evaluations on the impact of condition combinations on the generation process. The results demonstrated MaGRITTe's ability to generate realistic and contextually appropriate 3D scenes, outperforming benchmarks in many aspects, particularly in the reproduction of scenes that align with the given multimodal conditions.
MaGRITTe exemplifies how combining multimodal conditions can enrich 3D scene generation, offering potential benefits across virtual reality, augmented reality, and more. The method presents a pathway to reduce the dependency on extensive datasets without compromising the diversity and richness of generated content. Future research may explore the integration of more dynamic and interactive elements within generated scenes, enhancing the realism and applicability of generated 3D environments in real-world applications.
This study presents MaGRITTe, a method for generating and controlling 3D scenes under multimodal conditions, utilizing a mix of partial images, layout information, and text prompts. By leveraging the strengths of pretrained text-to-image models and NeRF, MaGRITTe demonstrates promising capabilities in generating detailed and contextually relevant 3D scenes from minimal input. This research contributes to the easing of content creation processes in 3D environments, pushing the boundaries of what is achievable with generative models and multimodal inputs.