A Recipe for Generating 3D Worlds From a Single Image (2503.16611v1)

Abstract: We introduce a recipe for generating immersive 3D worlds from a single image by framing the task as an in-context learning problem for 2D inpainting models. This approach requires minimal training and uses existing generative models. Our process involves two steps: generating coherent panoramas using a pre-trained diffusion model and lifting these into 3D with a metric depth estimator. We then fill unobserved regions by conditioning the inpainting model on rendered point clouds, requiring minimal fine-tuning. Tested on both synthetic and real images, our method produces high-quality 3D environments suitable for VR display. By explicitly modeling the 3D structure of the generated environment from the start, our approach consistently outperforms state-of-the-art, video synthesis-based methods along multiple quantitative image quality metrics. Project Page: https://katjaschwarz.github.io/worlds/

• The paper proposes a three-stage pipeline to generate navigable 3D worlds from a single image by leveraging pre-trained 2D diffusion models within a geometrically grounded framework.
• The method involves synthesizing a full 3600 panorama from the input image, lifting it to an initial 3D point cloud, and then using point cloud-conditioned inpainting to generate novel views.
• These generated views are finally used to train a 3D Gaussian Splatting (3DGS) model, achieving state-of-the-art quality for real-time 3D scene rendering compared to prior single-image methods.

Generating Immersive 3D Environments from 2D Inputs

Generating navigable 3D worlds from a single 2D image presents a significant challenge due to the inherent ambiguity and missing information. However, success in this area unlocks substantial potential for applications in virtual reality (VR), augmented reality (AR), game development, and content creation. The paper "A Recipe for Generating 3D Worlds From a Single Image" (2503.16611) proposes a practical pipeline to address this challenge, framing the task as an in-context learning problem for 2D generative models. This approach minimizes the need for extensive training by leveraging powerful pre-trained models, offering a novel path toward creating high-quality, immersive 3D experiences from a single photograph.

1. Context and Related Approaches

The task of generating 3D scenes from limited 2D information is inherently ill-posed. Existing methods often struggle with consistency, quality, or the need for multiple input views.

• Video Synthesis Methods: Approaches like WonderJourney and DimensionX attempt to generate novel views by leveraging the latent 3D understanding within video models. However, they often produce blurry results or struggle to maintain consistency across different viewpoints. The "Recipe" method contrasts this by explicitly modeling 3D structure early on via panorama lifting and depth estimation, aiming for greater consistency.
• Generative Models (GANs, Diffusion): While capable of generating 3D assets or scenes, purely generative approaches often struggle with creating large-scale, coherent, and detailed environments, especially when conditioned on a single input image. The "Recipe" leverages pre-trained 2D diffusion models for specific tasks (inpainting) within a structured 3D pipeline, rather than relying on end-to-end 3D generation.
• Traditional 3D Reconstruction (SfM, MVS): These techniques produce accurate models but require multiple input images or video, making them unsuitable for single-image input scenarios. They also struggle with textureless surfaces or occlusions. The "Recipe" overcomes the single-image limitation by synthesizing a full panorama first.

The proposed "Recipe" combines elements from generative modeling and 3D reconstruction, leveraging pre-trained 2D models for image synthesis within a geometrically grounded framework. It aims to achieve higher quality and consistency than video-based methods while requiring only a single input image.

2. The Generation Pipeline: A Three-Stage Recipe

The core contribution is a structured pipeline that decomposes the problem into three main stages:

Stage 1: 2D Panorama Synthesis

The goal is to generate a coherent 360° panorama from the single input image. This uses a pre-trained text-to-image diffusion model adapted for inpainting (using ControlNet), treating panorama creation as a zero-shot, in-context learning task.

1. Initialization: The input image is projected into an equirectangular layout. Its field-of-view is estimated (using Dust3R) to determine its placement within the 360° view.
2. Anchored Heuristic: To ensure global consistency, the input image is duplicated 180 degrees opposite its original position, serving as an "anchor".
3. VLM-Guided Prompting: A vision-LLM (Llama 3.2 Vision) generates distinct prompts for the overall scene, the sky/ceiling, and the ground/floor. This directional prompting guides the inpainting process.
4. Progressive Inpainting: The sky and ground regions are inpainted first using their specific prompts. The anchor image is then removed, and the remaining side views are progressively inpainted by rendering overlapping perspective views and feeding them to the ControlNet-inpainting model.
5. Refinement: The complete panorama undergoes a partial denoising process (last 30% of timesteps) using a standard text-to-image model, and the result is blended with the initial panorama for improved quality.

Stage 2: Lifting to 3D and Point Cloud-Conditioned Inpainting

This stage converts the 2D panorama into an initial 3D point cloud and fills in missing areas caused by occlusions.

1. Initial 3D Lifting: Multiple overlapping views are rendered from the panorama. A metric depth estimator is used to predict depth for these views. While Metric3Dv2 provides metric depth, the authors found MoGE (Mixture of Geometry Experts) more robust. MoGE's affine-invariant depth maps are scaled to approximate metric scale by aligning depth quantiles (20th and 80th) with Metric3Dv2's predictions, along with a minimum ground distance check. These depth maps are fused into an initial 3D point cloud.
2. Identifying Occlusions: Rendering this initial point cloud from novel viewpoints reveals gaps corresponding to occluded regions in the original panorama views.
3. Fine-tuning the Inpainter: Standard inpainters struggle with the fragmented masks produced by rendering point clouds. Therefore, the T2I diffusion + ControlNet inpainting model is minimally fine-tuned (approx. 5k iterations) specifically for this task.
4. Forward-Backward Warping: Training data for fine-tuning is generated using a forward-backward warping strategy. Points from an input view are warped to a novel view (using estimated depth) and then warped back. The diffusion loss is computed on the original view, masked by regions that successfully completed the round trip. This teaches the model to inpaint occluded regions conditioned on the visible 3D geometry rendered from that viewpoint.
5. Inpainting Novel Views: Cameras are placed around the scene (e.g., on a 2m cube), and views are rendered. The fine-tuned model inpaints the occluded regions in these novel views, using the rendered point cloud geometry as conditioning.

Stage 3: 3D Reconstruction with Gaussian Splatting

The final stage uses the generated views (original panorama perspectives + inpainted novel views) to create the final, high-fidelity 3D representation.

1. Representation Choice: 3D Gaussian Splatting (3DGS) is chosen for its ability to render high-quality scenes in real-time. The Splatfacto implementation is used.
2. Initialization and Training: The 3DGS model is initialized using the scaled point cloud from Stage 2, significantly accelerating training (to around 5k steps). Training uses images from the panorama synthesis (excluding anchor areas) and the inpainted regions from the novel views generated in Stage 2.
3. Trainable Image Distortion: To compensate for minor inconsistencies between the synthesized views, a small MLP-based trainable image distortion model is added to the 3DGS training. This learns per-image 2D warping fields, applied during rendering, resulting in sharper details and mitigating artifacts from view inconsistencies.

3. Implementation Details and Evaluation

The effectiveness of the pipeline was validated through quantitative and qualitative experiments on synthetic and real images.

• Setup: Pre-trained diffusion models (with ControlNet for inpainting), Llama 3.2 Vision for prompting, MoGE (scaled using Metric3Dv2) for depth, and Splatfacto for 3DGS were key components.
• Metrics: Panorama quality was assessed using BRISQUE, NIQE (lower is better perceptual quality), Q-Align (consistency), and CLIP-I (input image alignment). Inpainting adherence to geometry was measured with PSNR. Novel view rendering quality from the final 3DGS used BRISQUE, NIQE, and Q-Align.
• Quantitative Findings:
  • The "Anchored" panorama synthesis with VLM prompts significantly outperformed prior methods (MVDiffusion, Diffusion360) on all panorama quality metrics (Table 1 in the paper).
  • The forward-backward warping strategy for fine-tuning the inpainter yielded substantially higher PSNR scores compared to simpler strategies, indicating better geometric consistency (Table 2).
  • Novel views rendered from the final 3DGS model achieved superior image quality scores (BRISQUE, NIQE, Q-Align) compared to state-of-the-art video-based methods like WonderJourney and DimensionX (Table 3).
• Qualitative Findings: Visual comparisons (Figure 2 in the paper) showed the proposed method generating sharper, more consistent, and less artifact-prone 3D scenes compared to video-based competitors.
• Ablation Studies: These confirmed the benefits of key components:
  • The "Anchored" heuristic produced more coherent panoramas than alternative heuristics.
  • Forward-backward warping was crucial for effective point cloud-conditioned inpainting.
  • The trainable distortion model noticeably improved the sharpness and detail of the final 3DGS rendering (Figure 4).

4. Practical Implications, Limitations, and Future Directions

This "Recipe" offers a significant step towards accessible 3D content creation, particularly for VR/AR.

• Strengths:
  • Leverages powerful pre-trained 2D models, minimizing training needs.
  • Explicitly models 3D geometry early via panoramas and depth estimation, leading to better consistency.
  • The component-based approach (panorama, lifting/inpainting, reconstruction) allows for modular improvements.
  • Achieves state-of-the-art results for single-image 3D world generation compared to video-based methods.
  • Uses 3DGS for high-fidelity, real-time capable output.
• Limitations:
  • Performance may degrade for highly complex scenes with intricate geometry or severe occlusions.
  • The pipeline involves multiple stages (diffusion models, depth estimation, 3DGS training), which can be computationally intensive.
  • Results depend on the capabilities and potential biases of the underlying pre-trained models (diffusion, VLM, depth estimator).
• Future Work:
  • Improving robustness for more complex and diverse scene types.
  • Reducing computational requirements through model optimization or alternative representations.
  • Extending the framework to handle dynamic scenes (moving objects, changing lighting).
  • Enhancing texture and material realism.
  • Integrating user interaction for controllable or personalized 3D world generation.

In summary, the paper presents a practical and effective pipeline for generating immersive 3D worlds from single images. By cleverly combining pre-trained generative models with explicit 3D reasoning and reconstruction techniques like 3DGS, it achieves high-quality results and offers a promising direction for democratizing 3D content creation.