VideoScene: Distilling Video Diffusion Model to Generate 3D Scenes in One Step (2504.01956v2)

Abstract: Recovering 3D scenes from sparse views is a challenging task due to its inherent ill-posed problem. Conventional methods have developed specialized solutions (e.g., geometry regularization or feed-forward deterministic model) to mitigate the issue. However, they still suffer from performance degradation by minimal overlap across input views with insufficient visual information. Fortunately, recent video generative models show promise in addressing this challenge as they are capable of generating video clips with plausible 3D structures. Powered by large pretrained video diffusion models, some pioneering research start to explore the potential of video generative prior and create 3D scenes from sparse views. Despite impressive improvements, they are limited by slow inference time and the lack of 3D constraint, leading to inefficiencies and reconstruction artifacts that do not align with real-world geometry structure. In this paper, we propose VideoScene to distill the video diffusion model to generate 3D scenes in one step, aiming to build an efficient and effective tool to bridge the gap from video to 3D. Specifically, we design a 3D-aware leap flow distillation strategy to leap over time-consuming redundant information and train a dynamic denoising policy network to adaptively determine the optimal leap timestep during inference. Extensive experiments demonstrate that our VideoScene achieves faster and superior 3D scene generation results than previous video diffusion models, highlighting its potential as an efficient tool for future video to 3D applications. Project Page: https://hanyang-21.github.io/VideoScene

• The paper introduces a one-step, 3D-aware leap flow distillation technique that directly maps sparse views to 3D scene representations.
• It employs a dynamic denoising policy network that adaptively selects optimal noise levels to balance reconstruction quality and computational efficiency.
• Experiments demonstrate superior novel-view synthesis and faster inference, making 3D scene generation practical from limited input data.

Overview of VideoScene

VideoScene (2504.01956) presents a methodology for generating 3D scene representations directly from sparse input views by distilling knowledge from large pre-trained video diffusion models (VDMs). The primary goal is to overcome the inherent ill-posedness of sparse-view 3D reconstruction and the significant computational expense associated with iterative sampling in standard diffusion models. Existing approaches leveraging VDMs for 3D tasks often suffer from slow inference times and may produce geometrically inconsistent results due to the lack of explicit 3D constraints during generation. VideoScene aims to mitigate these issues by proposing a one-step generation process that distills the generative capabilities of a VDM while incorporating 3D awareness. This positions VideoScene as an efficient tool for bridging 2D video generation priors with 3D scene understanding and synthesis.

Core Methodology: Distillation and Dynamic Denoising

The central innovation of VideoScene lies in its distillation strategy, termed "3D-aware leap flow distillation." Unlike typical diffusion model distillation techniques that often progressively reduce sampling steps or learn a direct mapping from noise to data, VideoScene's approach is specifically designed for the video-to-3D context.

1. 3D-Aware Leap Flow Distillation: This strategy aims to directly map an initial state (potentially derived from sparse input views and noise) to a final 3D scene representation in a single computational step, effectively "leaping" over the intermediate denoising steps typical of diffusion models. The "3D-aware" aspect implies that the distillation process incorporates constraints or objectives related to geometric consistency across views or alignment with 3D structures. While the abstract doesn't specify the exact mechanism, this could involve integrating multi-view geometric principles, epipolar constraints, or losses based on rendered novel views during the distillation training phase. The objective is to ensure that the distilled model doesn't just generate plausible video frames but generates frames consistent with a single underlying 3D scene structure. This distillation process is trained to predict the outcome of the full diffusion process ($x_0$) directly from a specific noise level ($x_t$) or an initial condition derived from the sparse views.
2. Dynamic Denoising Policy Network: Standard one-step distillation often assumes a fixed starting noise level or timestep ($t$). However, the optimal starting point for distillation might vary depending on the input views or the complexity of the target scene. VideoScene introduces a dynamic denoising policy network trained to adaptively determine the most effective leap timestep during inference. This policy network likely takes features from the input views or intermediate generative states as input and outputs an optimal timestep $t$ from which the leap flow distillation should commence. This adaptive selection allows the model to potentially allocate more "denoising effort" (conceptually, by starting the leap from an earlier, noisier state) for more complex scenes or inputs, while using a faster leap (starting closer to the final state) for simpler cases, optimizing the trade-off between generation quality and computational efficiency. The training of this policy network could involve reinforcement learning or supervised learning based on metrics evaluating the quality of the final 3D reconstruction obtained from different leap timesteps.

The overall process aims to transform the iterative, computationally intensive sampling of a VDM into a single-pass feed-forward network evaluation, significantly accelerating inference while leveraging the strong generative priors learned by the VDM and enforcing 3D consistency.

Implementation and Training Considerations

Implementing VideoScene involves several key steps:

1. Base VDM: A large, pre-trained VDM serves as the "teacher" model. The quality and characteristics (e.g., resolution, frame rate, temporal consistency) of this base model will significantly influence the performance of the distilled VideoScene model.
2. Distillation Training Data: Training requires pairs of initial states (derived from sparse views and noise/timesteps) and target 3D scene representations. Generating these targets likely involves running the full inference process of the teacher VDM, potentially guided by the sparse input views, and potentially incorporating 3D reconstruction techniques (like NeRF fitting to the generated video) to obtain the ground truth for the 3D-aware aspect. Alternatively, synthetic datasets with known 3D ground truth could be used.
3. Network Architecture: The VideoScene model itself is likely a neural network architecture (e.g., a U-Net variant adapted for video/3D) trained to perform the one-step generation. The policy network is a separate, smaller network.
4. Loss Functions: The training objective for the distillation would include a loss comparing the output of the one-step model to the target 3D scene representation (e.g., rendering loss on novel views if using implicit representations like NeRF, or direct geometric losses if using explicit representations). An additional loss term related to the 3D-aware constraints would be crucial. The policy network would be trained with a separate objective, likely related to maximizing the quality of the final reconstruction given the chosen timestep.
5. Computational Requirements: Distillation training can still be computationally intensive, requiring significant GPU resources, although potentially less than training the base VDM from scratch. The major benefit is realized during inference, which is reduced to a single forward pass through the distilled model and the policy network, making it orders of magnitude faster than the iterative sampling of the original VDM.

The output 3D representation format is not explicitly stated in the abstract, but given the context of generating scenes from video priors, it is likely an implicit representation like a Neural Radiance Field (NeRF) or a related variant (e.g., Neural Implicit Surfaces, Tri-planes) that can be efficiently rendered from novel viewpoints and captures complex geometry and appearance.

Experimental Validation and Performance

The paper reports extensive experiments demonstrating VideoScene's efficacy. Key findings include:

• Superior Results: VideoScene is claimed to achieve superior 3D scene generation results compared to previous methods relying on VDMs without distillation or specialized 3D constraints. This superiority likely manifests as higher fidelity novel view synthesis (measured by PSNR, SSIM, LPIPS) and better geometric accuracy.
• Faster Inference: The core advantage is the significant speedup achieved through one-step generation. Inference time is substantially reduced compared to the iterative sampling required by the original VDM, making the approach more practical for real-time or interactive applications.
• Robustness: The method is suggested to handle challenging cases with minimal overlap between input views, leveraging the strong generative priors from the VDM to plausibly fill in missing information.
• Effectiveness of Dynamic Policy: Experiments likely validate the benefit of the dynamic denoising policy network compared to using a fixed leap timestep, showing improved quality or efficiency across diverse inputs.

The results position VideoScene as a promising approach that balances the generative power of large VDMs with the efficiency required for practical 3D scene generation applications, particularly from sparse inputs.

Practical Applications and Limitations

VideoScene offers potential benefits for various applications requiring fast 3D scene creation from limited visual data:

• Content Creation: Rapid generation of 3D environments for games, simulations, films, or virtual reality experiences from a few reference images or short video clips.
• Robotics and Autonomous Systems: Fast scene understanding and reconstruction for navigation or interaction tasks based on sparse sensor data.
• AR/VR: Real-time or near-real-time capture and reconstruction of environments.

However, potential limitations should be considered:

• Dependence on Teacher VDM: The quality ceiling is determined by the pre-trained VDM. Biases or artifacts present in the teacher model may be inherited or even amplified during distillation.
• Distillation Artifacts: The one-step approximation might introduce artifacts or reduce detail compared to the full iterative sampling process, especially for highly complex scenes.
• Nature of 3D Constraints: The effectiveness of the "3D-aware" distillation depends heavily on how these constraints are implemented and enforced. Implicitly learned geometry might not always be perfectly accurate or physically plausible.
• Generalization: The model's performance on scene types, lighting conditions, or camera trajectories significantly different from the distillation training data might degrade.
• View Consistency: While improved, ensuring perfect multi-view consistency, especially for dynamic elements if extending beyond static scenes, remains challenging.

Conclusion

VideoScene (2504.01956) introduces a novel distillation technique, "3D-aware leap flow distillation," combined with a dynamic denoising policy network, to transform large pre-trained video diffusion models into efficient one-step 3D scene generators. By explicitly incorporating 3D considerations into the distillation process and adaptively selecting the optimal inference path, the method achieves significant speedups and improved 3D reconstruction quality compared to prior VDM-based approaches, particularly for challenging sparse-view inputs. This work represents a notable step towards making the powerful generative capabilities of VDMs practical for demanding video-to-3D applications.