Controlling Space and Time with Diffusion Models

Abstract: We present 4DiM, a cascaded diffusion model for 4D novel view synthesis (NVS), supporting generation with arbitrary camera trajectories and timestamps, in natural scenes, conditioned on one or more images. With a novel architecture and sampling procedure, we enable training on a mixture of 3D (with camera pose), 4D (pose+time) and video (time but no pose) data, which greatly improves generalization to unseen images and camera pose trajectories over prior works that focus on limited domains (e.g., object centric). 4DiM is the first-ever NVS method with intuitive metric-scale camera pose control enabled by our novel calibration pipeline for structure-from-motion-posed data. Experiments demonstrate that 4DiM outperforms prior 3D NVS models both in terms of image fidelity and pose alignment, while also enabling the generation of scene dynamics. 4DiM provides a general framework for a variety of tasks including single-image-to-3D, two-image-to-video (interpolation and extrapolation), and pose-conditioned video-to-video translation, which we illustrate qualitatively on a variety of scenes. For an overview see https://4d-diffusion.github.io

• The paper introduces 4DiM, a cascaded diffusion model that advances 4D novel view synthesis with robust spatial and temporal control.
• It employs a multi-modal conditioning approach and calibrates SfM data using monocular depth estimators to effectively leverage heterogeneous training sources.
• Empirical results show that 4DiM outperforms baselines with improved metrics like FID, TSED, and SfM distances, affirming its practical impact.

Controlling Space and Time with Diffusion Models

The paper presents "4DiM," a cascaded diffusion model enabling 4D novel view synthesis (NVS), conditioned on one or more images, camera poses, and timestamps. This work advances the state of NVS by extending the capability of diffusion models to handle temporal dynamics and spatial consistency via multi-modal conditioning.

Key Contributions

1. The 4DiM Model

4DiM is a pixel-based diffusion model conditioned on arbitrary scenes, camera pose, and time. It comprises a base model generating 32 images at $64\times64$, and a super-resolution model up-sampling to $256\times256$. This architecture ensures fidelity and effective temporal-spatial conditioning. Training is achieved using a mixture of data sources including posed/unposed video of diverse scenes.

2. Data Mixture Strategy

One prominent challenge in 3D and 4D generative modeling is the limited availability of high-quality training data. To mitigate this, the authors propose joint training on a heterogeneous data mixture—3D (with camera pose), 4D (pose+time), and pure video (time without pose). This innovative use of mixed data sources optimally leverages the available data, aiding zero-shot application with fine-grained pose and temporal control.

3. Calibration of Pose Data

Another notable contribution is the calibration of SfM posed data using monocular metric depth estimators. This calibration process normalizes poses to metric scales, thereby facilitating more robust and physically interpretable control of the camera during both training and inference.

4. Evaluation Metrics

To accurately benchmark 4DiM's performance, the authors introduced novel metrics: SfM distances and keypoint distances. These metrics address the limitations in existing evaluation schemes. SfM distances measure the alignment of generated poses with ground truth, while keypoint distance evaluates the motion dynamics against reference video data.

Methodology

Architectural Innovations

The 4DiM utilizes a continuous-time diffusion model to learn the joint distribution over multiple views: $$p(\bm{x}_{C+1:N} \, |\, \bm{x}_{1:C},\: \bm{p}_{1:N},\: t_{1:N})$$ where $\bm{x}_{C+1:N}$ are the generated images, $\bm{x}_{1:C}$ are conditioning images, $\bm{p}_{1:N}$ are camera poses, and $t_{1:N}$ are timestamps. The training loss is the L1 distance between the predicted and actual noise, applied in the $v$-parametrization setup to stabilize training.

Conditioning and Sampling Techniques

Conditioning in 4DiM is achieved using masked FiLM layers incorporating per-pixel ray origins, directions, and timestamps. This enables effective training despite incomplete data. Sampling employs multi-guidance schemes allowing different weights for various conditioning variables like image and pose.

Calibrated RealEstate10K

An essential part of 4DiM's robustness is its reliance on a calibrated RealEstate10K dataset. By recalibrating pose scales via monocular depth estimators, the efficiency of metric regularities in-generated data improved notably, addressing practical challenges in specifying camera poses during inference.

Empirical Results

Quantitative Evaluation

4DiM consistently outperforms state-of-the-art models, such as PNVS and MotionCtrl, across multiple dimensions: Fidelity (FID, FDD, FVD), 3D consistency (TSED), and pose alignment (SfM distances). For example, on the RealEstate10K test split, 4DiM achieved a notable FID of 31.96 and a TSED of 0.9935, outperforming the baselines substantially.

Qualitative Analysis

Visual inspection of generated sequences revealed that 4DiM maintains high fidelity and consistency in object appearance and scene geometry. In applications like panorama stitching and space-time trajectory rendering, 4DiM produced seamless outputs, affirming both quality and practical applicability.

Practical and Theoretical Implications

4DiM has broad implications for fields requiring robust NVS, such as augmented reality, autonomous navigation, and synthetic data generation for robotics training. The inclusion of temporal dynamics in synthesis expands application boundaries into tasks like video-to-video translation and realistic simulation environments.

Theoretically, 4DiM provides a novel perspective on the importance of multi-modal conditioning and the benefits of calibration in learning structured scene representations. Future work may explore leveraging larger-scale pre-trained models, exploring finer temporal and spatial resolutions.

Conclusion

The 4DiM model sets a new benchmark in 4D novel view synthesis by addressing both spatial consistency and temporal dynamics, validated through strong empirical results and novel metrics. This paper contributes significantly to the growing body of work in generative models, proposing methods that can be further refined and extended across various domains in AI.