GS-DiT: Advancing Video Generation with Pseudo 4D Gaussian Fields through Efficient Dense 3D Point Tracking (2501.02690v1)

Abstract: 4D video control is essential in video generation as it enables the use of sophisticated lens techniques, such as multi-camera shooting and dolly zoom, which are currently unsupported by existing methods. Training a video Diffusion Transformer (DiT) directly to control 4D content requires expensive multi-view videos. Inspired by Monocular Dynamic novel View Synthesis (MDVS) that optimizes a 4D representation and renders videos according to different 4D elements, such as camera pose and object motion editing, we bring pseudo 4D Gaussian fields to video generation. Specifically, we propose a novel framework that constructs a pseudo 4D Gaussian field with dense 3D point tracking and renders the Gaussian field for all video frames. Then we finetune a pretrained DiT to generate videos following the guidance of the rendered video, dubbed as GS-DiT. To boost the training of the GS-DiT, we also propose an efficient Dense 3D Point Tracking (D3D-PT) method for the pseudo 4D Gaussian field construction. Our D3D-PT outperforms SpatialTracker, the state-of-the-art sparse 3D point tracking method, in accuracy and accelerates the inference speed by two orders of magnitude. During the inference stage, GS-DiT can generate videos with the same dynamic content while adhering to different camera parameters, addressing a significant limitation of current video generation models. GS-DiT demonstrates strong generalization capabilities and extends the 4D controllability of Gaussian splatting to video generation beyond just camera poses. It supports advanced cinematic effects through the manipulation of the Gaussian field and camera intrinsics, making it a powerful tool for creative video production. Demos are available at https://wkbian.github.io/Projects/GS-DiT/.

Efficient Video Generation Using Pseudo 4D Gaussian Fields

The paper discusses GS-DiT, a novel framework designed to enhance video generation capabilities by integrating pseudo 4D Gaussian fields with video diffusion transformers. The core premise is to enable multi-camera video generation and advanced video effects such as dolly zoom, which require 4D control, without the need for costly multi-view datasets.

The framework introduces a pseudo 4D Gaussian representation to the video generation process. This is achieved through a new method for dense 3D point tracking (D3D-PT) that significantly outperforms existing solutions like SpatialTracker in both accuracy and computational speed. By optimizing the dense 3D point trajectories directly from monocular videos, GS-DiT sidesteps the resource-intensive process of capturing multi-view synchronized videos, traditionally necessary for training video Diffusion Transformers (DiT) in a 4D controlled context.

The D3D-PT method is a key technical contribution, offering improvements of up to two orders of magnitude in inference times while achieving superior accuracy. It leverages temporal and spatial frame information to map and track dense 3D point positions and visibilities over time efficiently. This method provides the basis for constructing the pseudo 4D Gaussian fields inherently used to guide the subsequent video generation process.

Moreover, GS-DiT utilizes a finetuned pre-trained video Diffusion Transformer model to generate videos guided by the constructed pseudo 4D Gaussian fields. Consequently, this allows the model to extrapolate videos with precise control over camera parameters, content perspective, and dynamic scene changes, allowing extensive creative applications. The results indicate that the generated videos maintain high fidelity with the intended effects, which are pivotal for cinematic productions.

Experimental results are robust, showcasing GS-DiT's capacity to produce high-quality synchronized multi-camera shooting effects and other sophisticated techniques without reliance on synthetically generated dataset limitations present in prior methods like GCD. Notably, the rendered videos from GS-DiT align more accurately with the original content’s dynamics, while simultaneously allowing for novel views.

The implications of this research are substantial for both theoretical explorations and practical implementations in video generation. The framework potentially reduces resource dependencies by allowing for effective training on standard monocular video datasets rather than synchronous multi-view data sets, thereby offering scalability. Theoretically, it propels the envelope in 4D video content manipulation, suggesting new trajectories for diffusion-based generative models to incorporate spatial-temporal control mechanisms efficiently.

Future directions may involve further aligning and optimizing these Gaussian fields with real-time dynamic rendering fields or capitalizing on the advancements in dense 3D tracking for other applications such as VR environments or interactive media. This work paves a foundational step in advancing autonomous content generation techniques, with potential ripple effects across entertainment and virtual production domains.