Splatter a Video: Video Gaussian Representation for Versatile Processing (2406.13870v2)

Abstract: Video representation is a long-standing problem that is crucial for various down-stream tasks, such as tracking,depth prediction,segmentation,view synthesis,and editing. However, current methods either struggle to model complex motions due to the absence of 3D structure or rely on implicit 3D representations that are ill-suited for manipulation tasks. To address these challenges, we introduce a novel explicit 3D representation-video Gaussian representation -- that embeds a video into 3D Gaussians. Our proposed representation models video appearance in a 3D canonical space using explicit Gaussians as proxies and associates each Gaussian with 3D motions for video motion. This approach offers a more intrinsic and explicit representation than layered atlas or volumetric pixel matrices. To obtain such a representation, we distill 2D priors, such as optical flow and depth, from foundation models to regularize learning in this ill-posed setting. Extensive applications demonstrate the versatility of our new video representation. It has been proven effective in numerous video processing tasks, including tracking, consistent video depth and feature refinement, motion and appearance editing, and stereoscopic video generation. Project page: https://sunyangtian.github.io/spatter_a_video_web/

• The paper introduces a Video Gaussian Representation that explicitly models video content in 3D using Gaussian proxies and dynamic motion vectors.
• It integrates 2D priors like optical flow and depth with robust 3D motion regularization to achieve superior tracking and depth prediction.
• Experimental results on benchmarks reveal enhanced PSNR in video reconstruction, enabling advanced video editing and synthesis applications.

Video Gaussian Representation for Versatile Processing

The paper "Splatter a Video: Video Gaussian Representation for Versatile Processing" presents a novel approach to video representation, addressing key challenges in video processing tasks such as tracking, depth prediction, and editing. The authors propose a Video Gaussian Representation (VGR), leveraging 3D Gaussians to encode video content intrinsically in a canonical 3D space.

Key Contributions

The paper identifies limitations in current video representation methods, which either lack 3D structural modeling or leverage implicit 3D representations unsuitable for manipulation. To mitigate these, VGR incorporates explicit Gaussians as proxies, associating each with 3D motion vectors.

Methodology

The VGR framework involves:

• 3D Gaussian Representation: Video content is represented using Gaussians with attributes of position, orientation, scale, opacity, and spherical harmonics for appearance modeling. Time-dependent motion attributes enable dynamic representation.
• Camera Coordinate System: To obviate complex camera pose estimation from monocular views, the authors utilize an orthographic camera space, combining camera and object motions in 3D.
• Integration of 2D Priors: Optical flow and depth from foundation models are distilled into this 3D representation. Combined with 3D motion regularization, they ensure an accurate fit to real-world dynamics, supporting robust and coherent video processing.

Experimental Insights

The VGR method's efficacy is showcased across multiple tasks:

• Dense Tracking: The projection of Gaussian dynamics facilitates efficient tracking across frames.
• Depth and Feature Consistency: The unified 3D model offers consistent depth and feature predictions, surpassing traditional frame-by-frame estimations.
• Editing and Synthesis: The representation simplifies geometric and appearance editing, video interpolation, and novel view synthesis.

Quantitative and Qualitative Results

The evaluation, conducted on datasets like DAVIS, reveals superior PSNR values in video reconstruction over methods like OmniMotion and CoDeF. Despite dynamic scene complexities, VGR maintains high fidelity, supporting advanced manipulations.

Implications and Future Directions

The introduction of VGR marks progress toward more flexible and precise video representations, facilitating diverse practical applications from filmmaking to AR/VR. Future enhancements might aim at integrating more complex temporal dynamics and expanding to more extensive datasets. Further exploration could involve reducing reliance on existing priors, enhancing robustness to rapid, non-rigid scene changes.

In summary, the proposed VGR model represents a significant stride toward efficient video processing, offering a robust solution across applications with its explicit 3D modeling and incorporation of 2D priors. The paper contributes to the theoretical and practical advancements in video processing technologies, marking a valuable addition to the field.