MoSca: Dynamic Gaussian Fusion from Casual Videos via 4D Motion Scaffolds (2405.17421v2)

Abstract: We introduce 4D Motion Scaffolds (MoSca), a modern 4D reconstruction system designed to reconstruct and synthesize novel views of dynamic scenes from monocular videos captured casually in the wild. To address such a challenging and ill-posed inverse problem, we leverage prior knowledge from foundational vision models and lift the video data to a novel Motion Scaffold (MoSca) representation, which compactly and smoothly encodes the underlying motions/deformations. The scene geometry and appearance are then disentangled from the deformation field and are encoded by globally fusing the Gaussians anchored onto the MoSca and optimized via Gaussian Splatting. Additionally, camera focal length and poses can be solved using bundle adjustment without the need of any other pose estimation tools. Experiments demonstrate state-of-the-art performance on dynamic rendering benchmarks and its effectiveness on real videos.

• The paper introduces 4D Motion Scaffolds, a novel method that disentangles scene geometry from motion deformations using dynamic Gaussian fusion.
• It leverages pretrained 2D vision models to integrate per-frame depth, pixel trajectories, and epipolar cues for robust scene reconstruction.
• The approach eliminates external pose estimators, achieving superior LPIPS scores on benchmarks like DyCheck and NVIDIA while handling in-the-wild data.

Overview of MoSca: Dynamic Gaussian Fusion from Casual Videos via 4D Motion Scaffolds

Introduction

The paper introduces a novel approach termed 4D Motion Scaffolds (MoSca) for dynamic scene reconstruction and neural rendering from casual monocular videos. The problem addressed is significantly challenging due to the ill-posed nature of reconstructing dynamic scenes with unknown camera poses, especially from in-the-wild data. The authors leverage the advancements in pretrained vision models and propose a unique Motion Scaffold representation to efficiently encode the underlying motions and deformations. This representation aids in disentangling scene geometry and appearance from the deformation field, optimized through Gaussian Splatting.

Method

The methodology is built upon several key insights and innovations:

1. Leveraging 2D Vision Foundation Models:
   • The approach exploits various 2D priors such as per-frame metric depth estimations, long-term 2D pixel trajectories, and epipolar error maps from pretrained models.
   • These priors help disambiguate the partial observations typical in monocular video setups.
2. Dynamic Scene Representation with 4D Motion Scaffolds:
   • The core idea involves representing the motion dynamics using a sparse graph of motion trajectories, which is interpolated to derive a dense SE(3) deformation field.
   • The deformation field enables the transformation and fusion of observations across different timesteps, ensuring a compact, low-rank representation that efficiently handles the complexities of dynamic scenes.
3. Geometric and Photometric Optimization:
   • The initial motion graph is derived from lifting 2D predictions to 3D, followed by geometric optimization guided by physics-inspired constraints such as as-rigid-as-possible (ARAP) deformations.
   • A photometric rendering loss is used to refine the dynamic Gaussian representations, ensuring high-fidelity reconstructions.
4. COLMAP-Free Pose Estimation:
   • The framework eliminates the dependency on external pose estimators like COLMAP by utilizing static background tracks and depth alignments to solve for camera poses and intrinsic parameters. This aspect significantly enhances the robustness of the system for real-world usage.

Results

The empirical evaluation demonstrates the efficacy of MoSca across various benchmarks:

• On the DyCheck dataset, MoSca achieves superior performance, particularly on the LPIPS metric, highlighting its capability to handle significant view changes and occlusions effectively.
• On the NVIDIA dataset, MoSca shows competitive results with state-of-the-art LPIPS scores, reaffirming its robustness even in less challenging scenarios.
• Qualitative results on in-the-wild videos further showcase its practical applicability.

Implications and Future Work

The introduction of MoSca has several theoretical and practical implications:

• Theoretical: The proposed 4D Motion Scaffolds provide a new perspective on dynamic scene representations, offering a structured, interpretable approach to motion modeling.
• Practical: The framework's ability to work directly with casual videos without strict assumptions on camera movements or additional hardware makes it highly accessible for various real-world applications, such as AR/VR content creation and video editing.

Future developments could explore integrating large-scale 2D diffusion models to hallucinate completely occluded areas, pushing the boundaries toward even more comprehensive scene understanding. Enhancing the foundational 2D models' accuracy will also directly benefit the performance of MoSca, making it a continually evolving area of research.

Conclusion

MoSca presents a noteworthy contribution to the field of dynamic scene reconstruction from casual videos, leveraging innovations in 2D vision models, and introducing a structured, physically-inspired motion representation. While it establishes a new benchmark in several aspects, the ongoing research and improvements in foundational models promise even better performance and broader applications in the future.