Dynamic Gaussians Mesh: Consistent Mesh Reconstruction from Monocular Videos (2404.12379v2)

Abstract: Modern 3D engines and graphics pipelines require mesh as a memory-efficient representation, which allows efficient rendering, geometry processing, texture editing, and many other downstream operations. However, it is still highly difficult to obtain high-quality mesh in terms of structure and detail from monocular visual observations. The problem becomes even more challenging for dynamic scenes and objects. To this end, we introduce Dynamic Gaussians Mesh (DG-Mesh), a framework to reconstruct a high-fidelity and time-consistent mesh given a single monocular video. Our work leverages the recent advancement in 3D Gaussian Splatting to construct the mesh sequence with temporal consistency from a video. Building on top of this representation, DG-Mesh recovers high-quality meshes from the Gaussian points and can track the mesh vertices over time, which enables applications such as texture editing on dynamic objects. We introduce the Gaussian-Mesh Anchoring, which encourages evenly distributed Gaussians, resulting better mesh reconstruction through mesh-guided densification and pruning on the deformed Gaussians. By applying cycle-consistent deformation between the canonical and the deformed space, we can project the anchored Gaussian back to the canonical space and optimize Gaussians across all time frames. During the evaluation on different datasets, DG-Mesh provides significantly better mesh reconstruction and rendering than baselines. Project page: https://www.liuisabella.com/DG-Mesh/

• The paper introduces DG-Mesh, a framework for high-fidelity mesh reconstruction from monocular videos using advanced Gaussian splatting techniques.
• It employs Gaussian-Mesh Anchoring to uniformly distribute 3D Gaussians, ensuring stable vertex tracking across dynamic scenes.
• DG-Mesh outperforms prior methods by achieving lower Chamfer and Earth Mover's distances alongside higher PSNR in reconstructed meshes.

Dynamic Gaussians Mesh: A Method for Consistent Mesh Reconstruction from Monocular Videos

Introduction to Dynamic Gaussians Mesh (DG-Mesh)

The presented paper introduces a novel framework termed Dynamic Gaussians Mesh (DG-Mesh), designed for reconstructing high-quality, time-consistent meshes from monocular video data. This development is pertinent in the field of computer vision and 3D reconstruction, where deriving detailed and dynamic 3D models from single-camera footage remains a significant challenge.

Core Contributions and Methodology

DG-Mesh leverages advancements in 3D Gaussian Splatting to establish a base for mesh reconstruction that accurately captures the dynamics of moving scenes. The primary contributions of this framework can be summarized as follows:

• High-quality Mesh Reconstruction: The framework is capable of reconstructing meshes with high fidelity, addressing common issues in dynamic scene capture such as topology changes and complex motion patterns.
• Time-consistent Vertex Tracking: By maintaining a consistent mesh topology across frames, DG-Mesh facilitates vertex tracking over time, simplifying tasks such as texture mapping and dynamic simulations in post-processing stages.
• Gaussian-Mesh Anchoring: A novel technique introduced within this framework that ensures even distribution of Gaussian distributions across the mesh surface, thereby enhancing the mesh quality and stability across frames.

Technical Approach

The process involved in the DG-Mesh framework begins with the construction of deformable 3D Gaussians to represent the dynamic scenes. These Gaussians are then transformed across different frames using a deformation module. Subsequently, the mesh is reconstructed using a combination of Poisson solvers and a marching cube algorithm, ensuring that the vertices of these meshes can be consistently tracked and aligned through successive frames.

The innovation of Gaussian-Mesh Anchoring addresses the uneven distribution of Gaussians, a common issue with prior techniques. By anchoring and uniformly distributing Gaussian points on the mesh surface for each frame, the reconstruction performance is noticeably improved, especially in terms of dealing with topology changes and ensuring consistency.

Evaluation and Results

DG-Mesh was rigorously evaluated against various baseline models across multiple datasets that included challenging dynamic scenes like flapping bird wings and walking horses. The results demonstrated superiority in achieving lower Chamfer distances and Earth Mover's distances, as well as improved rendering quality evidenced by higher PSNR values on reconstructed mesh surfaces when compared to other state-of-the-art methods.

Future Outlook and Applications

The introduction of DG-Mesh opens several pathways for future research and application. While the current implementation focuses on foreground object reconstruction, expanding this to handle entire scenes with multiple interacting objects could greatly increase its utility. Moreover, integrating DG-Mesh with real-time video processing tools could revolutionize fields such as virtual reality, animation, and live-event broadcasting by providing a means to generate real-time 3D content from conventional video sources.

Concluding Remarks

In conclusion, Dynamic Gaussians Mesh presents a significant step forward in the reconstruction of dynamic meshes from monocular video feeds. By effectively addressing the challenges related to vertex tracking and mesh quality over time, this framework sets a new standard for future developments in the domain of dynamic 3D reconstruction.