GauFRe: Gaussian Deformation Fields for Real-time Dynamic Novel View Synthesis (2312.11458v2)

Abstract: We propose a method that achieves state-of-the-art rendering quality and efficiency on monocular dynamic scene reconstruction using deformable 3D Gaussians. Implicit deformable representations commonly model motion with a canonical space and time-dependent backward-warping deformation field. Our method, GauFRe, uses a forward-warping deformation to explicitly model non-rigid transformations of scene geometry. Specifically, we propose a template set of 3D Gaussians residing in a canonical space, and a time-dependent forward-warping deformation field to model dynamic objects. Additionally, we tailor a 3D Gaussian-specific static component supported by an inductive bias-aware initialization approach which allows the deformation field to focus on moving scene regions, improving the rendering of complex real-world motion. The differentiable pipeline is optimized end-to-end with a self-supervised rendering loss. Experiments show our method achieves competitive results and higher efficiency than both previous state-of-the-art NeRF and Gaussian-based methods. For real-world scenes, GauFRe can train in ~20 mins and offer 96 FPS real-time rendering on an RTX 3090 GPU. Project website: https://lynl7130.github.io/gaufre/index.html

• The paper introduces GauFRe, employing a time-dependent MLP to deform canonical Gaussians for dynamic scene reconstruction.
• It separates dynamic and static regions using distinct Gaussian sets, enhancing accuracy and optimizing computational resources.
• Results show that GauFRe attains state-of-the-art quality with faster optimization and real-time rendering for interactive applications.

Introduction

The field of 3D reconstruction from 2D images poses numerous challenges, particularly when reconstructing dynamic scenes from a single, moving camera's footage. Previous methods for addressing this issue rely on a diverse range of techniques, many of which come with their own sets of trade-offs concerning quality, optimization speed, and rendering speed. A new method, GauFRe, leverages the efficiency of Gaussian splatting extended with deformable 3D Gaussians, to accommodate dynamic changes, offering a favorable balance of high-quality reconstruction with real-time rendering capabilities.

Dynamic Scene Reconstruction Methodology

GauFRe differentiates itself by utilizing a multi-layer perceptron (MLP) to define a time-dependent deformation field, transforming a canonical arrangement of Gaussians to represent movement and deformation within a scene. Furthermore, the method acknowledges that natural scenes often contain large static regions. By employing both dynamic and static Gaussians—where static regions are represented by a separate, undeformable set—the MLP can concentrate on representing dynamic elements more accurately. The process involves optimizing the dynamic and static Gaussians with a self-supervised rendering loss, allowing for more efficient computing resource allocation and improved final image quality.

Performance and Validation

When evaluating GauFRe against several baselines on both synthetic and real-world datasets, it achieves quality on par with state-of-the-art alternatives while ensuring faster optimization and real-time rendering. These attributes suggest notable advantages in scenarios requiring rapid deployment or interactive applications. The deformation of Gaussians, coupled with the incorporation of a static Gaussian point cloud, delivers higher quality reconstructions with less computational expense.

Conclusion on GauFRe's Implications

The innovation brought forward by GauFRe marks a significant contribution. Not only does it provide a tool for rapid and high-quality reconstruction of dynamic scenes from monocular video inputs, but it also opens doors for applications in virtual reality, gaming, and video editing where real-world dynamic events need to be recreated and manipulated with high fidelity and efficiency. The dynamic/static separation is a particularly clever feature that allows more focused processing on movement within a scene, highlighting the potential for even more nuanced adaptations and extensions of this method in the future.