Fast View Synthesis of Casual Videos with Soup-of-Planes (2312.02135v2)

Abstract: Novel view synthesis from an in-the-wild video is difficult due to challenges like scene dynamics and lack of parallax. While existing methods have shown promising results with implicit neural radiance fields, they are slow to train and render. This paper revisits explicit video representations to synthesize high-quality novel views from a monocular video efficiently. We treat static and dynamic video content separately. Specifically, we build a global static scene model using an extended plane-based scene representation to synthesize temporally coherent novel video. Our plane-based scene representation is augmented with spherical harmonics and displacement maps to capture view-dependent effects and model non-planar complex surface geometry. We opt to represent the dynamic content as per-frame point clouds for efficiency. While such representations are inconsistency-prone, minor temporal inconsistencies are perceptually masked due to motion. We develop a method to quickly estimate such a hybrid video representation and render novel views in real time. Our experiments show that our method can render high-quality novel views from an in-the-wild video with comparable quality to state-of-the-art methods while being 100x faster in training and enabling real-time rendering.

• The paper introduces a hybrid explicit representation that separates static and dynamic scene content to achieve over 100x acceleration.
• It employs extended plane-based models with spherical harmonics and displacement maps to render complex, view-dependent effects.
• By integrating per-frame point clouds for dynamic content, the method delivers state-of-the-art quality on benchmarks like NVIDIA and DAVIS.

Fast View Synthesis for Casual Videos

The paper "Fast View Synthesis of Casual Videos" presents a novel method for efficiently generating high-quality novel views from monocular video, addressing the challenges of scene dynamics and lack of parallax that complicate such tasks. At the core, this research revisits explicit video representations to overcome the limitations of Neural Radiance Fields (NeRFs), which, although effective, notoriously require considerable time to train and render.

Methodological Overview

The proposed framework is based on a hybrid explicit representation that separately handles static and dynamic content. The static content is modeled using an extended plane-based scene representation, employing a set of flexible 3D oriented planes. This representation is enriched with spherical harmonics and displacement maps to enhance view-dependent effects and capture intricate surface geometries beyond simple planar approximations.

For dynamic video content, per-frame point clouds are employed, capitalizing on the inherent motion in videos to perceptually mask temporal inconsistencies. This distinction allows the system to achieve real-time rendering speeds and substantially reduces training times — achieving a claimed 100x speed improvement over traditional NeRF-based methods. The methodology involves a per-video optimization strategy that converges rapidly, within approximately 15 minutes on a GPU.

A significant merit of this approach lies in its capacity to deliver this efficiency without compromising quality, producing results comparable to current state-of-the-art methods on standard datasets like the NVIDIA and DAVIS datasets. These datasets pose varying degrees of challenges, from controlled environments to more real-world, complex scenes.

Numerical Results and Claims

In quantitative assessments, the proposed method closely matches the rendering quality of NeRF-based approaches, particularly highlighted through superior LPIPS scores which suggest perceptually high-fidelity outputs. Notably, this method leverages a much shorter training pipeline, asserting over a 100-fold acceleration comparatively.

Practical and Theoretical Implications

On a practical front, this technique shows promise for applications requiring rapid deployment and synthesis, where training time and computational efficiency are critical. This could include content creation for virtual reality environments, video editing, and interactive media applications.

Theoretically, the research opens discussions on the balance between implicit and explicit representations for video synthesis tasks. While neural methods like NeRFs have revolutionized view synthesis, this work suggests that revisiting explicit representations with carefully engineered enhancements can yield equal, if not superior, results concurrently with efficiency gains.

Future Directions

Several avenues for future exploration arise from this research. Extending these methods to accommodate more complex dynamic scenes with higher degrees of motion and occlusions could prove beneficial. Additionally, the integration of more advanced scene understanding techniques to improve initial depth and pose estimations could further enhance the consistency and accuracy of the synthesized views.

The paper positions itself as a meaningful exploration into efficient video synthesis, advancing the intersection of traditional graphics representations and modern computational techniques in a manner that is both innovative and practical for real-world applications.