3DGStream: On-the-Fly Training of 3D Gaussians for Efficient Streaming of Photo-Realistic Free-Viewpoint Videos (2403.01444v4)

Abstract: Constructing photo-realistic Free-Viewpoint Videos (FVVs) of dynamic scenes from multi-view videos remains a challenging endeavor. Despite the remarkable advancements achieved by current neural rendering techniques, these methods generally require complete video sequences for offline training and are not capable of real-time rendering. To address these constraints, we introduce 3DGStream, a method designed for efficient FVV streaming of real-world dynamic scenes. Our method achieves fast on-the-fly per-frame reconstruction within 12 seconds and real-time rendering at 200 FPS. Specifically, we utilize 3D Gaussians (3DGs) to represent the scene. Instead of the na\"ive approach of directly optimizing 3DGs per-frame, we employ a compact Neural Transformation Cache (NTC) to model the translations and rotations of 3DGs, markedly reducing the training time and storage required for each FVV frame. Furthermore, we propose an adaptive 3DG addition strategy to handle emerging objects in dynamic scenes. Experiments demonstrate that 3DGStream achieves competitive performance in terms of rendering speed, image quality, training time, and model storage when compared with state-of-the-art methods.

• The paper introduces a two-stage method that trains 3D Gaussians on-the-fly to efficiently stream free-viewpoint videos in dynamic scenes.
• The paper achieves rapid rendering at 200 FPS and completes per-frame reconstruction in 12 seconds while reducing storage overhead.
• The paper’s approach adapts transformations and adds new 3D Gaussians to manage scene changes in real time, enhancing visual fidelity.

Efficient Free-Viewpoint Video Streaming with 3DGStream

Introduction

The advent of photo-realistic Free-Viewpoint Videos (FVVs) has significantly impacted computer vision and graphics, particularly within VR/AR/XR applications. Despite various traditional and neural rendering methods developed to construct FVVs, real-world dynamic scenes pose unique challenges due to their complex geometries and the need for real-time rendering capabilities. This paper introduces 3DGStream, a novel approach utilizing 3D Gaussians for efficient streaming of FVVs, which not only facilitates on-the-fly per-frame reconstruction but also achieves unparalleled rendering speeds.

Background and Related Work

Existing methodologies largely struggle with the heavy computational and time requirements for rendering FVVs, with most solutions requiring complete video sequences for offline training, thus hindering their real-time application. However, recent works have shown promise in employing 3D Gaussians for static scenes, achieving rapid training and high-quality rendering. Building upon this foundation, 3DGStream addresses the challenge of dynamic scenes by initially training 3D Gaussians and iteratively applying transformations and additions to accommodate scene changes, thus enabling efficient and real-time FVV streaming without comprehensive offline training.

Methodology

3DGStream operates through a two-stage process to accommodate dynamic scene alterations efficiently. The first stage involves training a Neural Transformation Cache (NTC) to model the translations and rotations of 3DGs, effectively capturing object movements with minimal storage requirements. The second stage introduces an adaptive strategy for adding 3DGs to the scene, specifically targeting the appearance of new objects. This approach not only reduces the complexity associated with dynamic object incorporation but also significantly optimizes storage and training time by eliminating the need to train from scratch for each frame. By rendering with both transformed and newly added 3DGs, 3DGStream maintains high fidelity in scene representation across frames.

Experiments and Results

Extensive experiments demonstrate 3DGStream's effectiveness over current state-of-the-art methods in rendering speed, image quality, and efficiency. With capabilities to render at 200 FPS and complete per-frame reconstruction within 12 seconds, this method introduces significant improvements in the streaming of FVVs. Additionally, compared to existing techniques, 3DGStream requires substantially less model storage, showcasing its practicality for real-world applications.

Implications and Future Directions

3DGStream's innovative framework for efficient FVV streaming promises wide-ranging implications for virtual reality, telepresence, and interactive media. By mitigating the limitations of offline training and rendering speeds, this method stands at the forefront of advancing real-time, high-quality 3D content creation and streaming. Looking forward, the methodology paves the way for further research in dynamic scene representation and real-time streaming, potentially extending its application beyond FVV to other areas of computer vision and graphics requiring efficient, high-fidelity 3D model reconstruction and rendering.

Conclusion

In summary, 3DGStream advances the construction and streaming of Free-Viewpoint Videos through a novel, efficient approach leveraging 3D Gaussians. Its ability to handle dynamic scene alterations on-the-fly without compromising rendering quality or efficiency marks a significant step forward in real-time neural rendering. As this field continues to evolve, 3DGStream's contributions offer a solid foundation for future explorations into more sophisticated and scalable solutions for photo-realistic FVV streaming and beyond.