MoDec-GS: Global-to-Local Motion Decomposition and Temporal Interval Adjustment for Compact Dynamic 3D Gaussian Splatting (2501.03714v3)

Abstract: 3D Gaussian Splatting (3DGS) has made significant strides in scene representation and neural rendering, with intense efforts focused on adapting it for dynamic scenes. Despite delivering remarkable rendering quality and speed, existing methods struggle with storage demands and representing complex real-world motions. To tackle these issues, we propose MoDecGS, a memory-efficient Gaussian splatting framework designed for reconstructing novel views in challenging scenarios with complex motions. We introduce GlobaltoLocal Motion Decomposition (GLMD) to effectively capture dynamic motions in a coarsetofine manner. This approach leverages Global Canonical Scaffolds (Global CS) and Local Canonical Scaffolds (Local CS), extending static Scaffold representation to dynamic video reconstruction. For Global CS, we propose Global Anchor Deformation (GAD) to efficiently represent global dynamics along complex motions, by directly deforming the implicit Scaffold attributes which are anchor position, offset, and local context features. Next, we finely adjust local motions via the Local Gaussian Deformation (LGD) of Local CS explicitly. Additionally, we introduce Temporal Interval Adjustment (TIA) to automatically control the temporal coverage of each Local CS during training, allowing MoDecGS to find optimal interval assignments based on the specified number of temporal segments. Extensive evaluations demonstrate that MoDecGS achieves an average 70% reduction in model size over stateoftheart methods for dynamic 3D Gaussians from realworld dynamic videos while maintaining or even improving rendering quality.

• The paper presents a two-stage global-to-local motion decomposition that separates large-scale and fine-grained motions to enhance storage efficiency and rendering quality.
• The methodology uses an anchor-based global deformation and a localized Gaussian adjustment to model dynamic scenes with high precision.
• Temporal interval adjustment dynamically allocates model capacity, achieving up to 70% smaller models without compromising visual fidelity.

Overview of MoDec-GS: A Dynamic 3D Gaussian Splatting Framework

The paper introduces MoDec-GS, a novel framework designed for dynamic 3D Gaussian splatting, which addresses the challenges of modeling complex motions in real-world videos using a compact representation. This framework builds on the strengths of existing 3D Gaussian splatting methods but introduces significant advancements aimed at improving both storage efficiency and rendering quality.

Key Contributions

MoDec-GS's primary innovation lies in its Global-to-Local Motion Decomposition (GLMD) approach, which is a two-stage process that efficiently models dynamic scenes. The first stage, Global Anchor Deformation (GAD), captures large-scale global motions by deforming anchor attributes and positions using a compact global hexplane. This anchor-based technique effectively streamlines global motion representation, reducing complexity and storage demands. Subsequently, the Local Gaussian Deformation (LGD) uses a shared local hexplane to finely adjust smaller-scale, local motions, thereby enhancing the overall rendering quality by targeting residuals left after GAD.

Another noteworthy contribution is the Temporal Interval Adjustment (TIA), which dynamically adjusts temporal segments during training to better match the scene's motion characteristics. This innovative approach enables MoDec-GS to more effectively allocate its limited model capacity, leading to superior visual fidelity even with intricate motion patterns.

Methodological Significance

MoDec-GS employs an anchor-based representation to extend the canonical 3D Gaussian splatting into dynamic scenes, a technique known for its compactness. This approach not only reduces model size but also enhances rendering quality by efficiently handling both global and local motion types through the GLMD process. The integration of TIA further optimizes this process by allowing the temporal intervals to adapt during training, potentially minimizing model complexity while maximizing performance.

This framework's ability to significantly reduce model size—up to 70% compared to other state-of-the-art methods—without sacrificing rendering quality is a notable achievement, highlighting its potential utility in applications where storage and computational resources are limited.

Implications and Future Developments

The implications of MoDec-GS are particularly relevant for fields reliant on real-time rendering and dynamic scene reconstruction, such as virtual reality (VR), augmented reality (AR), and immersive media. The framework's compact design facilitates deployment on consumer-grade hardware, broadening the applicability of high-quality 3D rendering to more ubiquitous devices.

Future research could explore the integration of more sophisticated deformation models or learning strategies to further refine motion capture and rendering fidelity. Moreover, adapting MoDec-GS for other types of scene representations or input modalities could extend its utility beyond video-based NVS to more diverse real-world applications.

Conclusion

MoDec-GS represents a significant advancement in the domain of dynamic 3D Gaussian splatting. Its innovative approach to motion decomposition and adaptive temporal segmentation offers a compelling solution to the dual challenges of storage efficiency and rendering quality, paving the way for more resource-effective applications in dynamic 3D scene modeling. As such, it holds the promise of advancing the state-of-the-art in real-time dynamic scene rendering.