High-Speed Dynamic 3D Imaging with Sensor Fusion Splatting

Abstract: Capturing and reconstructing high-speed dynamic 3D scenes has numerous applications in computer graphics, vision, and interdisciplinary fields such as robotics, aerodynamics, and evolutionary biology. However, achieving this using a single imaging modality remains challenging. For instance, traditional RGB cameras suffer from low frame rates, limited exposure times, and narrow baselines. To address this, we propose a novel sensor fusion approach using Gaussian splatting, which combines RGB, depth, and event cameras to capture and reconstruct deforming scenes at high speeds. The key insight of our method lies in leveraging the complementary strengths of these imaging modalities: RGB cameras capture detailed color information, event cameras record rapid scene changes with microsecond resolution, and depth cameras provide 3D scene geometry. To unify the underlying scene representation across these modalities, we represent the scene using deformable 3D Gaussians. To handle rapid scene movements, we jointly optimize the 3D Gaussian parameters and their temporal deformation fields by integrating data from all three sensor modalities. This fusion enables efficient, high-quality imaging of fast and complex scenes, even under challenging conditions such as low light, narrow baselines, or rapid motion. Experiments on synthetic and real datasets captured with our prototype sensor fusion setup demonstrate that our method significantly outperforms state-of-the-art techniques, achieving noticeable improvements in both rendering fidelity and structural accuracy.

• The paper presents a sensor fusion approach that integrates RGB, event, and depth cameras for real-time 3D scene reconstruction.
• It introduces a deformable 3D Gaussian splatting framework that captures rapid motion with high temporal and spatial fidelity.
• Extensive tests on synthetic and real-world datasets show improved PSNR, LPIPS, and temporal consistency over current methods.

High-Speed Dynamic 3D Imaging with Sensor Fusion Splatting

In the field of computer graphics and vision, capturing fast and dynamically evolving 3D scenes presents a profound challenge, underscored by the limitations inherent in conventional imaging modalities such as RGB cameras. The paper authored by Zihao Zou et al. introduces an innovative method for high-speed dynamic 3D scene reconstruction by leveraging sensor fusion. This technique ingeniously integrates RGB, event, and depth cameras, each offering distinct data advantages, to overcome the shortcomings of traditional methods. The authors propose a framework underpinned by Gaussian splatting, effectively blending these imaging modalities to enable high-resolution, temporally-consistent scene reconstructions under challenging conditions.

Methodology

The cornerstone of this research is the utilization of a sensor fusion approach that leverages the strengths of RGB, event, and depth cameras. RGB cameras, while proficient in capturing detailed color information, struggle with high-speed scenes due to low frame rates. Event cameras, providing data with microsecond resolution, are more adept at capturing sudden movements by recording only the changes in brightness at the pixel level. Depth cameras supply critical spatial geometry information necessary for accurate 3D scene reconstruction.

The authors propose a cohesive representation of the scene using deformable 3D Gaussians. This representation is pivotal in unifying the data across the three sensor modalities, which is then processed using a Gaussian splatting framework. This process involves optimizing the parameters of these Gaussians jointly to capture the temporal deformation fields dynamically and accurately. This approach enables detailed, real-time reconstructions that were previously unattainable, especially in conditions characterized by rapid motion or low light.

Experimental Results

Extensive experiments were conducted on both synthetic and real-world datasets to evaluate the performance of the proposed approach. The experimental setup for synthetic scenes used simulators to render scenes with varied characteristics such as lighting and object speed. The results demonstrated that the proposed method outperformed state-of-the-art techniques in metrics like PSNR and LPIPS, highlighting significant improvements in rendering fidelity and structural accuracy.

For real-world experiments, a hardware prototype consisting of high-resolution RGB cameras, event cameras, and depth sensors was established. The resulting reconstructions consistently showed superior visual quality and temporal consistency compared to baseline methods. These demonstrations underscore the practical applicability of the method in capturing high-speed and dynamically complex scenes.

Implications and Future Directions

The implications of this research are manifold. Practically, the ability to reconstruct high-speed dynamic 3D scenes accurately can significantly impact industries such as automotive safety, animation, and virtual reality, where capturing the nuances of rapid motion is critical. Theoretically, this research paves the way for further exploration into sensor fusion and the use of deformable models in dynamic scene understanding.

Future developments could focus on enhancing the scalability of the approach for even more complex scenes and exploring real-time applications in diverse environmental conditions. Additionally, further integration of machine learning frameworks with sensor fusion techniques could yield more sophisticated models capable of understanding and predicting dynamic scene changes.

In summary, this paper makes a substantial contribution to the field of dynamic 3D scene reconstruction by presenting a method that effectively harnesses the complementary strengths of multiple imaging modalities. The introduction of deformable 3D Gaussians as a shared representation across sensor data marks a significant advancement, offering a promising direction for future research and application in high-speed imaging technologies.