IncEventGS: Pose-Free Gaussian Splatting from a Single Event Camera (2410.08107v4)

Abstract: Implicit neural representation and explicit 3D Gaussian Splatting (3D-GS) for novel view synthesis have achieved remarkable progress with frame-based camera (e.g. RGB and RGB-D cameras) recently. Compared to frame-based camera, a novel type of bio-inspired visual sensor, i.e. event camera, has demonstrated advantages in high temporal resolution, high dynamic range, low power consumption and low latency. Due to its unique asynchronous and irregular data capturing process, limited work has been proposed to apply neural representation or 3D Gaussian splatting for an event camera. In this work, we present IncEventGS, an incremental 3D Gaussian Splatting reconstruction algorithm with a single event camera. To recover the 3D scene representation incrementally, we exploit the tracking and mapping paradigm of conventional SLAM pipelines for IncEventGS. Given the incoming event stream, the tracker firstly estimates an initial camera motion based on prior reconstructed 3D-GS scene representation. The mapper then jointly refines both the 3D scene representation and camera motion based on the previously estimated motion trajectory from the tracker. The experimental results demonstrate that IncEventGS delivers superior performance compared to prior NeRF-based methods and other related baselines, even we do not have the ground-truth camera poses. Furthermore, our method can also deliver better performance compared to state-of-the-art event visual odometry methods in terms of camera motion estimation. Code is publicly available at: https://github.com/wu-cvgl/IncEventGS.

• The paper introduces a pose-free incremental 3D reconstruction method using Gaussian splatting with event camera data, eliminating dependence on ground-truth poses.
• It employs a SLAM-like framework that processes event streams in chunks and continuously interpolates camera motion in SE(3) space to refine 3D Gaussian scene representations.
• Experimental results demonstrate enhanced novel view synthesis and improved trajectory estimation, outperforming established NeRF-based and odometry methods in dynamic conditions.

Overview of IncEventGS: Pose-Free Gaussian Splatting from a Single Event Camera

The paper "IncEventGS: Pose-Free Gaussian Splatting from a Single Event Camera" introduces IncEventGS, a novel algorithm for incremental 3D reconstruction using a single event camera. This approach diverges from traditional frame-based camera systems, exploiting the advantages of event cameras such as high temporal resolution, dynamic range, and reduced latency. Key to this research is the concept of Gaussian splatting, a method traditionally used with known camera poses. IncEventGS innovatively eliminates the need for ground-truth poses, presenting a robust alternative for scenarios where such data is unavailable.

Technical Approach

IncEventGS leverages Gaussian splatting and implicitly integrates it with event camera data by adopting a continuous trajectory model within a SLAM-like framework. The technique involves:

• Event Stream Processing: Incoming event data is divided into chunks, with an innovative method to treat each as a distinct “image.” The system uses a continuous interpolation of the camera motion trajectory in $\mathfrak{se}3$ space to compute poses.
• 3D Gaussian Representation: The scene is expressed using 3D Gaussian splatting, building upon previous advancements in radiance field representations. This choice enhances image rendering quality and computational efficiency.
• SLAM Paradigm: The tracking and mapping stages mirror conventional SLAM methodologies. During tracking, camera poses are iteratively refined, while mapping focuses on optimizing the 3D Gaussian scene representation and capturing incremental scene changes.

Experimental Results

The paper reports superior performance of IncEventGS across both synthetic and real-world datasets. The algorithm's ability to recover accurate 3D scene representations without ground-truth poses is underscored by comparisons with NeRF-based methods and two-stage systems that integrate external odometry solutions. Key findings include:

• Novel View Synthesis: IncEventGS significantly outperforms baselines, achieving higher PSNR and SSIM values alongside lower LPIPS scores, reflecting superior image quality. The robust handling of challenging lighting conditions demonstrates the effectiveness of event-based approaches over traditional frame-based methods.
• Camera Trajectory Estimation: The new methodology provides better absolute trajectory error (ATE) metrics compared to existing event-based and visual odometry techniques. This highlights the enhanced flexibility and application potential in real-world settings where pose estimation is crucial.

Implications and Future Work

From a practical standpoint, IncEventGS opens new avenues for applications in augmented reality, robotics, and autonomous navigation. By removing dependencies on ground-truth pose data, the method can be deployed in environments with dynamic or unknown lighting conditions where traditional sensors fail.

Theoretically, this work sets a precedent for further exploration into the integration of neural representations with event camera data. Future research directions may focus on optimizing computational efficiency, exploring additional sensor integrations, and expanding the approach to support multiple event sources or collaborative settings.

IncEventGS represents a step towards more adaptive and robust computer vision systems, leveraging the unique characteristics of event cameras. It exemplifies how novel combinations of artificial intelligence and sensor technologies can overcome longstanding challenges in real-time 3D scene reconstruction.