EventNeRF: Neural Radiance Fields from a Single Colour Event Camera (2206.11896v3)

Abstract: Asynchronously operating event cameras find many applications due to their high dynamic range, vanishingly low motion blur, low latency and low data bandwidth. The field saw remarkable progress during the last few years, and existing event-based 3D reconstruction approaches recover sparse point clouds of the scene. However, such sparsity is a limiting factor in many cases, especially in computer vision and graphics, that has not been addressed satisfactorily so far. Accordingly, this paper proposes the first approach for 3D-consistent, dense and photorealistic novel view synthesis using just a single colour event stream as input. At its core is a neural radiance field trained entirely in a self-supervised manner from events while preserving the original resolution of the colour event channels. Next, our ray sampling strategy is tailored to events and allows for data-efficient training. At test, our method produces results in the RGB space at unprecedented quality. We evaluate our method qualitatively and numerically on several challenging synthetic and real scenes and show that it produces significantly denser and more visually appealing renderings than the existing methods. We also demonstrate robustness in challenging scenarios with fast motion and under low lighting conditions. We release the newly recorded dataset and our source code to facilitate the research field, see https://4dqv.mpi-inf.mpg.de/EventNeRF.

• The paper introduces EventNeRF, the first method to reconstruct photorealistic 3D scenes solely from a single-colour event camera using neural radiance fields.
• It employs a self-supervised training strategy with a custom ray sampling process that efficiently leverages asynchronous event data.
• Experimental results demonstrate that EventNeRF outperforms conventional RGB-based NeRF methods with higher PSNR, SSIM, and LPIPS metrics across diverse scenarios.

EventNeRF: A New Approach for Neural Radiance Fields Using Single Colour Event Cameras

The paper "EventNeRF: Neural Radiance Fields from a Single Colour Event Camera" by Viktor Rudnev et al. introduces a novel methodology to generate photorealistic 3D-consistent scene representations using neural radiance fields (NeRF) from data obtained solely through a single-colour event camera. The motivation behind this research is the inherent advantages of event cameras, such as their high dynamic range, low latency, and absence of motion blur, which make them highly suitable for real-time applications but their potential for dense 3D reconstruction remains largely unexplored.

Key Contributions

1. EventNeRF Framework: This approach proposes the first method to reconstruct dense and photorealistic novel views using solely a monocular event camera input. The method does not rely on additional inputs such as RGB images, which are typically required by existing NeRF systems.
2. Self-Supervised Learning: The paper presents a self-supervised training framework that utilizes the asynchronous nature of event data. This involves a custom ray sampling strategy that is tailored to the unique properties of event streams, enhancing data efficiency during training.
3. Experimental Evaluation: The authors introduce a dataset comprising synthetic and real-world scenes, demonstrating the potential of EventNeRF to manage scenarios involving fast motion and low lighting.

Results and Observations

The research provides compelling numerical and qualitative evidence of EventNeRF's efficacy, particularly when compared against conventional methods combined with NeRF using reconstructed frames (e.g., E2VID + NeRF). EventNeRF consistently delivers higher image quality, as evidenced by metrics such as PSNR, SSIM, and LPIPS, across various scenes with complex lighting and geometries.

Implications for Future Research

The implications of this work are considerable, both practically and theoretically. Practically, it opens up new possibilities for deploying event cameras in applications requiring high-quality 3D models, such as interactive environments and augmented reality. Theoretically, it challenges the traditional dependency on full-frame RGB images for neural radiance fields, potentially reducing the data storage and processing burdens associated with these applications.

Speculations on Future Developments

The research lays groundwork for further exploration into reducing the reliance on camera features traditionally used in computer vision by maximizing asynchronous event data properties. Future developments might focus on improving the data efficiency, extending the scalability of the method for larger and more dynamic scenes, and exploring hybrid systems that integrate event-based data with other modalities to refine the reconstruction processes further. Additionally, the release of the dataset and source code will likely spur continued advancements in event-based vision research.

In conclusion, the paper makes significant strides in the field of event camera vision, applying neural radiance fields to a data modality that is inherently different from conventional video streams, and providing a robust framework for future exploration and technological advances in this field.