Noise2Image: Noise-Enabled Static Scene Recovery for Event Cameras (2404.01298v2)

Abstract: Event cameras, also known as dynamic vision sensors, are an emerging modality for measuring fast dynamics asynchronously. Event cameras capture changes of log-intensity over time as a stream of 'events' and generally cannot measure intensity itself; hence, they are only used for imaging dynamic scenes. However, fluctuations due to random photon arrival inevitably trigger noise events, even for static scenes. While previous efforts have been focused on filtering out these undesirable noise events to improve signal quality, we find that, in the photon-noise regime, these noise events are correlated with the static scene intensity. We analyze the noise event generation and model its relationship to illuminance. Based on this understanding, we propose a method, called Noise2Image, to leverage the illuminance-dependent noise characteristics to recover the static parts of a scene, which are otherwise invisible to event cameras. We experimentally collect a dataset of noise events on static scenes to train and validate Noise2Image. Our results provide a novel approach for capturing static scenes in event cameras, solely from noise events, without additional hardware.

• The paper introduces Noise2Image, a novel method that leverages noise events from event cameras to reconstruct static scene intensity images.
• The paper develops an illuminance-dependent noise model that inverts noise generation dynamics to recover quality images solely from noise events.
• The paper validates the approach using the NE2I dataset, demonstrating that noise can be repurposed as a valuable signal for static scene reconstruction.

Noise2Image: Leveraging Noise Events in Event Cameras for Static Scene Recovery

Introduction to Event Cameras and Noise Events

Event cameras, an innovation in the field of vision sensors, operate distinctly from traditional frame-based cameras by capturing changes in brightness at each pixel asynchronously. These changes, termed as 'events', are recorded with their temporal coordinates and binary polarity, indicating an increase or decrease in brightness. This unique operation mode allows event cameras to excel in high-speed applications by providing data at rates conventional cameras cannot match, albeit with a significant limitation: an inability to capture static scenes due to the absence of brightness changes over time. The noise events, usually considered undesirable and filtered out, are generated even in static scenes due to random fluctuations in photon arrival, offering an innovative avenue for capturing static aspects of a scene.

Modeling the Generation of Noise Events

Contrary to previous efforts that focus on removing noise to improve signal quality, our approach analyzes the generation of noise events and their correlation with static scene intensity. We propose a statistical noise model that elucidates the relationship between noise events and scene illuminance. This model reveals an inversely proportional relationship, where the number of noise events, predominantly triggered due to photon noise in low to moderate brightness settings, decreases with increasing illuminance. This foundational understanding challenges the notion of noise as merely a detrimental aspect to be eliminated, positioning it instead as a valuable source of information for static scene reconstruction.

Noise2Image Method

Building on our noise event generation model, we introduce a method named Noise2Image. This method innovatively exploits the illuminance-dependent characteristics of noise events to reconstruct intensity images of static scenes from noise events alone. By modeling and inverting the process of noise event generation, albeit facing the challenges of its non-monotonicity and the necessity for a learned prior to resolve ambiguities, Noise2Image marks a significant step forward. It enables the capture of static scenes using event cameras without requiring additional hardware, thereby simplifying the setup and reducing both cost and computational requirements.

Experimental Validation and NE2I Dataset

To validate Noise2Image, we collect and analyze a dataset of noise events in static scenes, termed NE2I, through both experimental acquisition and synthetic noise generation. Our results demonstrate the capability of Noise2Image to robustly recover high-quality intensity images based solely on noise events. The method also proves to be resilient when applied to live scenes, showcasing its practicality beyond controlled environments.

Implications and Future Directions

The exploration of noise events for static scene recovery introduces a paradigm shift in how event cameras can be utilized, extending their application scope to encompass static environments. The theoretical modeling and practical realization of Noise2Image pave the way for future advancements in sensor technology and computational photography. Additionally, the compatibility of Noise2Image with dynamic scene reconstruction methods suggests promising avenues for comprehensive scene recovery, blending motion and stasis seamlessly. Future work may explore refining the noise model, exploring the spatial correlations of noise events, and integrating these insights into existing computational frameworks for enhanced visual sensing.

Conclusion

Noise2Image represents a transformative approach to leveraging noise events in event cameras for static scene recovery. By reconceptualizing noise as an asset rather than a hindrance, we unveil new potentials for event-based vision sensing. This research not only broadens the usability of event cameras but also contributes significantly to the theoretical understanding of noise event generation, offering novel perspectives for future innovations in the field.