Real-Time Intensity-Image Reconstruction for Event Cameras Using Manifold Regularisation (1607.06283v2)

Abstract: Event cameras or neuromorphic cameras mimic the human perception system as they measure the per-pixel intensity change rather than the actual intensity level. In contrast to traditional cameras, such cameras capture new information about the scene at MHz frequency in the form of sparse events. The high temporal resolution comes at the cost of losing the familiar per-pixel intensity information. In this work we propose a variational model that accurately models the behaviour of event cameras, enabling reconstruction of intensity images with arbitrary frame rate in real-time. Our method is formulated on a per-event-basis, where we explicitly incorporate information about the asynchronous nature of events via an event manifold induced by the relative timestamps of events. In our experiments we verify that solving the variational model on the manifold produces high-quality images without explicitly estimating optical flow.

• The paper introduces a variational energy minimization framework that leverages manifold regularisation for real-time event-based intensity image reconstruction.
• It employs a Kullback-Leibler divergence data term and a variant of total variation on manifolds to robustly model noise while preserving spatial-temporal features.
• Achieving over 500 FPS on standard hardware, the method offers significant advancements for dynamic applications in robotics and autonomous driving.

Real-Time Intensity-Image Reconstruction for Event Cameras

The paper by Reinbacher, Graber, and Pock presents a method for reconstructing intensity images from event cameras, specifically focusing on real-time applications. Event cameras, unlike traditional CMOS digital cameras, do not measure absolute light intensity but rather the changes in intensity at each pixel, thus providing a higher temporal resolution while reducing data transfer requirements. This paper advances the field by introducing a variational model leveraging manifold regularisation, which enables real-time image reconstruction without needing to estimate optical flow explicitly.

Overview

Neuromorphic cameras operate asynchronously, recording only intensity changes when a pixel's intensity differs by a predefined threshold. Despite their advantages, such as high temporal resolution and reduced latency, the events generated by these cameras cannot be directly used in frame-based computer vision applications. The authors aim to bridge this gap by developing a model that reconstructs intensity images from the detected events using a manifold induced by the event timestamps.

Methodology

The authors propose a variational energy minimization framework formulated on an event-basis, avoiding the complexities of simultaneous optical flow estimation. Key components of the framework include:

• Manifold Regularisation: The method applies manifold regularisation to incorporate the asynchronous nature of events directly into the reconstruction process. By constructing an event manifold based on time stamps, the algorithm guides the image reconstruction through this manifold, offering a novel approach to dealing with the sparsity and timing of event data.
• Data Term using Kullback-Leibler Divergence: The noise model takes into account the Poisson nature of camera noise, translating into a generalized Kullback-Leibler divergence as the data term in the energy minimisation framework, providing robust estimates under realistic imaging conditions.
• Total Variation on Manifolds: The method utilizes a variant of total variation, adapted for manifold structures, which allows for the smoothing of intensity images while preserving the spatial and temporal characteristics induced by the event data.

Strong Numerical Results

The implementation achieves real-time performance on standard hardware, achieving over 500 frames per second, by incorporating efficient utilization of GPUs for the optimization process. This performance is a standout result, underscoring the practical applicability of the proposed method in scenarios requiring rapid image updates.

Implications and Future Prospects

This research has several implications. Practically, it offers a live-intensity reconstruction framework that can enhance the utility of event cameras in dynamic environments like robotics and autonomous driving. Theoretically, it presents a promising avenue for further exploring manifold-based processing in neuromorphic vision systems, an area which could leverage the high temporal resolution for real-time decision-making and control applications.

Future developments could involve refining the noise models specific to event cameras and exploring more complex hierarchical manifold structures to enhance the quality and speed of reconstruction further. Additionally, integration with broader computer vision frameworks could open new paths for hybrid systems combining the strengths of both event-based and frame-based methods.

Conclusion

In summary, the authors present a sophisticated framework for real-time intensity image reconstruction from event camera data, circumventing traditional limitations by employing manifold regularisation and a robust variational approach. Their work establishes a robust method for transforming sparse and asynchronous event data into useful image reconstructions, offering significant contributions to the practical capabilities and application scope of event cameras.