Event-based Vision: A Survey (1904.08405v3)

Abstract: Event cameras are bio-inspired sensors that differ from conventional frame cameras: Instead of capturing images at a fixed rate, they asynchronously measure per-pixel brightness changes, and output a stream of events that encode the time, location and sign of the brightness changes. Event cameras offer attractive properties compared to traditional cameras: high temporal resolution (in the order of microseconds), very high dynamic range (140 dB vs. 60 dB), low power consumption, and high pixel bandwidth (on the order of kHz) resulting in reduced motion blur. Hence, event cameras have a large potential for robotics and computer vision in challenging scenarios for traditional cameras, such as low-latency, high speed, and high dynamic range. However, novel methods are required to process the unconventional output of these sensors in order to unlock their potential. This paper provides a comprehensive overview of the emerging field of event-based vision, with a focus on the applications and the algorithms developed to unlock the outstanding properties of event cameras. We present event cameras from their working principle, the actual sensors that are available and the tasks that they have been used for, from low-level vision (feature detection and tracking, optic flow, etc.) to high-level vision (reconstruction, segmentation, recognition). We also discuss the techniques developed to process events, including learning-based techniques, as well as specialized processors for these novel sensors, such as spiking neural networks. Additionally, we highlight the challenges that remain to be tackled and the opportunities that lie ahead in the search for a more efficient, bio-inspired way for machines to perceive and interact with the world.

• The paper details event camera technology, emphasizing asynchronous operation, high temporal resolution (~1µs), and low power consumption.
• The paper reviews both classical and novel event processing algorithms, including spiking neural networks for HDR and optical flow tasks.
• The paper showcases practical implementations in robotics, gesture recognition, and SLAM, demonstrating significant performance gains over conventional imaging.

Event-based Vision: A Survey

Event-based vision has garnered significant attention within the fields of robotics and computer vision due to its asynchronous nature and promising applications. The paper "Event-based Vision: A Survey," authored by Gallego et al., provides a detailed examination of event cameras, their operational principles, associated algorithms, and potential applications. This essay aims to offer a structured overview of the paper, elucidating key findings and discussing the broader implications for both practical and theoretical advancements in this domain.

Event cameras diverge fundamentally from traditional frame-based cameras. Instead of capturing entire images at a fixed rate, they output a stream of events that encode changes in pixel brightness asynchronously. This allows them to offer high temporal resolution (~1 microsecond), extremely high dynamic range (140 dB), and low power consumption, making them particularly suitable for dynamic and challenging environments. These properties significantly reduce motion blur and latency, offering substantial advantages over conventional imaging technologies in specific applications.

The paper is structured into several core sections, beginning with the working principles of event cameras. These sensors, inspired by biological vision systems, detect changes in the logarithmic intensity of light rather than absolute brightness values, resulting in a highly efficient manner of information capture. This section also covers available sensor technologies, highlighting innovations such as back-illuminated sensor stacks and motion artifact minimization techniques.

The next section explores event processing, exploring both classical and novel methods. Addressing the unique data format produced by event cameras, the survey discusses conventional algorithms adapted for event streams, such as the High Dynamic Range (HDR) imaging and optical flow estimation. Furthermore, it examines machine learning-driven methods, including spiking neural networks (SNNs) which are highly effective due to their event-based architecture compatible with neuromorphic sensors.

Subsequent sections review the vast array of algorithms devised for event camera data, spanning low-level vision tasks (e.g., feature detection, tracking, optic flow) to high-level vision tasks (e.g., object recognition, reconstruction, and segmentation). The paper notes substantial improvements in tasks like visual-inertial odometry and simultaneous localization and mapping (SLAM)—fields where the low latency and high dynamic range of event cameras can be fully leveraged.

A section on systems and demonstrators showcases practical implementations and demonstrations using event cameras. Specific case studies are detailed, including high-speed robotic navigation, gesture recognition systems, and HDR video generation. These examples illustrate the versatility and performance enhancements achievable via event-based vision technology.

Challenging issues are also brought to light, including the integration of event-based data with traditional frame data to enhance overall system robustness. The discussion prompts further research into hybrid systems that capture both conventional frames and event streams, enabling a richer, more comprehensive visual understanding.

In exploring future developments, the paper speculates on advances in event-based neuromorphic computing and their implications for broader AI and machine learning applications. The integration of event-based vision with emerging hardware architectures such as neuromorphic processors (e.g., IBM's TrueNorth, Intel's Loihi) could enable ultra-efficient, low-power processing, further propelling advancements in autonomous systems and intelligent sensing.

In conclusion, the survey by Gallego et al. compiles significant research contributions, addressing the unique capabilities and challenges of event-based vision. Theoretical advancements in event processing algorithms and practical insights into system deployments offer a comprehensive perspective on both current state and future directions. This work serves as a valuable resource for researchers seeking to explore the potential of event-based sensors in advancing intelligent autonomous systems and provides a foundation for continued innovation in this evolving field.