DDD17: End-To-End DAVIS Driving Dataset (1711.01458v1)

Abstract: Event cameras, such as dynamic vision sensors (DVS), and dynamic and active-pixel vision sensors (DAVIS) can supplement other autonomous driving sensors by providing a concurrent stream of standard active pixel sensor (APS) images and DVS temporal contrast events. The APS stream is a sequence of standard grayscale global-shutter image sensor frames. The DVS events represent brightness changes occurring at a particular moment, with a jitter of about a millisecond under most lighting conditions. They have a dynamic range of >120 dB and effective frame rates >1 kHz at data rates comparable to 30 fps (frames/second) image sensors. To overcome some of the limitations of current image acquisition technology, we investigate in this work the use of the combined DVS and APS streams in end-to-end driving applications. The dataset DDD17 accompanying this paper is the first open dataset of annotated DAVIS driving recordings. DDD17 has over 12 h of a 346x260 pixel DAVIS sensor recording highway and city driving in daytime, evening, night, dry and wet weather conditions, along with vehicle speed, GPS position, driver steering, throttle, and brake captured from the car's on-board diagnostics interface. As an example application, we performed a preliminary end-to-end learning study of using a convolutional neural network that is trained to predict the instantaneous steering angle from DVS and APS visual data.

• The paper presents a novel dataset that fuses asynchronous DVS with synchronous APS outputs, enhancing temporal resolution and dynamic range for autonomous driving.
• The paper integrates detailed telemetry such as steering angle and speed to provide a rich resource for sensor fusion and real-time control applications.
• The paper’s experimental results indicate that combining event-based vision with traditional imaging can improve neural network performance for steering angle prediction.

DDD17: End-To-End DAVIS Driving Dataset

The paper "DDD17: End-To-End DAVIS Driving Dataset" introduces a novel autonomous driving dataset captured using concurrent DAVIS sensor technology, which incorporates both Dynamic Vision Sensors (DVS) and Active Pixel Sensors (APS). The authors, Binas, Neil, Liu, and Delbruck from the Institute of Neuroinformatics, University of Zurich and ETH Zurich, have compiled the DDD17 dataset to facilitate the exploration of sensor fusion for advanced driver-assistance systems (ADAS), particularly under challenging conditions.

Insight into Sensor Technology

Event cameras such as the DVS are advantageous in automotive environments due to their ability to output temporal contrast events that signify changes in brightness, offering millisecond-level temporal accuracy. This is inherently more efficient than traditional frame-based systems, decreasing data rates while increasing temporal resolution. The DAVIS sensor extends this capability by concurrently providing APS frames, which are standard grayscale images, with a dynamic range exceeding 120 dB and effective frame rates over 1 kHz. In essence, DAVIS technology enhances the sensor's capability under varying lighting and environmental conditions, which is crucial for autonomous driving.

Dataset Composition and Accessibility

DDD17 presents over 12 hours of driving data recorded across diverse conditions, capturing highway and city driving in variable weather and lighting environments. The recording setup utilizes a DAVIS346B camera prototype, which significantly improves upon previous models with enhanced pixel count and low-light performance. The dataset is comprehensively annotated with vehicle telemetry data such as steering angle, speed, and diagnostic metrics through the OpenXC interface. This detailed amalgamation of visual and control data establishes DDD17 as a pivotal resource for researchers aiming to develop robust, low-latency ADAS systems.

Experimental Application and Implications

An application of the dataset involves training convolutional neural networks (CNNs) to predict steering angles purely from DVS and APS data streams. Preliminary experiments demonstrate the feasibility and efficiency gains of utilizing combined sensor data as opposed to solely frame-based imagery. While the initial results are considered inconclusive quantitatively, they suggest that event-based vision can complement traditional image sensors in vehicular control tasks effectively. This opens avenues for machine learning models designed to leverage the high resolution and asynchronous properties of event cameras for improved real-time decision-making in autonomous vehicles.

Future Prospects

The introduction of DDD17 paves the way for expansive research into sensor fusion methodologies applicable to real-world autonomous driving scenarios. The dataset serves as a foundation for examining the capabilities of event cameras in diverse environments, potentially reducing the reliance on complex, expensive sensors like LIDAR. Future developments will likely focus on optimizing deep learning architectures to better handle the unique data characteristics presented by DVS and APS, and exploring integration with existing automated vehicular platforms. Additionally, ongoing work seeks to address dataset limitations, such as data imbalance and the exclusion of complementary sensor data, to enhance predictive accuracy and robustness.

Conclusion

The DDD17 dataset highlighted in this paper is a significant contribution to the advancement of sensor fusion in autonomous driving. It presents unique opportunities for the development of efficient machine learning algorithms that embrace the asynchronous nature of event-driven cameras. This work not only broadens the scope of ADAS research but also aligns with future trends in artificial intelligence where computational efficiency and sensor synergy are pivotal in enhancing automation technology.