The TUM VI Benchmark for Evaluating Visual-Inertial Odometry (1804.06120v3)

Abstract: Visual odometry and SLAM methods have a large variety of applications in domains such as augmented reality or robotics. Complementing vision sensors with inertial measurements tremendously improves tracking accuracy and robustness, and thus has spawned large interest in the development of visual-inertial (VI) odometry approaches. In this paper, we propose the TUM VI benchmark, a novel dataset with a diverse set of sequences in different scenes for evaluating VI odometry. It provides camera images with 1024x1024 resolution at 20 Hz, high dynamic range and photometric calibration. An IMU measures accelerations and angular velocities on 3 axes at 200 Hz, while the cameras and IMU sensors are time-synchronized in hardware. For trajectory evaluation, we also provide accurate pose ground truth from a motion capture system at high frequency (120 Hz) at the start and end of the sequences which we accurately aligned with the camera and IMU measurements. The full dataset with raw and calibrated data is publicly available. We also evaluate state-of-the-art VI odometry approaches on our dataset.

• The paper introduces a comprehensive dataset for visual-inertial odometry evaluation, featuring 1024x1024 images at 20 Hz and 3-axis IMU data at 200 Hz with hardware-level synchronization.
• The paper demonstrates that state-of-the-art odometry algorithms show varied tracking performance, emphasizing the benefits of advanced sensor calibration and photometric accuracy.
• The paper establishes robust evaluation metrics, including absolute trajectory error and relative pose error, to guide future improvements in visual-inertial SLAM and augmented reality applications.

Evaluating Visual-Inertial Odometry with the TUM VI Benchmark

The Technical University of Munich (TUM VI) dataset provides a robust and comprehensive tool for evaluating visual-inertial odometry systems. Addressing a critical gap in available resources, this dataset responds to the growing interest in the integration of vision and inertial sensors, crucial for applications such as augmented reality and autonomous robotics. This dataset serves as an incomparable benchmark by facilitating a diverse set of sequences, extensive sensor data, and accurate ground truth information.

Dataset Characteristics

The TUM VI dataset is advanced in its approach to visual-inertial (VI) odometry evaluation. It includes high-resolution 1024x1024 camera images operating at 20 Hz, with a high dynamic range and photometric calibration, which is particularly appealing for applications that require precise photographic output. The dataset is complemented by an inertial measurement unit (IMU) providing 3-axis accelerometer and gyroscope measurements at 200 Hz, with time-synchronization at a hardware level. Importantly, high-accuracy ground truth data is provided, aligned with IMU and camera metrics, to ensure the precision of trajectory evaluation and to support robust research development in slamming algorithms.

Comparison with Existing Datasets

While older datasets suffice for many evaluations, they come with obvious limitations such as lower resolution imagery and suboptimal synchronization parameters, which this dataset duly addresses. When juxtaposed with leading competitors like the EuRoC MAV dataset, TUM VI emerges superior with its enhanced image resolution, photometric calibration, and diverse scene variety. Moreover, the synchronization of IMU and camera data at a hardware level ensures the precision of data alignment, a crucial aspect for accurate evaluation of odometry algorithms.

Calibration and Evaluation

The TUM VI dataset ensures precise sensor calibration, which includes both geometric aspects and timing relationships among the sensor components. The calibration comprehensively involves camera, IMU, and the motion capture system. Advanced calibration methods are applied to account for IMU axis scaling and misalignment, providing a solid foundation necessary for unbiased evaluation of odometry algorithms.

In terms of evaluation metrics, the dataset facilitates the calculation of both absolute trajectory error (ATE) and relative pose error (RPE), facilitating a more nuanced evaluation of algorithmic performance, particularly for odometry algorithms without global optimization components.

Algorithm Evaluation

Within this paper, state-of-the-art VI odometry algorithms such as OKVIS, ROVIO, VINS-Mono, and BASALT have been evaluated using the TUM VI dataset. The results reveal varying degrees of tracking accuracy across different algorithm implementations, with distinct performance weak points, especially in long duration or visually demanding sequences. Notably, ROVIO, which utilizes a Kalman filter approach, was outperformed by optimization-based methods. The benchmarking indicates critical opportunities for future algorithmic advancements, while reinforcing the TUM VI dataset's role as an empirical foundation for these developments.

Implications and Future Directions

The TUM VI dataset sets new standards for the evaluation of visual-inertial odometry methods. By providing a higher level of imaging detail and superior data synchronization, it not only enhances the precision of current research but also encourages developments in both predictive modeling and real-world applications. Future work may involve extending this benchmark framework to include additional environmental settings or to support novel visual-inertial algorithm configurations. As automation and augmented reality technologies continue to evolve, resources like TUM VI will be indispensable in bridging the gap between academic research and practical deployment.

In summary, the TUM VI benchmark embodies a significant step forward in visual-inertial odometry research, providing the tools necessary to spearhead future innovations in the field.