SOAC: Spatio-Temporal Overlap-Aware Multi-Sensor Calibration using Neural Radiance Fields (2311.15803v3)

Abstract: In rapidly-evolving domains such as autonomous driving, the use of multiple sensors with different modalities is crucial to ensure high operational precision and stability. To correctly exploit the provided information by each sensor in a single common frame, it is essential for these sensors to be accurately calibrated. In this paper, we leverage the ability of Neural Radiance Fields (NeRF) to represent different sensors modalities in a common volumetric representation to achieve robust and accurate spatio-temporal sensor calibration. By designing a partitioning approach based on the visible part of the scene for each sensor, we formulate the calibration problem using only the overlapping areas. This strategy results in a more robust and accurate calibration that is less prone to failure. We demonstrate that our approach works on outdoor urban scenes by validating it on multiple established driving datasets. Results show that our method is able to get better accuracy and robustness compared to existing methods.

• The paper introduces SOAC, which uses multiple NeRFs to achieve precise spatio-temporal calibration across diverse sensors.
• It employs an alternating optimization strategy that refines spatial transformations and temporal offsets, outperforming existing methods.
• Testing on datasets like KITTI-360, nuScenes, and Pandaset demonstrates SOAC’s enhanced accuracy and stability for autonomous driving applications.

Overview

A new approach titled SOAC, which stands for Spatio-Temporal Overlap-Aware Multi-Sensor Calibration, offers a robust solution for sensor calibration in applications such as autonomous driving where multiple sensors with varying modalities need to be accurately synchronized and aligned. To ensure the reliability and precision of operational systems, a correct sensor frame alignment is paramount, and SOAC capitalizes on the capabilities of Neural Radiance Fields (NeRF) to create a common volumetric representation for various sensor modalities, thus facilitating sensor calibration.

Background

Autonomous systems, particularly self-driving cars, rely heavily on data from various sensors to accurately perceive and interact with their environment. The alignment of these sensors, both spatially and temporally, needs to be precise to correctly interpret the data. Calibration methodologies can be grouped into two primary categories: target-based and targetless. While target-based methods involve physical calibration objects, which can be precise but are not scalable for mass production, targetless methods offer an adaptable solution for large-scale application, as they can operate without manually placed targets and are hence better suited for calibration in real-world conditions.

Deep learning models have been introduced into the calibration process, providing fast and accurate results that enable real-time recalibration, a necessary feature in autonomous driving cars. The introduction of NeRF as an implicit scene representation has led to further advances in the field. NeRF provides a differentiable framework that can be utilized for self-supervised, targetless calibration. Some methods that use NeRF manage to perform both spatial and temporal calibration but may struggle with ensuring consistent accuracy across all observed regions.

Spatio-Temporal Overlap-Aware Calibration

The SOAC technique differs from previous approaches by representing the scene with multiple NeRFs, one for each camera in the sensor array. This partitioning by camera ensures that the NeRF model does not overfit to regions observed by only one sensor and neglect the consistency required in overlapping regions. SOAC employs an alternating optimization procedure that trains the NeRFs and refines both the spatial transformation matrices and temporal offsets via gradient descent, improving calibration robustness and accuracy.

Implementation and Results

SOAC was tested on several well-regarded driving datasets such as KITTI-360, nuScenes, and Pandaset. Compared to existing methods, SOAC has shown enhanced accuracy and stability. It can perform the calibration task more effectively, both quantitatively and qualitatively, even allowing for dynamic environments by using semantic segmentation to remove moving elements that can complicate the calibration process. Additionally, SOAC demonstrates the ability to calibrate nearly complete 360-degree camera rigs, which is essential for sensor arrays in autonomous vehicles. The performance was validated with calibration experiments, and the training process for SOAC was found to be efficient, requiring less time to achieve better results compared to other methods such as MOISST.

Future Directions and Challenges

While SOAC represents a significant advancement in multi-sensor calibration, it does rely on a reference sensor with known trajectory, and the presence of nearby structures for effective calibration. These factors limit its applicability when these conditions are not met. Further research could aim to eliminate these restrictions to broaden the technology's applicability. In addition, the scalability of the system in terms of training time needs further improvement, as the addition of more cameras increases training time significantly. Despite these challenges, SOAC stands as a promising development for the autonomous driving industry, offering a more reliable and efficient calibration process for the multi-sensor systems that vehicles depend on.