- The paper introduces an online calibration algorithm that recovers exposure times, camera response, and vignetting factors using gain-robust KLT tracking.
- It employs a decoupled approach with an adaptive tracking frontend and a nonlinear optimization backend for real-time correction.
- Evaluations on synthetic and real datasets demonstrate reduced trajectory errors and improved VO and SLAM accuracy.
Photometric Calibration of Auto Exposure Video for Real-Time Visual Odometry and SLAM
The paper "Online Photometric Calibration of Auto Exposure Video for Realtime Visual Odometry and SLAM" addresses a significant challenge in visual odometry (VO) and simultaneous localization and mapping (SLAM): the need for photometric camera calibration. Direct methods like Direct Sparse Odometry (DSO) and LSD-SLAM rely on consistent scene brightness across multiple frames, but real-world auto-exposure cameras violate this assumption. This paper introduces an innovative algorithm for online photometric calibration, enhancing the effectiveness of VO and SLAM systems when using auto-exposure videos.
The core of the proposed method is the real-time recovery of exposure times, camera response function, and vignetting factors. The authors employ gain-robust KLT feature tracking to gather correspondences of scene points that are subsequently used in a nonlinear optimization framework. Noteworthy is the emphasis on efficiently handling vignetting effects — a common issue where pixel intensities diminish toward the image edges.
The authors validate their approach on datasets with available photometric ground truth, demonstrating their method's capability to approximate photometric parameters reliably. The algorithm achieves precision in visual odometry similar to that of manually calibrated video inputs, effectively calibrating video sequences on-the-fly. This is instrumental for scenarios where photometric calibration prior to processing is impractical.
Methodological Insights
The proposed pipeline is modular, consisting of a tracking frontend and an optimization backend. The tracking frontend leverages an adaptive KLT tracker that remains robust to lighting variations caused by exposure adjustments. The optimization backend, drawing on prior work by Goldman, employs a robust formulation with the Huber norm to mitigate the impact of outliers.
Tracking scene points over time enables the recovery of photometric parameters through alternating optimization. The inverse camera response function, vignetting model, and exposure changes are estimated iteratively, with emphasis on radial motion necessary for accurate vignetting recovery.
The algorithm's decoupling into offline and online calibration modes is strategic. Offline calibration leverages entire sequences for exhaustive parameter estimation, whereas online calibration prioritizes rapid exposure adjustment for immediate VO or SLAM processing. The proposed online method decouples exposure estimation from other photometric factors, enabling real-time correction and application.
Evaluation and Results
The empirical evaluation showcases the calibration's efficacy across synthetic and real datasets: the ICL-NUIM dataset (artificial disturbances) and the TUM Mono VO dataset (natural scenes with ground truth). For each dataset, the algorithm robustly recovers photometric parameters, significantly improving the performance metrics of direct methods like DSO. Additionally, the EuRoC Mav dataset tests confirm the system’s utility in dynamic illumination settings.
The results highlight marked reductions in trajectory error and improvements in 3D reconstructions, reinforcing the importance of photometric calibration in real-world deployments of visual SLAM systems. The real-time capabilities of the method provide substantial contributions, enabling its integration with existing VO methods to circumvent photometric inconsistencies.
Implications and Future Prospects
The implications of this work extend beyond traditional SLAM and VO applications, promising enhancements in any vision-based task requiring photometric constancy, such as optical flow computation. The authors suggest future work integrating their calibration approach directly with VO/SLAM systems. Such integration holds potential for simultaneous optimization of photometric parameters, camera poses, and scene geometry, possibly advancing the state-of-the-art in direct computational methods.
Overall, this paper presents a compelling solution to a well-recognized problem in computer vision, enhancing the reliability and applicability of direct VO and SLAM paradigms in variable lighting environments. The real-time capabilities and accuracy of the proposed calibration method position it as a valuable component for next-generation autonomous and robotic systems.