LiFCal: Online Light Field Camera Calibration via Bundle Adjustment

Abstract: We propose LiFCal, a novel geometric online calibration pipeline for MLA-based light field cameras. LiFCal accurately determines model parameters from a moving camera sequence without precise calibration targets, integrating arbitrary metric scaling constraints. It optimizes intrinsic parameters of the light field camera model, the 3D coordinates of a sparse set of scene points and camera poses in a single bundle adjustment defined directly on micro image points. We show that LiFCal can reliably and repeatably calibrate a focused plenoptic camera using different input sequences, providing intrinsic camera parameters extremely close to state-of-the-art methods, while offering two main advantages: it can be applied in a target-free scene, and it is implemented online in a complete and continuous pipeline. Furthermore, we demonstrate the quality of the obtained camera parameters in downstream tasks like depth estimation and SLAM. Webpage: https://lifcal.github.io/

• The paper introduces LiFCal, a novel target-free calibration method that uses bundle adjustment to simultaneously optimize intrinsic and extrinsic parameters.
• It refines an initial pinhole model with a thin lens correction, achieving intrinsic parameter accuracy with relative errors below 0.6% compared to reference calibrations.
• LiFCal enhances applications like metric depth estimation and ORB-SLAM3 by generating undistorted depth maps and accurate scene trajectories in dynamic environments.

Online Light Field Camera Calibration via Bundle Adjustment

The paper presents LiFCal, a novel geometric calibration pipeline for MLA-based light field cameras. Unlike traditional methods, LiFCal performs online calibration without the need for precise calibration targets, utilizing sequences from a moving camera. The proposed method integrates arbitrary metric scaling constraints and optimizes both intrinsic and extrinsic camera parameters via bundle adjustment. The intrinsic camera parameters, 3D scene point coordinates, and camera poses are refined in a single adjustment process based on micro image points.

Methodology

The core innovation in LiFCal lies in its ability to calibrate light field cameras in target-free scenes, a significant departure from traditional methods requiring intricate 3D calibration targets. The pipeline begins with raw image acquisition from the plenoptic camera. An initial calibration phase simplifies the camera model to a pinhole representation, enabling feature extraction and pose estimation via SIFT feature matching and RANSAC-based outlier rejection.

Following initialization, intrinsic parameters from a pinhole model are refined using the plenoptic camera model. This model employs a thin lens model for the main lens and a pinhole approximation for each micro lens, accounting for lens distortion directly on raw image coordinates. This correction simplifies downstream tasks like depth estimation and image synthesis, which traditionally require complex virtual depth distortion models.

The plenoptic bundle adjustment is a nonlinear optimization problem, solved using the Levenberg-Marquardt algorithm. The optimization minimizes reprojection errors of camera coordinates projected to micro image centers, allowing for the simultaneous refinement of all intrinsic and extrinsic parameters as well as 3D object points.

Results

LiFCal was evaluated against a state-of-the-art calibration pipeline utilizing professional photogrammetric software on a Raytrix R5 camera setup with three different lenses. Across various scenes and configurations, LiFCal generated intrinsic parameters with relative errors below 0.6% compared to the reference calibration, even when applied to target-free scenes.

The performance and repeatability of LiFCal were further validated through a series of calibration experiments using sequences from a public plenoptic VO dataset. The parameters estimated in these scenarios showed excellent consistency with sub-millimeter deviations from the reference values, despite the more challenging conditions of target-free calibration.

Practical Applications

The practical value of LiFCal is showcased in downstream tasks such as metric depth estimation and SLAM. Integrating the LiFCal-derived camera model into existing depth estimation pipelines yielded undistorted metric depth maps and totally focused images, eliminating the lens distortion visible in uncalibrated data. Moreover, ORB-SLAM3 was adapted to process RGB-D data synthesized from the plenoptic camera, resulting in accurate scene scaling and trajectory estimation with centimeter-level precision.

Future Directions

The results imply that LiFCal's target-free calibration methodology could be extended to various real-world applications requiring adaptive and precise camera calibration. Potential areas for future research include:

1. Enhanced Feature Detection Algorithms: Developing advanced algorithms to improve feature detection on low-texture surfaces could enhance LiFCal's performance in challenging scenes.
2. Generalization Across Camera Models: Extending the method to other plenoptic camera models and configurations will help validate its robustness and adaptability.
3. Integration with Deep Learning: Incorporating deep learning for feature extraction and scene understanding might further automate and improve the calibration process.

Conclusion

LiFCal represents a significant step forward for the field of light field camera calibration, providing a robust online calibration method that can be executed in dynamic, target-free environments. The demonstrated accuracy and practicality of the derived camera models in depth estimation and SLAM illustrate the method's potential to streamline and enhance various computer vision applications. Given its theoretical foundation and empirical validation, LiFCal lays the groundwork for more adaptive and intelligent camera systems capable of functioning in diverse real-world scenarios.