Learning Optical Flow and Scene Flow with Bidirectional Camera-LiDAR Fusion (2303.12017v2)

Abstract: In this paper, we study the problem of jointly estimating the optical flow and scene flow from synchronized 2D and 3D data. Previous methods either employ a complex pipeline that splits the joint task into independent stages, or fuse 2D and 3D information in an early-fusion'' orlate-fusion'' manner. Such one-size-fits-all approaches suffer from a dilemma of failing to fully utilize the characteristic of each modality or to maximize the inter-modality complementarity. To address the problem, we propose a novel end-to-end framework, which consists of 2D and 3D branches with multiple bidirectional fusion connections between them in specific layers. Different from previous work, we apply a point-based 3D branch to extract the LiDAR features, as it preserves the geometric structure of point clouds. To fuse dense image features and sparse point features, we propose a learnable operator named bidirectional camera-LiDAR fusion module (Bi-CLFM). We instantiate two types of the bidirectional fusion pipeline, one based on the pyramidal coarse-to-fine architecture (dubbed CamLiPWC), and the other one based on the recurrent all-pairs field transforms (dubbed CamLiRAFT). On FlyingThings3D, both CamLiPWC and CamLiRAFT surpass all existing methods and achieve up to a 47.9\% reduction in 3D end-point-error from the best published result. Our best-performing model, CamLiRAFT, achieves an error of 4.26\% on the KITTI Scene Flow benchmark, ranking 1st among all submissions with much fewer parameters. Besides, our methods have strong generalization performance and the ability to handle non-rigid motion. Code is available at https://github.com/MCG-NJU/CamLiFlow.

• The paper presents a novel bidirectional fusion module that jointly integrates dense image features with sparse LiDAR data.
• It introduces two fusion pipelines, CamLiPWC and CamLiRAFT, demonstrating significant improvements in 2D and 3D motion metrics.
• Experimental results show up to a 47.9% reduction in 3D error on FlyingThings3D and top performance on KITTI benchmarks.

A Technical Overview of "Learning Optical Flow and Scene Flow with Bidirectional Camera-LiDAR Fusion"

The paper "Learning Optical Flow and Scene Flow with Bidirectional Camera-LiDAR Fusion" by Haisong Liu et al. presents a novel methodology for jointly estimating optical flow and scene flow using simultaneous data from 2D cameras and 3D LiDAR sensors. This approach primarily targets improvements in the accuracy and efficiency of motion estimation critical for autonomous driving and other scene understanding tasks.

Methodological Contributions

The authors introduce an end-to-end framework that fuses 2D and 3D information through a bidirectional fusion mechanism, overcoming the limitations of traditional early or late fusion methods. A standout feature of this framework is the use of a learnable operator, termed the Bidirectional Camera-LiDAR Fusion Module (Bi-CLFM), which effectively integrates dense image features with sparse LiDAR point cloud features. This approach is implemented across various stages of the neural network architecture, enhancing the model's capacity to leverage complementary information from both modalities.

Two specific instantiations of the fusion pipeline are proposed:

• CamLiPWC: Based on the pyramidal coarse-to-fine strategy, it performs multi-stage fusion across different levels of a deep network.
• CamLiRAFT: Utilizing recurrent all-pairs field transforms, this model iteratively refines estimates in a recurrent architecture.

Both models demonstrate robust performance improvements over existing baselines, both in 2D and 3D metrics, via their unique application of bidirectional and multi-stage fusion.

Experimental Results

Numerical results on datasets such as FlyingThings3D and KITTI Scene Flow highlight the efficacy of the proposed methods. On FlyingThings3D, CamLiRAFT achieves an impressive 47.9% reduction in 3D end-point-error compared to the best published result, while ranking first on the KITTI Scene Flow benchmark with a scene flow error of just 4.26%—outperforming all previous methods with significantly fewer parameters.

These results indicate the proposed strategies are not only effective but also lightweight, enhancing their feasibility for real-time applications in autonomous vehicles. Interestingly, the approach also generalizes well to non-rigid motion, as evidenced by superior performance on the Sintel dataset without fine-tuning.

Implications and Future Directions

The development of such fusion pipelines introduces significant advancements in processing accuracy and network efficiency for optical and scene flow estimation. The use of bidirectional multi-stage fusion ensures that both 2D and 3D modalities contribute optimally to the network's predictions. This paper sets a precedent for future research exploring the integration of multi-modal data, suggesting that inter-modality complementarity can be maximized through carefully structured fusion processes.

One potential area for future research is the exploration of adaptive alignment strategies to manage sensor misalignment, ensuring robustness in real-world applications where perfect sensor calibration is not feasible. Moreover, extending these methodologies to include other types of sensors and modalities, such as radar or sound-based systems, could potentially offer further improvements in complex dynamic environments.

In summary, Liu et al.'s work provides valuable insights and methodologies for advancing multi-modal sensor fusion strategies in complex motion estimation tasks, paving the way for more accurate and reliable autonomous systems. The proposed frameworks represent a significant step forward in the efficient integration of visual and geometric data for scene dynamics understanding.