Towards Better Generalization: Joint Depth-Pose Learning without PoseNet (2004.01314v2)

Abstract: In this work, we tackle the essential problem of scale inconsistency for self-supervised joint depth-pose learning. Most existing methods assume that a consistent scale of depth and pose can be learned across all input samples, which makes the learning problem harder, resulting in degraded performance and limited generalization in indoor environments and long-sequence visual odometry application. To address this issue, we propose a novel system that explicitly disentangles scale from the network estimation. Instead of relying on PoseNet architecture, our method recovers relative pose by directly solving fundamental matrix from dense optical flow correspondence and makes use of a two-view triangulation module to recover an up-to-scale 3D structure. Then, we align the scale of the depth prediction with the triangulated point cloud and use the transformed depth map for depth error computation and dense reprojection check. Our whole system can be jointly trained end-to-end. Extensive experiments show that our system not only reaches state-of-the-art performance on KITTI depth and flow estimation, but also significantly improves the generalization ability of existing self-supervised depth-pose learning methods under a variety of challenging scenarios, and achieves state-of-the-art results among self-supervised learning-based methods on KITTI Odometry and NYUv2 dataset. Furthermore, we present some interesting findings on the limitation of PoseNet-based relative pose estimation methods in terms of generalization ability. Code is available at https://github.com/B1ueber2y/TrianFlow.

• The paper introduces a novel architecture that decouples depth and relative pose estimation without using PoseNet, effectively addressing scale inconsistency.
• It employs two-view triangulation and dense optical flow to align up-to-scale 3D structures, achieving state-of-the-art performance on benchmarks like KITTI and NYUv2.
• The approach enhances visual odometry and SLAM systems in challenging environments, paving the way for more robust autonomous navigation and robotics applications.

Towards Better Generalization: Joint Depth-Pose Learning without PoseNet

The paper "Towards Better Generalization: Joint Depth-Pose Learning without PoseNet" offers a significant improvement in addressing challenges in self-supervised learning for depth and pose estimation in visual odometry tasks. Specifically, it tackles the issue of scale inconsistency that hampers the performance of many existing approaches. Most traditional methods rely on learning a consistent scale of depth and pose across all input samples, a hypothesis that introduces complexities and limits generalization, particularly in indoor environments and for elongated visual odometry sequences.

The authors propose a novel architecture that explicitly disentangles the scale factor from network estimation. This system diverges from architectures like PoseNet by recovering relative pose via solving the fundamental matrix obtained from dense optical flow correspondence. A crucial component of the method is a two-view triangulation module that facilitates the recovery of an up-to-scale 3D structure. The approach aligns the scale of depth predictions with the triangulated point cloud, thus resolving the scale inconsistency problem central to the paper's claims.

Technical Insights and Numerical Results

The innovative angle of the proposed system lies in not requiring a PoseNet for relative pose estimation, instead opting for traditional two-view geometry principles. This allows the network to benefit from conventional strength while addressing neural network scale ambiguities. By doing so, the authors effectively sidestep the need to learn the implicit scale priors that hinder previous designs. The relative pose estimation is decoupled from depth prediction, simplifying the joint learning process.

The paper provides solid empirical evidence showcasing the efficiency and robustness of their approach. The proposed system achieves state-of-the-art results on the KITTI depth and flow estimation benchmarks. It also significantly outperforms existing models in terms of generalization ability, tested across various challenging scenarios such as differing camera ego-motions and indoor settings. Experiments on KITTI Odometry and NYUv2 datasets further reinforce these claims. Notably, the system displays robust performance even when tested on sequences with unseen camera movements, offering a promising alternative to conventional learning systems that rely on PoseNet.

Implications and Future Directions

The implications for practical applications are promising, especially for autonomous vehicles and robotics where visual SLAM (Simultaneous Localization and Mapping) systems are imperative. By resolving scale inconsistency issues, this research paves the way for more robust, scalable systems capable of functioning in varying environments. The dual decoupling design potentially minimizes prediction drift in long-sequence applications, a common challenge in visual odometry.

Theoretically, this work suggests a return to classical geometric principles can enhance the capabilities of deep learning frameworks. The engaging of two-view triangulation and explicit scale alignment highlights an encouraging move towards hybrid systems that leverage both learned and engineered components.

Future research might focus on overcoming limitations such as handling pure rotational motion or expanding to multi-view settings. Further exploring the integration of backend optimization into this framework could also enhance its portability and efficiency across broader applications.

In summary, the paper makes critical advancements in joint depth-pose learning by innovatively addressing scale inconsistency, providing empirical evidence of improved performance, and suggesting a roadmap for future developments in this area of AI research.