Real-time Vision-based Depth Reconstruction with NVidia Jetson

Abstract: Vision-based depth reconstruction is a challenging problem extensively studied in computer vision but still lacking universal solution. Reconstructing depth from single image is particularly valuable to mobile robotics as it can be embedded to the modern vision-based simultaneous localization and mapping (vSLAM) methods providing them with the metric information needed to construct accurate maps in real scale. Typically, depth reconstruction is done nowadays via fully-convolutional neural networks (FCNNs). In this work we experiment with several FCNN architectures and introduce a few enhancements aimed at increasing both the effectiveness and the efficiency of the inference. We experimentally determine the solution that provides the best performance/accuracy tradeoff and is able to run on NVidia Jetson with the framerates exceeding 16FPS for 320 x 240 input. We also evaluate the suggested models by conducting monocular vSLAM of unknown indoor environment on NVidia Jetson TX2 in real-time. Open-source implementation of the models and the inference node for Robot Operating System (ROS) are available at https://github.com/CnnDepth/tx2_fcnn_node.

• The paper evaluates fully-convolutional neural network architectures for real-time vision-based depth reconstruction on the NVidia Jetson platform, targeting mobile robotics applications.
• Through empirical evaluations on the NYU Depth v2 dataset, architectures achieved over 16FPS on Jetson for 320x240 images, demonstrating real-time capability and identifying a best-performing model.
• This research enables sophisticated visual SLAM operations on compact, low-cost embedded systems by providing robust FCNN architectures for depth estimation, advancing mobile robot autonomy.

Vision-based Depth Reconstruction for Mobile Robotics Applications

The paper presents an examination of vision-based depth reconstruction techniques suitable for real-time applications on embedded systems, specifically the NVidia Jetson platform. Vision-based depth estimation from a single image is a task essential to applications like mobile robotics and augmented reality. Despite its inherent challenges compared to more straightforward multi-camera or LIDAR-based systems, monocular depth reconstruction offers a cost-effective and practical solution for small robots equipped with just a single camera.

Approach and Methodology

The study compares several fully-convolutional neural network (FCNN) architectures designed to balance the trade-off between accuracy and computational efficiency. In particular, enhancements such as optimized encoder-decoder structures and novel loss functions are investigated to improve performance and reduce the computational load. The encoders primarily utilize variants of ResNet50, and the decoders exploit various configurations, including deconvolution and upsampling strategies paired with innovative interleaving techniques.

One noteworthy aspect of the approach is the introduction of a specialized interleaving implementation for up-convolutions. By optimizing convolutional operations, the authors report increased computational efficiency, advantageous for deployment on resource-limited hardware like the NVidia Jetson TX2.

The study utilizes an adaptive BerHu loss function, which dynamically adjusts during training to provide robustness in handling depth estimation errors across varying environmental distances. This loss function aims to address the critical requirement in robotics applications of maintaining high accuracy in close-range depth estimations to avoid collisions.

Results and Performance Metrics

Through empirical evaluations on the NYU Depth v2 dataset, the paper delineates the performance of several network architectures. The authors achieved framerates exceeding 16FPS on the Jetson platform for 320x240 input images, thereby ensuring applicability for real-time visual SLAM (vSLAM) systems. The architectural design effectively balances inference speed and predictive fidelity, with Basic + SC encoder and Upsampling + nonbt decoder emerging as the best-performing model concerning accuracy and efficiency.

Practical and Theoretical Implications

The article emphasizes the practicality of deploying FCNN-based depth estimation in autonomous systems that lack the resources to handle heavyweight computational loads typically necessary for state-of-the-art depth reconstruction methods. This stands to impact mobile robotics by enabling sophisticated vSLAM operations on compact, low-cost embedded systems, thereby advancing the autonomy and adaptability of mobile robotic units.

Future research may focus on enhancing pose and map accuracy assessments, an effort that could refine SLAM algorithms and facilitate the seamless integration of FCNN-based depth maps into the path planning algorithms critical for autonomous navigation.

Conclusion and Future Directions

In summary, the research presents robust FCNN architectures suited for real-time depth estimation on embedded platforms—endorsing their use in mobile robotics. The open-source nature of their implementations provides an accessible resource for further exploration and community-driven enhancements. Future work is suggested to expand on vSLAM evaluations and to leverage the detailed maps generated for complex navigational tasks, thereby enriching path-planning capabilities in dynamic environments.