Visual Semantic Navigation with Real Robots (2311.16623v2)

Abstract: Visual Semantic Navigation (VSN) is the ability of a robot to learn visual semantic information for navigating in unseen environments. These VSN models are typically tested in those virtual environments where they are trained, mainly using reinforcement learning based approaches. Therefore, we do not yet have an in-depth analysis of how these models would behave in the real world. In this work, we propose a new solution to integrate VSN models into real robots, so that we have true embodied agents. We also release a novel ROS-based framework for VSN, ROS4VSN, so that any VSN-model can be easily deployed in any ROS-compatible robot and tested in a real setting. Our experiments with two different robots, where we have embedded two state-of-the-art VSN agents, confirm that there is a noticeable performance difference of these VSN solutions when tested in real-world and simulation environments. We hope that this research will endeavor to provide a foundation for addressing this consequential issue, with the ultimate aim of advancing the performance and efficiency of embodied agents within authentic real-world scenarios. Code to reproduce all our experiments can be found at https://github.com/gramuah/ros4vsn.

• The paper introduces a ROS4VSN framework that enables VSN models to be deployed on real robots, highlighting a performance gap between simulation and actual environments.
• It compares state-of-the-art models, showing that modular approaches like VLV outperform end-to-end methods when transferred from virtual to real-world tasks.
• Real-world experiments, which include navigation tasks with specific target proximity requirements, validate the need for adapting VSN strategies to practical robotic settings.

Introduction to Visual Semantic Navigation

Visual Semantic Navigation (VSN) represents a robot's capability to interpret and navigate an environment using visual information and understanding of that environment's semantics. Traditionally, VSN models rely heavily on reinforcement learning techniques and are mostly tested in virtual simulations which they were designed for. Understanding how these models operate outside of these training confines, in actual real-world settings, is crucial for advancing robotics.

ROS-Based Framework for Real Robots

To address the gap between simulated and real-world environments, a ROS (Robot Operating System)-based framework termed ROS4VSN was developed. The framework aims to facilitate the deployment of any VSN model onto a ROS-compatible robot and test it in real scenarios. ROS4VSN stands out for its agnosticism towards the VSN model used, making the integration process relatively straightforward regardless of the VSN agent type.

Experimentation with State-of-the-Art VSN Models

ROS4VSN was employed to test two advanced VSN models—PIRLNav and VLV—within authentic real-world scenarios. These models, trained originally with images from real-world sources, were recalibrated to work with actual robots' inputs rather than simulated data. Experiments consisted of navigating to specific objects in a house, using a set of predefined starting points and considering a success if the robot could identify and stop within one meter of the target object within a specified number of actions.

Insights from Real-World Testing

The research revealed a noticeable difference in the performance of VSN solutions when tested in real-world setups compared to simulated environments. For instance, PIRLNav witnessed a significant drop in success rate, suggesting the difficulties VSN models face in adapting learned behaviors from virtual simulations to diverse conditions found in real settings. Moreover, the paper also affirmed the trend that modular learning approaches like VLV, which include specific components like an object detector, tend to perform better than end-to-end learning approaches when deployed in the physical world.

Conclusion and Future Outlook

This paper underlines the significance of further research for the improvement of VSN systems in actual robots. The ROS4VSN framework serves as a foundation for such work, offering a means to analyze and enhance the performance of VSN agents outside simulations. The hope is that ROS4VSN and similar endeavors will spur advancements and narrow the performance gap between robots’ simulated training and their operational reality.