OpenIN: Open-Vocabulary Instance-Oriented Navigation in Dynamic Domestic Environments

Abstract: In daily domestic settings, frequently used objects like cups often have unfixed positions and multiple instances within the same category, and their carriers frequently change as well. As a result, it becomes challenging for a robot to efficiently navigate to a specific instance. To tackle this challenge, the robot must capture and update scene changes and plans continuously. However, current object navigation approaches primarily focus on the semantic level and lack the ability to dynamically update scene representation. In contrast, this paper captures the relationships between frequently used objects and their static carriers. It constructs an open-vocabulary Carrier-Relationship Scene Graph (CRSG) and updates the carrying status during robot navigation to reflect the dynamic changes of the scene. Based on the CRSG, we further propose an instance navigation strategy that models the navigation process as a Markov Decision Process. At each step, decisions are informed by the LLM's commonsense knowledge and visual-language feature similarity. We designed a series of long-sequence navigation tasks for frequently used everyday items in the Habitat simulator. The results demonstrate that by updating the CRSG, the robot can efficiently navigate to moved targets. Additionally, we deployed our algorithm on a real robot and validated its practical effectiveness. The project page can be found here: https://OpenIN-nav.github.io.

• The paper presents the Carrier-Relationship Scene Graph (CRSG) to dynamically capture and update object relationships during navigation.
• It leverages multi-modal inputs—including textual, RGB, and image-level features along with LLMs and VLMs—for open-vocabulary recognition and instance-level differentiation.
• Experimental results in simulated and real environments demonstrate that dynamic scene updates significantly improve navigation efficiency.

OpenIN: A Framework for Navigating Domestic Environments with Dynamic Scene Updates

OpenIN presents a novel approach to object navigation in dynamic domestic environments, focusing on the challenges posed by frequently used objects that often change locations. The authors address the inadequacies of current methods, which typically lack the capability to update scene representations dynamically. The proposed method introduces the Carrier-Relationship Scene Graph (CRSG), enabling robots to capture and update the relationships between objects and their carriers (e.g., tables and cups) throughout the navigation process. This paper is supported by the National Natural Science Foundation of China, emphasizing its scientific significance.

Methodological Framework

The central innovation of OpenIN is the CRSG, which provides an evolving structure that represents the spatial and relational dynamics within a scene. The CRSG captures the 'carried-by' relationships, allowing the system to adapt to changes and update the positions of objects as environments evolve. The navigation process is further modeled using a Markov Decision Process (MDP), where decisions are aided by both the commonsense knowledge from LLMs and the visual-language feature similarity.

Multi-Modal Object Navigation

OpenIN handles navigation by supporting various instruction types, enhancing its flexibility over existing methods constrained by predefined object classes. The methodology includes several key components:

• Open-Vocabulary Recognition: By leveraging advancements in VLMs and LLMs, the system supports open-set object recognition, crucial for dynamic scenes where new object categories may appear.
• Instance-Level Differentiation: The paper underscores the importance of precise target identification in cluttered environments, achieved through multi-modal inputs combining textual, RGB, and image-level similarities.
• Dynamic Memory Updates: As objects move within the environment, the CRSG is continuously updated to reflect their new states, ensuring that navigation decisions are based on the latest scene configuration.

Experimental Validation

Experiments were conducted using both simulated environments in the Habitat simulator and on a real robotic platform. The tasks involved navigating to frequently used items in long sequences, reflecting real-world scenarios of erratic item placement. The results revealed that updating the CRSG significantly improved navigation efficiency, supporting the hypothesis that dynamic scene representation is crucial for effective object navigation.

Implications and Future Directions

The implications of this research extend to various fields where autonomous navigation is critical, such as service robotics in domestic settings or logistical operations in warehouses. By enabling robots to efficiently adapt to changes, the system promises enhanced autonomy and effectiveness.

Future research directions could explore integrating more advanced perception models or refining the MDP strategies for even more efficient decision-making. The continual evolution of LLMs will likely offer new opportunities for enhancing the semantic understanding that underpins CRSG updates.

Conclusively, OpenIN is a substantial step toward intelligent and adaptable robotic navigation systems capable of functioning effectively in dynamic real-world environments. The approach is characterized by its novel use of scene graphs and multi-modal navigation, setting a foundation for future studies in intelligent navigation.