ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning (2309.16650v1)

Abstract: For robots to perform a wide variety of tasks, they require a 3D representation of the world that is semantically rich, yet compact and efficient for task-driven perception and planning. Recent approaches have attempted to leverage features from large vision-LLMs to encode semantics in 3D representations. However, these approaches tend to produce maps with per-point feature vectors, which do not scale well in larger environments, nor do they contain semantic spatial relationships between entities in the environment, which are useful for downstream planning. In this work, we propose ConceptGraphs, an open-vocabulary graph-structured representation for 3D scenes. ConceptGraphs is built by leveraging 2D foundation models and fusing their output to 3D by multi-view association. The resulting representations generalize to novel semantic classes, without the need to collect large 3D datasets or finetune models. We demonstrate the utility of this representation through a number of downstream planning tasks that are specified through abstract (language) prompts and require complex reasoning over spatial and semantic concepts. (Project page: https://concept-graphs.github.io/ Explainer video: https://youtu.be/mRhNkQwRYnc )

ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning

The paper introduces ConceptGraphs, an innovative method for generating open-vocabulary 3D scene graphs aimed at improving robot perception and planning. ConceptGraphs leverages large vision-LLMs (LVLMs) and LLMs to create semantically rich, object-centric maps of 3D scenes. This approach addresses several limitations of traditional semantic mapping techniques, including scalability issues, insufficient semantic relationships, and poor adaptability to novel object classes.

Key Innovations

1. Object-Centric Mapping:
   • ConceptGraphs employ a class-agnostic segmentation model to identify objects in RGB-D images, which are then fused into a 3D point cloud. This results in a compact representation that associates each object with a geometric point cloud and a semantic feature vector.
   • Objects are tagged using LVLMs, enabling detailed descriptions and the ability to handle a wide range of novel classes without additional training data.
2. Open-Vocabulary Scene Graph Generation:
   • Relationships among objects are encoded in edges, which are derived from geometric proximity and semantic similarity measures. An MST (minimum spanning tree) approach is used to efficiently capture these relationships.
   • By leveraging LLMs, the system infers and labels spatial relationships, creating a flexible, semantically rich 3D scene graph that supports complex, language-based queries.
3. LLM Integration for Task Planning:
   • The LLM-based planner uses the scene graph to interpret and execute a wide variety of natural language queries. By converting scene graph data into a structured text format, the LLM can identify relevant objects and provide actionable plans.
   • This capability is demonstrated in several robotic tasks, such as navigation to specific objects, manipulation, and dynamic updates to the map as objects move or change.

Methodology

Object-Based 3D Mapping:

• The semantic feature vectors are derived using embeddings from a vision model like CLIP.
• Multi-view association techniques are applied to ensure that segmented object data from different perspectives are coherently integrated.

Node and Edge Generation:

• Nodes in the scene graph are created for each object, with captions refined and summarized using GPT-4 to ensure coherence and accuracy.
• Edges are determined based on a combination of geometric overlap and semantic similarity, inferred using LLMs, and labeled with spatial relationships.

Experimental Validation:

• Extensive evaluations were conducted on both simulated datasets (e.g., Replica) and real-world scenarios.
• The scene graph was assessed through human evaluations for node and edge accuracy, showing high precision in object detection and relationship inference.
• The system was further validated through multiple real-world robotic platforms, including a mobile manipulator and a wheeled robot, showcasing its applicability across diverse tasks such as object retrieval, navigation, and complex scene queries.

Implications and Future Directions

Implications:

• Scalability: The object-centric and graph-based approach significantly reduces memory usage, allowing for efficient mapping and querying in large environments.
• Flexibility: The integration with LVLMs and LLMs allows the system to handle a wide range of objects and relationships, making it versatile for real-world applications where predefined object classes are insufficient.
• Human-Robot Interaction: The ability to handle natural language queries and dynamically update the scene graph enhances the interactivity and usability of robotic systems in diverse settings.

Future Developments:

• Model Enhancements: Future work may involve integrating more advanced LVLMs to improve object captioning accuracy and reduce errors in smaller or ambiguous object detections.
• Dynamic Environments: Enhancing the system’s ability to handle temporal dynamics, such as moving objects or real-time scene updates, could further broaden its applications in dynamic and unstructured environments.
• Task-Specific Optimizations: Customizing the LLM planning component to leverage hierarchical structures in scene graphs can optimize task planning efficiency, especially for complex, multi-step tasks.

ConceptGraphs sets a new standard for robot perception and planning by providing a scalable, efficient, and semantically rich representation of 3D scenes. By leveraging state-of-the-art vision and LLMs, it offers a robust solution to some of the most pressing challenges in robotic perception and interaction.