Discovering Object Attributes by Prompting Large Language Models with Perception-Action APIs (2409.15505v2)

Abstract: There has been a lot of interest in grounding natural language to physical entities through visual context. While Vision LLMs (VLMs) can ground linguistic instructions to visual sensory information, they struggle with grounding non-visual attributes, like the weight of an object. Our key insight is that non-visual attribute detection can be effectively achieved by active perception guided by visual reasoning. To this end, we present a perception-action API that consists of VLMs and LLMs as backbones, together with a set of robot control functions. When prompted with this API and a natural language query, an LLM generates a program to actively identify attributes given an input image. Offline testing on the Odd-One-Out dataset demonstrates that our framework outperforms vanilla VLMs in detecting attributes like relative object location, size, and weight. Online testing in realistic household scenes on AI2-THOR and a real robot demonstration on a DJI RoboMaster EP robot highlight the efficacy of our approach.

• The paper's main contribution is a framework that integrates LLMs, VLMs, and robot control APIs for active object attribute detection.
• It converts natural language queries into executable programs that analyze visual and sensor data using programmatic reasoning.
• Experimental results show marked improvements in spatial reasoning and weight estimation compared to standalone visual models.

Discovering Object Attributes by Prompting LLMs with Perception-Action APIs

The paper "Discovering Object Attributes by Prompting LLMs with Perception-Action APIs" by Angelos Mavrogiannis, Dehao Yuan, and Yiannis Aloimonos addresses a crucial problem in the intersection of robotics and natural language understanding: the grounding of non-visual attributes. Current Vision LLMs (VLMs) perform well in mapping visual information to linguistic instructions but struggle with attributes that are not directly perceivable, such as the weight of objects.

Core Contribution

The authors propose a novel framework that integrates VLMs and LLMs with a set of robot control functions in a perception-action API. This framework enables a robot to identify object attributes actively by generating programs through LLMs based on input from natural language queries. The generated programs utilize both VLMs for visual reasoning and robot sensors for active perception, making significant strides in contexts where non-visual attributes must be inferred or directly measured.

Experimental Results

The efficacy of the proposed framework is demonstrated through extensive offline and online evaluations. Offline tests on the Odd-One-Out (O\textsuperscript{3}) dataset show that the framework significantly outperforms standalone VLMs in detecting attributes such as relative object location, size, and weight. For example, in spatial reasoning tasks, the combination of VLMs and LLMs improves accuracy in locating the second object in a sequence or identifying the largest object in a set.

Moreover, the framework's performance extends to real-life scenarios tested using AI2-THOR and a DJI RoboMaster EP robot. These online evaluations underscore the framework's robustness in dynamic, real-world environments, illustrating its practical utility.

Technical Insights

Several technical insights underpin the framework:

1. Programmatic Reasoning: The conversion of natural language queries into executable programs via LLMs introduces a systematic approach to attribute detection. This methodology leverages control flow tools, data structures, and utility functions inherent in programming languages to perform complex operations such as sorting detected objects and computing their relative attributes.
2. Combined Model Utility: The interaction between VLMs and LLMs allows complementary strengths to be leveraged, especially for non-visual attributes. For instance, VLMs can conduct preliminary visual inspections, while LLMs contextualize and refine these observations using domain knowledge.
3. Embodied Interaction: By incorporating a robot control API into the framework, the authors facilitate active perception. Robots can perform actions like navigating to an object or measuring its weight using onboard sensors, which is pivotal in embodied intelligence.

Comparative Analysis

The framework's performance was benchmarked against standalone models like OVD (Open Vocabulary Detection) and VQA (Visual Question Answering):

• Location and Size Queries: The proposed method demonstrated an accuracy improvement of 134% in location queries and 67% in size-related queries over OVD alone.
• Weight Estimation: The combination of VLM and LLM in the perception-action API outperformed individual models, increasing accuracy by 121% compared to OVD-only and 72% compared to VQA-only solutions.

Practical Implications

The implications of this work are multifaceted:

• Robust Household Robots: Integrating this framework into household robots can significantly enhance their ability to follow complex verbal instructions, thereby improving their functionality as domestic aides.
• Advanced HRI (Human-Robot Interaction): By enabling robots to understand and reason about a broader set of attributes, this research bridges a gap toward more intuitive and effective human-robot communication.
• Sensor Fusion: The framework highlights the importance of sensor fusion, combining visual data with other sensory inputs to form a comprehensive understanding of the environment.

Future Directions

Future research could extend the framework by:

• Enhanced Sensor Integration: Incorporating additional sensors such as IMUs or temperature sensors could expand the range of detectable attributes.
• Error Propagation Mitigation: Developing mechanisms to handle and correct errors in the initial stages of model calls could improve the reliability of downstream operations.
• Contextual Learning: Implementing mechanisms that allow the system to learn from repeated interactions and improve its performance over time.

In summary, the paper makes a significant academic contribution by demonstrating how integrating LLMs with VLMs and active perception mechanisms can elevate the functional capabilities of robots in detecting and reasoning about object attributes, particularly those that are non-visual. This integrated framework represents a valuable advancement in the field of embodied AI and robotics.