DANLI: Deliberative Agent for Following Natural Language Instructions (2210.12485v1)

Abstract: Recent years have seen an increasing amount of work on embodied AI agents that can perform tasks by following human language instructions. However, most of these agents are reactive, meaning that they simply learn and imitate behaviors encountered in the training data. These reactive agents are insufficient for long-horizon complex tasks. To address this limitation, we propose a neuro-symbolic deliberative agent that, while following language instructions, proactively applies reasoning and planning based on its neural and symbolic representations acquired from past experience (e.g., natural language and egocentric vision). We show that our deliberative agent achieves greater than 70% improvement over reactive baselines on the challenging TEACh benchmark. Moreover, the underlying reasoning and planning processes, together with our modular framework, offer impressive transparency and explainability to the behaviors of the agent. This enables an in-depth understanding of the agent's capabilities, which shed light on challenges and opportunities for future embodied agents for instruction following. The code is available at https://github.com/sled-group/DANLI.

An Analysis of DANLI: Deliberative Agent for Following Natural Language Instructions

Introduction

The paper "DANLI: Deliberative Agent for Following Natural Language Instructions" presents a novel approach to the continuous challenge in embodied AI: enabling agents to follow human language instructions for task execution. While previous agents predominantly focus on reactive strategies, imitating pre-encountered behaviors in training data, DANLI brings forward the concept of a neuro-symbolic deliberative agent. This agent integrates neural and symbolic representations from past experiences to reason and plan proactively.

Core Contributions

The primary contribution of DANLI lies in its ability to outperform reactive approaches by leveraging a combination of symbolic and neural representations for planning and reasoning. Specifically, DANLI achieves more than a 70% improvement over reactive baselines in the TEACh benchmark, which consists of hierarchical, long-horizon tasks. Several notable features of the DANLI system are as follows:

1. Hierarchical Task Monitoring: The task monitor of DANLI predicts sequences of high-level subgoals, capturing the hierarchical structure of tasks. This is achieved by a sequence-to-sequence LLM, which uses both dialog and action history to predict completed and upcoming subgoals in symbolic form.
2. Rich Semantic Map Representation: DANLI constructs a unique 3D semantic voxel map from egocentric vision and depth perception. This map encodes precise locations and states of object instances and their spatial relations, which enhances navigation and manipulation.
3. Symbolic Planning: Symbolic planning over the constructed representations enables DANLI to handle unforeseen circumstances and recover from failures by replanning. This robust planning facilitates the completion of complex tasks reliant on multiple intermediate subgoals.
4. Transparency and Debugging: The modular framework used in DANLI provides high transparency and explainability, crucial for understanding agent behaviors and improving strategies based on observed exceptions and failures.

Key Numerical Results

DANLI's superiority is underscored by significant improvements in task completion metrics on the TEACh benchmark. A comparison with several baseline models reveals the following:

• Success Rate Improvements:

DANLI achieves a success rate of 16.89% on the validation unseen split, outperforming the best reactive baseline HET-ON by approximately 4.37%.

• Efficiency Gains:

In path-length-weighted (PLW) goal condition success, DANLI demonstrates enhanced efficiency, reducing unnecessary actions and demonstrating near-human efficiency in 26% of tasks.

Implications

The implications of this work are manifold:

1. Practical Advancement in Embodied AI: DANLI's success rates and efficiency gains suggest that neuro-symbolic systems could be pivotal in advancing practical AI applications requiring natural language understanding and interaction with physical environments.
2. Interpretability in AI Systems: By integrating explicit symbolic representations, DANLI offers a degree of interpretability and debugging capabilities that reactive models lack. This transparency is essential for developing trustworthy and reliable AI systems.
3. Limitations and Future Directions: Despite its advances, DANLI operates within a closed domain of objects and actions, necessitating manual updates for new object affordances and actions. Future work will need to focus on developing methods for automatic acquisition of new symbolic knowledge and enhancing exception handling policies.

Conclusion

DANLI represents a significant step forward in the development of AI agents capable of following natural language instructions. By blending neural networks with symbolic reasoning, it addresses the limitations of reactive systems and sets the stage for more advanced, interpretable artificial intelligence in embodied contexts. As AI research progresses, integrating even tighter neuro-symbolic connections could offer robust, adaptive, and transparent solutions capable of handling real-world complexities in instructional tasks.