RAP: Retrieval-Augmented Planning with Contextual Memory for Multimodal LLM Agents (2402.03610v1)
Abstract: Owing to recent advancements, LLMs can now be deployed as agents for increasingly complex decision-making applications in areas including robotics, gaming, and API integration. However, reflecting past experiences in current decision-making processes, an innate human behavior, continues to pose significant challenges. Addressing this, we propose Retrieval-Augmented Planning (RAP) framework, designed to dynamically leverage past experiences corresponding to the current situation and context, thereby enhancing agents' planning capabilities. RAP distinguishes itself by being versatile: it excels in both text-only and multimodal environments, making it suitable for a wide range of tasks. Empirical evaluations demonstrate RAP's effectiveness, where it achieves SOTA performance in textual scenarios and notably enhances multimodal LLM agents' performance for embodied tasks. These results highlight RAP's potential in advancing the functionality and applicability of LLM agents in complex, real-world applications.
- Textworld: A learning environment for text-based games. In International Joint Conference on Artificial Intelligence (IJCAI), 2018.
- GenAI, M. Llama 2: Open foundation and fine-tuned chat models. In arXiv, 2023.
- Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. In Conference on Robot Learning (CoRL), 2019.
- Retrieval-augmented generation for knowledge-intensive nlp tasks. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS’20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829546.
- Visual instruction tuning. In ArXiv, volume abs/2304.08485, 2023.
- What makes good in-context examples for gpt-3333? In arXiv, 2021.
- Memory-assisted prompt editing to improve gpt-3 after deployment. In Empirical Methods in Natural Language Processing (EMNLP), 2022.
- Large language models as general pattern machines. In Conference on Robot Learning (CoRL), 2023.
- OpenAI. Gpt-4 technical report. In arXiv, 2023.
- Training language models to follow instructions with human feedback. In Neural Information Processing Systems (NeurIPS), 2022.
- Adapt: As-needed decomposition and planning with language models. In arXiv, 2023.
- Learning transferable visual models from natural language supervision. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 8748–8763. PMLR, 18–24 Jul 2021.
- Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In arXiv, 2019.
- Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Gurevych, I. and Miyao, Y. (eds.), Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2556–2565, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/v1/P18-1238. URL https://aclanthology.org/P18-1238.
- Reflexion: Language agents with verbal reinforcement learning. In Neural Information Processing Systems (NeurIPS), 2023.
- Alfworld: Aligning text and embodied environments for interactive learning. In International Conference on Learning Representations (ICLR), 2021.
- Selective annotation makes language models better few-shot learners. In arXiv, 2022.
- A survey on large language model based autonomous agents. In arXiv, 2023a.
- A Survey on Large Language Model based Autonomous Agents. In arXiv, 2023b.
- CogVLM: Visual Expert for Pretrained Language Models. In ArXiv, volume abs/2311.03079, 2023c.
- Chain-of-thought prompting elicits reasoning in large language models. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 24824–24837. Curran Associates, Inc., 2022.
- The rise and potential of large language model based agents: A survey. In arXiv, 2023.
- Webshop: Towards scalable real-world web interaction with grounded language agents. In arXiv, 2022.
- React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023.
- A survey on multimodal large language models. In arXiv, 2023.
- Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning, 2021.
- ExpeL: LLM Agents Are Experiential Learners. In AAAI Conference on Artificial Intelligence (AAAI), 2023.
- Tomoyuki Kagaya (2 papers)
- Thong Jing Yuan (2 papers)
- Yuxuan Lou (6 papers)
- Jayashree Karlekar (4 papers)
- Sugiri Pranata (12 papers)
- Akira Kinose (4 papers)
- Koki Oguri (2 papers)
- Felix Wick (8 papers)
- Yang You (173 papers)