AvaTaR: Optimizing LLM Agents for Tool Usage via Contrastive Reasoning (2406.11200v3)

Abstract: LLM agents have demonstrated impressive capabilities in utilizing external tools and knowledge to boost accuracy and reduce hallucinations. However, developing prompting techniques that enable LLM agents to effectively use these tools and knowledge remains a heuristic and labor-intensive task. Here, we introduce AvaTaR, a novel and automated framework that optimizes an LLM agent to effectively leverage provided tools, improving performance on a given task. During optimization, we design a comparator module to iteratively deliver insightful and comprehensive prompts to the LLM agent by contrastively reasoning between positive and negative examples sampled from training data. We demonstrate AvaTaR on four complex multimodal retrieval datasets featuring textual, visual, and relational information, and three general question-answering (QA) datasets. We find AvaTaR consistently outperforms state-of-the-art approaches across all seven tasks, exhibiting strong generalization ability when applied to novel cases and achieving an average relative improvement of 14% on the Hit@1 metric for the retrieval datasets and 13% for the QA datasets. Code and dataset are available at https://github.com/zou-group/avatar.

• The paper introduces AVATAR, a framework that uses contrastive reasoning to optimize LLM agents' tool usage, enhancing adaptability and generalization.
• The methodology features a two-phase process that refines prompt generation and strategic actions, resulting in improvements up to 14% in Hit@1 metrics.
• Experimental evaluations demonstrate AVATAR’s superiority over existing techniques, showcasing its potential for scalable multi-modal retrieval and QA applications.

Optimizing LLM Agents for Tool Usage with AVATAR

The paper under review presents AVATAR, a novel framework designed to optimize LLM agents, primarily for tool utilization in complex retrieval and question-answering (QA) tasks. By employing contrastive reasoning, AVATAR seeks to improve both task performance and generalization capabilities of LLM agents, addressing notable challenges faced by existing prompting techniques. This work is particularly significant for scenarios involving multi-stage problem-solving where effective tool integration is crucial.

Summary of AVATAR Framework

AVATAR introduces an automated framework comprising two primary components: an actor LLM and a comparator LLM. The actor LLM initiates actions based on initial prompts and updates actions according to optimized instructions, whereas the comparator LLM automates prompt generation by using contrastive reasoning to discern between positive and negative examples.

The framework is executed in two phases:

1. Optimization Phase: Through batch-wise sampling, the comparator LLM identifies systematic weaknesses across multiple examples. Positive and negative samples are contrasted to refine tool usage and strategy, producing adaptive and generalized prompts that are systematically better informed than per-sample instructions, which are prone to overfitting.
2. Deployment Phase: Optimized instructions and action sequences are applied to new instances, demonstrating enhanced generalization across novel queries. Notable techniques such as precise problem decomposition, strategic tool usage, and synthesis optimization are emphasized to ensure comprehensive responses.

Experimental Evaluation and Results

AVATAR's efficacy is substantiated through extensive experiments on four multimodal retrieval datasets and three QA datasets. The retrieval datasets include textual, visual, and relational elements, while the QA datasets focus on natural language answering capabilities.

The empirical results underscore AVATAR's superior performance over state-of-the-art baselines. For retrieval datasets, AVATAR achieves a 14% improvement in the Hit@1 metric and a 13% improvement for QA datasets. Specifically, the framework enhances the Hit@1 metric from 5.1% to 28.6% on the FLICKR30K-ENTITIES dataset and improves Recall@20 from 30.3% to 39.3% on the STARK-PRIME dataset. AVATAR’s capability to perform robust optimization within 25 iterations reflects the advantage of utilizing the comparator module.

Comparative Analysis

A deeper examination reveals that AVATAR consistently outperforms existing approaches like ReAct and Reflexion, which rely on multi-shot reasoning and self-reflective learning strategies, respectively. Despite these methods' novelty, they fail to achieve the optimization depth of AVATAR's holistic prompt generation. Reflexion and ExpeL, for instance, show limitations in strategy refinement and generalization across different domains, which AVATAR soundly addresses through its contrastive reasoning mechanism.

Theoretical and Practical Implications

Theoretically, AVATAR advances the understanding of contrastive reasoning as a compelling strategy for multi-level optimization in LLM agents. By focusing on holistic prompt generation, AVATAR addresses systemic errors prevalent in multi-stage tasks and propels the development of adaptive learning mechanisms for future iterations of LLM agent frameworks.

Practically, the adaptability of AVATAR to real-world, tool-utilizing scenarios highlights its potential for broader applications, particularly in dynamic knowledge retrieval environments and complex QA systems. The framework's ability to generalize from a constrained training set to broader contexts—the leave-out queries of the STARK benchmark, for example—illustrates a significant stride towards scalable AI deployments in industry and research.

Conclusion

AVATAR represents a significant advancement in the optimization of tool-utilizing LLM agents. By automating the generation of adaptive and effective prompts through contrastive reasoning, the framework significantly outperforms existing methods in both retrieval and QA tasks. Its implications for enhancing agent generalization and strategic refinement offer promising pathways for future AI research, particularly in enhancing the use of LLMs in complex, real-world scenarios. Future developments may involve extending AVATAR's methodologies to more dynamic and varied contexts, potentially integrating finer-tuned memory structures for sustained learning and applicability.