Synatra: Turning Indirect Knowledge into Direct Demonstrations for Digital Agents at Scale (2409.15637v2)

Abstract: LLMs can now act as autonomous agents that interact with digital environments and complete specific objectives (e.g., arranging an online meeting). However, accuracy is still far from satisfactory, partly due to a lack of large-scale, direct demonstrations for digital tasks. Obtaining supervised data from humans is costly, and automatic data collection through exploration or reinforcement learning relies on complex environmental and content setup, resulting in datasets that lack comprehensive coverage of various scenarios. On the other hand, there is abundant knowledge that may indirectly assist task completion, such as online tutorials that were created for human consumption. In this work, we present Synatra, an approach that effectively transforms this indirect knowledge into direct supervision at scale. We define different types of indirect knowledge, and carefully study the available sources to obtain it, methods to encode the structure of direct demonstrations, and finally methods to transform indirect knowledge into direct demonstrations. We use 100k such synthetically-created demonstrations to finetune a 7B CodeLlama, and demonstrate that the resulting agent surpasses all comparably sized models on three web-based task benchmarks Mind2Web, MiniWoB++ and WebArena, as well as surpassing GPT-3.5 on WebArena and Mind2Web. In addition, while synthetic demonstrations prove to be only 3% the cost of human demonstrations (at $0.031 each), we show that the synthetic demonstrations can be more effective than an identical number of human demonstrations collected from limited domains.

• The paper presents Synatra, a framework that converts indirect procedural knowledge from sources like online tutorials into direct, synthetic demonstrations for digital agent training.
• It leverages LLM capabilities to fine-tune a 7B CodeLlama model, achieving superior performance on benchmarks such as Mind2Web, MiniWoB++, and WebArena with a 3% cost advantage over human demonstrations.
• The study demonstrates that synthetic demonstrations can generalize across diverse web environments, offering a scalable, cost-effective approach to training digital agents and inspiring further research on real-world adaptability.

Synatra: Leveraging Indirect Knowledge for Training Digital Agents

The paper presents Synatra, an innovative framework for converting indirect knowledge into practical demonstrations to train digital agents. The challenge of limited availability of direct, large-scale, high-quality supervised demonstrations for digital tasks forms the core motivation for this research. Traditionally, training autonomous agents in digital environments relies heavily on direct human demonstrations, reinforcement learning, or complex simulation environments, each bearing significant costs and limitations. Synatra addresses these challenges by converting abundant existing knowledge into a format suitable for training LLMs, specifically targeting web-based tasks.

The paper contrasts direct and indirect knowledge, defining indirect knowledge as resources useful for task execution but not in a sequence-action format. It explores various sources of indirect knowledge, such as online tutorials and web page observations, and suggests methods for converting this into direct supervision through synthetic data generation. The framework involves transforming procedural human-targeted instructions into executable demonstrations by leveraging advanced LLMs' capabilities.

Empirical results illustrate that Synatra can significantly improve the performance of a fine-tuned 7B CodeLlama model. The evaluations conducted over three web-based benchmarks—Mind2Web, MiniWoB++, and WebArena—indicate that Synatra-fine-tuned models surpass both comparably sized models and even outperform some larger models, including GPT-3.5, in select tasks. The synthesis of 100k tasks from 21 domains has furnished a robust dataset that enabled these performance gains.

Several insights arise from the experimental results. Synatra demonstrates critical improvements over traditional training methods by being 3% the cost of curated human demonstrations, and it convincingly outperformed similar-sized models trained on human-annotated data constrained by limited domains. These imply both an economic advantage and a modeling advantage, as synthetic demonstrations effectively generalize knowledge across different task settings.

The convincingly superior performance of Synatra-trained models in task comprehension and execution raises pertinent discussions about future developments. A noteworthy consideration is the feasibility of effectively scaling data synthesis, especially given that diverse, real-world HTML complexities are challenging to fully replicate synthetically. Although Synatra produces a synthetic dataset, its congruency with actual web environments remains a critical factor for generalization.

Moreover, the implications of this work extend broadly into the AI landscape, notably around the ethical and societal influences. The adeptness with which models handle digital tasks heralds efficiency improvements across numerous domains but also invites scrutiny surrounding digital autonomy and its impact on employment landscapes.

In conclusion, Synatra's prowess in converting indirect procedural information into valuable training data points towards a future where AI agents are able to navigate complex digital tasks with greater reliability and efficiency. This work signals a transformative step in data-efficient model training, with various practical and theoretical implications worthy of continued exploration, especially in optimizing the balance between synthetic training environments and real-world adaptability. Future research might consider broader deployment of such systems while deepening investigations into other domains where similar conversion mechanisms may yield substantial benefits.