AgentStore: Scalable Integration of Heterogeneous Agents As Specialized Generalist Computer Assistant

Abstract: Digital agents capable of automating complex computer tasks have attracted considerable attention due to their immense potential to enhance human-computer interaction. However, existing agent methods exhibit deficiencies in their generalization and specialization capabilities, especially in handling open-ended computer tasks in real-world environments. Inspired by the rich functionality of the App store, we present AgentStore, a scalable platform designed to dynamically integrate heterogeneous agents for automating computer tasks. AgentStore empowers users to integrate third-party agents, allowing the system to continuously enrich its capabilities and adapt to rapidly evolving operating systems. Additionally, we propose a novel core \textbf{MetaAgent} with the \textbf{AgentToken} strategy to efficiently manage diverse agents and utilize their specialized and generalist abilities for both domain-specific and system-wide tasks. Extensive experiments on three challenging benchmarks demonstrate that AgentStore surpasses the limitations of previous systems with narrow capabilities, particularly achieving a significant improvement from 11.21\% to 23.85\% on the OSWorld benchmark, more than doubling the previous results. Comprehensive quantitative and qualitative results further demonstrate AgentStore's ability to enhance agent systems in both generalization and specialization, underscoring its potential for developing the specialized generalist computer assistant. All our codes will be made publicly available in https://chengyou-jia.github.io/AgentStore-Home.

• The paper demonstrates a scalable platform that integrates heterogeneous agents and significantly improves performance, achieving a 23.85% success rate on the OSWorld benchmark.
• It employs an innovative AgentToken strategy and automated SELF-INSTRUCT training to dynamically assign tasks with high accuracy.
• The framework offers a forward-thinking approach to creating specialized generalist assistants, enhancing digital automation in evolving computing environments.

Insights on AgentStore: A Scalable Platform for Heterogeneous Agent Integration

The authors present AgentStore, a platform designed to integrate heterogeneous agents for automating complex tasks across operating systems. The substantial improvement in performance on the OSWorld benchmark, where AgentStore achieved a success rate of 23.85% compared to the previous best of 11.21%, highlights the efficacy of this approach. AgentStore's development is driven by the limitations present in existing agent methodologies, particularly their struggles with generalization and specialization when confronted with open-ended tasks in real-world computing environments. The concept draws inspiration from the App Store's model for integrating diverse functionalities into a cohesive system.

Key Components and Methodology

AgentStore is characterized by its architecture, which comprises three central components: AgentPool, AgentEnroll, and MetaAgent. The AgentPool houses feature-specific agents, while AgentEnroll offers a standardized protocol for incorporating new agents into the system. MetaAgent serves as the hub for task management, employing a novel AgentToken strategy for efficient coordination of these agents.

1. AgentToken Strategy: This innovation is pivotal in the MetaAgent's ability to dynamically handle and route tasks to the appropriate agent from an expanding catalog. AgentToken assignments enable MetaAgent to discern which agent is most suitable for a given task or how multiple agents might collaborate effectively. This tokenization method allows MetaAgent to predict the required agent with high accuracy, avoiding the complexities of retraining and lengthy contexts.
2. Training with SELF-INSTRUCT: The authors propose an automated self-instruct mechanism to generate training data for fine-tuning AgentTokens, thereby reducing reliance on pre-collected datasets. This automated process, leveraging BERTScore to refine generated outputs for quality and diversity, demonstrates efficiency in scaling AgentStore's capabilities.
3. Practical Implementation: The application of AgentStore within OSWorld demonstrates its ability to execute tasks that range from specialized operations, such as modifying VLC recording settings, to more integrated procedures encompassing multi-agent collaboration.

Implications and Future Directions

AgentStore's scalable integration of agents suggests significant implications for developing "specialized generalists," AI systems that capably perform specific tasks while remaining adaptable to broader challenges. This flexibility is critical as operating systems and associated applications continue to evolve, demanding agents capable of addressing novel and increasingly intricate tasks.

The concept of dynamically integrating diverse agents opens avenues for future exploration in AI, particularly in enhancing the robustness and comprehensiveness of digital assistants. This could include expanding AgentStore to incorporate even more heterogeneous agents, potentially improving its ability to handle complex, multi-step, and cross-application tasks. Moreover, the implementation of the AgentToken strategy in a wider variety of AI applications might offer new insights into efficient agent interaction models.

Overall, the authors contribute a forward-thinking approach to addressing the limitations in current digital agents, offering a scalable framework that leverages the specialized capabilities of individual agents while maintaining general applicability across tasks. AgentStore stands as a promising development towards realizing more capable and versatile AI-driven automation systems.