Automated Design of Agentic Systems

Abstract: Researchers are investing substantial effort in developing powerful general-purpose agents, wherein Foundation Models are used as modules within agentic systems (e.g. Chain-of-Thought, Self-Reflection, Toolformer). However, the history of machine learning teaches us that hand-designed solutions are eventually replaced by learned solutions. We describe a newly forming research area, Automated Design of Agentic Systems (ADAS), which aims to automatically create powerful agentic system designs, including inventing novel building blocks and/or combining them in new ways. We further demonstrate that there is an unexplored yet promising approach within ADAS where agents can be defined in code and new agents can be automatically discovered by a meta agent programming ever better ones in code. Given that programming languages are Turing Complete, this approach theoretically enables the learning of any possible agentic system: including novel prompts, tool use, workflows, and combinations thereof. We present a simple yet effective algorithm named Meta Agent Search to demonstrate this idea, where a meta agent iteratively programs interesting new agents based on an ever-growing archive of previous discoveries. Through extensive experiments across multiple domains including coding, science, and math, we show that our algorithm can progressively invent agents with novel designs that greatly outperform state-of-the-art hand-designed agents. Importantly, we consistently observe the surprising result that agents invented by Meta Agent Search maintain superior performance even when transferred across domains and models, demonstrating their robustness and generality. Provided we develop it safely, our work illustrates the potential of an exciting new research direction toward automatically designing ever-more powerful agentic systems to benefit humanity.

• The paper presents a novel ADAS framework that uses Meta Agent Search to iteratively program and refine code-based agents.
• The methodology employs iterative solution generation, simulated human feedback, and ensemble evaluation on benchmarks like ARC, DROP, and MGSM.
• Results demonstrate agents outperforming baselines with up to 13.6 F1 points and 25.9% accuracy improvement, indicating strong transferability.

Automated Design of Agentic Systems

The paper presents a novel research area termed Automated Design of Agentic Systems (ADAS), which is poised to innovate the creation of agentic systems using Foundation Models (FMs) like GPT and Claude as building blocks. ADAS focuses on automating the design of powerful agentic systems by employing a meta agent programming new agents in code, allowing them to iteratively discover and improve upon previous solutions.

Figure 1: Overview of the proposed algorithm Meta Agent Search and examples of discovered agents.

Meta Agent Search

The algorithm introduced, Meta Agent Search, utilizes a meta agent to iteratively program new agents and test their performance. The core idea is to maintain an archive of previously discovered agents, which informs the meta agent in subsequent iterations. Programming languages serve as the search space, exploiting their Turing Completeness to theoretically enable the discovery of any possible agentic system. This approach benefits from the heavy lifting done by recent FMs proficient in coding, enabling automated agent discovery without extensive manual tuning.

Implementation Details

Meta Agent Search involves five key steps:

• Generate Initial Solutions: Leveraging multiple FM modules to propose candidate solutions.
• Simulate Human-Like Feedback: Assign roles to modules to simulate human reviews, enhancing robustness.
• Expert Feedback Evaluation: Modules evaluate solutions based on efficiency, readability, and simplicity.
• Iterative Refinement: Solving sub-problems repeatedly in search for improvements.
• Ensemble Approach: Aggregating the top solutions to achieve a final robust system.

  Figure 2: The three key components of Automated Design of Agentic Systems (ADAS).

Experiments and Results

Extensive experiments were conducted across domains including ARC Challenge, Reading Comprehension (DROP), and Multilingual Grade School Math Benchmark (MGSM), showcasing the algorithm's effectiveness.

• ARC Challenge: Meta Agent Search discovered agents that substantially outperformed state-of-the-art baselines. Notably, the iterative design allowed the emergence of sophisticated feedback mechanisms, resulting in higher accuracy rates.
• Reasoning and Problem-Solving: Agents developed via Meta Agent Search demonstrated superior performance in benchmarks like DROP and MGSM, showing improvements of up to 13.6 F1 points and 25.9% accuracy over traditional methods.
• Transferability: Demonstrated strong generalization capability when transferring agents across multiple AM models (e.g., GPT-3.5 to Claude-Sonnet), maintaining robustness and outperforming baselines significantly.

Implications of ADAS

The findings suggest that ADAS could lead to more efficient and effective discovery of agentic systems, reducing human resource expenditure and potentially unveiling novel design patterns rooted in machine discovery. However, issues such as model-generated code safety and the ethical pursuit of AGI capabilities must be carefully considered.

Future Directions

Key future research avenues include:

• Higher-order ADAS: Introduce meta-self-referential systems to evolve meta agents.
• Multi-objective ADAS: Balance performance, cost, and latency in agent designs.
• Novel Evaluation Functions: Develop intelligent functions to analyze execution logs, improving evaluation accuracy while reducing associated costs.
• Complex Domains: Expand ADAS scope to include real-world applications and multi-step interactions.

Conclusion

This paper proposes a transformative approach in designing agentic systems by leveraging machine learning's progression toward automated, learned solutions. Through ADAS and Meta Agent Search, the capability to systematically generate agents significantly enhances machine learning's application to complex problem-solving tasks across domains. The integration of such systems potentially accelerates the path to Artificial General Intelligence.