Aviary: training language agents on challenging scientific tasks (2412.21154v1)

Abstract: Solving complex real-world tasks requires cycles of actions and observations. This is particularly true in science, where tasks require many cycles of analysis, tool use, and experimentation. Language agents are promising for automating intellectual tasks in science because they can interact with tools via natural language or code. Yet their flexibility creates conceptual and practical challenges for software implementations, since agents may comprise non-standard components such as internal reasoning, planning, tool usage, as well as the inherent stochasticity of temperature-sampled LLMs. Here, we introduce Aviary, an extensible gymnasium for language agents. We formalize agents as policies solving language-grounded partially observable Markov decision processes, which we term language decision processes. We then implement five environments, including three challenging scientific environments: (1) manipulating DNA constructs for molecular cloning, (2) answering research questions by accessing scientific literature, and (3) engineering protein stability. These environments were selected for their focus on multi-step reasoning and their relevance to contemporary biology research. Finally, with online training and scaling inference-time compute, we show that language agents backed by open-source, non-frontier LLMs can match and exceed both frontier LLM agents and human experts on multiple tasks at up to 100x lower inference cost.

• The paper introduces the Aviary framework that leverages language decision processes to train agents for complex scientific tasks.
• It applies a novel methodology integrating large language models and expert iteration to achieve competitive performance with modest computational resources.
• Empirical evaluations across environments like DNA manipulation, literature Q&A, and protein engineering demonstrate the framework's versatility and efficiency.

Analysis of "Aviary: Training Language Agents on Challenging Scientific Tasks"

The paper "Aviary: Training Language Agents on Challenging Scientific Tasks" introduces a framework for training language agents, termed "Aviary," designed for environments that require intricate cyclical processes of actions and observations, specifically in scientific domains. The authors formalize the concept of language agents as policies for solving what they refer to as "language decision processes (LDPs)," which integrate elements of traditional partially observable Markov decision processes (POMDPs). This novel approach is applied within five distinct environments, including three representing challenging tasks in contemporary biological research: manipulating DNA constructs, answering scientific literature-based research questions, and engineering protein stability.

Methodological Framework

The authors provide a comprehensive framework that outlines the operational dynamics of language agents. These agents can integrate LLMs as core components, allowing them to generalize zero-shot across versatile problem domains more adeptly than traditional AI methodologies reliant on predefined rules. The report identifies the intrinsic stochastic nature of LLM sampling as a conceptual and implementation challenge, yet also as an opportunity, given the capability of LLMs to leverage feedback from dynamic, observation-based environments.

Aviary Environments

The work implements five environments:

1. DNA Construction Manipulation: This environment is geared towards tasks related to molecular cloning.
2. Scientific Literature Question Answering: It involves extracting information from scientific texts to solve predefined queries.
3. Protein Stability Engineering: Focused on proposing mutations to improve protein stability, hence addressing key concerns in biochemistry and biophysics.
4. GSM8K: It assesses general mathematical reasoning abilities.
5. HotpotQA: It entails answering complex factual questions using a diverse array of sources.

These environments are chosen for their ability to simulate the multi-step reasoning necessary for real-world scientific discoveries.

Results and Implications

The paper reveals that language agents, when equipped with open-source, smaller-scale LLMs, can achieve and sometimes surpass the performance of more advanced LLMs and expert humans in these scientific tasks, realized with lower computational costs. Notably, the work demonstrates that trained LLMs utilizing expert iteration techniques can converge toward high-performance thresholds even given modest resource constraints. Majority voting across multiple inference iterations, termed "consensus@$k$," further bolsters the agents' reliability in delivering accurate solutions.

Theoretical and Practical Significance

From a theoretical perspective, the work proposes a robust formalism for reasoning about agentic learning processes through SCGs (Stochastic Computation Graphs), which can model both deterministic and stochastic processes within a unified framework. This structured approach allows for efficient optimization across varying LDP environments.

Practically, the Aviary framework presents a scalable solution for advancing AI-driven scientific research by lowering entry barriers for implementing sophisticated agents for complex task-solving without significant investment in high-capacity computational infrastructure. This democratization of access potentially accelerates innovation in domains where automated reasoning and decision-making processes can supplement human expertise.

Future Developments in AI

The insights from this research may influence future developments in AI, particularly concerning the integration and training of LLMs within dynamic, feedback-rich environments. The synergy of advanced language processing abilities with tailored tool usage opens avenues for autonomous systems capable of executing highly specialized tasks, likely ushering a new era of scientific exploration and technological innovation facilitated by AI. These advancements could lead to more inclusive utilization of AI in various scientific and industrial sectors, promoting cross-disciplinary collaborations autonomously mediated by adaptable LLMs.

In conclusion, "Aviary" exemplifies the promising horizon for language agents in scientific inquiry, supporting enhanced precision and efficiency in tasks previously constrained by the limitations of traditional AI methodologies. The paper sets a foundation for ongoing research into optimizing agent frameworks and defining more robust interaction protocols between AI and systematic scientific methodologies.