BioDiscoveryAgent: An AI Agent for Designing Genetic Perturbation Experiments (2405.17631v3)

Abstract: Agents based on LLMs have shown great potential in accelerating scientific discovery by leveraging their rich background knowledge and reasoning capabilities. In this paper, we introduce BioDiscoveryAgent, an agent that designs new experiments, reasons about their outcomes, and efficiently navigates the hypothesis space to reach desired solutions. We demonstrate our agent on the problem of designing genetic perturbation experiments, where the aim is to find a small subset out of many possible genes that, when perturbed, result in a specific phenotype (e.g., cell growth). Utilizing its biological knowledge, BioDiscoveryAgent can uniquely design new experiments without the need to train a machine learning model or explicitly design an acquisition function as in Bayesian optimization. Moreover, BioDiscoveryAgent, using Claude 3.5 Sonnet, achieves an average of 21% improvement in predicting relevant genetic perturbations across six datasets, and a 46% improvement in the harder task of non-essential gene perturbation, compared to existing Bayesian optimization baselines specifically trained for this task. Our evaluation includes one dataset that is unpublished, ensuring it is not part of the LLM's training data. Additionally, BioDiscoveryAgent predicts gene combinations to perturb more than twice as accurately as a random baseline, a task so far not explored in the context of closed-loop experiment design. The agent also has access to tools for searching the biomedical literature, executing code to analyze biological datasets, and prompting another agent to critically evaluate its predictions. Overall, BioDiscoveryAgent is interpretable at every stage, representing an accessible new paradigm in the computational design of biological experiments with the potential to augment scientists' efficacy.

• The paper demonstrates that BioDiscoveryAgent improves gene hit ratios by 18-29% through iterative experimental design using an LLM.
• The agent integrates literature search, gene feature analysis, and an AI critic to deliver transparent and interpretable experimental decisions.
• Experimental results validate the agent’s scalability and robustness, outperforming baseline methods by 130% in two-gene perturbations.

BioDiscoveryAgent: An AI Agent for Designing Genetic Perturbation Experiments

Introduction

The paper "BioDiscoveryAgent: An AI Agent for Designing Genetic Perturbation Experiments" presents a novel approach to accelerating the design of genetic perturbation experiments using a specialized AI agent, BioDiscoveryAgent. This AI agent leverages a LLM to generate hypotheses, plan experiments, and interpret results with the ultimate goal of identifying genes that, when perturbed, result in a specific phenotype.

Methodology

BioDiscoveryAgent addresses the inefficiencies and complexities involved in traditional genetic perturbation experiment designs that employ bespoke machine learning models and acquisition functions. The agent integrates domain-specific knowledge, interpretable decision-making, and a suite of external tools, thus creating a robust framework for experimental design.

The core of BioDiscoveryAgent is powered by an LLM. Unlike conventional machine learning approaches requiring specific training, the LLM within BioDiscoveryAgent leverages pre-existing biological knowledge and reasoning capabilities. The agent functions in a closed-loop setup, iteratively designing experiments based on the results from previous ones. It operates through a structured prompt-response mechanism, processing experimental observations and forming hypotheses for subsequent rounds of gene perturbations.

Tools and Features

BioDiscoveryAgent incorporates multiple tools to enhance its performance:

1. Literature Search: The agent can query biomedical literature databases and summarize findings to inform its gene selection. This feature adds verifiable citations to its decisions and enriches its knowledge base.
2. Gene Search Based on Features: By evaluating the similarity or dissimilarity of gene features obtained from tabular datasets, the agent can identify genes that may not be well-represented in the textual data the LLM was trained on.
3. AI Critic: A secondary LLM acts as a critic, evaluating and refining the primary agent's gene selection. This iterative feedback loop improves decision-making quality and gene selection diversity.

Experimental Results

BioDiscoveryAgent’s performance was validated using data from multiple genetic perturbation experiments across different contexts and cell types. The evaluation employed a comparative analysis against baseline methods that utilize conventional machine learning models with various acquisition functions.

Numerical Results:

• In single-gene perturbation experiments, BioDiscoveryAgent demonstrated an average of 18% improvement in hit ratio over baseline methods across five datasets.
• For non-essential genes, which represent a more challenging subset, the agent reported a 29% higher hit detection rate than baselines, underscoring its capability to predict challenging yet biologically significant hits.
• In the two-gene perturbation task, the agent outperformed the random sampling baseline by 130%.

Robustness and Interpretability

The paper emphasizes the interpretability of BioDiscoveryAgent’s decision-making process. Each stage of the agent's reasoning, from hypothesis formation to gene selection, is transparent and can be traced back to specific literature references or experimental observations. This traceability is crucial for practical applications where experimental results must be reproducible and scientifically valid.

Future Implications

BioDiscoveryAgent’s architecture and functionality suggest significant implications for future developments in artificial intelligence and biological research:

1. Scalability: The framework is scalable to other types of high-dimensional biological data and can be adapted for various experimental contexts beyond gene perturbation.
2. Human-AI Collaboration: By providing an interpretable and efficient experimental design process, the agent facilitates closer collaboration between human researchers and AI, potentially shifting the paradigm of how biological experiments are conceived and conducted.
3. Generalizability: The method can be generalized to other domains requiring iterative experimentation and complex hypothesis testing, thus broadening the impact of LLMs in scientific discovery.

Conclusion

The paper showcases BioDiscoveryAgent as an effective and interpretable AI agent for designing genetic perturbation experiments. By integrating LLMs with specialized tools for literature search, gene feature analysis, and AI-based critique, the agent overcomes the limitations of traditional machine learning-based experimental designs. This innovative approach significantly enhances the efficiency and accuracy of identifying genes that influence specific phenotypes, opening new avenues for biological research and interdisciplinary applications of AI.