The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery

Abstract: One of the grand challenges of artificial general intelligence is developing agents capable of conducting scientific research and discovering new knowledge. While frontier models have already been used as aides to human scientists, e.g. for brainstorming ideas, writing code, or prediction tasks, they still conduct only a small part of the scientific process. This paper presents the first comprehensive framework for fully automatic scientific discovery, enabling frontier LLMs to perform research independently and communicate their findings. We introduce The AI Scientist, which generates novel research ideas, writes code, executes experiments, visualizes results, describes its findings by writing a full scientific paper, and then runs a simulated review process for evaluation. In principle, this process can be repeated to iteratively develop ideas in an open-ended fashion, acting like the human scientific community. We demonstrate its versatility by applying it to three distinct subfields of machine learning: diffusion modeling, transformer-based language modeling, and learning dynamics. Each idea is implemented and developed into a full paper at a cost of less than $15 per paper. To evaluate the generated papers, we design and validate an automated reviewer, which we show achieves near-human performance in evaluating paper scores. The AI Scientist can produce papers that exceed the acceptance threshold at a top machine learning conference as judged by our automated reviewer. This approach signifies the beginning of a new era in scientific discovery in machine learning: bringing the transformative benefits of AI agents to the entire research process of AI itself, and taking us closer to a world where endless affordable creativity and innovation can be unleashed on the world's most challenging problems. Our code is open-sourced at https://github.com/SakanaAI/AI-Scientist

• The paper presents an end-to-end system that autonomously generates research ideas, implements experiments, writes manuscripts, and conducts reviews.
• It demonstrates near-human performance with balanced accuracy of 0.65 and superhuman F1 scores at a cost of under $15 per paper.
• The framework integrates idea generation, experiment iteration, and automated review, paving the way for scalable and democratized scientific discovery.

The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery

Introduction and Motivation

The paper presents a comprehensive framework for fully automating the scientific discovery process in machine learning, leveraging recent advances in LLMs and agentic coding assistants. The AI Scientist system is designed to autonomously generate novel research ideas, implement code-level changes, execute experiments, analyze results, write scientific manuscripts, and conduct peer review, all without human intervention. This approach aims to transcend the limitations of prior research automation efforts, which have typically been restricted to narrow domains or hand-crafted search spaces, and to democratize research by drastically reducing the cost and time required to produce publishable scientific work.

Figure 1: Conceptual illustration of The AI Scientist, an end-to-end LLM-driven scientific discovery process.

System Architecture and Workflow

The AI Scientist operates in three main phases: idea generation, experimental iteration, and paper write-up, followed by an automated review process. The system is initialized with a lightweight codebase and a LaTeX template, enabling it to explore a broad range of research directions within a given domain. The workflow is as follows:

1. Idea Generation: The system uses LLMs with chain-of-thought and self-reflection prompting to propose diverse, novel research directions. Each idea is scored for interestingness, feasibility, and novelty, and filtered using literature search via the Semantic Scholar API and web access tools to avoid duplication with existing work.
2. Experiment Iteration: The AI Scientist employs the Aider coding assistant to implement code modifications, plan and execute experiments, and iteratively refine its approach based on experimental outcomes. Robust error handling and up to four retries per experiment ensure resilience to implementation failures.
3. Paper Write-up: Experimental results and notes are used to generate a full scientific manuscript in LaTeX, section by section, with automated citation search and insertion. The system performs multiple rounds of self-reflection to streamline the text and correct compilation errors.
4. Automated Reviewing: A GPT-4o-based reviewer agent evaluates the generated papers using standard conference guidelines, producing numerical scores and accept/reject decisions. The reviewer incorporates self-reflection, few-shot prompting, and ensembling to improve robustness and reduce variance.

Figure 2: Evaluation of The AI Scientist's paper reviewing process on ICLR 2022 OpenReview Data using GPT-4o. Reflexion and one-shot prompting improve accuracy; ensembling reduces variance.

Empirical Evaluation and Results

The framework was applied to three distinct ML subfields: diffusion modeling, transformer-based language modeling, and learning dynamics (grokking). Hundreds of papers were autonomously generated and evaluated, with a per-paper cost of less than \$15. The automated reviewer achieved near-human performance on ICLR 2022 OpenReview data, with balanced accuracy of 0.65 (vs. 0.66 for humans), superhuman F1 scores (0.57 vs. 0.49), and comparable AUC (0.65). The false negative rate was lower than the human baseline, indicating fewer high-quality papers were rejected, though the false positive rate was higher.

Figure 3: Violin plots showing the distribution of scores generated by the reviewer for AI-generated papers across three domains and four foundation models.

Qualitative analysis of generated papers revealed that the system can produce manuscripts with precise mathematical descriptions, comprehensive experimental write-ups, and novel visualizations. For example, the "Adaptive Dual-Scale Denoising" paper proposed a dual-branch denoiser for diffusion models, implemented the idea in code, and produced both quantitative and qualitative improvements over baselines.

Figure 4: Preview of the "Adaptive Dual-Scale Denoising" paper, entirely autonomously generated by The AI Scientist.

However, several pathologies were observed, including hallucinated experimental details, positive spin on negative results, minimal references, and presentation of intermediate results atypical for standard conference papers. The system's performance was comparable to an early-stage ML researcher, capable of executing ideas but sometimes lacking deep domain insight.

Implementation Details and Trade-offs

The AI Scientist leverages open-source tools and APIs, with modular prompts for each stage. The use of Aider enables robust code editing and error correction, but implementation success rates vary across LLMs. Claude Sonnet 3.5 produced the highest quality papers, while GPT-4o struggled with LaTeX compilation. Open-weight models (DeepSeek Coder, Llama-3.1 405b) were more cost-effective but less reliable. The system is compute-efficient, with most experiments running on a single 8xH100 node over a week.

Key implementation considerations include:

• Sandboxing and Safety: Minimal guardrails can lead to undesirable outcomes (e.g., uncontrolled process spawning, excessive storage usage). Strict containerization and resource limits are recommended.
• Vision Capabilities: The current system cannot interpret figures or plots, relying solely on textual descriptions. Integrating multimodal models would address this limitation.
• Result Verification: Hallucination of results and metrics remains a concern. Linking code, logs, and outputs for reproducibility is essential.
• Scalability: The framework is model-agnostic and can be parallelized for large-scale idea generation and evaluation.

Limitations and Ethical Considerations

The AI Scientist exhibits several limitations:

• Idea Redundancy: Generated ideas can be repetitive across runs and models.
• Implementation Failures: A significant fraction of ideas are not successfully implemented or compiled.
• Result Trustworthiness: Generated papers should be treated as promising leads rather than definitive scientific contributions.
• Reviewer Limitations: The automated reviewer cannot ask clarifying questions or interpret visual data.

Ethical risks include potential misuse for generating low-quality or unethical research, overwhelming peer review systems, and the possibility of unsafe experimentation if integrated with physical labs. Transparency in AI-generated research and reviews is critical.

Implications and Future Directions

The AI Scientist framework demonstrates the feasibility of fully automating the scientific discovery process in ML, with potential for extension to other domains given appropriate experimental automation. The system democratizes research by lowering barriers to entry and enabling rapid iteration. Future work should focus on:

• Integrating multimodal capabilities for figure and plot interpretation.
• Enhancing reliability and automatic verification of results.
• Incorporating human feedback and interaction for higher-quality outputs.
• Expanding to other scientific fields via cloud robotics and automated labs.
• Addressing alignment and safety challenges as capabilities scale.

The framework paves the way for a fully AI-driven scientific ecosystem, including autonomous researchers, reviewers, and conferences. While current systems excel at incremental innovation, it remains an open question whether they can generate paradigm-shifting ideas.

Conclusion

The AI Scientist represents a significant advance in automating the end-to-end scientific discovery process, integrating LLMs, agentic coding assistants, and automated reviewing. Empirical results demonstrate near-human review accuracy and the ability to generate hundreds of medium-quality papers at low cost. While limitations and ethical risks remain, the framework offers a scalable, interpretable, and democratized approach to research. As foundation models and agentic systems continue to improve, the potential for fully autonomous, open-ended scientific discovery across disciplines becomes increasingly tangible.