NovelSeek: When Agent Becomes the Scientist -- Building Closed-Loop System from Hypothesis to Verification (2505.16938v2)

Abstract: AI is accelerating the transformation of scientific research paradigms, not only enhancing research efficiency but also driving innovation. We introduce NovelSeek, a unified closed-loop multi-agent framework to conduct Autonomous Scientific Research (ASR) across various scientific research fields, enabling researchers to tackle complicated problems in these fields with unprecedented speed and precision. NovelSeek highlights three key advantages: 1) Scalability: NovelSeek has demonstrated its versatility across 12 scientific research tasks, capable of generating innovative ideas to enhance the performance of baseline code. 2) Interactivity: NovelSeek provides an interface for human expert feedback and multi-agent interaction in automated end-to-end processes, allowing for the seamless integration of domain expert knowledge. 3) Efficiency: NovelSeek has achieved promising performance gains in several scientific fields with significantly less time cost compared to human efforts. For instance, in reaction yield prediction, it increased from 27.6% to 35.4% in just 12 hours; in enhancer activity prediction, accuracy rose from 0.65 to 0.79 with only 4 hours of processing; and in 2D semantic segmentation, precision advanced from 78.8% to 81.0% in a mere 30 hours.

• The paper introduces a unified multi-agent framework that automates the entire research cycle from idea generation to experimental validation.
• It employs specialized agents for literature review, code analysis, and adaptive experimentation to convert hypotheses into executable methodologies.
• Experimental results demonstrate significant improvements over baselines, with enhanced metrics across diverse scientific and technical tasks.

NovelSeek (2505.16938) presents a unified closed-loop multi-agent framework designed to automate and accelerate scientific research across diverse fields. The paper addresses key challenges in Autonomous Scientific Research (ASR), namely generating effective and novel research proposals and implementing robust closed-loop feedback for experimental validation.

The core of NovelSeek is a multi-agent system that facilitates the entire research cycle, from hypothesis generation to experimental verification. It comprises four main modules:

1. Self-Evolving Idea Generation with Human-interactive Feedback: This module focuses on creating and refining research ideas.
   • Survey Agent: Explores existing literature in two modes (literature review and deep research) by generating and refining keyword combinations to identify relevant scientific papers and their methodologies.
   • Code Review Agent: Analyzes user-provided or public baseline code repositories to understand structures, dependencies, and identify areas for improvement. It uses static analysis and LLMs to generate documentation.
   • Idea Innovation Agent: Generates novel ideas based on task definitions, baseline methods, and literature insights using an LLM with a higher temperature. It also evolves existing ideas by incorporating critiques and literature insights.
   • Assessment Agent: Evaluates generated ideas using multidimensional scoring (coherence, credibility, verifiability, novelty, alignment) and provides detailed narratives. It aims to ensure diversity among top-ranked ideas.
   • Human-interactive Feedback: Allows human experts or automated agents to provide feedback on ideas, guiding their refinement.
   • Orchestration Agent: Coordinates the interactions and workflows among all other agents, manages data flow, and determines optimal points for human feedback.
2. Comprehensive Idea-to-Methodology Construction: This module translates high-level research ideas into detailed, executable methodologies.
   • Methodology Development Agent: Initializes a basic method structure by integrating the idea with baseline code analysis and relevant literature. It then iteratively refines this structure based on automated assessments and human feedback, ensuring rigor and completeness.
3. Evolutionary Experimental Planning and Execution: This module implements the refined methodology and validates it through experiments.
   • Exception-Guided Debugging Framework: Converts methodological text descriptions into executable code. It systematically captures runtime exceptions, analyzes tracebacks, and uses LLMs to formulate targeted fixes iteratively. It uses Aider for single-file tasks and OpenHands for complex repository-level tasks.
   • Experimental Planning and Adaptive Evolution: Plans implementation at multiple levels (architectural, algorithmic, optimization) and employs an adaptive evolution approach. This involves structured iterations of implementation, performance assessment, and refinement, maintaining records of decisions and their effects.

The paper validates NovelSeek across 12 diverse scientific research tasks spanning science (Reaction Yield Prediction, Molecular Dynamics, Power Flow Estimation, Transcription Prediction for Perturbation Response, Enhancer Activity Prediction), time series (Time Series Forecasting), natural language (Sentiment Analysis), image (2D Image Classification, 2D Semantic Segmentation, Large Vision-LLM Fine-tuning), and point cloud (3D Point Cloud Classification, 3D Point Cloud Autonomous Driving).

Experimental results demonstrate that NovelSeek consistently improves baseline performance across these tasks and outperforms existing auto-research systems like Dolphin and AI-Researcher. For example, it increased Reaction Yield Prediction R² from 27.6% to 35.4%, Enhancer Activity Prediction HK-PCC from 0.52 to 0.79, and 2D Semantic Segmentation mIoU from 78.8% to 81.0%. NovelSeek also shows a higher success rate for generating executable code and achieving performance gains compared to baselines and other systems. A key finding is NovelSeek's ability to handle complex, multi-file (repo-level) codebase modifications, a limitation of some prior work.

Human evaluation of generated ideas indicates that NovelSeek produces ideas with higher soundness, contribution, and overall rating compared to AI-Scientist-V2, suggesting greater novelty and effectiveness.

The system utilizes powerful LLMs like GPT-4o for idea generation and assessment, and Claude-3.7-Sonnet for code generation and debugging. Tools like Aider and OpenHands are integrated for code implementation depending on complexity. Cost analysis shows that idea generation per idea is around \$0.6, and coder-debug costs vary but are generally reasonable, especially for complex repo-level tasks (\$1.1 - \$1.2 per run).

Case studies illustrate specific novel methods discovered by NovelSeek, such as "Adaptive Dual-Attention Graph-Transformer" for Reaction Yield Prediction and "Hierarchical Equivariant Directional Graph Encoder" for Molecular Dynamics. Visualizations highlight the iterative experimental planning and adaptive evolution process, showing how complex methods are decomposed and implemented step-by-step.

The authors acknowledge several technical challenges and future directions, including enhancing knowledge retrieval and representation from scientific literature, improving agent adaptability through feedback loops, and developing better benchmarks for evaluating the value and generalization of AI-generated scientific discoveries. NovelSeek is presented as a step towards more autonomous, scalable, and efficient scientific research, aiming to reduce dependence on human effort and accelerate discovery. The project provides open-source code and baseline implementations for reproducibility.