SciAgent: Tool-augmented Language Models for Scientific Reasoning (2402.11451v2)

Abstract: Scientific reasoning poses an excessive challenge for even the most advanced LLMs. To make this task more practical and solvable for LLMs, we introduce a new task setting named tool-augmented scientific reasoning. This setting supplements LLMs with scalable toolsets, and shifts the focus from pursuing an omniscient problem solver to a proficient tool-user. To facilitate the research of such setting, we construct a tool-augmented training corpus named MathFunc which encompasses over 30,000 samples and roughly 6,000 tools. Building on MathFunc, we develop SciAgent to retrieve, understand and, if necessary, use tools for scientific problem solving. Additionally, we craft a benchmark, SciToolBench, spanning five scientific domains to evaluate LLMs' abilities with tool assistance. Extensive experiments on SciToolBench confirm the effectiveness of SciAgent. Notably, SciAgent-Mistral-7B surpasses other LLMs with the same size by more than 13% in absolute accuracy. Furthermore, SciAgent-DeepMath-7B shows much superior performance than ChatGPT.

Tool-Augmented LLMs for Scientific Reasoning

The paper presents "SciAgent," a novel approach aimed at enhancing LLMs to tackle scientific reasoning tasks across various domains by incorporating specialized toolsets. Recognizing the inherent challenges that scientific reasoning poses even for state-of-the-art LLMs, the researchers propose a paradigm shift from developing a catch-all problem-solving model to creating a proficient tool-user model. This approach leverages external toolset collections specifically designed to augment the reasoning capabilities of LLMs, allowing them to apply domain-specific knowledge effectively.

Core Contributions

1. Tool-Augmented Scientific Reasoning Framework: The authors introduce a new framework that supplements LLMs with a variety of tools, allowing for enhanced scientific reasoning. This shifts the focus from creating an all-knowing model to one that effectively utilizes specialized tools for problem-solving.
2. Dataset and Toolset Development: A significant contribution is the construction of “MathFunc,” a comprehensive tool-augmented training corpus containing over 30,000 samples and nearly 6,000 tools. This corpus allows LLMs to learn and practice integrating tools into their analytical processes. Also, the paper introduces “SciToolBench,” a benchmark designed to evaluate LLMs' tool-assisted reasoning within five scientific domains.
3. SciAgent Model Implementation: The development of “SciAgent” builds on the MathFunc corpus, capable of retrieving and employing relevant tools effectively. Notably, it includes SciAgent-Mistral-7B, which demonstrated a substantial improvement over other models in the same class by showing over a 13% increase in absolute accuracy.

Experimental Findings

The paper details extensive experimentation using SciToolBench to evaluate the efficacy of the SciAgent model. The SciAgent-Mistral-7B outperformed existing models by more than 13% in accuracy on SciToolBench. Additionally, the SciAgent-DeepMath-7B surpassed ChatGPT, highlighting the benefits of integrating domain-specific tools into LLMs. These results underscore the potential of this tool-augmented framework to address and navigate the complexities of STEM problem-solving, where traditional LLMs have struggled.

Implications and Future Directions

The implications of this research are substantial for both theoretical advancements and practical applications in AI. By equipping LLMs with external tools, the paper opens pathways for creating more adaptable and capable AI systems capable of diverse and complex reasoning tasks. Practically, this framework could revolutionize how AI is applied in scientific research, education, and industry-specific problem-solving.

Future work suggested by the paper includes refining the toolsets to cover more domains and further enhancing the capability of LLMs to select and apply the most relevant tools autonomously. Additionally, the challenge remains to expand the corpus of training data to provide a more robust foundation for developing AI systems that are both general-purpose and capable of domain-specific expertise.

In conclusion, the SciAgent represents a significant step forward in the application of LLMs to scientific reasoning tasks, leveraging toolsets to provide new insights and improved performance in STEM domains. As this field evolves, the integration of advanced external tools could further empower AI systems to tackle more intricate challenges across various scientific and technical fields.