Artificial Scientific Discovery (2411.11672v2)

Abstract: Rooted in the explosion of deep learning over the past decade, this thesis spans from AlphaGo to ChatGPT to empirically examine the fundamental concepts needed to realize the vision of an artificial scientist: a machine with the capacity to autonomously generate original research and contribute to the expansion of human knowledge. The investigation begins with Olivaw, an AlphaGo Zero-like agent that discovers Othello knowledge from scratch but is unable to communicate it. This realization leads to the development of the Explanatory Learning (EL) framework, a formalization of the problem faced by a scientist when trying to explain a new phenomenon to their peers. The effective EL prescriptions allow us to crack Zendo, a popular board game simulating the scientific endeavor. This success comes with a fundamental insight: an artificial scientist must develop its own interpretation of the language used to explain its findings, and not rely on a rigid existing interpreter. Questioning the very process of learning an interpreter, we turn our attention to the inner functioning of modern multimodal models. This culminates in a simple idea to build CLIP-like models where interpretation and perception are explicitly disentangled: a cost-effective approach that couples two unimodal models using little multimodal data and no further training. Finally, we discuss what ChatGPT and its siblings are still missing to become artificial scientists, and introduce the Big-Bench Symbol Interpretation Task, a benchmark about interpreting Zendo-like explanations that sees LLMs going no further than random chance while being instead fully solved by humans.

• The paper analyzes the potential of Large Language Models (LLMs) to function as artificial scientists, exploring their capabilities in knowledge discovery, communication, and interpretation.
• Empirical evaluation and conceptual analysis reveal that current LLMs lack critical features for scientific reasoning, such as a model of reality, critical data assessment, and the ability for symbol interpretation and emergent reasoning.
• The work suggests future directions for creating artificial scientists, emphasizing the need for AI models with dynamic world understanding, critical data evaluation, and reasoning abilities akin to human scientific inquiry.

Analyzing the Potential of LLMs as Artificial Scientists

The paper under review presents a multifaceted exploration into the potential of creating an artificial scientist. This endeavor bridges several domains within AI and robotics, focusing on the fusion of knowledge discovery, communication, and interpretation. It embarks on this journey through a comprehensive series of studies and experiments, highlighting both current advancements and limitations in AI systems, particularly focusing on LLMs.

The authors initiate their inquiry with the development of {\sc Olivaw}, an AI system inspired by AlphaGo Zero, designed to master the game of Othello. This system underscores a key challenge in AI: while machines can autonomously discover and utilize knowledge, conveying these insights to humans remains challenging. It emphasizes that, akin to human scientists, AI must not only accrue knowledge but also effectively communicate it.

A central contribution of this work is the introduction of Explanatory Learning, where machines are tasked to interpret symbols autonomously. By devising Critical Rationalist Networks (CRNs), the authors propose a mechanism to achieve this. CRNs are notable for their emphasis on explanation over mere prediction, supporting both interpretability and adjustment in machine learning, which fosters a deeper intersection between human language and AI reasoning.

Moving beyond traditional AI learning paradigms, the authors explore the creation of unified spaces for visual and linguistic modalities without explicit multimodal training through ASIF, a procedural innovation. This work demonstrates an alternative avenue for aligning image and text data, challenging the necessity to train extensive multimodal models and urging the importance of retrieval methods and data efficiency.

The crux of the paper explores evaluating LLMs like GPT and PaLM as potential progenitors of artificial scientific reasoning. Despite their impressive capabilities, the authors present a critical examination of their shortcomings. LLMs currently lack a model of reality, making them susceptible to hallucinations and misinformation. They cannot critically assess the credibility or novelty of input but instead update uniformly. This fundamentally diverges from traditional scientific methodologies which prioritize evidence-based reasoning and skepticism towards new data.

Furthermore, the empirical evaluation through the Odeen environment, reminiscent of human scientific discovery, highlights LLMs' inability to perform tasks requiring symbol interpretation and emergent reasoning. In the Big-Bench collaboration, LLMs like PaLM failed to surpass random guessing in symbol interpretation tasks, further underscoring the gap between human scientific reasoning capabilities and those of current LLMs.

In summary, while the paper showcases significant advancements in AI, particularly in leveraging LLMs for various tasks, it underlines critical areas that need refinement for LLMs to be genuinely integrated as artificial scientists. The necessity to imbue AI models with a dynamic understanding of the world, critical assessment of data, and an awareness of their limitations remains paramount. Future directions suggested include integrating multimodal data to refine world models and incorporating reasoning abilities that parallel scientific inquiry. Overall, this paper serves as a reflective and insightful contribution towards AI models capable of advancing scientific knowledge autonomously.