Sparks of Artificial General Intelligence: Early experiments with GPT-4 (2303.12712v5)

Abstract: AI researchers have been developing and refining LLMs that exhibit remarkable capabilities across a variety of domains and tasks, challenging our understanding of learning and cognition. The latest model developed by OpenAI, GPT-4, was trained using an unprecedented scale of compute and data. In this paper, we report on our investigation of an early version of GPT-4, when it was still in active development by OpenAI. We contend that (this early version of) GPT-4 is part of a new cohort of LLMs (along with ChatGPT and Google's PaLM for example) that exhibit more general intelligence than previous AI models. We discuss the rising capabilities and implications of these models. We demonstrate that, beyond its mastery of language, GPT-4 can solve novel and difficult tasks that span mathematics, coding, vision, medicine, law, psychology and more, without needing any special prompting. Moreover, in all of these tasks, GPT-4's performance is strikingly close to human-level performance, and often vastly surpasses prior models such as ChatGPT. Given the breadth and depth of GPT-4's capabilities, we believe that it could reasonably be viewed as an early (yet still incomplete) version of an artificial general intelligence (AGI) system. In our exploration of GPT-4, we put special emphasis on discovering its limitations, and we discuss the challenges ahead for advancing towards deeper and more comprehensive versions of AGI, including the possible need for pursuing a new paradigm that moves beyond next-word prediction. We conclude with reflections on societal influences of the recent technological leap and future research directions.

• The paper presents evidence that GPT-4 exhibits AGI traits by excelling in language, coding, mathematics, and reasoning tasks.
• The paper employs innovative evaluation methods inspired by psychology to overcome the limitations of traditional benchmarks.
• The paper highlights GPT-4's limitations, including hallucinations, arithmetic errors, and potential societal impacts like bias and job displacement.

Sparks of Artificial General Intelligence: Early Experiments with GPT-4

The paper "Sparks of Artificial General Intelligence: Early experiments with GPT-4" (2303.12712) explores the capabilities and limitations of an early, text-only version of OpenAI's GPT-4. It presents evidence suggesting that GPT-4 exhibits a level of general intelligence exceeding previous AI models, demonstrating proficiency across diverse domains and tasks. The paper emphasizes a qualitative, exploratory approach to evaluation, focusing on novel and challenging tasks to probe GPT-4's understanding and reasoning abilities.

Assessment Methodology

The standard methodology of evaluating AI systems on established benchmarks is deemed inadequate for assessing GPT-4 due to the model's extensive pre-training on web-scale data, which likely includes these benchmarks. To address this limitation, the authors adopt an approach closer to traditional psychology, emphasizing novel and difficult tasks crafted to gauge GPT-4's true understanding and flexible application of knowledge. This method relies on human creativity to generate unique prompts and probe the model's responses for consistency, coherence, and correctness.

Evidence for General Intelligence

The paper provides numerous examples to support the claim that GPT-4 demonstrates traits of AGI. These examples span a wide range of domains, including:

• Language: GPT-4 exhibits mastery of natural language, generating fluent and coherent text, summarizing, translating, and answering questions with impressive accuracy.
• Coding and Mathematics: GPT-4 solves complex coding problems and performs mathematical reasoning at a level comparable to human experts.
• Vision: Despite being a text-only model, GPT-4 demonstrates an understanding of visual concepts, generating SVG code for recognizable images and manipulating visual features based on natural language descriptions.
• Music: GPT-4 composes original tunes in ABC notation, showcasing an understanding of musical structure and patterns.
• Reasoning and Planning: GPT-4 plays games, interacts with tools, and simulates environments, indicating an ability to plan and learn from experience.
• Understanding Humans: GPT-4 demonstrates common sense and an understanding of human motives and emotions, enabling it to engage in hypothetical dialogues and address explainability challenges.

(Figure 1)

Figure 1: Preliminary examples of {GPT-4}'s capabilities in language, vision, coding, and mathematics.

Limitations and Biases

The paper acknowledges that GPT-4 is not without limitations. These include:

• Hallucinations: GPT-4 can generate incorrect or nonsensical information, particularly in open-domain contexts. (Figure 2)
• Arithmetic Errors: GPT-4 sometimes makes mistakes in basic arithmetic calculations.
• Lack of Planning: GPT-4's autoregressive architecture can hinder its ability to plan ahead and solve problems requiring multiple steps.

  Figure 3: GPT-4 passes mock technical interviews on LeetCode. GPT-4\ could potentially be hired as a software engineer\protect\footnotemark.

The authors also note that GPT-4's patterns of intelligence are not always human-like, and the model may exhibit biases present in its training data.

Societal Influences and Future Directions

The paper reflects on the potential societal influences of GPT-4 and similar AGI systems. These include:

• Transformation of Occupations: GPT-4's capabilities may automate tasks currently performed by humans, potentially leading to job displacement.
• New Tools for Disinformation: GPT-4 could be used by malicious actors to generate disinformation and manipulate public opinion.
• Algorithmic Bias: GPT-4 may perpetuate or amplify existing societal biases present in its training data.

The paper emphasizes the need for ongoing research and monitoring to understand the benefits and risks of AGI systems, and to develop policies and guidelines for their responsible development and deployment. Key challenges include defining AGI, building missing components in LLMs, and gaining a better understanding of the origins of intelligence in these models.

Conclusion

The paper "Sparks of Artificial General Intelligence: Early experiments with GPT-4" (2303.12712) presents a compelling case that GPT-4 represents a significant step towards AGI. While acknowledging the model's limitations and biases, the authors highlight its remarkable capabilities across a wide range of domains and tasks. The paper concludes by emphasizing the need for continued research and responsible development to ensure that AGI systems benefit society as a whole.