How to Measure the Intelligence of Large Language Models?

Abstract: With the release of ChatGPT and other LLMs the discussion about the intelligence, possibilities, and risks, of current and future models have seen large attention. This discussion included much debated scenarios about the imminent rise of so-called "super-human" AI, i.e., AI systems that are orders of magnitude smarter than humans. In the spirit of Alan Turing, there is no doubt that current state-of-the-art LLMs already pass his famous test. Moreover, current models outperform humans in several benchmark tests, so that publicly available LLMs have already become versatile companions that connect everyday life, industry and science. Despite their impressive capabilities, LLMs sometimes fail completely at tasks that are thought to be trivial for humans. In other cases, the trustworthiness of LLMs becomes much more elusive and difficult to evaluate. Taking the example of academia, LLMs are capable of writing convincing research articles on a given topic with only little input. Yet, the lack of trustworthiness in terms of factual consistency or the existence of persistent hallucinations in AI-generated text bodies has led to a range of restrictions for AI-based content in many scientific journals. In view of these observations, the question arises as to whether the same metrics that apply to human intelligence can also be applied to computational methods and has been discussed extensively. In fact, the choice of metrics has already been shown to dramatically influence assessments on potential intelligence emergence. Here, we argue that the intelligence of LLMs should not only be assessed by task-specific statistical metrics, but separately in terms of qualitative and quantitative measures.

• The paper introduces a dual-metric framework for measuring LLM intelligence by separating quantitative data recall from qualitative reasoning skills.
• It details a methodology using extensive tests and controlled trials to assess specialized knowledge ranges and reasoning capabilities.
• Implications include improved evaluation techniques that inform future training paradigms while highlighting current limits in qualitative advancements.

Measuring Intelligence in LLMs

Introduction

The paper "How to Measure the Intelligence of LLMs?" (2407.20828) addresses the crucial issue of intelligence evaluation within LLMs, highlighting the deficiencies of current assessment paradigms primarily designed for human intelligence. In a landscape dominated by LLMs, such as ChatGPT, the paper discusses the critical need for robust, dual-metric evaluation systems, distinguishing between quantitative and qualitative intelligence in these models.

Quantitative vs. Qualitative Intelligence

To effectively measure intelligence in LLMs, the authors propose a bifurcation into quantitative and qualitative intelligence assessments. Quantitative intelligence pertains to the model's data storage capacity and the ability to maneuver and amalgamate stored information, akin to human knowledge. Unlike humans, LLMs exhibit persistent access to concentrated, voluminous data across diverse domains, ranging from mundane topics to advanced scientific concepts. Quantitative assessments involve extensive testing across numerous questions to delineate the LLM's specialization spectrum and verification of information retrieval efficacy, reflecting the breadth of the model's base training data (Figure 1).

Figure 1: Quantitative (left) and qualitative (right) performance of LLMs.

On the other hand, qualitative intelligence deals with reasoning, strategic planning, and novel problem-solving capabilities, necessitating distinct evaluation methodologies. Qualitative assessments face challenges due to proprietary model architectures and comprehensive training data volumes, demanding innovative experimentation techniques such as randomized control trials (RCTs) to evaluate machine persuasiveness alongside human capabilities. While qualitative advancements in LLMs have improved, they remain incremental, notably when juxtaposed against the exponential growth in model size and data.

Computational and Intelligence Growth

The paper contemplates the theoretical ceiling of computational growth vis-à-vis intelligence growth. It postulates that while the computational scope of LLMs may encompass exhaustive human knowledge data, qualitative advancements remain tethered to human linguistic and cognitive paradigms. This inherent limitation suggests that while LLMs may exhibit superior quantitative intelligence, qualitatively transcending human intellect significantly is improbable within prevailing self-supervised training frameworks. The discourse emphasizes the need for innovative training paradigms that transcend current cognitive-modeling limitations to achieve substantial qualitative intelligence growth.

Societal and Research Implications

The paper recognizes the societal implications of burgeoning LLM capabilities. Despite advancements in quantitative intelligence, the qualitative capacity to transform foundational human concepts remains constrained. Consequently, the paper suggests that the immediate risks posed by LLMs include societal issues like job displacement and misinformation, rather than existential threats of uncontrolled superintelligence. The imperative remains to develop comprehensive evaluation frameworks integrating both quantitative and qualitative metrics to reliably gauge and address emerging intelligence properties in LLMs.

Conclusion

In conclusion, the paper asserts the necessity for dual-focused intelligence metrics to assess the evolution in LLMs. While advancements in quantitative capacities are evident, qualitative measures still lag, questioning the potential of LLMs to surpass human cognitive faculties fundamentally. Future frameworks must address these distinct intelligence aspects to foster nuanced, reliable assessments of potential "super-human" AI characteristics and societal impacts.