Dynamic Intelligence Assessment: Benchmarking LLMs on the Road to AGI with a Focus on Model Confidence (2410.15490v3)

Abstract: As machine intelligence evolves, the need to test and compare the problem-solving abilities of different AI models grows. However, current benchmarks are often simplistic, allowing models to perform uniformly well and making it difficult to distinguish their capabilities. Additionally, benchmarks typically rely on static question-answer pairs that the models might memorize or guess. To address these limitations, we introduce Dynamic Intelligence Assessment (DIA), a novel methodology for testing AI models using dynamic question templates and improved metrics across multiple disciplines such as mathematics, cryptography, cybersecurity, and computer science. The accompanying dataset, DIA-Bench, contains a diverse collection of challenge templates with mutable parameters presented in various formats, including text, PDFs, compiled binaries, visual puzzles, and CTF-style cybersecurity challenges. Our framework introduces four new metrics to assess a model's reliability and confidence across multiple attempts. These metrics revealed that even simple questions are frequently answered incorrectly when posed in varying forms, highlighting significant gaps in models' reliability. Notably, API models like GPT-4o often overestimated their mathematical capabilities, while ChatGPT-4o demonstrated better performance due to effective tool usage. In self-assessment, OpenAI's o1-mini proved to have the best judgement on what tasks it should attempt to solve. We evaluated 25 state-of-the-art LLMs using DIA-Bench, showing that current models struggle with complex tasks and often display unexpectedly low confidence, even with simpler questions. The DIA framework sets a new standard for assessing not only problem-solving but also a model's adaptive intelligence and ability to assess its limitations. The dataset is publicly available on the project's page: https://github.com/DIA-Bench.

Dynamic Intelligence Assessment: Advancing LLM Evaluation on the Path to AGI

The paper, "Dynamic Intelligence Assessment: Benchmarking LLMs on the Road to AGI with a Focus on Model Confidence," presents the Dynamic Intelligence Assessment (DIA) framework, offering a novel methodology for evaluating LLMs across multiple disciplines. This paper advocates for a more rigorous and dynamic approach to benchmarking, addressing critical gaps in existing static datasets and focusing on model reliability and confidence.

Motivation and Novel Contributions

As LLMs evolve, distinguishing their capabilities through conventional benchmarks becomes increasingly challenging. Static question-answer pairs allow models to achieve artificially high scores through memorization or guessing. The authors propose the Dynamic Intelligence Assessment (DIA) framework, alongside the DIA-Bench dataset, to counteract these limitations. By employing dynamic question templates that span various fields such as mathematics, cryptography, and cybersecurity, the framework provides a more robust evaluation of LLM problem-solving capabilities. These templates include mutable parameters that generate diverse challenges in multiple formats, from text and PDFs to binary compilations and visual puzzles.

The introduction of four innovative metrics—Reliability Score, Task Success Rate, Confidence Index, and Near Miss Score—serves to evaluate models not just on accuracy, but on consistency and reliability. These metrics offer a nuanced view of a model's performance across different instances, thus prioritizing adaptive intelligence over mere successful completion of tasks.

Strong Numerical Insights

The paper provides a comprehensive evaluation of eight state-of-the-art LLMs against the DIA-Bench dataset. Notable observations highlight the discrepancies between models with and without tool-using capabilities. For instance, ChatGPT-4o, equipped to use external tools and execute code, significantly outperforms API-only models like GPT-4o in complex tasks. This distinction is captured in the high disparity between their Reliability Scores and Confidence Indices. ChatGPT-4o's tool utilization translated into superior performance and better task-skipping decisions, underlining the importance of tool-using capabilities in adaptive intelligence.

Implications and Future Directions

The DIA framework and its findings have significant implications for both theoretical and practical AI developments. The identified limitations in current LLMs, particularly regarding consistent problem-solving and self-assessment of their abilities, underscore the distance yet to be covered towards achieving AGI. Models like ChatGPT-4o illustrate that while improvements have been made, current models still struggle with maintaining reliability, especially in assessing and skipping tasks beyond their reach.

Speculating on future advancements, the research suggests that enhancing self-awareness capabilities in models may play a critical role in closing the gap to AGI. Moreover, the development of more sophisticated dynamic benchmarks, including broader disciplinary ranges and more complex task structures, will be crucial in driving LLMs toward more generalizable and reliable AI systems.

The public availability of the DIA-Bench dataset provides a valuable resource for future research aimed at evolving AI evaluation methods. By embracing dynamic and adaptive benchmarking, the paper sets a new standard for assessing LLM capabilities that align closely with real-world application needs and expectations.