GRACE: A Granular Benchmark for Evaluating Model Calibration against Human Calibration

Abstract: LLMs are often miscalibrated, leading to confidently incorrect answers. We introduce GRACE, a benchmark for LLM calibration that incorporates comparison with human calibration. GRACE consists of question-answer pairs, in which each question contains a series of clues that gradually become easier, all leading to the same answer; models must answer correctly as early as possible as the clues are revealed. This setting permits granular measurement of model calibration based on how early, accurately, and confidently a model answers. After collecting these questions, we host live human vs. model competitions to gather 1,749 data points on human and model teams' timing, accuracy, and confidence. We propose a metric, CalScore, that uses GRACE to analyze model calibration errors and identify types of model miscalibration that differ from human behavior. We find that although humans are less accurate than models, humans are generally better calibrated. Since state-of-the-art models struggle on GRACE, it effectively evaluates progress on improving model calibration.

Evaluating Model Calibration with GRACE

The paper introduces GRACE, a granular benchmark designed for the evaluation of model calibration, focusing particularly on the performance disparity between LLMs and human subjects. Calibration, in this context, pertains to the alignment of a model’s confidence with its prediction accuracy. Notably, LLMs have exhibited tendencies toward miscalibration, often providing incorrect answers with unwarranted confidence. This benchmark serves as a platform to scrutinize and compare the calibration of these models against human-level calibration, highlighting the peculiarities and challenges inherent in current state-of-the-art models.

GRACE comprises a structured set of question-answer pairs, each crafted to provide incrementally revealing clues, aimed at the same conclusive answer. This controlled revelation of information enables the precise assessment of a model’s calibration, based on its promptness and the accuracy-confident correlation of its responses. By simulating conditions wherein both human participants and models are subjected to identical questions, GRACE allows for the collection of robust comparative data on model versus human performance.

The authors have implemented live competitions between human teams and model counterparts, culminating in a substantial dataset of 1,749 data points detailing timing, accuracy, and confidence parameters. Analysis of this dataset reveals that, while state-of-the-art models exhibit superior accuracy, humans display markedly better calibration capabilities. This contrast underscores an essential dimension of artificial intelligence that remains underexplored: the calibration of predictive confidence alongside sheer accuracy.

To quantify these findings, the paper introduces CalScore, a novel metric devised to dissect model calibration errors. CalScore provides insights into specific miscalibration patterns, distinguishing them from typical human calibration behaviors. The results indicate that current leading models struggle substantially when evaluated with GRACE, underlining the benchmark’s efficacy in pinpointing the nuances of calibration that need enhancement for these systems.

The implications of this research are significant for advancing AI applications where reliability and confidence in decision-making are paramount. The nuanced evaluation facilitated by GRACE might catalyze the development of improved calibration techniques, potentially leading to models that mirror human judgment reliability more closely. Moreover, as AI systems continue to permeate decision-critical domains, the importance of robust calibration cannot be overstated, emphasizing the necessity for ongoing research and benchmarking akin to what GRACE proposes.

Looking forward, the application of GRACE could pivot research trajectories towards developing methods that enhance both the accuracy and the calibration of AI models. Researchers might explore augmentation techniques or new model architectures better equipped to balance these dual objectives. This could lead to more reliable AI systems, with calibration capabilities that align more closely with or surpass those of human counterparts. GRACE thus presents a meaningful stride in addressing these challenges, offering a framework to catalyze future advancements in AI calibration.