Lost in Inference: Rediscovering the Role of Natural Language Inference for Large Language Models (2411.14103v1)

Abstract: In the recent past, a popular way of evaluating natural language understanding (NLU), was to consider a model's ability to perform natural language inference (NLI) tasks. In this paper, we investigate if NLI tasks, that are rarely used for LLM evaluation, can still be informative for evaluating LLMs. Focusing on five different NLI benchmarks across six models of different scales, we investigate if they are able to discriminate models of different size and quality and how their accuracies develop during training. Furthermore, we investigate the extent to which the softmax distributions of models align with human distributions in cases where statements are ambiguous or vague. Overall, our results paint a positive picture for the NLI tasks: we find that they are able to discriminate well between models at various stages of training, yet are not (all) saturated. Furthermore, we find that while the similarity of model distributions with human label distributions increases with scale, it is still much higher than the similarity between two populations of humans, making it a potentially interesting statistic to consider.

• The paper demonstrates that NLI benchmarks provide meaningful differentiation across LLMs via few-shot evaluations.
• It employs comprehensive analysis of five NLI tasks on various model scales, revealing non-linear training progress and improving accuracy with additional examples.
• The study underscores the need for better LLM calibration to align model judgments more closely with human reasoning in ambiguous cases.

Examination of Natural Language Inference in Evaluating LLMs

The paper "Lost in Inference: Rediscovering the Role of Natural Language Inference for LLMs" offers a nuanced examination of the applicability and utility of Natural Language Inference (NLI) tasks in evaluating LLMs. This research addresses a pertinent gap, as the focus on NLI has diminished with the rise of LLMs, raising questions about the continued relevance of NLI benchmarks in contemporary AI research.

The paper systematically investigates five distinct NLI benchmarks across six models, varying in architectural complexity and size. The primary aim is to assess whether these benchmarks can reliably discriminate between models based on size and quality, understand their performance trajectory during model training, and explore the extent of alignment between model outputs and human interpretations in cases of ambiguous or vague statements.

Methodology and Results

• Benchmarks and Models: The analysis includes well-known NLI benchmarks such as SNLI, MNLI, HANS, ANLI, and $\alpha$NLI. Models examined span Llama 3.1's variants (8B, 70B, 405B) and the Mistral family (7B, Mixtral 8x7B, and 8x22B). These models were evaluated on five NLI tasks, both fully pre-trained and in various stages of training up to 2 trillion tokens.
• Performance Across Shots: The findings indicate that NLI benchmarks indeed provide meaningful differentiation across models of various scales when few-shot examples are introduced. The results highlight a poor zero-shot performance improving significantly with as few as one additional example, although there remains considerable room for improvement, particularly on challenging benchmarks like ANLI, which even large models struggle to surpass 70% accuracy on.
• Training Dynamics: Analysis of training progress suggests that benchmark scores improve over time, especially for larger models. However, the progression exhibits non-linearities, and accuracy gains are not consistently monotonic at smaller scales, implying limited utility for granular monitoring during training at this scale.
• Contamination Analysis: Through a contamination score assessment, it was determined that the model performances were largely unaffected by contamination from training data, indicating that results from these NLI benchmarks are credible and not simply artifacts of model exposure to test data during training.
• Potential for Model Improvement: A detailed examination involving ChaosNLI, an NLI dataset with high annotation entropy due to human disagreement, underscores the complexity of the tasks. This analysis indicates that while larger models show lower divergence from human judgments than previous generations, the distribution alignment remains significantly distant from that of human annotators, suggesting fertile ground for future research in model calibration and consumer preference alignment.

Implications and Future Directions

This paper reaffirms the role of NLI benchmarks as a potent tool for evaluating and advancing LLMs. The notion that NLI benchmarks can discriminate between models and offer meaningful signal through model training is a key takeaway. Moreover, the paper suggests that current benchmarks still reflect areas where models fall short of human-like understanding, particularly in tasks that involve nuanced interpretation beyond binary correctness.

The paper proposes a pivotal direction for future research: improving the calibration of LLMs to more closely resemble human judgment distributions, which could enhance the application of LLMs in complex decision-making scenarios where human-like reasoning and consensus are valued. Researchers may explore methods to finetune or architect these models to better encapsulate human disagreement patterns and improve model interpretability as AI systems increasingly partake in human-centric environments.

In conclusion, this research contributes significantly to our understanding of NLI tasks within the context of LLM evaluation. It highlights the continued relevance and vitality of these benchmarks, not just as evaluative tools, but as integral components for driving future innovations in large-scale LLMing and understanding.