Can Large Language Models Understand Context? (2402.00858v1)

Abstract: Understanding context is key to understanding human language, an ability which LLMs have been increasingly seen to demonstrate to an impressive extent. However, though the evaluation of LLMs encompasses various domains within the realm of Natural Language Processing, limited attention has been paid to probing their linguistic capability of understanding contextual features. This paper introduces a context understanding benchmark by adapting existing datasets to suit the evaluation of generative models. This benchmark comprises of four distinct tasks and nine datasets, all featuring prompts designed to assess the models' ability to understand context. First, we evaluate the performance of LLMs under the in-context learning pretraining scenario. Experimental results indicate that pre-trained dense models struggle with understanding more nuanced contextual features when compared to state-of-the-art fine-tuned models. Second, as LLM compression holds growing significance in both research and real-world applications, we assess the context understanding of quantized models under in-context-learning settings. We find that 3-bit post-training quantization leads to varying degrees of performance reduction on our benchmark. We conduct an extensive analysis of these scenarios to substantiate our experimental results.

Introduction

LLMs have been increasingly employed for a variety of NLP applications, displaying impressive linguistic comprehension and world knowledge. While their performance on various benchmarks is noteworthy, these evaluations may not sufficiently address the models' ability to understand contextual nuances in language. This paper introduces a benchmark specifically crafted to probe LLMs' contextual understanding, comprising four tasks and nine datasets adapted for generative models.

Model Evaluation and Compression

The paper first assesses LLM performance under in-context learning (ICL) settings, comparing pre-trained dense models and fine-tuned state-of-the-art models. Findings indicate dense models fall short in grasping complex contextual features. As LLMs become increasingly large, their resource demands grow, prompting research into model compression techniques like post-training quantization. The paper extends to examining how 3-bit quantization affects LLM performance on the established benchmark.

Extensive Analysis

In contexts rich with linguistic constructs, such as coreference resolution and discourse parsing, LLMs demonstrate variable performance. Larger models fare better on more straightforward tasks, yet struggle with more complex document-based coreferences or nuanced discourse relations, often falling short of the capabilities displayed by fine-tuned models. This suggests a resilience to model compression when it concerns understanding context and an area ripe for further optimization.

Implications and Insights

This paper presents an in-depth look at the current limitations of LLMs' contextual understanding, revealing a performance gap between pre-trained models employing ICL and fine-tuned equivalents. The reduction in performance observed due to quantization highlights a trade-off between model efficiency and linguistic capability. Through the lens of the newly introduced benchmark, the paper carves out a niche for improving the contextual acuity of LLMs and underscores the importance of developing models that balance performance with practicality for real-world deployment.