Estimating Knowledge in Large Language Models Without Generating a Single Token (2406.12673v2)

Abstract: To evaluate knowledge in LLMs, current methods query the model and then evaluate its generated responses. In this work, we ask whether evaluation can be done before the model has generated any text. Concretely, is it possible to estimate how knowledgeable a model is about a certain entity, only from its internal computation? We study this question with two tasks: given a subject entity, the goal is to predict (a) the ability of the model to answer common questions about the entity, and (b) the factuality of open-ended responses generated by the model about the entity. Experiments with a variety of LLMs show that KEEN, a simple probe trained over internal subject representations, succeeds at both tasks - correlating with both the QA accuracy of the model per-subject and FActScore, a recent factuality metric in open-ended generation. Moreover, KEEN naturally aligns with the model's hedging behavior and faithfully reflects changes in the model's knowledge after fine-tuning. Lastly, we show a more interpretable yet equally performant variant of KEEN, which highlights a small set of tokens indicative of clusters and gaps in the model's knowledge. Being simple and lightweight, KEEN can be leveraged to guide decisions such as when it is appropriate to apply further training or augment queries with retrieval.

• The paper presents KEEN, a novel method that estimates a model's entity knowledge by probing hidden representations.
• It leverages features like hidden states and vocabulary projections to predict accuracy in question answering and factuality in generated content.
• Empirical results show strong correlations (0.58–0.77) between KEEN estimates and actual performance, suggesting potential for efficient model evaluation and fine-tuning.

Estimating Knowledge in LLMs Without Generating a Single Token

The paper "Estimating Knowledge in LLMs Without Generating a Single Token" introduces a novel methodology for evaluating the knowledge encapsulated within LLMs without necessitating the generation of textual responses. Traditional approaches to this evaluation involve querying models and assessing their generated outputs, but this work proposes an intrinsic, query-free technique.

Core Contributions and Approach

The primary objective is to determine if it is feasible to gauge a model's knowledge on specific entities, leveraging only its internal calculations. The authors present KEEN (Knowledge Estimation of ENtities), a probing mechanism trained over internal representations of subject entities within models. KEEN aims to predict two key facets:

1. The model's ability to accurately answer common questions about an entity.
2. The factual accuracy of generated responses related to the entity.

To achieve this, the authors formalize the task into two specific settings:

• Question Answering (QA): Estimating the accuracy of answers to entity-specific questions.
• Open-Ended Generation (OEG): Predicting the factuality rate of model-generated content about an entity.

Experimental Framework

The evaluation involves multiple LLMs, including GPT-2, Pythia, LLaMA2, and Vicuna. KEEN probes, trained over hidden representations corresponding to entity tokens, are assessed for their efficacy in predicting model performance without the need for generating responses. The results consistently show strong correlations between KEEN estimates and actual model accuracy/factuality, with Pearson correlation values ranging from 0.58 to 0.77 across different models and tasks.

Detailed Observations

Evaluation of Knowledge Pre-Generation

The paper elucidates that KEEN successfully estimates the model's knowledge about an entity by solely analyzing how the model processes the entity's name within its hidden states. The approach capitalizes on interpretability research indicating hidden representations capture extensive entity-specific attributes during inference.

KEEN's design incorporates three types of features extracted from hidden states:

• Hidden States (HS): Averaging subject representations from upper-intermediate layers.
• Vocabulary Projections (VP): Projections of hidden states to the vocabulary space.
• Top-k Vocabulary Projections (VP-k): A refined subset of the most influential tokens.

Empirical Results

In both QA and OEG tasks, KEEN demonstrates its robustness by significantly outperforming intrinsic feature-based baselines, such as self-attention outputs and fully-connected scores, and external features like entity popularity. Notably, KEEN's VP and VP-50 variants are particularly effective, striking a balance between performance and interpretability. The VP-50 probes, which limit the feature space to 50 tokens, retain high correlation with factuality scores while enhancing interpretability.

Probing Outputs and Model Behavior

KEEN's utility extends beyond static evaluations:

• Hedging Behavior: The KEEN scores correlate inversely with the model's tendency to hedge responses, implying that models inherently assess their uncertainty and hedge more on less-known entities.
• Knowledge Shift Post-Fine-Tuning: The probes accurately reflect shifts in knowledge when models are fine-tuned on specific entities, highlighting KEEN's sensitivity to dynamic changes in the knowledge base.

Implications and Future Directions

KEEN presents a compelling case for intrinsic, efficient knowledge estimation in LLMs. Practically, it can guide several decision-making scenarios—whether to augment queries with retrieval, identify knowledge gaps for targeted fine-tuning, or enhance the factuality of generated content. The ability to estimate knowledge from internal states without iterated querying reduces computational overhead and can streamline model evaluation processes.

Future work could explore fine-grained knowledge estimates, discerning detailed attributes or subject areas within an entity's knowledge scope. Additionally, expanding KEEN's applicability to non-entity-centric questions and evaluating its performance across diverse model architectures could further validate and extend its utility.

In conclusion, KEEN offers a scalable, interpretable, and robust methodology for pre-generation knowledge estimation in LLMs, with significant potential for advancing the reliability and efficiency of LLM assessments.