Fundamental Problems With Model Editing: How Should Rational Belief Revision Work in LLMs? (2406.19354v1)

Abstract: The model editing problem concerns how LLMs should learn new facts about the world over time. While empirical research on model editing has drawn widespread attention, the conceptual foundations of model editing remain shaky -- perhaps unsurprisingly, since model editing is essentially belief revision, a storied problem in philosophy that has eluded succinct solutions for decades. Model editing nonetheless demands a solution, since we need to be able to control the knowledge within LLMs. With this goal in mind, this paper critiques the standard formulation of the model editing problem and proposes a formal testbed for model editing research. We first describe 12 open problems with model editing, based on challenges with (1) defining the problem, (2) developing benchmarks, and (3) assuming LLMs have editable beliefs in the first place. Many of these challenges are extremely difficult to address, e.g. determining far-reaching consequences of edits, labeling probabilistic entailments between facts, and updating beliefs of agent simulators. Next, we introduce a semi-synthetic dataset for model editing based on Wikidata, where we can evaluate edits against labels given by an idealized Bayesian agent. This enables us to say exactly how belief revision in LLMs falls short of a desirable epistemic standard. We encourage further research exploring settings where such a gold standard can be compared against. Our code is publicly available at: https://github.com/peterbhase/LLM-belief-revision

Fundamental Problems With Model Editing: How Should Rational Belief Revision Work in LLMs?

Model editing in NLP addresses the necessity of updating LLMs (LMs) with new information to reflect the current state of the world accurately. However, this paper, "Fundamental Problems With Model Editing: How Should Rational Belief Revision Work in LLMs?" by Hase et al., critically examines the theoretical underpinnings and practical approaches to model editing, proposing a shift towards a more formal, controlled evaluation framework. This essay unpacks the paper's core arguments, critiques, and contributions, emphasizing the challenges that linger and exploring the experimental methodology and results drawn from a newly introduced semi-synthetic dataset.

Theoretical Challenges in Model Editing

The paper outlines twelve core challenges in model editing, categorized into three domains: defining the model editing problem, developing appropriate benchmarks, and assuming that LLMs have editable beliefs. Each category is fraught with conceptual, methodological, and theoretical obstacles that impede progress in formulating robust model editing techniques.

Defining the Model Editing Problem

1. Problem of Background Beliefs: Rational interpretation of new information heavily depends on prior beliefs. This challenge implies that an LLM's response to new evidence is influenced by its pre-existing knowledge, making it essential to understand and evaluate these background beliefs when implementing model edits.
2. Problem of Many Possible Worlds: New facts can imply multiple possible states of the world, complicating the determination of which state is most likely. This problem points to the inherent difficulty of specifying correct model behavior without an absolute frame of reference.
3. Problem of Complete Corrigibility: Ideally, LLMs should accept any update to their knowledge, but sweeping changes can have unforeseen, complex consequences. This highlights the need for practical methods to handle broad, impactful belief updates.
4. Problem of Missing Context: Model edits often lack conversational or broader contextual information, which is critical for interpreting updates accurately. This problem highlights the deficiency in current datasets that only provide decontextualized input-output pairs for editing.
5. Problem of Coherence At All Cost: Balancing the computational cost of maintaining coherent beliefs with achieving other practical goals poses a significant challenge, especially for agentic LLMs operating within bounded resources.

Developing Reliable Benchmarks

1. Factual Entailment Is Hard to Annotate: Properly labeling entailments between facts is difficult due to epistemic uncertainty and human cognitive biases. This issue complicates the creation of reliable training and evaluation datasets.
2. Vague and Ambiguous Factual Claims: Many factual claims used in datasets are imprecise, leading to ambiguity in their truth values. This necessitates careful selection and rigorous validation of dataset items.
3. Error Correction Requires Targeted, Model-Dependent Testing Strategies: Fixing errors in LLM outputs requires benchmarks that specify clear expectations for corrections. Generic factual claims are insufficient for evaluating how well editing methods rectify model errors.

Editable Beliefs in LLMs

1. LLMs as Agents or Agent Simulators: It is unclear whether LLMs simulate beliefs based on their training data or possess coherent, consistent beliefs of their own. This ambiguity affects the interpretation of belief updates and their expected outcomes.
2. LLMs as Agents or Databases: There is a debate on whether LLMs should be seen as passive knowledge repositories or active epistemic agents. RLHF and similar processes partially shape LLMs to be truth-oriented, yet they retain characteristics of both frameworks.
3. No Learned Belief Update Mechanism: The optimization processes used in model editing may not correspond to any innate belief revision mechanism within LLMs. This disconnect raises concerns about the efficacy of current edit strategies.
4. Unclear Mechanism to Edit Credences: LLMs express uncertainty through token probabilities and output semantics, complicating the adjustment of belief credences. Model edits must navigate these dual channels to achieve coherent belief updates.

Contributions and Findings

To address these issues, the paper introduces a formal testbed for model editing based on a semi-synthetic dataset with a predefined structure derived from Wikidata. This dataset allows for controlled evaluation by comparing an edited LM's outputs to the expected outcomes from a Bayesian model, which serves as an idealized rational agent.

Experimental Setup and Results

In terms of practical implementation, the authors:

1. Created a structured, semi-synthetic pretraining corpus ensuring coherent fact relationships.
2. Introduced a Bayesian evaluation framework providing exact posterior probabilities for test cases.
3. Trained an 83m parameter autoregressive Transformer on this corpus, demonstrating its ability to memorize and generate appropriate outputs from the data.

The experimental results focused on measuring three primary aspects post-editing:

1. Generative Accuracy: Evaluating whether the edited LM provides the correct outputs for updated inputs.
2. Probabilistic Coherence: Assessing the consistency between the LM's probabilities and Bayesian posteriors.
3. Logical Coherence: Checking adherence to logical principles in the LM's generated probabilities.

The key findings revealed that while the LM could learn specific updates (100% accuracy for direct edit requests), it generally failed to propagate these updates appropriately across related facts and maintain logical and probabilistic coherence. This underscores the gap between current model editing techniques and the ideal of rational belief revision.

Implications and Future Directions

The theoretical and empirical insights from this paper underline the necessity for more rigorous definitions and methodologies in model editing. Future research must focus on:

1. Refining the conceptual framework for model editing, specifying clearer goals and mechanisms for belief updates.
2. Developing robust, context-aware datasets that enable reliable evaluation of belief coherence and correction.
3. Identifying and leveraging potential innate mechanisms for belief revision within LLMs to enhance editing efficacy.
4. Exploring scalable solutions that balance computational costs with the demand for coherent, consistent model updates.

In conclusion, this paper sets the stage for a deeper understanding and more precise formulation of the model editing problem, advocating for a shift towards formalized, structured evaluation frameworks that can faithfully reflect the complexities of rational belief revision in LLMs.