KEditVis: A Visual Analytics System for Knowledge Editing of Large Language Models

Abstract: LLMs demonstrate exceptional capabilities in factual question answering, yet they sometimes provide incorrect responses. To address this issue, knowledge editing techniques have emerged as effective methods for correcting factual information in LLMs. However, typical knowledge editing workflows struggle with identifying the optimal set of model layers for editing and rely on summary indicators that provide insufficient guidance. This lack of transparency hinders effective comparison and identification of optimal editing strategies. In this paper, we present KEditVis, a novel visual analytics system designed to assist users in gaining a deeper understanding of knowledge editing through interactive visualizations, improving editing outcomes, and discovering valuable insights for the future development of knowledge editing algorithms. With KEditVis, users can select appropriate layers as the editing target, explore the reasons behind ineffective edits, and perform more targeted and effective edits. Our evaluation, including usage scenarios, expert interviews, and a user study, validates the effectiveness and usability of the system.

• The paper introduces KEditVis, a visual analytics system that enables efficient, targeted knowledge editing in large language models to minimize unintended model drift.
• The system employs interactive layer selection using cosine similarity and token projection analysis along with multi-metric scheme comparisons for precise edits.
• User studies and scenario evaluations confirm that KEditVis outperforms automated methods, delivering statistically significant improvements in editing accuracy and usability.

KEditVis: Visual Analytics for Knowledge Editing in LLMs

Motivation and Context

The proliferation of LLMs has brought unprecedented advances in factual question answering, yet the persistence of outdated or erroneous factual knowledge remains an unsolved challenge. Traditional fine-tuning for knowledge updates is resource-intensive and prone to undesirable side effects such as overfitting and knowledge drift. Knowledge editing methods—particularly local weight modification approaches—have emerged as efficient solutions for targeted factual correction. However, current automated editing workflows lack transparency, especially regarding layer selection and granular evaluation, hindering the identification of optimal editing strategies and leading to issues such as over-editing and insufficient edit efficacy.

System Design and Methodology

KEditVis introduces a comprehensive visual analytics environment for knowledge editing of LLMs. The system is architected around several interlinked components: an interactive LLM chat view for fact exploration and prompt generation, automated and manual management tools for facts and prompts, and a suite of visualizations that expose internal model behaviors both before and after editing. The interface integrates knowledge graphs for semantic navigation and guides users through the editing process using layered visualizations of token probabilities and cosine similarity metrics for model layer selection.

Figure 1: The KEditVis interface combines LLM chat, knowledge graph navigation, edit management, layered internal model visualizations, set-based scheme comparison, detailed output analysis, and global impact drift visualization.

Edit View and Interactive Layer Selection

Central to KEditVis is the edit view, which facilitates dynamic layer selection informed by both cosine similarity and token projection analysis. The visualization encodes per-layer cosine similarity between input and hidden state vectors, leveraging prior interpretability research suggesting that lower similarity correlates with layers critical for knowledge transformation [hong2024interpretability]. Complementary token ranking charts reveal the probability trajectories of key tokens across layers, highlighting where factual content is integrated and propagated. Users iteratively select layer ranges for editing, supported by system-generated recommendations that identify promising sub-ranges within broader selection schemes.

Scheme Comparison and Multi-Criteria Analysis

KEditVis integrates set visualization via wireframes and tables to compare multiple editing schemes in terms of both layer overlap and performance across several metrics: efficacy (ES), paraphrase success (PS), neighborhood success (NS), reference score (RS), and generation entropy (GE). This supports nuanced decision-making and fosters exploration beyond fixed or purely automated approaches. Detailed results for prompts are rendered alongside bolded output diffs, facilitating granular comparison of editing impact.

Figure 2: Set visualization and SUS user study outcomes, revealing both intuitive scheme relationships and high user satisfaction scores.

Evaluation of Global Edit Impact

The system provides a drift analysis scatter plot using t-SNE projection of hidden states, enabling global assessment of the edit’s semantic footprint across test prompts. This ensures that targeted edits do not induce catastrophic knowledge drift or degrade model performance outside the intended scope.

Scenario-Based Evaluation

Two practical scenarios were executed: counterfactual knowledge editing and factual error correction, both demonstrating the iterative analysis and refinement enabled by KEditVis.

Scenario I: Counterfactual Editing

An experiment involved editing the fact “iPhone is developed by Microsoft.” By leveraging the knowledge graph and layered visualizations, users identified over-editing spanning too many layers, manifesting as incoherent output repetitions and semantic confusion. Subsequent scheme refinement focused on narrower layer ranges, validated through multi-metric and detailed output comparisons, ultimately yielding precise edits without damaging similar knowledge.

Figure 3: Layer-wise and scheme-wise analytics for counterfactual editing on “iPhone,” including knowledge graph navigation and drift evaluation.

Scenario II: Error Correction

An edit targeting the first recipient of the Turing Award illustrated that single-layer modifications, even with prominent cosine similarity, are insufficient for comprehensive knowledge representation updates. Optimal schemes spanned multiple contiguous layers exhibiting correlated trends in both metrics, producing outputs with increased confidence and minimal global disruption.

Figure 4: Layer selection guided by knowledge graph and ranking chart analysis for Turing Award error correction, contrasting pre- and post-edit model behavior.

Quantitative and Qualitative Outcomes

User studies (n=12) and expert interviews corroborate the effectiveness and usability of KEditVis. Participants consistently outperformed fixed-layer and automated selection baselines, achieving statistically significant improvements in editing metric scores across multiple tasks. The System Usability Scale (SUS) averaged 86.25, placing KEditVis in the top decile for usability. Detailed analysis revealed that human-in-the-loop decision-making—supported by visual analytics—achieves more precise and robust edits, particularly where automated metrics or selection methods fail.

Implications and Future Directions

KEditVis demonstrates that interactive, multi-perspective visualization can overcome key bottlenecks in knowledge editing workflows. By making internal model dynamics interpretable and actionable, it enables users to circumvent the limitations of static or automated layer selection and guides context-aware intervention. The approach is generalizable across editing algorithms and LLM architectures, enabling its extension to broader tasks such as model distillation or neuron-level editing. Future work will address scalability in layer and fact selection, explore efficient recommendation systems for editing schemes, and deepen comparative studies across editing methods and LLM variants.

Conclusion

KEditVis advances knowledge editing in LLMs by integrating interactive visual analytics for layer selection, scheme comparison, and output evaluation. The system empowers practitioners to perform targeted, effective edits with minimal knowledge drift, substantiated by strong qualitative and quantitative results. As ascertained in scenarios and user studies, human-guided, visually informed editing surpasses automated layer selection, marking a significant step towards transparent, reliable LLM maintenance.

Figure 5: Example of knowledge editing, illustrating the targeted update of factual knowledge within an LLM.