KnowledgeVIS: Interpreting Language Models by Comparing Fill-in-the-Blank Prompts (2403.04758v1)

Abstract: Recent growth in the popularity of LLMs has led to their increased usage for summarizing, predicting, and generating text, making it vital to help researchers and engineers understand how and why they work. We present KnowledgeVis, a human-in-the-loop visual analytics system for interpreting LLMs using fill-in-the-blank sentences as prompts. By comparing predictions between sentences, KnowledgeVis reveals learned associations that intuitively connect what LLMs learn during training to natural language tasks downstream, helping users create and test multiple prompt variations, analyze predicted words using a novel semantic clustering technique, and discover insights using interactive visualizations. Collectively, these visualizations help users identify the likelihood and uniqueness of individual predictions, compare sets of predictions between prompts, and summarize patterns and relationships between predictions across all prompts. We demonstrate the capabilities of KnowledgeVis with feedback from six NLP experts as well as three different use cases: (1) probing biomedical knowledge in two domain-adapted models; and (2) evaluating harmful identity stereotypes and (3) discovering facts and relationships between three general-purpose models.

• The paper introduces KnowledgeVIS, a visual analytics system that interprets BERT-based models using dynamic fill-in-the-blank prompts.
• It employs semantic clustering and coordinated visualizations like heat maps and scatter plots to reveal underlying model associations.
• Evaluations in biomedical and bias contexts demonstrate how KnowledgeVIS improves understanding of model reasoning and ethical deployments.

Interpreting BERT-Based LLMs with KnowledgeVIS

The paper "KnowledgeVIS: Interpreting LLMs by Comparing Fill-in-the-Blank Prompts" presents a human-in-the-loop visual analytics system designed to scrutinize LLMs, specifically focusing on BERT-based models. This system, KnowledgeVIS, aims to bridge the gap in understanding what these models have learned and how they apply this knowledge to downstream tasks. Utilizing a fill-in-the-blank approach, KnowledgeVIS provides researchers and engineers with interactive visualizations that illuminate the associations these models have ingrained during their training processes.

Overview of KnowledgeVIS

KnowledgeVIS integrates multi-dimensional interaction techniques with visual analytics to enhance the interpretability of LLMs like BERT. Key features include:

1. Prompt Interface: Users can create and modify fill-in-the-blank prompts. The system allows for flexible subject inputs within templates to generate diverse prompt variations, which is crucial for probing different types of relationships and associations within the model.
2. Prediction Clustering: A novel semantic clustering technique groups predicted words based on taxonomic similarity, providing a clear structure for users to analyze relationships between predictions.
3. Visualization Tools: The system employs multiple coordinated views:
   • A Heat Map for broad visibility of prediction probabilities.
   • A Set View that aids in comparing set memberships and rank through parallel tag clouds.
   • A Scatter Plot using a dust-and-magnet metaphor to examine the relationships across multiple prompts.

Evaluative Use Cases

KnowledgeVIS's utility is demonstrated through several use cases with different models and tasks:

• Biomedical Knowledge: By evaluating domain-specific models like SciBERT and PubMedBERT, the system identifies how grammar and phrasing impact the models' understanding and association capabilities, particularly in sensitive domains like healthcare.
• Identity Stereotypes: It uncovers biases in general-purpose models such as BERT and RoBERTa, revealing contextual gender, racial, and political stereotypes that may not be apparent through standard evaluation benchmarks.
• Knowledge Probing: The system compares large and small models (BERT vs. DistilBERT) for grasping complex reasoning, revealing differences in handling verb-based versus noun-based associations.

Implications and Future Work

KnowledgeVIS underscores the potential for human-in-the-loop systems to complement existing quantitative benchmarks, providing nuanced, qualitative insights into LLM performance. By surfacing and visualizing predictive associations, it allows practitioners to evaluate and iteratively improve model outcomes, especially for applications demanding high reliability and ethical considerations.

Future research could expand this approach to include other transformer-based models, explore automated prompt generation, and tackle multi-modal inputs for comprehensive interpretability. Additionally, embedding the system within model training processes could further elucidate model learning dynamics, potentially guiding the development of less biased and more robust LLMs.

In conclusion, KnowledgeVIS represents a significant step towards demystifying the logic and learning within BERT-based LLMs. Its focus on interpretability could inform better engineering practices, enhancing model transparency and trustworthiness, especially in complex, real-world applications where understanding emergent model behaviors is crucial.