Token Erasure as a Footprint of Implicit Vocabulary Items in LLMs (2406.20086v3)

Abstract: LLMs process text as sequences of tokens that roughly correspond to words, where less common words are represented by multiple tokens. However, individual tokens are often semantically unrelated to the meanings of the words/concepts they comprise. For example, Llama-2-7b's tokenizer splits the word "northeastern" into the tokens ['_n', 'ort', 'he', 'astern'], none of which correspond to semantically meaningful units like "north" or "east." Similarly, the overall meanings of named entities like "Neil Young" and multi-word expressions like "break a leg" cannot be directly inferred from their constituent tokens. Mechanistically, how do LLMs convert such arbitrary groups of tokens into useful higher-level representations? In this work, we find that last token representations of named entities and multi-token words exhibit a pronounced "erasure" effect, where information about previous and current tokens is rapidly forgotten in early layers. Using this observation, we propose a method to "read out" the implicit vocabulary of an autoregressive LLM by examining differences in token representations across layers, and present results of this method for Llama-2-7b and Llama-3-8B. To our knowledge, this is the first attempt to probe the implicit vocabulary of an LLM.

• The paper experimentally demonstrates that token erasure reveals implicit lexical units by showing diminished probe accuracy in last token positions.
• It introduces a novel heuristic to score the lexicality of token sequences, validated using Llama-2-7b and Llama-3-8B models.
• Findings improve model interpretability and tokenization strategies, offering practical tools for enhanced NLP applications.

A Comprehensive Analysis of "Token Erasure as a Footprint of Implicit Vocabulary Items in LLMs"

The paper "Token Erasure as a Footprint of Implicit Vocabulary Items in LLMs" authored by Sheridan Feucht, David Atkinson, Byron C. Wallace, and David Bau from Northeastern University, offers a nuanced investigation into the mechanisms underlying lexical representation in LLMs. Specifically, the authors introduce the phenomenon of "token erasure" and its implications for understanding how LLMs encode sequences of tokens into meaningful lexical units. This summary aims to encapsulate the salient points and contributions of the paper, with an emphasis on the empirical and theoretical implications for the field of NLP.

Central Hypothesis and Contributions

The paper hypothesizes that LLMs, through pretraining, develop an implicit vocabulary that enables them to map arbitrary token sequences to semantically meaningful units. These lexical items could include multi-token words, named entities, and idiomatic expressions, which the authors argue are treated as single units of meaning by LLMs despite their non-compositional nature.

Key contributions include:

1. Empirical Identification of the Token Erasure Effect: The paper presents evidence showing that the last tokens in multi-token words and named entities exhibit an "erasure" effect where information about preceding and current tokens is diminished or forgotten in early network layers.
2. Development of A Lexicality Heuristic: The authors propose a novel heuristic for scoring the "lexicality" of token sequences based on this erasure effect. This heuristic is employed to identify implicit vocabulary items within LLMs.
3. Application across LLM Architectures: The methodology and findings are validated using two different models—Llama-2-7b and Llama-3-8B—demonstrating the robustness of the proposed approach.

Methodological Overview

Linear Probing of Hidden States

To ascertain what the last token positions in multi-token sequences encode, the authors use linear probes trained to predict neighboring token values from hidden representations. Probes are trained on both Llama-2-7b and Llama-3-8b across all model layers to predict offsets within $[-3, -2, -1, 0, 1]$.

Observed Results:

• A pronounced drop in accuracy for predicting preceding tokens when examining the last token position of multi-token sequences.
• This "erasure" effect is posited to arise from an implicit process in early layers that converts token embeddings into meaningful units.

Validation and Further Probing

Using the CounterFact dataset, which comprises entity-rich prompts, the paper validates the token erasure effect by scrutinizing test performances of token probes on last subject tokens versus all other tokens. Results showcase a significant degradation in probe accuracy for last tokens, reaffirming the hypothesis regarding token erasure.

Building the Implicit Vocabulary

Leveraging these insights, the authors introduce an erasure score ($\psi$) that quantifies the "erasing" behavior across layers. Documents are segmented by identifying high scoring, non-overlapping sequences of tokens which exhibit erasure characteristics. The method effectively enumerates the implicit vocabulary items present in LLMs, highlighting their potential as lexical units.

Implications and Future Directions

Practical Applications

The identification of implicit vocabularies can enhance:

1. Model Interpretability: Understanding which tokens models inherently treat as lexical units can elucidate model behavior and decision-making processes.
2. Robust Tokenization Strategies: Refining tokenization methods to align with implicit lexical items can potentially improve model performance on downstream tasks.
3. Error Analysis and Debugging: Detecting when and where token erasure fails can help identify weaknesses in model training or pre-processing phase.

Theoretical Contributions

From a theoretical standpoint, this research advances the understanding of token embeddings and their transformation into higher-level semantic representations. The notion of implicit vocabulary storage challenges existing perspectives on tokenization and invites further exploration into how neural networks interpret and represent complex linguistic constructs.

Prospective Developments

Future work could extend these findings by:

1. Expanding to Diverse Languages: Investigating whether implicit vocabularies and the token erasure effect are consistent across different languages and language families.
2. Scaling to Various Model Sizes: Analyzing models of other scales and architectures to determine the generality of the erasure effect.
3. Incorporating Contextual Factors: Examining how context and token position within larger discourse structures might influence implicit vocabulary formation.

Conclusion

The paper "Token Erasure as a Footprint of Implicit Vocabulary Items in LLMs" presents compelling evidence for the implicit lexical processing capabilities of LLMs via the novel concept of token erasure. The paper not only provides a deeper insight into the inner workings of LLMs but also proposes practical tools for probing and understanding these sophisticated models. As LLMs continue to evolve, such foundational research will be pivotal in shaping the approaches and methodologies used in future NLP endeavors. The establishment of implicit vocabularies as functional units within LLMs paves the way for both theoretical advancements and practical enhancements in the field.