Neurons in Large Language Models: Dead, N-gram, Positional (2309.04827v1)

Abstract: We analyze a family of LLMs in such a lightweight manner that can be done on a single GPU. Specifically, we focus on the OPT family of models ranging from 125m to 66b parameters and rely only on whether an FFN neuron is activated or not. First, we find that the early part of the network is sparse and represents many discrete features. Here, many neurons (more than 70% in some layers of the 66b model) are "dead", i.e. they never activate on a large collection of diverse data. At the same time, many of the alive neurons are reserved for discrete features and act as token and n-gram detectors. Interestingly, their corresponding FFN updates not only promote next token candidates as could be expected, but also explicitly focus on removing the information about triggering them tokens, i.e., current input. To the best of our knowledge, this is the first example of mechanisms specialized at removing (rather than adding) information from the residual stream. With scale, models become more sparse in a sense that they have more dead neurons and token detectors. Finally, some neurons are positional: them being activated or not depends largely (or solely) on position and less so (or not at all) on textual data. We find that smaller models have sets of neurons acting as position range indicators while larger models operate in a less explicit manner.

Exploring Neuronal Activity in LLMs: A Lightweight Study

Introduction

Researchers have found a resource-efficient way to probe the inner workings of LLMs, specifically the OPT model family, using a set-up that can be run on just a single GPU. Their findings reveal interesting insights into the nature of neuronal activity across different scales of model complexity, ranging from 125 million to 66 billion parameters. This blog post explores the significance of these findings and their implications for the understanding and future development of LLMs.

Neuronal Sparsity and Activation

The researchers observed a consistent pattern across the models: larger models tended to have a higher proportion of non-responsive, or "dead", neurons. Surprisingly, over 70% of neurons in some layers of the largest models never activated regardless of the data presented. This sparsity is mostly prevalent in the first half of the models while the latter half remains highly activated.

• Sparse early layers: The early layers of these models show significant sparsity, indicating that they might be encoding simpler and more discrete features, such as specific tokens or n-grams.
• Larger models, more sparsity: As the size of the model increases, the sparsity also increases. The implication here is that larger models seem to afford the luxury of dedicating neurons to very specific functions, which may not be necessary in most contexts.

The Role of Active Neurons

The research further distinguishes the functions of these active neurons. Many of these are strongly responsive to very specific types of inputs, particularly tokens or n-grams.

• Token and N-gram Detectors: Large numbers of neurons are dedicated to recognizing specific tokens or sequences of tokens. Interestingly, the function of these neurons extends beyond simple detection— they also actively suppress the representation of the tokens they detect from the ongoing informational flow within the model. This dual functionality suggests a sophisticated mechanism whereby the model not only recognizes but also dynamically regulates the influence of specific input features on its output.

Positional Encoding and Its Implications

Apart from feature detection, some neurons are specialized in encoding positional information. Their activation is heavily dependent on the position of tokens in the input sequence rather than the textual content itself.

• Position-dependency Varies by Model Size: Smaller models tend to use positional information more explicitly. In contrast, larger models encode position in a more abstracted manner, which might help in handling more complex contextual relationships that are less about token position and more about inter-token dynamics.
• Collaborative Encoding: Neurons appear to operate in a sort of team-like manner with overlapping responsibilities for covering various position ranges, ensuring comprehensive sensitivity to positional cues across the model.

The Path Forward

These findings offer a richer understanding of how LLMs manage and utilize their expansive neural landscapes. They reveal a nuanced picture of information processing where different types of neurons have specialized roles that contribute to the model's overall functionality. Such insights are invaluable for both theoretical and practical advancements in AI. They help in refining model architectures and could guide efforts in making these models more efficient and interpretable.

Analyzing models on single GPUs makes this type of research highly accessible, encouraging broader participation in the field and potentially accelerating innovation. Future research could build on these findings to explore how these specialized neurons contribute to the emergent behavior of LLMs in complex tasks and whether similar patterns exist across other families of models. This can lead to more targeted and efficient model designs that capitalize on these inherent efficiencies and specializations.