---
title: Token Frequency Neurons in Transformers
url: https://www.emergentmind.com/topics/token-frequency-neurons
type: topic
---

# Token Frequency Neurons in Transformers

Token frequency neurons, also known as rare-token neurons, are specialized hidden units in the final MLP layer of transformer-based language models that exert a disproportionate causal effect on the prediction of low-frequency—or "rare"—tokens. These neurons have been systematically identified and analyzed in large language models (LLMs), revealing a distinctive, emergent organization and a mechanistic role in regulating rare-token outputs and fallback distributions. The study of token frequency neurons elucidates core aspects of sparse specialization, activation geometry, and heavy-tailed synaptic statistics within neural architectures [2505.12822, 2509.21163, 2406.16254].

## 1. Definition and Mathematical Characterization

A token frequency neuron is defined as an individual hidden unit in a transformer's final MLP layer that, when intervened upon, induces a large expected change in the model’s output probability or loss for rare target tokens. A token's frequency $f(r)$ is quantified as its unigram count divided by the total token count in the training corpus; rare tokens are those in the lower percentiles (e.g., below the median) of $f(r)$ [2505.12822, 2406.16254].

Mathematically, let $x$ denote the residual stream before the last MLP, with neuron $i$’s activation $n_i$. The mean-ablation intervention sets $n_i$ to its dataset mean $\bar n_i$, updating the downstream state:
$$
\tilde x^{(i)} = x + (\bar n_i - n_i) w^{(i)}_{\rm out}
$$
where $w^{(i)}_{\rm out}$ is the $i$-th output weight. The causal effect on the token-level loss is measured by
$$
\Delta \text{loss}(i) = \mathbb{E}_{x \sim D} \left| \mathcal{L}(\mathrm{LM}(x), y) - \mathcal{L}(\mathrm{LM}(\tilde x^{(i)}), y) \right|
$$
while per-token influence
$$
I_{i, r} = \mathbb{E}_{x: y=r} [-\log p(r | x) + \log p(r | \tilde x^{(i)})]
$$
quantifies neuron $i$’s direct impact on rare token $r$ [2505.12822, 2509.21163].

In an alternative formulation, token frequency neurons are defined as those whose MLP output weights, passed through the unembedding matrix $W_U$, are approximately parallel to the "log-frequency direction" $v_{\text{freq}}$ given by
$$
v_{\text{freq},j} = \log p_{\text{freq},j} - \text{mean}_k[\log p_{\text{freq},k}]
$$
Thus, the neuron’s contribution to logit $l_j$ is
$$
\Delta l_j \approx \alpha n_i v_{\text{freq},j}
$$
with $\alpha$ a proportionality constant. Such neurons adjust the model’s pre-softmax logits toward or away from the empirical unigram distribution, effectively serving as calibration mechanisms [2406.16254].

## 2. Influence Measurement and Discovery Methodology

The influence of a candidate neuron on rare tokens is quantified by comparing the model’s output distribution before and after mean ablation, typically using either per-token logit differences, changes in output probabilities, or token-level cross-entropy change:
$$
I_n(t) = \mathbb{E}_{x \sim \mathcal{D}} \left[ z_t(x) - z_t(x; n~\text{ablated}) \right]
$$
for token $t$, or
$$
I_n(t) = \mathbb{E}_{x \sim \mathcal{D}} \left[ p(y = t | x) - p(y = t | x; n~\text{ablated}) \right]
$$
[2509.21163].

Directional probes identify neurons whose output weights are highly aligned, in the unembedding space, with the corpus log-frequency vector $v_{\text{freq}}$, using cosine similarity:
$$
\cos \theta_i = \frac{W_U w^{(i)}_{\text{out}} \cdot v_{\text{freq}}}{\| W_U w^{(i)}_{\text{out}} \|~ \| v_{\text{freq}} \|}
$$
[2406.16254]. Combined with ablation-based mediation analysis—partitioning neuron effect into components along $v_{\text{freq}}$ (“direct effect”) and orthogonal subspaces (“total effect”)—this yields both statistical and causal signatures for token frequency neurons.

Experimentally, large language models such as Pythia-410M and GPT-2 exhibit small subsets of final-layer MLP units (typically 1–2% of total) whose ablation disproportionately shifts rare-token output probabilities and KL-divergence with respect to the unigram distribution [2505.12822, 2406.16254].

## 3. Three-Regime Specialization in Neuron Influence

When neurons are ranked by decreasing rare-token influence and plotted in log-log coordinates, three persistent regimes emerge during training [2505.12822, 2509.21163]:
- **Plateau regime:** The highest-influence (top $\sim$1–2%) neurons exhibit nearly flat influence, forming a plateau above the fitted power law. These are the rare-token neurons.
- **Power-law decay:** The next $\sim$10% of neurons descend according to $\log |\Delta \text{loss}(r)| \approx -\kappa \log r + \beta$, where $\kappa$ is the decay exponent.
- **Rapid-decay tail:** The remaining vast majority ($\sim$87%) of neurons exhibit influence that drops off faster than the power law.

These regimes are detected using segmented linear fits on log-log plots and change-point methods for local slope estimation. The plateau’s upward deviation $\delta(r)$ quantifies its separation from the power-law bulk and grows throughout training. Notably, common-token prediction lacks this structured hierarchy, following a simple power-law decay [2509.21163].

## 4. Subnetwork Structure and Activation Geometry

Despite being distributed across the final MLP layer, rare-token neurons form a coordinated functional subnetwork:
- **Low Effective Dimensionality:** Principal component analysis (PCA) of their activation matrices across diverse contexts reveals that rare-token neurons inhabit a low-dimensional subspace, as quantified by participation ratio $D_{\text{eff}}$:
$$
D_{\text{eff}} = \frac{\left( \sum_{i=1}^{K} \lambda_i \right)^2 }{ \sum_{i=1}^{K} \lambda_i^2 }
$$
or by the rank at which cumulative eigenvalue mass passes a threshold (e.g., 95%).
- **Co-activation and Avoidance:** Mean pairwise cosine similarity between rare-token neurons’ activations is high (≈0.41), while it is near zero (≈0.03) between rare-token and random neurons. Graph connectivity (edges for $\cos \theta_{ij}>0.5$) exposes a densely-connected rare-token subnetwork module, suggesting tightly coordinated specialization [2505.12822].

Importantly, community detection and spatial analysis show that rare-token neurons are not physically clustered; instead, they are spatially dispersed and do not form discrete layer-wise modules [2509.21163].

## 5. Heavy-Tailed Weight Statistics and Self-Organized Criticality

Under Heavy-Tailed Self-Regularization (HT-SR) theory, rare-token neuron groups exhibit weight-correlation matrices with heavy-tailed eigenvalue spectra. The tail index $\alpha_{\text{Hill}}$ is computed by the Hill estimator:
$$
\alpha_{\mathrm{Hill}} = \left[ \frac{1}{k} \sum_{i=1}^k \ln \left( \frac{\lambda_i}{\lambda_k} \right) \right]^{-1}
$$
with $k$ chosen to capture the power-law regime. Plateaus of rare-token neurons systematically have lower $\alpha_{\text{Hill}}$ (<2) compared to random neurons, compatible with self-organized criticality and the formation of sparse, specialist subnetworks in the absence of explicit architectural modularity [2505.12822, 2509.21163].

## 6. Training Dynamics and Functional Emergence

Longitudinal analyses during pretraining reveal that the specialization of token frequency neurons is not present at initialization. Instead:
- Early in training, influence curves lack a plateau, descending smoothly.
- Around the midpoint, a nascent plateau emerges among the top few percent of neurons, which grows in both height and separation from the bulk as training advances.
- Simultaneously, the Hill exponent for plateau neurons declines (from ∼3 to <2), and their effective activation dimension shrinks, indicating the crystallization of functional coordination and heavy-tailed specialization [2505.12822, 2509.21163].

This emergence aligns with distributed specialization rather than mixture-of-experts or modular routing: rare tokens access the coordinated subnetwork via standard attention circuits, as evidenced by similar attention patterns and low modularity $Q$ compared to random baselines.

## 7. Functional Significance, Calibration, and Practical Implications

Token frequency (rare-token) neurons implement a mechanistically interpretable fallback system for low-frequency tokens:
- **Calibration and Confidence Regulation:** These neurons shift the model’s output distribution toward the unigram (“default”) distribution, especially in high-uncertainty or low-signal contexts, mediating confidence hedging and error avoidance [2406.16254].
- **Editing and Robustness:** Selectively intervening on the plateau subnetwork supports model editing for rare-token behaviors and enhances reliability—pruning strategies can remove the rapid-decay neurons with little loss of rare-token fidelity, while plateau neurons should be preserved [2509.21163].
- **Avoidance of Dedicated Modules:** Token frequency neurons are universal, distributed, and do not rely on mixture-of-experts-style architectures or specialized routing pathways.

The functional behavior of token frequency neurons unifies theories of complementary learning systems and sparse coding, with rare-token handling localized to a compact, co-activated, heavy-tailed subnetwork emergent from standard training [2505.12822, 2509.21163, 2406.16254]. These findings have direct implications for interpretability, model calibration, and the optimization of computational resources in large-scale neural language models.

Source: https://www.emergentmind.com/topics/token-frequency-neurons