Neuron Populations Exhibit Divergent Selectivity with Scale

Abstract: We investigate whether neuron populations within neural networks evolve predictably with scale, extending scaling laws beyond macroscopic observables such as loss. To probe this question, we study Rosetta Neurons, a previously characterized class of neurons whose activation patterns are similar across independently trained models (Dravid et al., 2023). In separate analyses of LLMs up to 30B parameters and vision models up to 5B parameters, we observe that the population of Rosetta Neurons follows a sublinear power law in model size, growing in absolute number but occupying a shrinking fraction of the total neuron count. We further observe a Neuron Polarization Effect: Rosetta Neurons become more selective and increasingly monosemantic with scale, separating from a growing non-Rosetta population that remains less selective. An analytical model balancing feature utility against limited neuron capacity explains the sublinear power-law scaling and this polarization effect. Finally, we find that Rosetta Neurons become more domain-specialized with scale and illustrate their selectivity through a targeted data-filtering case study for continued pretraining. Our results point to a scaling law for interpretable, shared neuron-level structure, linking model size to systematic changes in neuron universality, selectivity, and specialization.

• The paper introduces a predictive framework demonstrating that recurring 'Rosetta Neurons' follow a sublinear power law with increasing model size.
• The methodology employs mutual nearest-neighbor matching via Pearson correlation to reliably identify monosemantic neurons across independently trained models.
• The findings offer practical insights for enhancing model interpretability and targeted data filtering by bridging scaling laws with internal neuron specialization.

Divergent Selectivity and Scaling Laws in Neuron Populations

Introduction

"Neuron Populations Exhibit Divergent Selectivity with Scale" (2606.03990) systematically investigates how neuron-level representational structure in neural networks changes with scale, introducing a predictive framework for understanding interpretable feature emergence and selectivity among neuron populations. Focusing on the population termed "Rosetta Neurons"—individual MLP units that are recurrently discovered across independently trained models—the paper extends the paradigm of scaling laws to the internal organization and selectivity of neuron cohorts, offering both empirical findings and an analytical model that together elucidate the interplay between capacity, monosemanticity, and specialization.

Figure 1: Neuron populations across scale: As model size grows, Rosetta Neurons increase in absolute count but decrease as a fraction, with boosted selectivity and specialization.

Methodology and Formalization of Neuron Recurrence

The analysis is anchored on Rosetta Neurons, identified as mutual nearest-neighbor neurons across two (or more) independently-initialized models trained on the same data. The paper quantitatively compares neuron activations using Pearson correlation across highly-aligned input positions, utilizing canonicalized spans for both text and image domains to ensure robust cross-model comparison independent of tokenizer or grid specifics.

To ensure the reliability and interpretability of matches, only neurons that are each other's closest neighbors (mutual k-nearest, typically $k=1$) are retained. This process is visualized in the pipeline figure below, which underscores the cross-modal applicability and the coherence of matched neuron firing:

Figure 2: Rosetta Neuron identification method—matched neurons display high activations for the same input features across models, indicating shared conceptual alignment.

Empirical Scaling Laws for Recurring Neuron Populations

The central empirical finding is that the number of cross-model recurring Rosetta Neurons, $|\mathcal{R}|$, follows a robust sublinear power law with respect to the total neuron count, $N$, in both language and vision models:

$$|\mathcal{R}| \propto N^\alpha,\ \ \text{with}\ \ 0.5 < \alpha < 0.7,$$

across all examined families in both modalities. The power law fit achieves $R^2 \approx 0.99$, confirming the regularity and predictive nature of this phenomenon.

This scaling is not an artifact of architectural scale or correlation structure in untrained random networks; the trend is absent when either training is omitted or token alignment is broken, indicating that the Rosetta population reflects learned, data-driven internal organization rather than mechanistic chance.

Analytical Model: Capacity Allocation and the Neuron Polarization Effect

To explain the empirical sublinearity and population polarization, the work introduces a capacity-allocation model motivated by feature superposition theory. Features are ordered by importance with a power-law spectrum, and neuron capacity is allocated so that only high-importance features are isolated monosemantically, while lower-importance features are subject to distributed, polysemantic superposition.

The model predicts the emergence of a "Neuron Polarization Effect": the selective Rosetta population becomes more specialized and interpretable with scale, while the non-Rosetta majority becomes increasingly polysemantic, mixing many features without clean isolation. The Rosetta frontier (the feature importance cutoff for clean isolation) expands sublinearly with neuron count. This is depicted in the following schematic, clarifying the origin of divergent selectivity.

Figure 3: Feature-isolation frontiers: Only top-ranked features (by importance) are monosemantically represented as Rosetta Neurons; their count grows as a sublinear power law with network size.

Quantitative and Qualitative Evidence for Polarization and Specialization

Neuron Polarization Across Scale

Empirical analyses validate the model's prediction of growing selectivity. In LLMs (e.g., Pythia), the excess kurtosis of output token projections (a proxy for monosemantic selectivity) of Rosetta Neurons increases rapidly with scale, while non-Rosetta neurons stay near the baseline. In vision models (OpenCLIP), monosemanticity judged by a multimodal LLM grows for Rosetta Neurons and declines for the non-Rosetta background as model size increases.

Figure 4: Vocabulary-space neuron selectivity in Pythia increases with scale for Rosetta Neurons, indicating greater monosemanticity.

Emergence of Domain Specialization

In addition to pure selectivity, the Rosetta set exhibits progressive domain specialization. As LLMs scale, the top-activating contexts for Rosetta Neurons are increasingly concentrated in specialized domains such as code and mathematics, in contrast to their smaller or non-Rosetta counterparts. This finding is robust against differences in dataset exposure, reflecting internal representational restructuring and the capacity to isolate rarer, more specific features.

Figure 5: Rosetta Neuron top activations in Pythia shift toward code/math with network scale, exceeding their baseline dataset frequency.

Case Study: Single-Neuron Domain Filtering

A controlled experiment on CodeSearchNet reveals that an individual Rosetta Neuron can be sufficiently selective to act as a functional domain filter. Filtering with the top-activating Rosetta Neuron for JavaScript recovers the target domain with F1=0.98, enabling continued pretraining that matches oracle-level test perplexity. Non-Rosetta neurons do not approach this effectiveness, underlining the practical downstream utility of polarization and selectivity.

Qualitative Visualizations

The qualitative analysis of top activations (Figures 7–17) further substantiates the findings. Rosetta Neurons at all scales consistently activate for semantically coherent and interpretable concepts—e.g., function signatures in Python, Markdown syntax, or specific object categories in vision models—while non-Rosetta neurons fire on heterogeneous or mixed contexts, reinforcing the quantized separation between monosemantic and polysemantic populations imposed by scaling.

Implications and Theoretical Significance

These results have several implications:

• Predictive Model Internal Structure: Unlike classical scaling laws that only address loss or perplexity, this work shows that interpretable, universal feature representations undergo their own scaling dynamics, leading to predictable enrichment of monosemantic, specialized neurons.
• Interpretable Model Editing and Data Filtering: The emergence of robust, domain-specialized units offers paths toward targeted model editing, domain-specific data curation, and enhanced mechanistic interpretability.
• Limits of Neuron Basis Sparsity: The finding that the Rosetta fraction decreases with scale, and that a majority of representational capacity remains polysemantic, highlights nontrivial challenges for basis decomposition and monosemantic interpretability at extreme model scales.
• Theory/Empiricism Synthesis: The analytical framework aligns with the empirical scaling law, providing a capacity-constrained, feature-competition-based explanation for the selective frontier, and motivating the development of future theories on representational specialization and the modularity of internal circuits.

Conclusion

This paper rigorously demonstrates that the population of recurring, interpretable neurons exhibits systematic scaling behavior, characterized by sublinear power-law growth and sharply increasing selectivity and specialization with network scale. Theoretical modeling based on superposition and feature importance accurately predicts both the magnitude and qualitative direction of these effects. These findings bridge macroscopic model scaling laws and microscopic representational organization, pointing to foundational regularities in how deep learning models allocate and specialize capacity as they scale, with substantial implications for interpretability, transfer, and future network design.