Language Neurons in Multilingual Models

• Language neurons are specific hidden units in transformer models that exhibit selective activation patterns tied to one or more languages.
• They are classified into language-specific, language-related, and language-agnostic types using metrics like LAPE and causal relevance tests.
• Interventions such as deactivation and amplification validate their roles in multilingual inference, enhancing controlled language output and translation.

Language neurons are a functional and mechanistic construct emerging from empirical investigations of LLMs, referring to distinct subsets of network hidden units whose activation is tightly coupled to language identity or language-specific linguistic phenomena. The language neuron paradigm encompasses language-specific, language-related, and language-agnostic neurons, each defined by precise statistical or causal criteria regarding their firing patterns and contribution to multilingual processing. Their identification, functional dissection, and targeted manipulation yield new explanatory power for how LLMs internalize and execute language discrimination, multilingual transfer, and cross-linguistic reasoning.

1. Formal Definitions and Characterizations

Language neurons are defined with respect to their selective activation across languages, typically in the feed-forward sublayers of transformer models. For neuron $(i,j)$ in layer $i$ and language $k$, the activation probability is computed as $$p_{(i,j)}^k = \mathbb{E}_{x \in D_k} [I(\text{SiLU}(x^{(i)} W^{(i)})_j > 0)]$$, where $x^{(i)}$ is the input, $W^{(i)}$ the layer weights, and $I$ the indicator function (Zhang et al., 27 May 2025).

The canonical classification partitions neurons into three non-overlapping types:

• Language-specific neurons: Active above a threshold ($\tau$) in exactly one language.
• Language-related neurons: Active above $\tau$ in several, but not all, languages.
• Language-agnostic neurons: Active above $\tau$ in every language.

A widely used quantification is Language Activation Probability Entropy (LAPE) (Tang et al., 2024, Gurgurov et al., 30 Jul 2025), which normalizes language-wise activation probabilities into a probability vector and computes the entropy:

$\text{LAPE}_{i,j} = -\sum_{k=1}^L p'_{(i,j)}^k \log p'_{(i,j)}^k$

Neurons with lowest entropy (highest selectivity) are candidate language neurons. Further functional definition has been advanced by causal relevance: A neuron is language-specific only if its ablation induces a significant, selectively degradative effect on that language’s performance while minimally affecting others, as formalized by kurtosis-based relevance and the LangSpec-F1 metric (Le et al., 8 Jan 2026).

2. Identification Methodologies and Taxonomies

Multiple pipelines for language neuron identification have been rigorously developed and compared:

• Entropy-max thresholding: Select neurons with low-normalized entropy and high peak activation; assign types based on thresholded activation support sizes (Zhang et al., 27 May 2025).
• Kurtosis-based relevance (CRANE): Select neurons for which high-kurtosis in language-conditioned relevance attests to rare, functionally critical contributions; ablation tests provide causal grounding (Le et al., 8 Jan 2026).
• Average-precision ranking: Select neurons whose averaged activations best discriminate between a positive (target language) and negative (all others) class over evaluation data (Kojima et al., 2024).

Fine-grained algorithms (see Table below) output three sets: specific, related, agnostic. Intervention-free, these methods do not require gradient-based optimization and are robust to scaling across model and language inventory.

Method	Main Criterion	Intervention for Validation
LAPE	Entropy of activations	Perplexity, language scores (Tang et al., 2024)
CRANE	Relevance kurtosis	Targeted ablation, LangSpec-F1 (Le et al., 8 Jan 2026)
Average-Precision	Class discrimination	Forced activation, output change (Kojima et al., 2024)

Layer-wise analyses consistently show that language neurons are non-uniformly distributed—often peaking in the lowest and highest layers of decoder stacks for language-specific neurons, while the middle layers are enriched for language-agnostic (shared) units (Zhang et al., 27 May 2025, Tang et al., 2024, Kojima et al., 2024, Gurgurov et al., 30 Jul 2025, Wang et al., 2024, Rahmanisa et al., 30 Jul 2025).

3. Functional Roles in Multilingual Model Computation

Language neurons underpin a modular workflow in multilingual inference, as elucidated in transformers with hierarchical specialization:

1. Multilingual understanding (input projection): Early layers peak in language-sensitive neurons, mapping input tokens into a representation that encodes language membership and surface linguistic form (Zhang et al., 27 May 2025).
2. Shared semantic space reasoning: Middle layers lose language-specificity, instead concentrating language-agnostic units that support interlingual semantic abstraction.
3. Multilingual output space transformation: In later layers, language-sensitive neurons reassert, guiding shared semantics toward the output distribution of the target language.
4. Vocabulary mapping: Final layers see both related and agnostic neurons active, mediating the alignment of output into the full multilingual vocabulary.

Causal interventions validate these roles: Deactivation of language-specific neurons increases perplexity or impairs language accuracy only in the target language; activation or amplification can force or bias output language without loss of task utility (Tang et al., 2024, Kojima et al., 2024, Gurgurov et al., 30 Jul 2025, Rahmanisa et al., 30 Jul 2025, Saha et al., 1 Feb 2026).

Instruction alignment reduces extreme language-specificity and increases the population and influence of related and agnostic neurons, facilitating zero-shot transfer, especially for low-resource languages—the "spontaneous multilingual alignment" effect (Zhang et al., 27 May 2025).

4. Interventions, Steering, and Control

Empirical studies demonstrate robust manipulation protocols for language neurons:

• Deactivation (zeroing): Suppressing identified neurons disables the targeted language without broadly affecting others (Tang et al., 2024, Kojima et al., 2024, Le et al., 8 Jan 2026).
• Amplification (patching): Injecting precomputed median or max activations for identified neurons reliably steers model outputs toward the desired language (>90% success rate in LSS metric) (Rahmanisa et al., 30 Jul 2025).
• Arithmetic operations: Systematic additive or multiplicative activation editing ("language arithmetics") can be used to enforce, suppress, or swap languages (Gurgurov et al., 30 Jul 2025).

Self-steering (activating a language's own neurons) can marginally improve fluency and BLEU in translation, especially in low-resource scenarios; cross-steering is generally deleterious, supporting the view that language-specific neurons are not a handle for productive cross-lingual transfer (Rahmanisa et al., 30 Jul 2025, Mondal et al., 21 Mar 2025).

Recent advancements extend interventions to low-rank subsystems (Neural FOXP2), where language preference is controlled by a sparse, low-rank subspace learned via sparse autoencoders; targeted subspace shifts reweight defaultness toward non-English languages without deleterious impact on task accuracy (Saha et al., 1 Feb 2026).

5. Dynamics, Sharing, and Polysemanticity

The static "language neuron" definition is complicated by the dynamic, input- and task-dependence of neuron function. Recent work partitions neurons into all-shared (activate for all languages), partial-shared, specific, and non-activated, revealing that all-shared units dominate attribution and are crucial for robust cross-lingual reasoning and editing (Wang et al., 2024).

Overlap analyses show that high-resource and typologically similar languages share up to 75% of their most-informative neurons, while distant or low-resource languages have more dedicated, less overlapping circuits (Stańczak et al., 2022, Gurgurov et al., 30 Jul 2025). Orthographic distance (e.g., Latin vs. non-Latin scripts) predicts more segregated neuron populations.

Polysemanticity—where a neuron supports multiple functions or concepts—remains pervasive, motivating multi-modal interpretability frameworks such as NeuronScope, which decomposes activations into atomic semantic modes for precise explanation and targeted editability (Liu et al., 7 Jan 2026).

6. Broader Interpretability and Neurobiological Parallels

Psycholinguistic probing has directly linked the existence of task-competence to the localization of specialized "competence neurons" in LLMs: abilities such as implicit causality and gender-sound association are mediated by discrete, causally necessary neurons, while absent competences (e.g., sound-shape association) correlate with the lack of such neuron specialization (Duan et al., 2024).

Parallel investigations of human brain language-selective assemblies reveal that phonetic-selective "phoneme neurons" emerge early in auditory regions, while lexical-level ("word neurons") develop in higher associative cortices through childhood, following a trajectory echoed by LLMs' internalization of phonetic-to-lexical-to-semantic representations (Evanson et al., 5 Dec 2025).

Specialized subclasses such as "repetition neurons" instantiate skill-level circuits responsible for degenerate generation behavior, demonstrating the tractability of neuron-level explanations for emergent capabilities and failure modes (Hiraoka et al., 2024).

7. Implications, Limitations, and Future Directions

Language neuron research provides foundational mechanisms for interpretability, targeted model editing, and principled control of multilingual behavior (Tang et al., 2024, Zhang et al., 27 May 2025, Le et al., 8 Jan 2026, Gurgurov et al., 30 Jul 2025). However, interventions can fail when polysemanticity, subcircuit entanglement, or unintended off-target effects disrupt downstream utility (Mondal et al., 21 Mar 2025). Cross-lingual transfer remains only weakly accessible through language neurons, and true modularity is elusive except in high-resource scenarios.

Future efforts focus on: (i) refining causal identification across context and task; (ii) dissecting interactions with attention mechanisms; (iii) scaling to ultra-multilingual or code-mixed regimes; and (iv) bridging model and biological language assembly insights for unified computational neuro-linguistics (Zhang et al., 27 May 2025, Saha et al., 1 Feb 2026, Evanson et al., 5 Dec 2025).

Language neurons thus represent both a concrete mechanistic handle and a dynamic substrate for the ongoing elucidation and engineering of multilinguality in artificial and biological systems.