Language Overlap Neurons in Multilingual Models

Updated 17 July 2026

Language overlap neurons are groups in multilingual models that exhibit similar activation patterns across languages, blending language-specific and universal processing.
They are identified via methods like activation probability analysis and causal masking, which clarify their roles in semantic and syntactic tasks.
Empirical studies show that overlap varies by layer and task, enabling practical applications such as improved low-resource transfer and language control.

Searching arXiv for the cited work and closely related papers on neuron overlap in multilingual models. Language overlap neurons are neuron populations in multilingual LLMs whose functional or activation profiles are shared across more than one language. Across recent work, the term encompasses several related phenomena: neurons that are simultaneously labeled as language-specific for multiple languages under activation-based criteria, neurons whose masking causes asymmetric but non-zero degradation across languages, and neurons that act as shared detectors of semantic or syntactic structure across languages rather than as exclusive per-language modules (Le et al., 8 Jan 2026, Tang et al., 2024, Riemenschneider et al., 2 Jun 2025, Stańczak et al., 2022). The resulting picture is neither strict compartmentalization nor complete language agnosticism. Instead, multilingual models appear to contain a spectrum ranging from highly specialized neurons to broadly shared multilingual circuitry, with overlap patterns depending on the definition of specificity, the task, the model family, and the layer.

1. Definitions and conceptual scope

The most common early operationalization of language-specific neurons in multilingual LLMs is activation-based. In LAPE-style analyses, a neuron is regarded as language-specific when its activation probability is concentrated on one language or a small subset of languages, formalized through Language Activation Probability Entropy (LAPE) (Tang et al., 2024). In this framework, overlap neurons arise when a low-LAPE neuron has high activation probability for multiple languages; the Chinese–Japanese case is explicitly reported, with “approximately 25%” overlap for both languages in LLaMA-2-70B (Tang et al., 2024). Closely related activation-based work on decoder-only multilingual PLMs instead selects top- and bottom- $k$ neurons by average precision for language discrimination and defines overlap as the intersection of those sets, reporting “a slight overlap (< 5%) between languages” across English, German, French, Spanish, Chinese, and Japanese (Kojima et al., 2024).

A more recent line of work argues that activation preference is not sufficient to establish language specificity. CRANE explicitly criticizes the assumption that high activation on language $\ell$ implies functional importance for $\ell$ , stating that “activation does not imply functional necessity” (Le et al., 8 Jan 2026). In that framework, a neuron set $\mathcal{N}_\ell$ is language-specific to $\ell$ if masking it causes a larger relative performance degradation on $\ell$ than on any $\ell' \neq \ell$ , under a fixed intervention budget (Le et al., 8 Jan 2026). This redefinition moves the concept of overlap from co-activation toward shared causal contribution.

Several adjacent literatures generalize the overlap notion beyond language identity alone. In multilingual concept probing, overlap neurons are neurons whose average-precision expertise for the same semantic concept is shared across languages (Riemenschneider et al., 2 Jun 2025). In morphosyntactic probing, overlap is the intersection of top-50 neurons informative about a grammatical category in two languages (Stańczak et al., 2022). In task-conditioned multilingual analysis, overlap neurons are those activated across parallel inputs in multiple languages and are categorized as all-shared or partial-shared depending on whether they fire for all languages or only a subset (Wang et al., 2024). These definitions are not identical, but all of them instantiate a common idea: multilingual models reuse at least part of their neuron-level circuitry across languages.

2. From activation overlap to causal overlap

The strongest conceptual shift in the literature is the move from correlational overlap to causal overlap. CRANE, introduced as “Causal Relevance Analysis of Language-Specific Neurons in Multilingual LLMs,” analyzes standard Transformer architectures and treats each column of an MLP linear projection matrix as one neuron (Le et al., 8 Jan 2026). It uses Layer-wise Relevance Propagation and AttnLRP to obtain language-conditioned relevance distributions $\mathcal{R}_{n,\ell}$ for each neuron $n$ and language $\ell$ , then computes a layer-normalized kurtosis statistic to identify candidate neurons whose relevance is concentrated for one language more than for others (Le et al., 8 Jan 2026). Candidate selection is only a screening stage; language specificity is validated by inference-time masking.

Under this causal framing, overlap neurons are not defined by simultaneous firing alone. They are inferred when masking a set of neurons selected for one language also harms non-target languages, but less strongly. CRANE repeatedly describes this as a “consistent asymmetric pattern”: masking neurons relevant to a target language “selectively degrades performance on that language while preserving performance on other languages to a substantial extent” (Le et al., 8 Jan 2026). This implies language-selective but non-exclusive specialization, rather than disjoint partitions.

CRANE also introduces LangSpec-F1 to quantify this asymmetry under intervention. With target-language drop $\ell$ 0, maximum non-target drop $\ell$ 1, original score $\ell$ 2, and numerical constant $\ell$ 3, it defines

$\ell$ 4

$\ell$ 5

$\ell$ 6

High LangSpec-F1 indicates a neuron set whose causal effect is concentrated on the target language while collateral damage remains limited (Le et al., 8 Jan 2026). Values well below 1 are therefore direct evidence of overlap: the neurons are not exclusive to one language, even when their importance is asymmetric.

This causal perspective also clarifies a recurrent methodological disagreement in the literature. Activation-based methods such as LAPE can localize neurons that are statistically associated with a language, but CRANE reports that masking those neurons often barely degrades the corresponding language and can even improve it, whereas CRANE-selected neurons produce strong, targeted degradation (Le et al., 8 Jan 2026). A plausible implication is that some earlier reports of low overlap partly reflect the choice of activation-based identification criteria rather than purely intrinsic model structure.

3. Empirical patterns of overlap and specialization

Across the cited work, overlap is repeatedly observed, but its magnitude and interpretation depend on the object being probed. In decoder-only multilingual PLMs, activation-based average-precision probes found language-specific neurons to be “unique, with a slight overlap (< 5%) between languages,” and located them mainly in the first and last few layers (Kojima et al., 2024). This result supports a picture in which language identity is largely encoded by disjoint edge-layer circuits, with limited shared neuron membership at the level of top/bottom AP-ranked language discriminators.

By contrast, CRANE’s intervention-based results show that even neurons selected as language-specific typically remain functionally shared. On LLaMA2-7B-Base, masking Vietnamese CRANE neurons changes MMLU from 0.4579 to 0.3517, C-Eval from 0.3470 to 0.2816, and Belebele $\ell$ 7 from 0.3722 to 0.2233; the Vietnamese drop is largest, but English and Chinese also degrade (Le et al., 8 Jan 2026). Similarly, masking English CRANE neurons changes MMLU from 0.4579 to 0.3483, C-Eval from 0.3470 to 0.2801, and Belebele $\ell$ 8 from 0.3722 to 0.2811 (Le et al., 8 Jan 2026). These examples operationalize overlap as asymmetric non-zero transfer of damage under ablation.

A distinct but related line of evidence comes from morphosyntactic probing in multilingual encoders. For 43 languages and 14 morphosyntactic categories, the cross-lingual overlap between top-50 neurons is reported to be “about 20% of the top-50 neurons overlap on average” between two languages that express the same category (Stańczak et al., 2022). The extent varies by category: Number shows strong overlap, with 0.40 significant language-pair proportion in mBERT, 0.51 in XLM-R base, and 0.74 in XLM-R large, whereas Mood is much lower, with 0.00, 0.07, and 0.05 respectively (Stańczak et al., 2022). This supports the view that overlap is feature-specific: categories with more universal or smaller value inventories tend to share more neurons than categories with more diverse realizations.

Causal probing of subject–verb agreement neurons also distinguishes architectures. In multilingual autoregressive XGLM, cross-lingual neuron overlap for agreement is reported as significant for all structures, whereas in mBERT significant overlap is mostly limited to “across a PP” and other long-distance agreement structures (Mueller et al., 2022). At the same time, the paper concludes that in autoregressive multilingual LLMs most syntactic agreement neurons remain language-specific, even though some are cross-lingual (Mueller et al., 2022). This again points to a mixed regime rather than a single organizing principle.

4. Layer-wise organization and shared semantic space

A recurring result across disparate methodologies is that overlap is layer-dependent. Activation-based LAPE studies on LLaMA-2, BLOOM, and Mistral report that language-specific neurons are concentrated in top and bottom layers, with a pronounced U-shaped distribution in larger LLaMA-2 models (Tang et al., 2024). In LLaMA-2-70B, language-specific neurons are sparse in the middle layers, while cross-lingual sentence embedding similarity rises in the middle and falls in upper layers, suggesting a language-neutral semantic core bracketed by language-sensitive encoding and decoding regions (Tang et al., 2024).

CRANE does not claim a detailed hierarchical map, but it computes per-layer kurtosis contrasts and reports that some layers contain neurons with strong language-conditioned relevance concentration while others are more shared (Le et al., 8 Jan 2026). “Language Arithmetics” likewise finds language-specific neurons concentrated in deeper layers of Llama-3.1-8B, Mistral-Nemo-12B, and Aya-Expanse models, especially for non-Latin scripts, while related languages show overlapping neurons in their deep-layer FFNs (Gurgurov et al., 30 Jul 2025). In code LLMs, language-specific neurons are primarily in bottom layers, while neurons exclusive to each programming language tend to appear in the top layers, and middle layers are described as a shared concept space closer to English and programming-language keywords (Kargaran et al., 1 Jun 2025).

Independent evidence for a middle-layer shared space appears at representation level. Work on monolingual and multilingual BERTs, Whisper, and fMRI alignment reports that different languages converge on a similar embedding space especially in middle layers, with average cross-language sentence similarity in unilingual BERTs peaking around late-intermediate layers and mBERT peaking around layer 8 (Zada et al., 25 Jun 2025). The paper does not explicitly isolate neurons, but it interprets these middle-layer dimensions as cross-lingual conceptual overlap. A plausible implication is that some overlap neurons observed in multilingual LLM studies are local neuron-level manifestations of a broader shared semantic subspace.

Aya-23-8B provides a complementary decoder-only case. It exhibits language-agnostic processing through most layers, with strong language-specific specialization concentrated in final layers 27–31, while overlap between specialized neuron sets is shaped by typological relatedness and script similarity rather than a single English pivot (Trinley et al., 27 Jul 2025). This architecture therefore retains a strong middle-to-late shared multilingual regime while postponing sharper specialization to the end of the stack.

Overlap is not invariant across tasks. “Sharing Matters: Analysing Neurons Across Languages and Tasks in LLMs” classifies FFN neurons for parallel multilingual inputs as all-shared, partial-shared, specific, or non-activated (Wang et al., 2024). In XNLI and cross-lingual knowledge editing, all-shared neurons occupy a central functional role: when ablating all-shared neurons in BLOOMZ on XNLI, accuracy collapses from 53.8 to 16.7 in English, 41.8 to 3.5 in German, 50.3 to 10.1 in Spanish, 49.0 to 10.0 in French, and comparably for other languages, whereas removing specific neurons has minimal effect (Wang et al., 2024). This demonstrates that overlap neurons shared across all languages can dominate multilingual reasoning behavior.

The same paper shows that neuron behavior is highly input-dependent. The fraction of neurons that keep the same behavior type across all examples in fact probing is near zero, and even neurons classified as non-activated for one example rarely remain inactive across an entire dataset (Wang et al., 2024). This suggests that overlap should not be reified as a static global label on a neuron index; for many analyses it is a context-conditioned role.

Concept-level probing during multilingual pre-training offers another perspective. “Cross-Lingual Generalization and Compression: From Language-Specific to Shared Neurons” defines expertise vectors $\ell$ 9 over MLP neurons for concept $\ell$ 0 and language $\ell$ 1, where each entry is the average precision of a neuron at predicting concept presence (Riemenschneider et al., 2 Jun 2025). Cross-lingual overlap is then measured as the proportion of shared top- $\ell$ 2 concept experts: $\ell$ 3 The paper reports that by late training “about $\ell$ 4 of the top 500 concept-selective neurons are shared between any pair of languages,” and that these shared concept neurons concentrate in middle layers where language identification is suppressed (Riemenschneider et al., 2 Jun 2025). Here overlap is a mechanism of compression: language-specific concept encodings gradually converge into shared abstractions.

Overlap can also be measured for non-language axes. In “Isolating Culture Neurons in Multilingual LLMs,” overlap is formalized as $\ell$ 5, the intersection of language-specific and culture-specific neurons for a matched language–culture pair (Namazifard et al., 4 Aug 2025). The paper reports that on average 56.7% of culture-specific neurons are pure culture neurons, implying roughly 43.3% overlap with language neurons, and that pure culture neurons account for about 76.3% of the total effect of ablating culture-specific neurons (Namazifard et al., 4 Aug 2025). This indicates that language overlap need not mean inseparability: substantial overlap may coexist with identifiable language-independent subpopulations along other representational axes.

6. Applications, interventions, and open questions

Neuron-level overlap has already been used for control and transfer. Language-specific neuron activation and deactivation can steer output language in LLaMA-2, BLOOM, Mistral, and Aya models (Tang et al., 2024, Gurgurov et al., 30 Jul 2025). “Language Arithmetics” reports that additive activation plus deactivation of source-language neurons outperforms replacement-style interventions for language forcing, with 28.12% success at top 5% neurons on Llama-3.1 under the reported protocol (Gurgurov et al., 30 Jul 2025). The same study also finds that related languages benefit from activating each other’s neurons, which is consistent with overlap-mediated transfer (Gurgurov et al., 30 Jul 2025).

Bridge selection for low-resource transfer has likewise been tied explicitly to overlap neurons. BridgeX-ICL defines language overlap neurons as intersections $\ell$ 6 of high-frequency FFN neurons for two languages and builds an HSIC-based metric over their activations to select an optimal bridge language (Xu et al., 23 Aug 2025). Across 2 cross-lingual tasks and 15 language pairs, BridgeX-ICL improves bilingual lexicon induction performance by 6.02 percentage points on average for LLaMA-3-8B and 5.25 for Mistral-7B over zero-shot, and improves MRC accuracy by 6.03% and 4.48% respectively (Xu et al., 23 Aug 2025). This is one of the clearest practical uses of overlap neurons as an internal signal for multilingual transfer.

A broader interpretability implication comes from NeuronXA, which evaluates cross-lingual alignment through FFN neuron-state overlap rather than sentence embeddings (Huang et al., 20 Jul 2025). Using only 100 parallel sentence pairs, it achieves a Pearson correlation of 0.9556 with multilingual downstream-task performance and 0.8514 with transferability (Huang et al., 20 Jul 2025). The paper’s layer-wise analysis again locates maximum alignment in middle layers, reinforcing the claim that semantic overlap is concentrated there.

Several limitations recur across the literature. Many studies analyze only FFN/MLP neurons rather than full circuits; several cover only a small number of languages or one model family; and multiple papers note that neuron-level overlap may depend strongly on the probing task, thresholding scheme, or intervention granularity (Le et al., 8 Jan 2026, Xu et al., 23 Aug 2025, Wang et al., 2024, Stańczak et al., 2022). CRANE specifically notes that kurtosis is only one statistic and masking is a coarse intervention, while other work emphasizes that overlap patterns can be input-dependent rather than fixed neuron identities (Le et al., 8 Jan 2026, Wang et al., 2024). This suggests that “language overlap neurons” should not be treated as a single stable object independent of methodology.

A plausible synthesis is that multilingual models contain multiple forms of overlap simultaneously. There are activation-overlap neurons that respond to several languages, causal-overlap neurons whose ablation harms several languages asymmetrically, concept-overlap neurons that encode the same semantics across languages, and task-shared neurons that become multilingual hubs only under specific inference conditions. Across papers, the most stable regularities are that overlap is real, structured, and often strongest in middle layers, while strict language exclusivity is limited and increasingly hard to sustain once causal or semantic criteria are used (Le et al., 8 Jan 2026, Riemenschneider et al., 2 Jun 2025, Huang et al., 20 Jul 2025).