NeuronXA: Neuron State Cross-Lingual Alignment

• The paper introduces NeuronXA, a method that measures neuron-level activation overlaps to evaluate and improve multilingual LLM performance.
• It employs high-dimensional FFN activation patterns that strongly correlate with downstream tasks, outperforming traditional sentence-level benchmarks.
• The methodology, validated on various transformer models, offers actionable insights for enhancing cross-lingual transfer even in low-resource settings.

Neuron State-Based Cross-Lingual Alignment (NeuronXA) is a principled methodology for evaluating and enhancing the multilingual capabilities of LLMs by leveraging neuron-level activation patterns instead of conventional sentence-level embeddings. Drawing on neuroscientific principles, NeuronXA quantifies semantic alignment by measuring the state overlap of individual feed-forward network (FFN) neurons in transformer architectures across languages, yielding finer-grained and more semantically grounded metrics of cross-lingual transfer and understanding. Compared to previous benchmarks based on token-pooling or sentence encodings, NeuronXA exploits high-dimensional neuron-state vectors and their alignment, demonstrating strong correlations with downstream performance and robustness even for low-resource and long-tail languages (Huang et al., 20 Jul 2025).

1. Theoretical Foundations and Motivation

NeuronXA is inspired by the neuroscientific observation that semantically related inputs activate overlapping neural populations. Transformer LLMs are composed of stacked FFN layers, each comprising numerous neurons encoding diverse linguistic and factual features at varying levels of abstraction. While traditional alignment metrics collapse sentence representations into low-dimensional vectors—often suffering from anisotropy and representation collapse—NeuronXA instead examines the vector of activation states for individual neurons, treating it as an intrinsic and semantically meaningful representation of the linguistic input.

This neuron-centric approach extends the evaluation of cross-lingual alignment beyond pooled embeddings, offering an unbiased metric for alignment even on low-resource languages. In contrast to MEXA and similar benchmarks reliant on cosine similarity of sentence-level embeddings, NeuronXA uses an indicator-based pairwise matching criterion over raw neuron activations, mitigating performance biases and improving diagnostic sensitivity for semantic alignment (Huang et al., 20 Jul 2025).

2. Mathematical Formalization

NeuronXA's methodology centers on the activation profiles produced by the FFN of each transformer layer. For input hidden state $x \in \mathbb{R}^d$, the gated FFN is formalized as:

$$\mathrm{FFN}^I(x) = \sigma(W_G x + b_G) \odot (W^I x + b^I)$$

where $W_G, W^I \in \mathbb{R}^{d_{ff} \times d}$, $b_G, b^I \in \mathbb{R}^{d_{ff}}$, $\sigma(\cdot)$ is a nonlinearity, and $\odot$ denotes elementwise multiplication. The $i$-th neuron's activation value is:

$\alpha_i(x) = [\mathrm{FFN}^I(x)]_i$

For a $T$-token sentence at layer $\ell$, token-wise activations $n_{\ell t} \in \mathbb{R}^{d_{ff}}$ are combined into a weighted average:

$$N_\ell = \sum_{t=1}^{T} w_t n_{\ell t}, \quad w_t = \frac{t}{\sum_{k=1}^{T} k}$$

Parallel corpora provide $n$ sentence pairs per language pair, yielding a pairwise cosine similarity matrix $C_\ell$:

$$(C_\ell)_{ij} = \cos(N_\ell^{(1)}(i), N_\ell^{(2)}(j))$$

Alignment is scored as:

$$\mu(C_\ell) = \frac{1}{n}\sum_{i=1}^{n} \mathbb{1}[(C_\ell)_{ii} > \max_{j \neq i}\{(C_\ell)_{ij}, (C_\ell)_{ji}\}]$$

The aggregate NeuronXA score averages $\mu(C_\ell)$ across layers (Huang et al., 20 Jul 2025).

Variants such as NASCA (binary activation states) and NAVCA (absolute magnitude pooling) further refine the measure, with NASCA and weighted averaging delivering the best alignment–performance prediction across tasks.

3. Experimental Design and Neural Analysis

NeuronXA has been evaluated on open-source LLMs at the 7B–14B parameter scale, including LLaMA-2/3, Qwen, Mistral, GLM, and OLMo series (Huang et al., 20 Jul 2025). Datasets comprised high-quality parallel corpora: FLORES-200 (1012 manually validated sentences in 213 languages) and Tatoeba (up to 1000 aligned pairs in 112 languages). Experiments typically sample $n=100$ parallel pairs per language pair.

The neuron-state extraction and alignment metrics are computed for all FFN layers, and alignment dynamics are analyzed layerwise, revealing that $\mu(C_\ell)$ scores rise in early layers, peak in intermediate layers, and decline near model output, reflecting the transition from shared semantic mapping to language-specialized generation.

Empirical studies have also applied neuron overlap methods such as BridgeX-ICL, which probes language-overlap neurons and uses HSIC-based metrics for optimal bridge selection in cross-lingual transfer, with bridge neurons serving as lightweight, zero-shot adapters (Xu et al., 23 Aug 2025). Detailed analyses demonstrate that overlap patterns correlate with language taxonomy but can be influenced by model training data, with middle-layer overlap neurons forming the principal substrate for semantic alignment.

4. Evaluation Metrics and Correlation with Downstream Tasks

NeuronXA's principal evaluation involves computing Pearson correlation coefficients between NeuronXA scores and LLM performance on diverse downstream tasks:

• Zero-shot cross-lingual transfer (fine-tuning on English XNLI, evaluating in target languages)
• Cross-lingual factual consistency (BMLAMA-53)
• Multilingual reading comprehension (Belebele)
• Multilingual scientific reasoning (m-ARC)
• Multilingual knowledge benchmarks (m-MMLU)

NeuronXA achieves average Pearson $r=0.9556$ against three multilingual benchmarks and $r=0.8514$ for transfer tasks, outperforming alternative metrics including MEXA, CKA, SVCCA, and ANC. Notably, only 100 parallel sentences suffice for reliable alignment prediction, and random similarity matrices rarely achieve comparable alignment scores by chance ($p<0.00016$ for $\mu>0.05$ in 100$\times$100 matrices) (Huang et al., 20 Jul 2025).

BridgeX-ICL further demonstrates a $+6\%$ average accuracy gain over strong baselines on both bilingual lexicon induction and Belebele machine reading comprehension, including consistent outperformance of prior bridge heuristics (Xu et al., 23 Aug 2025).

5. Neuron Type Taxonomy and Inference Dynamics

Extending NeuronXA's interpretability, research has distinguished three neuron types in LLMs:

• Language-specific neurons: Activated predominantly in a single language
• Language-related neurons: Activated in several, but not all, languages
• Language-agnostic neurons: Robustly activated across all languages

A multi-stage inference workflow emerges:

1. Multilingual Understanding (Layers 1–4): Dominated by language-sensitive neurons, mapping input into shared representations.
2. Shared Semantic Space Reasoning (Layers $\sim$5–12): Language-agnostic neurons prevail, reflecting language-neutral cognitive operations.
3. Multilingual Output Space Transformation (Layers $\sim$13–end-1): Resurgence of language-sensitive neurons, projecting semantic reasoning back into target-language code.
4. Vocabulary Space Outputting (Final layer): Mixed increase in language-related and agnostic neurons, mapping logic to final token outputs (Zhang et al., 27 May 2025).

NeuronXA incorporates explicit alignment objectives—KL-divergence or $\ell_2$ loss—over activation probabilities, compelling non-English activation profiles toward English-like distributions during training.

6. Empirical Findings and Practical Guidance

Key empirical observations include:

• NeuronXA-based retrieval achieves up to $25\%$ higher accuracy and mitigates directional asymmetry for long-tail languages (Huang et al., 20 Jul 2025).
• Layer localization shows neuron overlap concentrated in middle layers and final layers, substantiating the role of shared semantic bridging and language output coding (Xu et al., 23 Aug 2025).
• Spontaneous multilingual alignment arises: aligning only a subset of languages (e.g. zh/de→en) propagates gains to held-out languages, and neuron-level distribution shifts accompany improved cross-lingual generalization (Zhang et al., 27 May 2025).

Best practices for model monitoring and alignment include:

• Tracking layerwise alignment dynamics ($\mu(C_\ell)$, overlap indices) during pre-training
• Diagnosing drops in alignment (e.g., $<0.05$ overlap) as signals for intervention by altering training schedules or model sizes (Wang et al., 2024)
• Using NeuronXA as an efficient proxy for predicting transferability with minimal parallel data

BridgeX-ICL additionally recommends probing bridge candidates via overlap neurons and HSIC scores, thereby selecting intermediary languages for improved zero-shot transfer without model retraining.

7. Limitations and Future Directions

NeuronXA depends on white-box model access to neuron activations, limiting applicability to closed-source LLMs. It constitutes only a partial measure of overall generative cross-lingual phenomena. Prospective research directions involve:

• Generalizing NeuronXA beyond FFN activations to incorporate attention mechanisms or contrasting architectures
• Incorporating differentiable NeuronXA objectives in multi-task or continual fine-tuning
• Investigating neuron-state alignment during various stages of pre-training
• Extending neuron-centric alignment metrics to open-vocabulary, machine translation quality estimation, and non-discriminative tasks (Huang et al., 20 Jul 2025)

These avenues suggest that neuron state-based alignment metrics may become integral to future LLM evaluation suites and multilingual diagnostic pipelines, providing both an interpretive lens and a practical proxy for zero-shot diagnostic and benchmarking tasks.