Brain–LLM Alignment

• Brain–LLM Alignment is the measurable similarity between LLM activations and human neural responses, providing a framework for linking AI and cognitive processing.
• Techniques like linear encoding, representational similarity analysis, and temporal mapping reveal that syntactic and architectural factors drive the observed alignment.
• Instruction tuning and multimodal integration enhance alignment, suggesting that model design inspired by brain dynamics can improve both performance and interpretability.

Brain–LLM Alignment refers to the systematic similarity between internal representations of LLMs and neural activity in the human brain, typically measured during natural language processing but now extended to multimodal and cognitive domains. This alignment is assessed both at the level of high-level representational geometry and through direct, predictive mappings (e.g., ridge regression) from model activations to neural recordings (fMRI, MEG, ECoG, EEG). The field quantitatively dissects which aspects of LLM computation mirror brain processing, identifies the linguistic and architectural factors driving this similarity, traces its developmental trajectory during LLM training, and explores the implications for model design and cognitive theory.

1. Foundations and Formal Metrics of Brain–LLM Alignment

Alignment between LLMs and brain activity is operationalized using metrics that quantify the predictability or similarity between model-derived representations and biological neural responses. The dominant metrics and procedures include:

• Linear Encoding (Linear Predictivity): Model representations $x$ (often from language-selective units) are mapped to brain responses $y$ via ridge regression: $\hat{y} = W x + b$. The alignment score is the Pearson correlation $r = \operatorname{corr}(y_{pred}, y_{true})$ on held-out data, typically normalized by the noise ceiling (cross-subject consistency in brain responses), i.e., $\text{normalized\_score} = r / c$ (AlKhamissi et al., 3 Mar 2025).
• Representational Similarity Analysis (RSA): For a set of stimuli, pairwise (dis)similarities between model embeddings and brain responses are calculated, producing representational dissimilarity matrices whose upper-triangle entries are compared via correlation: $$RSA\_score = \rho(\text{vec}(RDM^{\text{brain}}), \text{vec}(RDM^{\text{model}}))$$ (Ren et al., 28 Feb 2024).
• Temporal Alignment: Time-resolved brain data (e.g., MEG) are mapped to representations from successive model layers. The timing of maximal model–brain correspondence is statistically linked to model depth, with the "temporal score" defined as the Pearson correlation between layer depth and timing of best alignment: $$r_{\text{temp}} = \operatorname{corr}\{T_{\max}(\ell), \text{depth}(\ell)\}$$ (Raugel et al., 1 Dec 2025).
• Topological Summaries: The Brainscore metric can also be interpreted via multiscale topological features of the representational time-series (e.g., $q$–Wasserstein distances between persistence diagrams in homological dimensions), summed over model and brain representations by region and hemisphere (Li, 10 May 2024).

Alignment is usually assessed within functionally localized neural language networks but is now being extended to modality-agnostic concept regions and non-linguistic cognitive networks.

2. Neural and Computational Determinants of Alignment

Syntactic and Structural Drivers:

A principal result is that syntactic properties—especially top-constituent sequences and syntactic tree depth—account for the majority of model–brain alignment. Removing these features from model representations yields substantial decreases in alignment, particularly in middle layers and canonical language-related ROIs: Inferior Frontal Gyrus (IFG, Broca's area) for constituents, and Anterior/Posterior Temporal Lobe for tree depth (Oota et al., 2022). Surface features and semantic features (tense, subject/object number) incrementally contribute, with spatial specificity to regions such as PCC and ATL.

Trajectory and Layerwise Dynamics:

Alignment peaks in middle-to-high model layers that correspond temporally to known neurophysiological landmarks (e.g., the N400 ERP component, 300–400 ms post-word), indicating that LLMs and the brain both perform semantic integration at analogous depths/times (Xiao et al., 29 Sep 2025). Shallow layers in LLMs align with early sensory brain responses, deeper layers with later associative and integrative responses (Raugel et al., 1 Dec 2025).

Model Architecture and Inductive Biases:

Architectural priors—multihead attention, subword tokenization (BPE), and shallow recurrent depth—are necessary and sufficient for non-trivial alignment, even in untrained models. BPE tokenization imparts frequency sensitivity, while multihead attention enables integration over context (AlKhamissi et al., 21 Jun 2024). Recurrence (weight-tied or iterated passes) modestly boosts alignment, paralleling iterative cortical computations.

Model Scaling and Training:

Alignment increases logarithmically with model size up to $\sim$13B parameters, with diminishing returns beyond this point (Ren et al., 28 Feb 2024). Early in training, next-word prediction and formal linguistic competence (as measured by BLiMP and SyntaxGym) tightly track brain alignment, peaking at $\sim$4B tokens; functional (world-knowledge, reasoning) competence grows independently and is weakly predictive of further alignment (AlKhamissi et al., 3 Mar 2025). Untrained models already achieve $\sim$50% of the brain alignment of fully trained models due to inherent architectural properties.

Factor	Drives Brain Alignment?	Quantitative Effect
Syntactic features (syntax)	Strong	Drop of 0.02–0.03 $r$ when removed (Oota et al., 2022)
Semantic/discourse features	Moderate	PCC, ATL, MFG show effect
Model scaling	Positive, sublinear	$0.018\,\log_{10}(N_{\text{params}})$ (Ren et al., 28 Feb 2024)
Instruction tuning	Modest positive	$+6.2\%$ relative increase (Aw et al., 2023)
Context window	Positive, logarithmic	$r_\text{temp} \sim \ln$(context)(Raugel et al., 1 Dec 2025)

3. Impact of Instruction Tuning, Multimodality, and Training Paradigms

Instruction Tuning:

Instruction tuning consistently increases LLM–brain alignment by $\sim$6% (Δρ > 0) across diverse architectures and datasets (Aw et al., 2023). Improvements are strongest in higher layers and in models with enhanced world-knowledge representations. The mechanism is additive: both instruction format and additional training examples contribute. In multimodal settings, instruction tuning further enhances alignment and allows for task-specific differentiation of representational content, with instructions targeting recognition, counting, or scene description preferentially activating corresponding brain regions (e.g., object-count tasks in early visual cortex, scene-captioning in lateral occipital and language ROIs) (Oota et al., 26 May 2025, Oota et al., 9 Jun 2025).

Multimodal Alignment:

Instruction-tuned multimodal LLMs (MLLMs) significantly outperform vision-only or non-instruction-tuned models in both image and video–language alignment to human cortex. These gains are especially pronounced in high-level visual (e.g., floc-places/FFA), language-selective, and multimodal integration regions (e.g., angular gyrus, pSTS). Models capable of encoding fine-grained, instruction-driven features explain both shared and unique variances in brain responses, supporting a view in which modality-agnostic concept regions underlie the hierarchy of human semantic cognition (Ryskina et al., 15 Aug 2025).

Beyond Language Tasks:

LLM–brain alignment extends beyond linguistic processing: in carefully translated nonlinguistic sensory-motor tasks, LLM-internal dynamics can be linearly mapped to human neural activity in anticipatory, sensorimotor, and attention-related cortices, as evidenced by joint modeling of LLM hidden states and human intracranial EEG (Ngo et al., 26 Aug 2025). Behavioral outputs (e.g., reaction times) and internal representations (CKA~0.39) display similarity with human neural trajectories.

4. Evolution, Functional Organization, and Specialization

Temporal Evolution During Training:

Brain alignment is characterized by an early plateau (pretraining, up to $\sim$128M tokens), a sharp rise to peak (2–8B tokens), and a subsequent saturation or decline. Formal, rule-based competence and brain alignment peak far earlier than functional competence or continued next-word prediction improvements (AlKhamissi et al., 3 Mar 2025).

Functional Organization:

LLM internal structure exhibits brain-like functional organization. Artificial neuron (AN) sub-groups—identified via sparse coding over Transformer blocks—mirror the partitioning of the brain into functional networks (FBNs): language, visual, auditory, working memory, default-mode, and frontoparietal networks. Larger and more recent LLMs (e.g., Llama 3) display a more compact, consistent, and hierarchical functional specialization, mapping deeper layers to higher-order cognitive networks and shallower layers to sensory networks (Sun et al., 25 Oct 2024). This provides a computational substrate for both local specialization (e.g., Broca’s area↔syntax modules) and global integrative networks (domain-general processing).

Attribution and Feature Reliance:

Fine-grained attribution methods reveal that the subset of words most responsible for brain–LLM alignment is nearly disjoint from those critical for next-word prediction. While NWP relies on syntactic, local context (recency/primacy effects), brain alignment emphasizes semantic and discourse-level features with a more focused recency bias, paralleling human integration of meaning (Proietti et al., 14 Oct 2025). This provides a mechanistic basis for the unique mapping from LLMs to neurobiological data.

5. Applications, Model Design Implications, and Future Directions

Neuro-inspired Model Design:

Maximally brain-aligned LLMs are expected to combine explicit formal linguistic structure with targeted architectural extensions for functional reasoning and multi-demand integration. Inductive architectural priors such as shallow, localized multi-head attention with subword tokenization, and explicit recurrence, reproduce key aspects of brain computation even in the absence of training (AlKhamissi et al., 21 Jun 2024). Use of linear alignment metrics (e.g., encoding models) as heuristics for model selection, initialization, and early stopping may efficiently tune models toward neuro-alignment (AlKhamissi et al., 3 Mar 2025).

Interfacing Brain and LLMs:

End-to-end pipelines using optimal transport or multi-subject semantic alignment methods permit direct feeding of neuroimaging data into text-generating LLMs. These models achieve high performance in image description (“brain captioning”) and semantic reconstruction from fMRI, with explicit control for redundancy and synergy across brain regions (Xiao et al., 9 Mar 2025, Han et al., 28 May 2024, Ma et al., 8 Jan 2024, Zheng et al., 26 Sep 2024).

Cross-domain Generalization:

Alignment is not confined to language cortex but generalizes to modality-agnostic and action-concept regions, and even to tasks outside the standard linguistic repertoire, provided representations are formatted appropriately. Cross-modal and cross-linguistic studies further support the geometric and hierarchical congruence of LLM and human brain representations (Ryskina et al., 15 Aug 2025, Ngo et al., 26 Aug 2025).

Limitations and Open Problems:

Despite these advances, LLM–brain alignment remains partial. Model size, context, and fine-tuning produce sublinear improvements, and even state-of-the-art models leave “brain alignment benchmarks unsaturated” (AlKhamissi et al., 3 Mar 2025). Temporal dynamics differ: brains show continuous, iterative integration across distributed networks, while LLM computation is discretized, layer-wise, and predominantly feed-forward (Xiao et al., 29 Sep 2025). Further, current methods rely on linear mappings; the extent to which nonlinear or recurrent architectures could close remaining alignment gaps is an open question.

Future Directions:

Key research directions include (1) incorporating explicit syntactic/structural objectives during pretraining, (2) developing multi-network alignment procedures to model functional and domain-general networks, (3) extending metrics and models to real-time, interactive, and sensorimotor paradigms, (4) leveraging topological and attribution-based analyses for interpretability and safe deployment, and (5) feedback loops between neuroscience and LLM development to mutually inform architectures, benchmarks, and theoretical models (AlKhamissi et al., 3 Mar 2025, Li, 10 May 2024, Raugel et al., 1 Dec 2025).

6. Summary Table: Major Results, Metrics, and Implications

Study / Domain	Alignment Metric	Key Finding / Trend
(AlKhamissi et al., 3 Mar 2025)	Linear Predictivity	Brain alignment peaks early (2–8B tokens), most closely tracks formal linguistic competence (syntax); functional competence less predictive.
(Oota et al., 2022)	Ridge Regression (removal)	Knockout of syntactic features produces largest alignment drop; IFG and ATL/ PTL as key loci.
(Aw et al., 2023)	Pearson r (encoding)	Instruction-tuning yields ~6% improvement; effect grows with model size and world-knowledge.
(AlKhamissi et al., 21 Jun 2024)	Ridge regression (localization)	Shallow, untrained attention with BPE achieves strong alignment; recurrence and increased heads further help.
(Raugel et al., 1 Dec 2025)	Temporal score	Model depth mirrors brain timing; alignment improves with model scale and context window.

Alignment between LLMs and the brain is best viewed as a multi-dimensional, dynamic phenomenon. It encompasses anatomical specificity (syntactic, semantic, and domain-general networks), computational sequence (temporal unfolding and depth), architectural priors, and training trajectory. Advances in both measurement and model design continue to refine this convergence, informing the development of both artificial language systems and computational cognitive neuroscience.