fMRI predictors based on language models of increasing complexity recover brain left lateralization (2405.17992v2)

Abstract: Over the past decade, studies of naturalistic language processing where participants are scanned while listening to continuous text have flourished. Using word embeddings at first, then LLMs, researchers have created encoding models to analyze the brain signals. Presenting these models with the same text as the participants allows to identify brain areas where there is a significant correlation between the functional magnetic resonance imaging (fMRI) time series and the ones predicted by the models' artificial neurons. One intriguing finding from these studies is that they have revealed highly symmetric bilateral activation patterns, somewhat at odds with the well-known left lateralization of language processing. Here, we report analyses of an fMRI dataset where we manipulate the complexity of LLMs, testing 28 pretrained models from 8 different families, ranging from 124M to 14.2B parameters. First, we observe that the performance of models in predicting brain responses follows a scaling law, where the fit with brain activity increases linearly with the logarithm of the number of parameters of the model (and its performance on natural language processing tasks). Second, although this effect is present in both hemispheres, it is stronger in the left than in the right hemisphere. Specifically, the left-right difference in brain correlation follows a scaling law with the number of parameters. This finding reconciles computational analyses of brain activity using LLMs with the classic observation from aphasic patients showing left hemisphere dominance for language.

The paper presents a sophisticated paper investigating the relationship between the complexity of LLMs and brain activity patterns measured through functional magnetic resonance imaging (fMRI). This work focuses on reconciling the somewhat conflicting findings between computational models and classical neuropsychological observations regarding the lateralization of language processing in the brain.

Key Findings and Methodology

1. Naturalistic Language Processing and fMRI:
   • The paper builds on the growing body of research where participants are scanned while listening to continuous text.
   • Researchers have traditionally used word embeddings and LLMs to create encoding models, which are then presented with the same text as the participants. This allows for identifying brain areas where fMRI time series significantly correlate with the models' predicted neural activations.
2. Bilateral Activation Patterns Versus Left Lateralization:
   • Previous studies with these models have shown symmetric bilateral activation patterns during language processing, which contrasts with the well-established left hemispheric dominance for language.
3. Experiment with LLM Complexity:
   • The authors test 28 pretrained models from 8 different families, with sizes ranging from 124 million to 14.2 billion parameters.
   • They explore how the brain's response patterns change as a function of model complexity.
4. Scaling Law and Model Performance:
   • The performance in predicting brain responses follows a scaling law: the fit between the models and brain activity increases linearly with the logarithm of the number of parameters in the model, which also correlates with the model's performance on natural language processing tasks.
   • This indicates that as LLMs become more sophisticated, they increasingly accurately reflect how the human brain processes language.

Emergence of Left-Right Asymmetry

One of the most significant findings of the paper is the emergence of left-right asymmetry in brain activations as model complexity increases:

• Small Models: The smallest models do not show any significant asymmetry, meaning they do not preferentially fit either left or right hemispheric brain activations.
• Larger Models: As model size increases, there is a noticeable pattern where larger models increasingly fit left hemispheric activations better than right hemispheric ones. This difference also adheres to a scaling law.

Implications

This research has notable implications:

• Reconciliation with Neuropsychology: The findings provide a bridge between computational analyses using LLMs and classical neuropsychological observations (such as those derived from studies of aphasic patients) that demonstrate left hemisphere dominance for language.
• Future Research Directions: The paper's results suggest that further increasing the complexity and capacity of LLMs could bring even more insights into the neural underpinnings of language processing, potentially informing both artificial intelligence and cognitive neuroscience fields.

The paper highlights how advancing technologies in LLMs can offer profound insights into human brain function, particularly in a domain as complex and uniquely human as language processing.