Interpreting Brain Responses to Language with Sparse Features from Language Models

Abstract: A central goal of cognitive neuroscience is to characterize the features that are represented by human language cortex. Artificial LMs have emerged as a powerful tool to address this challenge, but studies relating biological and artificial representations are often criticized as relating one black box to another. The present work introduces Augmented Sparse Encoding Models, an encoding framework that replaces dense LM hidden states with hierarchically-organized sparse autoencoder (SAE) features, while explicitly including surprisal as a predictor. Using this approach, we (i) produce interpretations of neural responses and (ii) test whether model-brain alignment reflects primary or idiosyncratic variation in LM representations. Using a high-field 7T fMRI dataset of eight participants listening to 200 linguistically diverse sentences, we first validate our modeling framework by recovering previous interpretations of voxel populations tuned to processing difficulty and meaning abstractness. We then interpret a previously-uncharacterized (but reliable) voxel population and find that it is tuned to people-related content. Next, we show that the fronto-temporal human language network is predicted by a common set of features across its constituent regions, but find that frontal regions are relatively well-explained by surprisal alone, even in the absence of LM-based features. Finally, we show that brain responses during language processing are not merely predictable from an arbitrary set of LM features. Rather, brain responses are best explained by the features that tend to capture the most general information encoded in LM representations, suggesting a nontrivial correspondence between brain and LM language representation.

• The paper introduces augmented sparse encoding models that decompose dense LM representations into interpretable, monosemantic features while integrating surprisal to separate processing difficulty from content.
• It replaces standard dense activations with hierarchically structured SAEs, using LASSO selection and ridge regression on high-resolution fMRI data to achieve voxel-level interpretability.
• The results demonstrate that a small set of general features drives robust brain-LM alignment across language regions, while also revealing non-canonical, idiosyncratic voxel subpopulations.

Interpretable Sparse Features from LLMs Illuminate Human Brain Responses to Language

Introduction and Motivation

This work systematically addresses a central challenge in cognitive neuroscience: establishing an interpretable mapping between high-dimensional LLM (LM) representations and human brain responses during language comprehension. While prior LM-to-brain encoding studies quantitatively connected model activations to neural data, they often suffered from opacity—creating "black box to black box" mappings that limit theoretical traction and mechanistic interpretability. To move beyond this limitation, the authors introduce Augmented Sparse Encoding Models, leveraging sparse autoencoders (SAEs) with hierarchical structure to decompose dense LM representations into a high-dimensional, interpretable, and monosemantic basis. This SAE feature space is then combined with an explicit surprisal predictor to further distinguish processing difficulty from representational content. Analyses are performed on a high-resolution 7T fMRI dataset with participants listening to 200 diverse sentences, affording both fine-grained voxel-level and region-based analyses.

Methodological Framework

The methodological advance stems from replacing standard dense residual-stream LM features with feature sets constructed via either JumpReLU or Matryoshka SAEs trained on the Gemma-2-2b LLM. The Matryoshka SAE, in particular, is designed to offer a hierarchical structure—a nesting of feature bins ordered in granularity from highly general to highly idiosyncratic. Encoding models then perform sparse LASSO-based feature selection, followed by ridge regression to predict voxel-wise fMRI amplitude.

Critically, LM-derived surprisal (mean token-level negative log-probability) is included as a separate regressor in all models, enabling a clean separation of processing difficulty effects from content representation effects. Voxels are categorized both by correlation with previously identified principal components—capturing axes of processing difficulty and semantic abstractness/concreteness—and by canonical fronto-temporal language functional ROIs (fROIs), further enabling regionally and functionally stratified analysis.

Figure 1: Voxel selection and feature space overview, contrasting standard residual LM encoding with the proposed sparse SAE/augmented surprisal method.

Empirical Validation and Voxel Subtype Analysis

Recovery and Nuancing of Canonical Dimensions

Initially, the model is validated by its ability to recover and extend voxel tuning axes discovered in prior work: namely, a processing difficulty axis (quantified by SR surprisal) and an abstractness/concreteness axis. Sparse SAE encoding matches the predictivity of dense-residual LM features using substantially fewer features (mean of 18 Matryoshka SAE features per voxel). Importantly, in "hard-to-process" voxels, surprisal alone suffices, with no added benefit from SAE features. Conversely, for abstract and concrete content voxels, inclusion of SAE-derived content features yields significant predictivity gains beyond surprisal.

Figure 2: (A) Predictivity of four voxel subtypes using different feature sets, showing content-driven subtypes require SAE/LM features; (B) Distinct SAE features selected for abstract/concrete subtypes.

Feature-Level Interpretability and Asymmetry

Inspection of the sparse SAE features selected for abstract and concrete voxels reveals strong alignment with manually curated semantic axes (e.g., people-related, scene-related, question-related features from the Matryoshka basis). Notably, the feature selection displays an asymmetric pattern: certain people-event features suppress abstract voxels without commensurate enhancement of concrete voxels, challenging an overly symmetric view of content axes in prior PCA-based analyses.

Discovery and Characterization of Non-Canonical Voxel Populations

A case study focuses on the "Ghost" voxel subtype, defined by weak loadings on both established principal components. A subpopulation of Ghost voxels, found via sparse-feature generalization analysis and hierarchical clustering, is coherently explained by a small set of people-specific SAE features—distinct from canonical axes. Anatomically, this cluster is predominantly outside classical language ROIs, localized instead to the angular gyrus and adjacent areas associated with theory-of-mind.

Figure 3: (A) Generalization heatmap identifies a cluster of Ghost voxels sharing a predictive people-specific feature basis (red square); (B) Corresponding selected Matryoshka SAE features.

Network-Level Mapping: Fronto-Temporal Language Regions

Sparse encoding models are then generalized to five left-hemisphere language fROIs. Here, several strong and theoretically significant patterns emerge:

• Frontal regions (especially MFG) are largely explained by surprisal alone, while temporal regions require LM/SAE content features for optimal predictivity.
• The generalization heatmap demonstrates widespread feature sharing across regions, with no marked within-fROI specialization, indicating a core, network-shared representational basis in the human language system.
• Feature prevalence analysis reveals a small set of highly recurrent, broadly shared Matryoshka features, accompanied by a large tail of less prevalent features that are reused across multiple participants (but not all), suggesting both a common representational kernel and substantial individual/categorical diversity.

  Figure 4: (A) Predictivity breakdown by fROI and feature space; (B) Generalization heatmap showing cross-voxel feature reuse across the language network; (C) Feature prevalence across voxels and participants measures core and idiosyncratic sharing.

Hierarchical Feature Importance: General Versus Fine-Grained Dimensions

A critical analytic angle is the exploitation of the Matryoshka SAE's hierarchical ordering to test whether the feature dimensions most predictive of neural responses are general or idiosyncratic. The overwhelming majority of predictive power in both fROIs and principal component-based subtypes is concentrated within the most general, earliest SAE feature bin—despite its comprising only 0.4% of all features. Restricting regressions to these general features yields higher predictivity than using the union of all fine-grained bins.

Figure 5: (A) Feature selection histogram by Matryoshka bin; (B) Average number of features per bin for each fROI; (C) Predictivity restricted to individual bins, with general features dominating.

Theoretical and Practical Implications

1. Interpretability in LM–Brain Mapping:

This approach bridges the interpretability gap by enabling both the recovery of known semantic-processing axes and the characterization of previously unclassified neural populations. The method's ability to reliably select small, human-interpretable SAE features facilitates potential causal and intervention experiments.

2. Unified but Heterogeneous Language Network Code:

The human language network appears to encode linguistic content along a largely shared set of high-level content axes, with important regional variations—specifically, a gradient from processing-difficulty-based frontal cortex to content-driven temporal areas. However, individual brains also display substantial idiosyncratic feature usage, suggesting both universality and personalization.

3. Constraints on Brain–LM Representational Alignment:

Alignment between LMs and brain responses is not arbitrary, nor is it a byproduct of encoding accuracy; it is preferentially realized via the most generic, widely applicable dimensions of LM representational spaces. This observation supports strong claims about the nontrivial overlap in how biological and artificial systems decompose the language space and sets clear constraints for theoretical modeling.

4. Extension and Contrast with Prior Work:

These findings extend recent research that used low-dimensional embeddings and region averages for encoding, providing much finer-grained, voxel-level, and feature-atomic analytic resolution (2606.06857). They critically challenge the notion that fine-grained or idiosyncratic LM features drive brain-model alignment.

Conclusion

Augmented Sparse Encoding Models, leveraging hierarchical SAEs and explicit surprisal predictors, advance the field by enabling detailed, interpretable, and sparse mappings from LM representations to brain activity during language comprehension. A small, general subset of model-derived features underlies robust, widespread alignment across individuals, regions, and semantic axes, while also permitting the discovery of non-canonical, socially tuned neural populations. These results have direct methodological significance for both the interpretability of large models and for the cognitive neuroscience of language, as well as significant implications for future interventionist neuroscience and explainable AI. Future directions include expansion to richer, multimodal data, further individualized mapping, and the experimental manipulation of sparse linguistic features to causally probe representational structure.

---

References:

• Interpreting Brain Responses to Language with Sparse Features from LLMs (2606.06857)