Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
169 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Measuring Meaning Composition in the Human Brain with Composition Scores from Large Language Models (2403.04325v3)

Published 7 Mar 2024 in cs.CL and cs.AI

Abstract: The process of meaning composition, wherein smaller units like morphemes or words combine to form the meaning of phrases and sentences, is essential for human sentence comprehension. Despite extensive neurolinguistic research into the brain regions involved in meaning composition, a computational metric to quantify the extent of composition is still lacking. Drawing on the key-value memory interpretation of transformer feed-forward network blocks, we introduce the Composition Score, a novel model-based metric designed to quantify the degree of meaning composition during sentence comprehension. Experimental findings show that this metric correlates with brain clusters associated with word frequency, structural processing, and general sensitivity to words, suggesting the multifaceted nature of meaning composition during human sentence comprehension.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (45)
  1. Palm 2 technical report.
  2. Scaling laws for language encoding models in fMRI. In Thirty-seventh Conference on Neural Information Processing Systems.
  3. Douglas K Bemis and Liina Pylkkänen. 2011. Simple composition: A magnetoencephalography investigation into the comprehension of minimal linguistic phrases. Journal of Neuroscience, 31(8):2801–2814.
  4. Douglas K Bemis and Liina Pylkkänen. 2013. Flexible composition: Meg evidence for the deployment of basic combinatorial linguistic mechanisms in response to task demands. PloS one, 8(9):e73949.
  5. Language switching decomposed through meg and evidence from bimodal bilinguals. Proceedings of the National Academy of Sciences, 115(39):9708–9713.
  6. Abstract linguistic structure correlates with temporal activity during naturalistic comprehension. Brain and language, 157:81–94.
  7. Qing Cai and Marc Brysbaert. 2010. Subtlex-ch: Chinese word and character frequencies based on film subtitles. PloS one, 5(6):e10729.
  8. Deep language algorithms predict semantic comprehension from brain activity. Scientific reports, 12(1):16327.
  9. Charlotte Caucheteux and Jean-Rémi King. 2022. Brains and algorithms partially converge in natural language processing. Communications biology, 5(1):134.
  10. What does BERT look at? an analysis of BERT’s attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276–286, Florence, Italy. Association for Computational Linguistics.
  11. A magnetoencephalographic component whose latency reflects lexical frequency. Cognitive brain research, 10(3):345–348.
  12. Bruce Fischl. 2012. Freesurfer. Neuroimage, 62(2):774–781.
  13. Graham Flick and Liina Pylkkänen. 2020. Isolating syntax in natural language: Meg evidence for an early contribution of left posterior temporal cortex. Cortex, 127:42–57.
  14. L. Frazier. 1985. Syntactic complexity. In D. Dowty, L. Karttunen, and A. Zwicky, editors, Natural language parsing: Psychological, computational, and theoretical perspectives, pages 129–189. Cambridge, UK: Cambridge University Press.
  15. Roles of scaling and instruction tuning in language perception: Model vs. human attention. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 13042–13055, Singapore. Association for Computational Linguistics.
  16. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 30–45, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  17. Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5484–5495, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  18. Shared computational principles for language processing in humans and deep language models. Nature neuroscience, 25(3):369–380.
  19. Peter Hagoort. 2005. On broca, brain, and binding: a new framework. Trends in cognitive sciences, 9(9):416–423.
  20. John T. Hale. 2014. Automaton Theories of Human Sentence Comprehension. CSLI Publications, Center for the Study of Language and Information, Stanford, California.
  21. Intersubject synchronization of cortical activity during natural vision. science, 303(5664):1634–1640.
  22. John Hewitt and Christopher D. Manning. 2019. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4129–4138, Minneapolis, Minnesota. Association for Computational Linguistics.
  23. Le petit prince multilingual naturalistic fmri corpus. Scientific data, 9(1):530.
  24. Semantic composition in experimental and naturalistic paradigms. Imaging Neuroscience, 2:1–17.
  25. Jixing Li and Liina Pylkkänen. 2021. Disentangling semantic composition and semantic association in the left temporal lobe. Journal of Neuroscience, 41(30):6526–6538.
  26. Neural dynamics of semantic composition. Proceedings of the National Academy of Sciences, 116(42):21318–21327.
  27. Eric Maris and Robert Oostenveld. 2007. Nonparametric statistical testing of eeg-and meg-data. Journal of neuroscience methods, 164(1):177–190.
  28. William Matchin and Gregory Hickok. 2020. The cortical organization of syntax. Cerebral Cortex, 30(3):1481–1498.
  29. Same words, different structures: An fmri investigation of argument relations and the angular gyrus. Neuropsychologia, 125:116–128.
  30. Neurophysiological dynamics of phrase-structure building during sentence processing. Proceedings of the National Academy of Sciences, 114(18):E3669–E3678.
  31. Gpt-4 technical report.
  32. Alicia Parrish and Liina Pylkkänen. 2022. Conceptual combination in the latl with and without syntactic composition. Neurobiology of Language, 3(1):46–66.
  33. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227–2237, New Orleans, Louisiana. Association for Computational Linguistics.
  34. Converging evidence for the neuroanatomic basis of combinatorial semantics in the angular gyrus. Journal of Neuroscience, 35(7):3276–3284.
  35. Liina Pylkkänen. 2019. The neural basis of combinatory syntax and semantics. Science, 366(6461):62–66.
  36. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv e-prints.
  37. The neural architecture of language: Integrative modeling converges on predictive processing. Proceedings of the National Academy of Sciences, 118(45):e2105646118.
  38. Disambiguating form and lexical frequency effects in meg responses using homonyms. Language and Cognitive Processes, 27(2):275–287.
  39. End-to-end memory networks. Advances in neural information processing systems, 28.
  40. Combining computational controls with natural text reveals aspects of meaning composition. Nature computational science, 2(11):745–757.
  41. Llama 2: Open foundation and fine-tuned chat models.
  42. Neurons in large language models: Dead, n-gram, positional.
  43. The latl as locus of composition: Meg evidence from english and arabic. Brain and Language, 141:124–134.
  44. Victor H Yngve. 1960. A model and an hypothesis for language structure. Proceedings of the American philosophical society, 104(5):444–466.
  45. Linmin Zhang and Liina Pylkkänen. 2015. The interplay of composition and concept specificity in the left anterior temporal lobe: An meg study. NeuroImage, 111:228–240.
Citations (1)
View on Semantic Scholar

Summary

  • The paper introduces the Composition Score to quantify meaning composition in human sentence comprehension using transformer-based LLM insights.
  • It demonstrates that the Composition Score correlates with brain regions involved in word frequency, structural, and general linguistic processing.
  • This approach bridges computational models with neurolinguistic research, offering a new tool for investigating semantic processing in the brain.

The paper "Measuring Meaning Composition in the Human Brain with Composition Scores from LLMs" introduces an innovative approach to quantify the process of meaning composition in human sentence comprehension. This process is fundamental in linguistics, as it involves combining smaller language units into meaningful phrases and sentences, a subject extensively studied in neurolinguistics. However, quantifying this cognitive process computationally has proven challenging.

The authors propose a novel metric called the Composition Score. This metric leverages key insights from transformer architectures, specifically the key-value memory mechanism found in the feed-forward network blocks of LLMs. By interpreting these network blocks, the Composition Score is designed to evaluate the degree of meaning composition occurring during the comprehension of a sentence.

Key findings from the paper include:

  • Correlation with Brain Activity: The Composition Score was found to correlate with specific brain clusters that are involved in linguistic processing. These clusters are associated with:
    • Word Frequency: Areas sensitive to how often words occur in language use.
    • Structural Processing: Regions involved in understanding the grammar and syntax of sentences.
    • General Sensitivity to Words: Brain areas that respond to words in a broad linguistic context.

These correlations suggest that the Composition Score captures a multifaceted aspect of meaning composition linked to various cognitive processes in the brain. This underscores the complexity of meaning making in human language comprehension and opens new avenues for further research into the computational modeling of semantic processing in the brain.

Overall, the development of the Composition Score represents a significant step in bridging computational models and neurolinguistic research, providing a new tool for exploring how meaning is constructed and represented in the human brain.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown