Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
143 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

On the Shape of Brainscores for Large Language Models (LLMs) (2405.06725v3)

Published 10 May 2024 in q-bio.NC, cs.AI, cs.CL, and cs.LG

Abstract: With the rise of LLMs, the novel metric "Brainscore" emerged as a means to evaluate the functional similarity between LLMs and human brain/neural systems. Our efforts were dedicated to mining the meaning of the novel score by constructing topological features derived from both human fMRI data involving 190 subjects, and 39 LLMs plus their untrained counterparts. Subsequently, we trained 36 Linear Regression Models and conducted thorough statistical analyses to discern reliable and valid features from our constructed ones. Our findings reveal distinctive feature combinations conducive to interpreting existing brainscores across various brain regions of interest (ROIs) and hemispheres, thereby significantly contributing to advancing interpretable machine learning (iML) studies. The study is enriched by our further discussions and analyses concerning existing brainscores. To our knowledge, this study represents the first attempt to comprehend the novel metric brainscore within this interdisciplinary domain.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (80)
  1. A connectomic atlas of the human cerebrum—chapter 7: the lateral parietal lobe. Operative Neurosurgery, 15(Suppl 1):S295, 2018.
  2. Open llm leaderboard. https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard, 2023.
  3. Melinda Zsofia Bognar. Prospects of ai in architecture: Symbolicism, connectionism, actionism. Journal of Architectural Informatics Society, 2022.
  4. Geometric and topological inference, volume 57. Cambridge University Press, 2018.
  5. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712, 2023.
  6. The clinician’s guide to interpreting a regression analysis. Eye, 36(9):1715–1717, 2022.
  7. Gunnar Carlsson. Topology and data. Bulletin of the American Mathematical Society, 46(2):255–308, 2009.
  8. Topological data analysis with applications. Cambridge University Press, 2021.
  9. Nathan Carter. Data Science for Mathematicians. CRC Press, 2020.
  10. Brains and algorithms partially converge in natural language processing. Communications biology, 5(1):134, 2022.
  11. Evidence of a predictive coding hierarchy in the human brain listening to speech. Nature human behaviour, 7(3):430–441, 2023.
  12. The coefficient of determination r-squared is more informative than smape, mae, mape, mse and rmse in regression analysis evaluation. PeerJ Computer Science, 7:e623, 2021.
  13. Lipschitz functions have l p -stable persistence. Foundations of Computational Mathematics, 10(2):127–139, Jan 2010. doi: https://doi.org/10.1007/s10208-010-9060-6.
  14. Cross-lingual language model pretraining. Advances in neural information processing systems, 32, 2019.
  15. Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116, 2019.
  16. Proper and common names in the semantic system. Brain Structure and Function, 228(1):239–254, 2023.
  17. Automatic parcellation of human cortical gyri and sulci using standard anatomical nomenclature. Neuroimage, 53(1):1–15, 2010.
  18. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
  19. Computational topology for data analysis. Cambridge University Press, 2022.
  20. Computational topology: an introduction. American Mathematical Society, 2010.
  21. fmriprep: a robust preprocessing pipeline for functional mri. Nature methods, 16(1):111–116, 2019.
  22. New method for fmri investigations of language: defining rois functionally in individual subjects. Journal of neurophysiology, 104(2):1177–1194, 2010.
  23. Lexical and syntactic representations in the brain: an fmri investigation with multi-voxel pattern analyses. Neuropsychologia, 50(4):499–513, 2012.
  24. Neural correlate of the construction of sentence meaning. Proceedings of the National Academy of Sciences, 113(41):E6256–E6262, 2016.
  25. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323, 2022.
  26. OPTQ: accurate quantization for generative pre-trained transformers. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https://openreview.net/pdf?id=tcbBPnfwxS.
  27. Persistent homology: a step-by-step introduction for newcomers. In STAG, pp.  1–10, 2016.
  28. A multi-modal parcellation of human cerebral cortex. Nature, 536(7615):171–178, 2016.
  29. Textbooks are all you need. arXiv preprint arXiv:2306.11644, 2023.
  30. The human brain reactivates context-specific past information at event boundaries of naturalistic experiences. Nature neuroscience, 26(6):1080–1089, 2023.
  31. Alyssa Hughes. Phi-2: The surprising power of small language models, 12 2023. URL https://www.microsoft.com/en-us/research/blog/phi-2-the-surprising-power-of-small-language-models/.
  32. What does bert learn about the structure of language? In ACL 2019-57th Annual Meeting of the Association for Computational Linguistics, 2019.
  33. Mapping brains with language models: A survey. arXiv preprint arXiv:2306.05126, 2023.
  34. Iwan Kawrakow. k-quants by ikawrakow · pull request #1684 · ggerganov/llama.cpp, Jun 2023. URL https://github.com/ggerganov/llama.cpp/pull/1684.
  35. Ctrl: A conditional transformer language model for controllable generation. arXiv preprint arXiv:1909.05858, 2019.
  36. Gary Koplik. Persistent homology: A non-mathy introduction with examples, Jun 2022. URL https://towardsdatascience.com/persistent-homology-with-examples-1974d4b9c3d0.
  37. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942, 2019.
  38. Structural similarities between language models and neural response measurements. In NeurIPS 2023 Workshop on Symmetry and Geometry in Neural Representations, 2023a.
  39. Textbooks are all you need ii: phi-1.5 technical report. arXiv preprint arXiv:2309.05463, 2023b.
  40. Coupling artificial neurons in bert and biological neurons in the human brain. Proceedings of the AAAI Conference on Artificial Intelligence, 37(7):8888–8896, 2023.
  41. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019.
  42. Natural and artificial intelligence: A brief introduction to the interplay between ai and neuroscience research. Neural Networks, 144:603–613, 2021.
  43. Emergent linguistic structure in artificial neural networks trained by self-supervision. Proceedings of the National Academy of Sciences, 117(48):30046–30054, 2020.
  44. The gudhi library: Simplicial complexes and persistent homology. In Mathematical Software–ICMS 2014: 4th International Congress, Seoul, South Korea, August 5-9, 2014. Proceedings 4, pp.  167–174. Springer, 2014.
  45. Sensor modalities for brain-computer interface technology: A comprehensive literature review. Neurosurgery, 86(2), Jul 2019. doi: https://doi.org/10.1093/neuros/nyz286.
  46. Syntax-sensitive regions of the posterior inferior frontal gyrus and the posterior temporal lobe are differentially recruited by production and perception. Cerebral Cortex Communications, 1(1):tgaa029, 2020.
  47. The temporal dynamics of structure and content in sentence comprehension: Evidence from fmri-constrained meg. Human Brain Mapping, 40(2):663–678, 2019. doi: https://doi.org/10.1002/hbm.24403. URL https://onlinelibrary.wiley.com/doi/abs/10.1002/hbm.24403.
  48. Parcellation-based anatomic model of the semantic network. Brain and behavior, 11(4):e02065, 2021.
  49. Christoph Molnar. Interpretable machine learning : a guide for making black box models explainable. Leanpub, 2022. ISBN 9798411463330.
  50. Outlier-robust optimal transport. In International Conference on Machine Learning, pp.  7850–7860. PMLR, 2021.
  51. Geometrical aspects of entropy production in stochastic thermodynamics based on wasserstein distance. Physical Review Research, 3(4):043093, 2021.
  52. The “narratives” fmri dataset for evaluating models of naturalistic language comprehension. Scientific data, 8(1):250, 2021.
  53. Robert Nau. What’s a good value for r-squared?, 2019. URL https://people.duke.edu/~rnau/rsquared.htm.
  54. Outlier-robust optimal transport: Duality, structure, and statistical analysis. In International Conference on Artificial Intelligence and Statistics, pp.  11691–11719. PMLR, 2022.
  55. Brain hierarchy score: Which deep neural networks are hierarchically brain-like? IScience, 24(9), 2021.
  56. How does the brain process syntactic structure while listening? In Findings of the Association for Computational Linguistics: ACL 2023, pp.  6624–6647, 2023.
  57. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830, 2011.
  58. The refinedweb dataset for falcon llm: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.01116, 2023.
  59. Russell A Poldrack. Region of interest analysis for fmri. Social cognitive and affective neuroscience, 2(1):67–70, 2007.
  60. Language models are unsupervised multitask learners, 2019.
  61. Moklesur Rahman. Understanding wasserstein distance: A powerful metric in machine learning, Jun 2023. URL https://rmoklesur.medium.com/understanding-wasserstein-distance-a-powerful-metric-in-machine-learning-100a1ff46b66.
  62. Artificial intelligence a modern approach. London, 2010.
  63. Regions of interest as nodes of dynamic functional brain networks. Network Neuroscience, 2(4):513–535, 2018.
  64. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108, 2019.
  65. Brain-score: Which artificial neural network for object recognition is most brain-like? BioRxiv, pp.  407007, 2018.
  66. The neural architecture of language: Integrative modeling converges on predictive processing. Proceedings of the National Academy of Sciences, 118(45):e2105646118, 2021.
  67. giotto-tda: A topological data analysis toolkit for machine learning and data exploration. The Journal of Machine Learning Research, 22(1):1834–1839, 2021.
  68. MosaicML NLP Team. Introducing mpt-7b: A new standard for open-source, commercially usable llms, 2023. URL www.mosaicml.com/blog/mpt-7b. Accessed: 2023-05-05.
  69. P value interpretations and considerations. Journal of thoracic disease, 8(9):E928, 2016.
  70. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
  71. Increasing brain-llm alignment via information-theoretic compression. In UniReps: the First Workshop on Unifying Representations in Neural Models, 2023. URL https://openreview.net/forum?id=WcfVyzzJOS.
  72. Katharine Turner. Katharine turner (12/3/19): Why should q=p in the wasserstein distance between persistence diagrams?, 12 2019. URL https://www.youtube.com/watch?v=d_iqovKui6k.
  73. fMRI : Basics and Clinical Applications. Springer Berlin Heidelberg, Berlin, Heidelberg, 2010. ISBN 9783540681328.
  74. fMRI : Basics and Clinical Applications (Second Edition). Springer Berlin Heidelberg, Berlin, Heidelberg, 2013. ISBN 9783642343421.
  75. Attention is all you need. Advances in neural information processing systems, 30, 2017.
  76. Žiga Virk. Introduction to persistent homology, 2022. URL https://zalozba.fri.uni-lj.si/virk2022.pdf.
  77. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32, 2019.
  78. Query augmentation by decoding semantics from brain signals. arXiv preprint arXiv:2402.15708, 2024.
  79. Time series clustering with topological and geometric mixed distance. Mathematics, 9(9):1046, 2021.
  80. A survey on model compression for large language models. arXiv preprint arXiv:2308.07633, 2023.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now