Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
80 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Integrating LLM, EEG, and Eye-Tracking Biomarker Analysis for Word-Level Neural State Classification in Semantic Inference Reading Comprehension (2309.15714v2)

Published 27 Sep 2023 in cs.CL and cs.AI

Abstract: With the recent proliferation of LLMs, such as Generative Pre-trained Transformers (GPT), there has been a significant shift in exploring human and machine comprehension of semantic language meaning. This shift calls for interdisciplinary research that bridges cognitive science and NLP. This pilot study aims to provide insights into individuals' neural states during a semantic relation reading-comprehension task. We propose jointly analyzing LLMs, eye-gaze, and electroencephalographic (EEG) data to study how the brain processes words with varying degrees of relevance to a keyword during reading. We also use a feature engineering approach to improve the fixation-related EEG data classification while participants read words with high versus low relevance to the keyword. The best validation accuracy in this word-level classification is over 60\% across 12 subjects. Words of high relevance to the inference keyword had significantly more eye fixations per word: 1.0584 compared to 0.6576 when excluding no-fixation words, and 1.5126 compared to 1.4026 when including them. This study represents the first attempt to classify brain states at a word level using LLM knowledge. It provides valuable insights into human cognitive abilities and the realm of AGI, and offers guidance for developing potential reading-assisted technologies.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (43)
  1. H. Wang, T. Fu, Y. Du, W. Gao, K. Huang, Z. Liu, P. Chandak, S. Liu, P. Van Katwyk, A. Deac et al., “Scientific discovery in the age of artificial intelligence,” Nature, vol. 620, no. 7972, pp. 47–60, 2023.
  2. K. Singhal, S. Azizi, T. Tu, S. S. Mahdavi, J. Wei, H. W. Chung, N. Scales, A. Tanwani, H. Cole-Lewis, S. Pfohl et al., “Large language models encode clinical knowledge,” Nature, pp. 1–9, 2023.
  3. M. Abdullah, A. Madain, and Y. Jararweh, “Chatgpt: Fundamentals, applications and social impacts,” in 2022 Ninth International Conference on Social Networks Analysis, Management and Security (SNAMS).   IEEE, 2022, pp. 1–8.
  4. S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. Lundberg et al., “Sparks of artificial general intelligence: Early experiments with gpt-4,” arXiv preprint arXiv:2303.12712, 2023.
  5. D. Gunning, M. Stefik, J. Choi, T. Miller, S. Stumpf, and G.-Z. Yang, “Xai—explainable artificial intelligence,” Science robotics, vol. 4, no. 37, p. eaay7120, 2019.
  6. M. A. Just and P. A. Carpenter, “A theory of reading: from eye fixations to comprehension.” Psychological review, vol. 87, no. 4, p. 329, 1980.
  7. K. Rayner, “Eye movements in reading and information processing: 20 years of research.” Psychological bulletin, vol. 124, no. 3, p. 372, 1998.
  8. M. Binz and E. Schulz, “Using cognitive psychology to understand gpt-3,” Proceedings of the National Academy of Sciences, vol. 120, no. 6, p. e2218523120, 2023.
  9. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray et al., “Training language models to follow instructions with human feedback,” Advances in Neural Information Processing Systems, vol. 35, pp. 27 730–27 744, 2022.
  10. C. Pandarinath, P. Nuyujukian, C. H. Blabe, B. L. Sorice, J. Saab, F. R. Willett, L. R. Hochberg, K. V. Shenoy, and J. M. Henderson, “High performance communication by people with paralysis using an intracortical brain-computer interface,” Elife, vol. 6, p. e18554, 2017.
  11. D. Shawky and A. Badawi, “Towards a personalized learning experience using reinforcement learning,” Machine learning paradigms: Theory and application, pp. 169–187, 2019.
  12. M. Kutas and K. D. Federmeier, “Thirty years and counting: finding meaning in the n400 component of the event-related brain potential (erp),” Annual review of psychology, vol. 62, pp. 621–647, 2011.
  13. M. Kutas and S. A. Hillyard, “Reading senseless sentences: Brain potentials reflect semantic incongruity,” Science, vol. 207, no. 4427, pp. 203–205, 1980.
  14. W. Kintsch, “The role of knowledge in discourse comprehension: a construction-integration model.” Psychological review, vol. 95, no. 2, p. 163, 1988.
  15. J. S. B. Evans, “Dual-processing accounts of reasoning, judgment, and social cognition,” Annu. Rev. Psychol., vol. 59, pp. 255–278, 2008.
  16. D. E. Rumelhart, “Schemata: The building blocks of cognition,” in Theoretical issues in reading comprehension.   Routledge, 2017, pp. 33–58.
  17. R. C. Anderson, “Role of the reader’s schema in comprehension, learning, and memory,” in Theoretical models and processes of literacy.   Routledge, 2018, pp. 136–145.
  18. D. S. McNamara and J. Magliano, “Toward a comprehensive model of comprehension,” Psychology of learning and motivation, vol. 51, pp. 297–384, 2009.
  19. L. Baretta, L. M. B. Tomitch, V. K. Lim, and K. E. Waldie, “Investigating reading comprehension through eeg,” Ilha do Desterro: A Journal of English Language, Literatures in English and Cultural Studies, no. 63, pp. 69–99, 2012.
  20. M. F. Mridha, S. C. Das, M. M. Kabir, A. A. Lima, M. R. Islam, and Y. Watanobe, “Brain-computer interface: Advancement and challenges,” Sensors, vol. 21, no. 17, p. 5746, 2021.
  21. H. Zeng, C. Yang, G. Dai, F. Qin, J. Zhang, and W. Kong, “Eeg classification of driver mental states by deep learning,” Cognitive neurodynamics, vol. 12, pp. 597–606, 2018.
  22. R. J. Seitz, R. Schäfer, D. Scherfeld, S. Friederichs, K. Popp, H.-J. Wittsack, N. Azari, and M. Franz, “Valuating other people’s emotional face expression: a combined functional magnetic resonance imaging and electroencephalography study,” Neuroscience, vol. 152, no. 3, pp. 713–722, 2008.
  23. M. Tanaka, A. Ishii, and Y. Watanabe, “Neural effects of mental fatigue caused by continuous attention load: a magnetoencephalography study,” Brain research, vol. 1561, pp. 60–66, 2014.
  24. A. C. Jenkins, “Rethinking cognitive load: a default-mode network perspective,” Trends in Cognitive Sciences, vol. 23, no. 7, pp. 531–533, 2019.
  25. Q. Wang, S. Yang, M. Liu, Z. Cao, and Q. Ma, “An eye-tracking study of website complexity from cognitive load perspective,” Decision support systems, vol. 62, pp. 1–10, 2014.
  26. N. Hollenstein, J. Rotsztejn, M. Troendle, A. Pedroni, C. Zhang, and N. Langer, “Zuco, a simultaneous eeg and eye-tracking resource for natural sentence reading,” Scientific data, vol. 5, no. 1, pp. 1–13, 2018.
  27. H. Brouwer, H. Fitz, and J. Hoeks, “Getting real about semantic illusions: Rethinking the functional role of the p600 in language comprehension,” Brain research, vol. 1446, pp. 127–143, 2012.
  28. C. D. Manning, M. Surdeanu, J. Bauer, J. R. Finkel, S. Bethard, and D. McClosky, “The stanford corenlp natural language processing toolkit,” in Proceedings of 52nd annual meeting of the association for computational linguistics: system demonstrations, 2014, pp. 55–60.
  29. S. Aydore, D. Pantazis, and R. M. Leahy, “A note on the phase locking value and its properties,” Neuroimage, vol. 74, pp. 231–244, 2013.
  30. T.-P. Jung, S. Makeig, M. Westerfield, J. Townsend, E. Courchesne, and T. J. Sejnowski, “Analyzing and visualizing single-trial event-related potentials,” in Advances in Neural Information Processing Systems, M. Kearns, S. Solla, and D. Cohn, Eds., vol. 11.   MIT Press, 1998.
  31. M. Rubinov and O. Sporns, “Complex network measures of brain connectivity: uses and interpretations,” Neuroimage, vol. 52, no. 3, pp. 1059–1069, 2010.
  32. Y. Zhang, Y. Liao, Y. Zhang, and L. Huang, “Emergency braking intention detect system based on k-order propagation number algorithm: a network perspective,” Brain Sciences, vol. 11, no. 11, p. 1424, 2021.
  33. W. Ding, Y. Zhang, and L. Huang, “Using a novel functional brain network approach to locate important nodes for working memory tasks,” International journal of environmental research and public health, vol. 19, no. 6, p. 3564, 2022.
  34. Y. Chen, Y. Zhang, W. Ding, F. Cui, and L. Huang, “Research on working memory states based on weighted-order propagation number algorithm: An eeg perspective,” Journal of Sensors, vol. 2022, 2022.
  35. E. Bullmore and O. Sporns, “Complex brain networks: graph theoretical analysis of structural and functional systems,” Nature reviews neuroscience, vol. 10, no. 3, pp. 186–198, 2009.
  36. K.-J. Chiang, S. Dong, C.-K. Cheng, and T.-P. Jung, “Using eeg signals to assess workload during memory retrieval in a real-world scenario,” Journal of Neural Engineering, vol. 20, no. 3, p. 036010, 2023.
  37. F. Lotte, L. Bougrain, A. Cichocki, M. Clerc, M. Congedo, A. Rakotomamonjy, and F. Yger, “A review of classification algorithms for eeg-based brain–computer interfaces: a 10 year update,” Journal of Neural Engineering, vol. 15, no. 3, p. 031005, apr 2018. [Online]. Available: https://dx.doi.org/10.1088/1741-2552/aab2f2
  38. A. Craik, Y. He, and J. L. Contreras-Vidal, “Deep learning for electroencephalogram (eeg) classification tasks: a review,” Journal of Neural Engineering, vol. 16, no. 3, p. 031001, apr 2019. [Online]. Available: https://dx.doi.org/10.1088/1741-2552/ab0ab5
  39. Y. Li, R. Yin, H. Park, Y. Kim, and P. Panda, “Wearable-based human activity recognition with spatio-temporal spiking neural networks,” 2022.
  40. C. Tallon-Baudry, O. Bertrand, M.-A. Hénaff, J. Isnard, and C. Fischer, “Attention Modulates Gamma-band Oscillations Differently in the Human Lateral Occipital Cortex and Fusiform Gyrus,” Cerebral Cortex, vol. 15, no. 5, pp. 654–662, 09 2004. [Online]. Available: https://doi.org/10.1093/cercor/bhh167
  41. V. Wyart and C. Tallon-Baudry, “How ongoing fluctuations in human visual cortex predict perceptual awareness: baseline shift versus decision bias,” Journal of Neuroscience, vol. 29, no. 27, pp. 8715–8725, 2009.
  42. S. Palva, S. Kulashekhar, M. Hämäläinen, and J. M. Palva, “Localization of cortical phase and amplitude dynamics during visual working memory encoding and retention,” Journal of Neuroscience, vol. 31, no. 13, pp. 5013–5025, 2011.
  43. S.-C. Chen, H.-C. She, M.-H. Chuang, J.-Y. Wu, J.-L. Tsai, and T.-P. Jung, “Eye movements predict students’ computer-based assessment performance of physics concepts in different presentation modalities,” Computers & Education, vol. 74, pp. 61–72, 2014.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (7)
  1. Yuhong Zhang (27 papers)
  2. Qin Li (179 papers)
  3. Sujal Nahata (1 paper)
  4. Tasnia Jamal (1 paper)
  5. Shih-kuen Cheng (1 paper)
  6. Gert Cauwenberghs (25 papers)
  7. Tzyy-Ping Jung (23 papers)
Citations (1)
View on Semantic Scholar