Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
167 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Multimodal Brain-Computer Interface for In-Vehicle Driver Cognitive Load Measurement: Dataset and Baselines (2304.04273v2)

Published 9 Apr 2023 in cs.LG, cs.HC, and eess.SP

Abstract: Through this paper, we introduce a novel driver cognitive load assessment dataset, CL-Drive, which contains Electroencephalogram (EEG) signals along with other physiological signals such as Electrocardiography (ECG) and Electrodermal Activity (EDA) as well as eye tracking data. The data was collected from 21 subjects while driving in an immersive vehicle simulator, in various driving conditions, to induce different levels of cognitive load in the subjects. The tasks consisted of 9 complexity levels for 3 minutes each. Each driver reported their subjective cognitive load every 10 seconds throughout the experiment. The dataset contains the subjective cognitive load recorded as ground truth. In this paper, we also provide benchmark classification results for different machine learning and deep learning models for both binary and ternary label distributions. We followed 2 evaluation criteria namely 10-fold and leave-one-subject-out (LOSO). We have trained our models on both hand-crafted features as well as on raw data.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (88)
  1. M. Miyaji, H. Kawanaka, and K. Oguri, “Driver’s cognitive distraction detection using physiological features by the adaboost,” in 12th International IEEE Conference on Intelligent Transportation Systems.   IEEE, 2009, pp. 1–6.
  2. O. Palinko, A. L. Kun, A. Shyrokov, and P. Heeman, “Estimating cognitive load using remote eye tracking in a driving simulator,” in Proceedings of the Symposium on Eye-tracking Research & Applications, 2010, pp. 141–144.
  3. L. Fridman, B. Reimer, B. Mehler, and W. T. Freeman, “Cognitive load estimation in the wild,” in Proceedings of the Chi Conference on Human Factors in Computing Systems, 2018, pp. 1–9.
  4. A. Yüce, H. Gao, G. L. Cuendet, and J.-P. Thiran, “Action units and their cross-correlations for prediction of cognitive load during driving,” IEEE Transactions on Affective Computing, vol. 8, no. 2, pp. 161–175, 2016.
  5. E. Q. Wu, D. Hu, P.-Y. Deng, Z. Tang, Y. Cao, W.-M. Zhang, L.-M. Zhu, and H. Ren, “Nonparametric bayesian prior inducing deep network for automatic detection of cognitive status,” IEEE Transactions on Cybernetics, vol. 51, no. 11, pp. 5483–5496, 2020.
  6. E. Q. Wu, Z. Tang, Y. Yao, X.-Y. Qiu, P.-Y. Deng, P. Xiong, A. Song, L.-M. Zhu, and M. Zhou, “Scalable gamma-driven multilayer network for brain workload detection through functional near-infrared spectroscopy,” IEEE Transactions on Cybernetics, vol. 52, no. 11, pp. 12 464–12 478, 2021.
  7. J. Sweller, “Cognitive load theory,” in Psychology of Learning and Motivation.   Elsevier, 2011, vol. 55, pp. 37–76.
  8. R. Das, D. Chatterjee, D. Das, A. Sinharay, and A. Sinha, “Cognitive load measurement-a methodology to compare low cost commercial eeg devices,” in International Conference on Advances in Computing, Communications and Informatics.   IEEE, 2014, pp. 1188–1194.
  9. S. Schneegass, B. Pfleging, N. Broy, F. Heinrich, and A. Schmidt, “A data set of real world driving to assess driver workload,” in Proceedings of the 5th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, 2013, pp. 150–157.
  10. I. Mijić, M. Šarlija, and D. Petrinović, “Mmod-cog: A database for multimodal cognitive load classification,” in 11th International Symposium on Image and Signal Processing and Analysis.   IEEE, 2019, pp. 15–20.
  11. V. Markova, T. Ganchev, and K. Kalinkov, “Clas: A database for cognitive load, affect and stress recognition,” in International Conference on Biomedical Innovations and Applications.   IEEE, 2019, pp. 1–4.
  12. M. Gjoreski, T. Kolenik, T. Knez, M. Luštrek, M. Gams, H. Gjoreski, and V. Pejović, “Datasets for cognitive load inference using wearable sensors and psychological traits,” Applied Sciences, vol. 10, no. 11, p. 3843, 2020.
  13. A. Kalatzis, A. Teotia, V. G. Prabhu, and L. Stanley, “A database for cognitive workload classification using electrocardiogram and respiration signal,” in International Conference on Applied Human Factors and Ergonomics.   Springer, 2021, pp. 509–516.
  14. P. Sarkar, K. Ross, A. J. Ruberto, D. Rodenbura, P. Hungler, and A. Etemad, “Classification of cognitive load and expertise for adaptive simulation using deep multitask learning,” in 8th International Conference on Affective Computing and Intelligent Interaction.   IEEE, 2019, pp. 1–7.
  15. K. Ross, P. Sarkar, D. Rodenburg, A. Ruberto, P. Hungler, A. Szulewski, D. Howes, and A. Etemad, “Toward dynamically adaptive simulation: Multimodal classification of user expertise using wearable devices,” Journal of Sensors, vol. 19, no. 19, p. 4270, 2019.
  16. R. Brunken, J. L. Plass, and D. Leutner, “Direct measurement of cognitive load in multimedia learning,” Educational Psychologist, vol. 38, no. 1, pp. 53–61, 2003.
  17. R. E. Mayer and R. Moreno, “Aids to computer-based multimedia learning,” Learning and Instruction, vol. 12, no. 1, pp. 107–119, 2002.
  18. M. Klepsch, F. Schmitz, and T. Seufert, “Development and validation of two instruments measuring intrinsic, extraneous, and germane cognitive load,” Frontiers in Psychology, vol. 8, p. 1997, 2017.
  19. F. G. Paas, “Training strategies for attaining transfer of problem-solving skill in statistics: a cognitive-load approach,” Journal of Educational Psychology, vol. 84, no. 4, p. 429, 1992.
  20. S. G. Hart and L. E. Staveland, “Development of nasa-tlx (task load index): Results of empirical and theoretical research,” in Advances in Psychology.   Elsevier, 1988, vol. 52, pp. 139–183.
  21. R. Brünken, S. Steinbacher, J. L. Plass, and D. Leutner, “Assessment of cognitive load in multimedia learning using dual-task methodology.” Experimental Psychology, vol. 49, no. 2, p. 109, 2002.
  22. B. Park and R. Brünken, “The rhythm method: A new method for measuring cognitive load—an experimental dual-task study,” Applied Cognitive Psychology, vol. 29, no. 2, pp. 232–243, 2015.
  23. P. W. Van Gerven, F. Paas, J. J. Van Merriënboer, and H. G. Schmidt, “Memory load and the cognitive pupillary response in aging,” Psychophysiology, vol. 41, no. 2, pp. 167–174, 2004.
  24. S. Chen and J. Epps, “Using task-induced pupil diameter and blink rate to infer cognitive load,” Human–Computer Interaction, vol. 29, no. 4, pp. 390–413, 2014.
  25. F. G. Paas and J. J. Van Merriënboer, “Variability of worked examples and transfer of geometrical problem-solving skills: A cognitive-load approach,” Journal of Educational Psychology, vol. 86, no. 1, p. 122, 1994.
  26. P. Antonenko, F. Paas, R. Grabner, and T. Van Gog, “Using electroencephalography to measure cognitive load,” Educational Psychology Review, vol. 22, no. 4, pp. 425–438, 2010.
  27. S. Chisholm, J. K. Caird, and J. Lockhart, “The effects of practice with mp3 players on driving performance,” Accident Analysis & Prevention, vol. 40, no. 2, pp. 704–713, 2008.
  28. D. He, B. Donmez, C. C. Liu, and K. N. Plataniotis, “High cognitive load assessment in drivers through wireless electroencephalography and the validation of a modified n-back task,” IEEE Transactions on Human-Machine Systems, vol. 49, no. 4, pp. 362–371, 2019.
  29. S. Barua, M. U. Ahmed, and S. Begum, “Classifying drivers’ cognitive load using eeg signals.” in pHealth, 2017, pp. 99–106.
  30. N. M. Yusof, J. Karjanto, M. Z. Hassan, J. Terken, F. Delbressine, and M. Rauterberg, “Reading during fully automated driving: a study of the effect of peripheral visual and haptic information on situation awareness and mental workload,” IEEE transactions on intelligent transportation systems, vol. 23, no. 10, pp. 19 136–19 144, 2022.
  31. Z. Jiang, X. He, C. Lu, B. Zhou, X. Fan, C. Wang, X. Ma, E. C. Ngai, and L. Chen, “Understanding drivers’ visual and comprehension loads in traffic violation hotspots leveraging crowd-based driving simulation,” IEEE transactions on intelligent transportation systems, vol. 23, no. 12, pp. 23 369–23 383, 2022.
  32. J. Ayoub, N. Du, X. J. Yang, and F. Zhou, “Predicting driver takeover time in conditionally automated driving,” IEEE transactions on intelligent transportation systems, vol. 23, no. 7, pp. 9580–9589, 2022.
  33. B. Mehler, B. Reimer, J. F. Coughlin, and J. A. Dusek, “Impact of incremental increases in cognitive workload on physiological arousal and performance in young adult drivers,” Transportation Research Record, vol. 2138, no. 1, pp. 6–12, 2009.
  34. B. Mehler, B. Reimer, and J. F. Coughlin, “Sensitivity of physiological measures for detecting systematic variations in cognitive demand from a working memory task: an on-road study across three age groups,” Human Factors, vol. 54, no. 3, pp. 396–412, 2012.
  35. J. L. Kolodner, “An introduction to case-based reasoning,” Artificial Intelligence Review, vol. 6, no. 1, pp. 3–34, 1992.
  36. M. Braun, J. Schubert, B. Pfleging, and F. Alt, “Improving driver emotions with affective strategies,” Multimodal Technologies and Interaction, vol. 3, no. 1, p. 21, 2019.
  37. C. Nass, I.-M. Jonsson, H. Harris, B. Reaves, J. Endo, S. Brave, and L. Takayama, “Improving automotive safety by pairing driver emotion and car voice emotion,” in CHI’05 Extended Abstracts on Human Factors in Computing Systems, 2005, pp. 1973–1976.
  38. S. Zepf, J. Hernandez, A. Schmitt, W. Minker, and R. W. Picard, “Driver emotion recognition for intelligent vehicles: A survey,” ACM Computing Surveys (CSUR), vol. 53, no. 3, pp. 1–30, 2020.
  39. G. Zhang and A. Etemad, “Capsule attention for multimodal eeg-eog representation learning with application to driver vigilance estimation,” IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 29, pp. 1138–1149, 2021.
  40. L. M. Bergasa, J. Nuevo, M. A. Sotelo, R. Barea, and M. E. Lopez, “Real-time system for monitoring driver vigilance,” IEEE Transactions on Intelligent Transportation Systems, vol. 7, no. 1, pp. 63–77, 2006.
  41. C.-T. Lin, C.-H. Chuang, C.-S. Huang, S.-F. Tsai, S.-W. Lu, Y.-H. Chen, and L.-W. Ko, “Wireless and wearable eeg system for evaluating driver vigilance,” IEEE Transactions on Biomedical Circuits and Systems, vol. 8, no. 2, pp. 165–176, 2014.
  42. Z. Emami and T. Chau, “The effects of visual distractors on cognitive load in a motor imagery brain-computer interface,” Behavioural Brain Research, vol. 378, p. 112240, 2020.
  43. W. J. Chai, A. I. Abd Hamid, and J. M. Abdullah, “Working memory from the psychological and neurosciences perspectives: a review,” Frontiers in Psychology, vol. 9, p. 401, 2018.
  44. S. Siuly, Y. Li, and Y. Zhang, “Eeg signal analysis and classification,” IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 11, pp. 141–144, 2016.
  45. W. Klimesch, “Eeg alpha and theta oscillations reflect cognitive and memory performance: a review and analysis,” Brain Research Reviews, vol. 29, no. 2–3, pp. 169–195, 1999.
  46. K. Ross, P. Hungler, and A. Etemad, “Unsupervised multi-modal representation learning for affective computing with multi-corpus wearable data,” Journal of Ambient Intelligence and Humanized Computing, pp. 1–26, 2021.
  47. R. Xiong, F. Kong, X. Yang, G. Liu, and W. Wen, “Pattern recognition of cognitive load using eeg and ecg signals,” Journal of Sensors, vol. 20, no. 18, p. 5122, 2020.
  48. A. M. Hughes, G. M. Hancock, S. L. Marlow, K. Stowers, and E. Salas, “Cardiac measures of cognitive workload: a meta-analysis,” Human Factors, vol. 61, no. 3, pp. 393–414, 2019.
  49. E. Johannessen, A. Szulewski, N. Radulovic, M. White, H. Braund, D. Howes, D. Rodenburg, and C. Davies, “Psychophysiologic measures of cognitive load in physician team leaders during trauma resuscitation,” Computers in Human Behavior, vol. 111, p. 106393, 2020.
  50. J. Zagermann, U. Pfeil, and H. Reiterer, “Measuring cognitive load using eye tracking technology in visual computing,” in Proceedings of the 6th Workshop on Beyond Time and Errors on Novel Evaluation Methods for Visualization, 2016, pp. 78–85.
  51. S. T. Iqbal, P. D. Adamczyk, X. S. Zheng, and B. P. Bailey, “Understanding changes in mental workload during task execution,” Tech. Rep., 2004.
  52. J. Engström, G. Markkula, T. Victor, and N. Merat, “Effects of cognitive load on driving performance: The cognitive control hypothesis,” Human Factors, vol. 59, no. 5, pp. 734–764, 2017.
  53. P. Sena, M. d’Amore, M. Pappalardo, A. Pellegrino, A. Fiorentino, and F. Villecco, “Studying the influence of cognitive load on driver’s performances by a fuzzy analysis of lane keeping in a drive simulation,,” IFAC Proceedings Volumes, vol. 46, no. 21, pp. 151–156, 2013.
  54. L. Cabañero, R. Hervás, I. González, J. Fontecha, T. Mondéjar, and J. Bravo, “Analysis of cognitive load using eeg when interacting with mobile devices,” Multidisciplinary Digital Publishing Institute Proceedings, vol. 31, no. 1, p. 70, 2019.
  55. I. Volman, K. Roelofs, S. Koch, L. Verhagen, and I. Toni, “Anterior prefrontal cortex inhibition impairs control over social emotional actions,” Current biology, vol. 21, no. 20, pp. 1766–1770, 2011.
  56. G. Zhang and A. Etemad, “Deep recurrent semi-supervised eeg representation learning for emotion recognition,” in 9th International Conference on Affective Computing and Intelligent Interaction.   IEEE, 2021, pp. 1–8.
  57. G. Zhang, V. Davoodnia, and A. Etemad, “Parse: Pairwise alignment of representations in semi-supervised eeg learning for emotion recognition,” IEEE Transactions on Affective Computing, 2022.
  58. J. R. Stroop, “Studies of interference in serial verbal reactions.” Journal of Experimental Psychology, vol. 18, no. 6, p. 643, 1935.
  59. F. Schmiedek, M. Lövdén, and U. Lindenberger, “A task is a task is a task: putting complex span, n-back, and other working memory indicators in psychometric context,” Frontiers in Psychology, vol. 5, p. 1475, 2014.
  60. Y. Santiago-Espada, R. R. Myer, K. A. Latorella, and J. R. Comstock Jr, “The multi-attribute task battery ii (matb-ii) software for human performance and workload research: A user’s guide,” Tech. Rep., 2011.
  61. A. Morley, L. Hill, and A. Kaditis, “10-20 system eeg placement,” European Respiratory Society, European Respiratory Society, 2016.
  62. V. Jurcak, D. Tsuzuki, and I. Dan, “10/20, 10/10, and 10/5 systems revisited: their validity as relative head-surface-based positioning systems,” Neuroimage, vol. 34, no. 4, pp. 1600–1611, 2007.
  63. A. Burns, B. R. Greene, M. J. McGrath, T. J. O’Shea, B. Kuris, S. M. Ayer, F. Stroiescu, and V. Cionca, “SHIMMER–a wireless sensor platform for noninvasive biomedical research,” IEEE Sensors Journal, vol. 10, no. 9, pp. 1527–1534, 2010.
  64. M. Rizzo, P. A. Sheffield, L. Stierman, and J. Dawson, “Demographic and driving performance factors in simulator adaptation syndrome,” in Driving Assesment Conference, vol. 2, no. 2003.   University of Iowa, 2003.
  65. J. G. Reed-Jones, W. J. Reed-Jones, L. M. Trick, R. Toxopeus, and L. A. Vallis, “Comparing techniques to reduce simulator adaptation syndrome and improve naturalistic behaviour during simulated driving,” in Driving Assesment Conference, vol. 5, no. 2009.   University of Iowa, 2009.
  66. S. V. Cobb, S. Nichols, A. Ramsey, and J. R. Wilson, “Virtual reality-induced symptoms and effects (vrise),” Presence: Teleoperators & Virtual Environments, vol. 8, no. 2, pp. 169–186, 1999.
  67. G. Gálvez-García, J. Albayay, L. Rehbein, and F. Tornay, “Mitigating simulator adaptation syndrome by means of tactile stimulation,” Applied Ergonomics, vol. 58, pp. 13–17, 2017.
  68. L. Frank, R. S. Kennedy, R. S. Kellogg, and M. E. McCauley, “Simulator sickness: A reaction to a transformed perceptual world. 1. scope of the problem,” ESSEX CORP ORLANDO FL, Tech. Rep., 1983.
  69. R. S. Kennedy, N. E. Lane, K. S. Berbaum, and M. G. Lilienthal, “Simulator sickness questionnaire: An enhanced method for quantifying simulator sickness,” The International Journal of Aviation Psychology, vol. 3, no. 3, pp. 203–220, 1993.
  70. P. J. Gianaros, E. R. Muth, J. T. Mordkoff, M. E. Levine, and R. M. Stern, “A questionnaire for the assessment of the multiple dimensions of motion sickness,” Aviation, Space, and Environmental Medicine, vol. 72, no. 2, p. 115, 2001.
  71. F. G. Paas, “Training strategies for attaining transfer of problem-solving skill in statistics: a cognitive-load approach.” Journal of Educational Psychology, vol. 84, no. 4, p. 429, 1992.
  72. N. Thakor, J. Webster, and W. Tompkins, “Optimal qrs detector,” Medical and Biological Engineering and Computing, vol. 21, no. 3, pp. 343–350, 1983.
  73. H. G. Goovaerts, H. H. Ros, T. J. Van Den Akker, and H. Schneider, “A digital qrs detector based on the principle of contour limiting,” IEEE Transactions on Biomedical Engineering, no. 2, pp. 154–160, 1976.
  74. C. L. Lim, C. Rennie, R. J. Barry, H. Bahramali, I. Lazzaro, B. Manor, and E. Gordon, “Decomposing skin conductance into tonic and phasic components,” International Journal of Psychophysiology, vol. 25, no. 2, pp. 97–109, 1997.
  75. C. E. Shannon, “A mathematical theory of communication,” The Bell System Technical Journal, vol. 27, no. 3, pp. 379–423, 1948.
  76. B. Hjorth, “Eeg analysis based on time domain properties,” Electroencephalography and Clinical Neurophysiology, vol. 29, no. 3, pp. 306–310, 1970.
  77. A. Lempel and J. Ziv, “On the complexity of finite sequences,” IEEE Transactions on Information Theory, vol. 22, no. 1, pp. 75–81, 1976.
  78. F. Kaspar and H. Schuster, “Easily calculable measure for the complexity of spatiotemporal patterns,” Physical Review A, vol. 36, no. 2, p. 842, 1987.
  79. T. Higuchi, “Approach to an irregular time series on the basis of the fractal theory,” Physica D: Nonlinear Phenomena, vol. 31, no. 2, pp. 277–283, 1988.
  80. A. H. Al-Nuaimi, E. Jammeh, L. Sun, and E. Ifeachor, “Higuchi fractal dimension of the electroencephalogram as a biomarker for early detection of alzheimer’s disease,” in 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.   IEEE, 2017, pp. 2320–2324.
  81. E. Shamsi, M. A. Ahmadi-Pajouh, and T. S. Ala, “Higuchi fractal dimension: An efficient approach to detection of brain entrainment to theta binaural beats,” Biomedical Signal Processing and Control, vol. 68, p. 102580, 2021.
  82. F. Shaffer and J. P. Ginsberg, “An overview of heart rate variability metrics and norms,” Frontiers in Public Health, p. 258, 2017.
  83. P. Sarkar and A. Etemad, “Self-supervised ecg representation learning for emotion recognition,” IEEE Transactions on Affective Computing, 2020.
  84. I. Kalamaras, A. Zamichos, A. Salamanis, A. Drosou, D. D. Kehagias, G. Margaritis, S. Papadopoulos, and D. Tzovaras, “An interactive visual analytics platform for smart intelligent transportation systems management,” IEEE Transactions on Intelligent Transportation Systems, vol. 19, no. 2, pp. 487–496, 2017.
  85. Z. Li, G. Xiong, Y. Tian, Y. Lv, Y. Chen, P. Hui, and X. Su, “A multi-stream feature fusion approach for traffic prediction,” IEEE transactions on intelligent transportation systems, vol. 23, no. 2, pp. 1456–1466, 2020.
  86. Y. Chen, T. Shu, X. Zhou, X. Zheng, A. Kawai, K. Fueda, Z. Yan, W. Liang, I. Kevin, and K. Wang, “Graph attention network with spatial-temporal clustering for traffic flow forecasting in intelligent transportation system,” IEEE Transactions on Intelligent Transportation Systems, 2022.
  87. D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” International Conference on Learning Representations, 2015.
  88. W.-L. Zheng, W. Liu, Y. Lu, B.-L. Lu, and A. Cichocki, “Emotionmeter: A multimodal framework for recognizing human emotions,” IEEE Transactions on Cybernetics, vol. 49, no. 3, pp. 1110–1122, 2018.
Citations (4)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now