Papers
Topics
Authors
Recent
Search
2000 character limit reached

Deep Insights into Cognitive Decline: A Survey of Leveraging Non-Intrusive Modalities with Deep Learning Techniques

Published 24 Oct 2024 in cs.LG and cs.AI | (2410.18972v1)

Abstract: Cognitive decline is a natural part of aging, often resulting in reduced cognitive abilities. In some cases, however, this decline is more pronounced, typically due to disorders such as Alzheimer's disease. Early detection of anomalous cognitive decline is crucial, as it can facilitate timely professional intervention. While medical data can help in this detection, it often involves invasive procedures. An alternative approach is to employ non-intrusive techniques such as speech or handwriting analysis, which do not necessarily affect daily activities. This survey reviews the most relevant methodologies that use deep learning techniques to automate the cognitive decline estimation task, including audio, text, and visual processing. We discuss the key features and advantages of each modality and methodology, including state-of-the-art approaches like Transformer architecture and foundation models. In addition, we present works that integrate different modalities to develop multimodal models. We also highlight the most significant datasets and the quantitative results from studies using these resources. From this review, several conclusions emerge. In most cases, the textual modality achieves the best results and is the most relevant for detecting cognitive decline. Moreover, combining various approaches from individual modalities into a multimodal model consistently enhances performance across nearly all scenarios.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (298)
  1. D. Kiely, “Cognitive function in encyclopaedia of quality of life and well-being research (ed. michalos, ac) 974–978,” 2014.
  2. I. J. Deary, J. Corley, A. J. Gow, S. E. Harris, L. M. Houlihan, R. E. Marioni, L. Penke, S. B. Rafnsson, and J. M. Starr, “Age-associated cognitive decline,” British Medical Bulletin, vol. 92, no. 1, pp. 135–152, 09 2009. [Online]. Available: https://doi.org/10.1093/bmb/ldp033
  3. S. Duong, T. Patel, and F. Chang, “Dementia: What pharmacists need to know,” Canadian Pharmacists Journal / Revue des Pharmaciens du Canada, vol. 150, no. 2, pp. 118–129, 2017. [Online]. Available: https://doi.org/10.1177/1715163517690745
  4. F. Portet, P. J. Ousset, P. J. Visser, G. B. Frisoni, F. Nobili, P. Scheltens, B. Vellas, J. Touchon, and the MCI Working Group of the European Consortium on Alzheimer’s Disease (EADC), “Mild cognitive impairment (mci) in medical practice: a critical review of the concept and new diagnostic procedure. report of the mci working group of the european consortium on alzheimer’s disease,” Journal of Neurology, Neurosurgery & Psychiatry, vol. 77, no. 6, pp. 714–718, 2006. [Online]. Available: https://jnnp.bmj.com/content/77/6/714
  5. P. Celsis, “Age-related cognitive decline, mild cognitive impairment or preclinical alzheimer’s disease?” Annals of Medicine, vol. 32, no. 1, pp. 6–14, 2000, pMID: 10711572. [Online]. Available: https://doi.org/10.3109/07853890008995904
  6. World Health Organization, “Mental health action plan 2013-2020,” WHO Library Cataloguing-in-Publication DataLibrary Cataloguing-in-Publication Data, pp. 1–44, 2023.
  7. A. R. Damasio, “Aphasia,” New England Journal of Medicine, vol. 326, no. 8, pp. 531–539, 1992.
  8. C. Cabello-Collado, J. Rodriguez-Juan, D. Ortiz-Perez, J. Garcia-Rodriguez, D. Tomás, and M. F. Vizcaya-Moreno, “Automated generation of clinical reports using sensing technologies with deep learning techniques,” Sensors, vol. 24, no. 9, 2024. [Online]. Available: https://www.mdpi.com/1424-8220/24/9/2751
  9. R. Zhao, R. Yan, Z. Chen, K. Mao, P. Wang, and R. X. Gao, “Deep learning and its applications to machine health monitoring,” Mechanical Systems and Signal Processing, vol. 115, pp. 213–237, 2019.
  10. J. Venugopalan, L. Tong, H. R. Hassanzadeh, and M. D. Wang, “Multimodal deep learning models for early detection of alzheimer’s disease stage,” Scientific reports, vol. 11, no. 1, p. 3254, 2021.
  11. S. Liu, S. Liu, W. Cai, S. Pujol, R. Kikinis, and D. Feng, “Early diagnosis of alzheimer’s disease with deep learning,” in 2014 IEEE 11th international symposium on biomedical imaging (ISBI).   IEEE, 2014, pp. 1015–1018.
  12. M. Di Luca, D. Nutt, W. Oertel, P. Boyer, J. Jaarsma, F. Destrebecq, G. Esposito, and V. Quoidbach, “Towards earlier diagnosis and treatment of disorders of the brain,” Bulletin of the World Health Organization, vol. 96, no. 5, p. 298, 2018.
  13. D. Wen, Z. Wei, Y. Zhou, G. Li, X. Zhang, and W. Han, “Deep learning methods to process fmri data and their application in the diagnosis of cognitive impairment: a brief overview and our opinion,” Frontiers in neuroinformatics, vol. 12, p. 23, 2018.
  14. H. Taheri Gorji and N. Kaabouch, “A deep learning approach for diagnosis of mild cognitive impairment based on mri images,” Brain sciences, vol. 9, no. 9, p. 217, 2019.
  15. L. Kang, J. Jiang, J. Huang, and T. Zhang, “Identifying early mild cognitive impairment by multi-modality mri-based deep learning,” Frontiers in aging neuroscience, vol. 12, p. 206, 2020.
  16. X. Feng, Z. C. Lipton, J. Yang, S. A. Small, F. A. Provenzano, A. D. N. Initiative, F. L. D. N. Initiative et al., “Estimating brain age based on a uniform healthy population with deep learning and structural magnetic resonance imaging,” Neurobiology of aging, vol. 91, pp. 15–25, 2020.
  17. S. Chauhan, L. Vig, M. De Filippo De Grazia, M. Corbetta, S. Ahmad, and M. Zorzi, “A comparison of shallow and deep learning methods for predicting cognitive performance of stroke patients from mri lesion images,” Frontiers in neuroinformatics, vol. 13, p. 53, 2019.
  18. G. Zhu, B. Jiang, L. Tong, Y. Xie, G. Zaharchuk, and M. Wintermark, “Applications of deep learning to neuro-imaging techniques,” Frontiers in neurology, vol. 10, p. 869, 2019.
  19. T. Jo, K. Nho, and A. J. Saykin, “Deep learning in alzheimer’s disease: diagnostic classification and prognostic prediction using neuroimaging data,” Frontiers in aging neuroscience, vol. 11, p. 220, 2019.
  20. E. Pellegrini, L. Ballerini, M. d. C. V. Hernandez, F. M. Chappell, V. González-Castro, D. Anblagan, S. Danso, S. Muñoz-Maniega, D. Job, C. Pernet et al., “Machine learning of neuroimaging for assisted diagnosis of cognitive impairment and dementia: a systematic review,” Alzheimer’s & Dementia: Diagnosis, Assessment & Disease Monitoring, vol. 10, pp. 519–535, 2018.
  21. S. A. Graham, E. E. Lee, D. V. Jeste, R. Van Patten, E. W. Twamley, C. Nebeker, Y. Yamada, H.-C. Kim, and C. A. Depp, “Artificial intelligence approaches to predicting and detecting cognitive decline in older adults: A conceptual review,” Psychiatry research, vol. 284, p. 112732, 2020.
  22. H. Choi, K. H. Jin, A. D. N. Initiative et al., “Predicting cognitive decline with deep learning of brain metabolism and amyloid imaging,” Behavioural brain research, vol. 344, pp. 103–109, 2018.
  23. S. Grueso and R. Viejo-Sobera, “Machine learning methods for predicting progression from mild cognitive impairment to alzheimer’s disease dementia: a systematic review,” Alzheimer’s research & therapy, vol. 13, pp. 1–29, 2021.
  24. S. L. Warren and A. A. Moustafa, “Functional magnetic resonance imaging, deep learning, and alzheimer’s disease: A systematic review,” Journal of Neuroimaging, vol. 33, no. 1, pp. 5–18, 2023.
  25. M. Ansart, S. Epelbaum, G. Bassignana, A. Bône, S. Bottani, T. Cattai, R. Couronné, J. Faouzi, I. Koval, M. Louis et al., “Predicting the progression of mild cognitive impairment using machine learning: a systematic, quantitative and critical review,” Medical Image Analysis, vol. 67, p. 101848, 2021.
  26. A. Alberdi, A. Aztiria, and A. Basarab, “On the early diagnosis of alzheimer’s disease from multimodal signals: A survey,” Artificial Intelligence in Medicine, vol. 71, pp. 1–29, 2016. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0933365716300732
  27. V. Skaramagkas, A. Pentari, Z. Kefalopoulou, and M. Tsiknakis, “Multi-modal deep learning diagnosis of parkinson’s disease—a systematic review,” IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 31, pp. 2399–2423, 2023.
  28. Q. Yang, X. Li, X. Ding, F. Xu, and Z. Ling, “Deep learning-based speech analysis for alzheimer’s disease detection: a literature review,” Alzheimer’s Research & Therapy, vol. 14, no. 1, p. 186, 2022.
  29. X. Qi, Q. Zhou, J. Dong, and W. Bao, “Noninvasive automatic detection of alzheimer’s disease from spontaneous speech: a review,” Frontiers in Aging Neuroscience, vol. 15, 2023. [Online]. Available: https://www.frontiersin.org/journals/aging-neuroscience/articles/10.3389/fnagi.2023.1224723
  30. S. Gauthier, B. Reisberg, M. Zaudig, R. C. Petersen, K. Ritchie, K. Broich, S. Belleville, H. Brodaty, D. Bennett, H. Chertkow et al., “Mild cognitive impairment,” The lancet, vol. 367, no. 9518, pp. 1262–1270, 2006.
  31. R. C. Petersen, “Mild cognitive impairment,” CONTINUUM: lifelong Learning in Neurology, vol. 22, no. 2, pp. 404–418, 2016.
  32. P. B. Rosenberg, M. M. Mielke, B. Appleby, E. Oh, J.-M. Leoutsakos, and C. G. Lyketsos, “Neuropsychiatric symptoms in mci subtypes: the importance of executive dysfunction,” International journal of geriatric psychiatry, vol. 26, no. 4, pp. 364–372, 2011.
  33. Z. Arvanitakis, R. C. Shah, and D. A. Bennett, “Diagnosis and management of dementia,” Jama, vol. 322, no. 16, pp. 1589–1599, 2019.
  34. D. S. Geldmacher and P. J. Whitehouse, “Evaluation of dementia,” New England Journal of Medicine, vol. 335, no. 5, pp. 330–336, 1996.
  35. S. H. Ferris and M. Farlow, “Language impairment in alzheimer’s disease and benefits of acetylcholinesterase inhibitors,” Clinical interventions in aging, pp. 1007–1014, 2013.
  36. H. S. Kirshner and S. M. Wilson, “Aphasia and aphasic syndromes,” Bradley’s Neurology in Clinical Practice E-Book, vol. 133, 2021.
  37. M. P. Alexander and A. E. Hillis, “Aphasia,” Handbook of clinical neurology, vol. 88, pp. 287–309, 2008.
  38. L. V. Kalia and A. E. Lang, “Parkinson’s disease,” The Lancet, vol. 386, no. 9996, pp. 896–912, 2015.
  39. D. Aarsland, K. Andersen, J. P. Larsen, R. Perry, T. Wentzel-Larsen, A. Lolk, and P. Kragh-Sørensen, “The rate of cognitive decline in parkinson disease,” Archives of neurology, vol. 61, no. 12, pp. 1906–1911, 2004.
  40. D. Aarsland, B. Creese, M. Politis, K. R. Chaudhuri, D. H. Ffytche, D. Weintraub, and C. Ballard, “Cognitive decline in parkinson disease,” Nature Reviews Neurology, vol. 13, no. 4, pp. 217–231, 2017.
  41. R. Van Reekum, D. T. Stuss, and L. Ostrander, “Apathy: why care?” The Journal of neuropsychiatry and clinical neurosciences, vol. 17, no. 1, pp. 7–19, 2005.
  42. G. Montoya-Murillo, N. Ibarretxe-Bilbao, J. Peña, and N. Ojeda, “The impact of apathy on cognitive performance in the elderly,” International Journal of Geriatric Psychiatry, vol. 34, no. 5, pp. 657–665, 2019.
  43. G. Konstantakopoulos, D. Ploumpidis, P. Oulis, P. Patrikelis, A. Soumani, G. N. Papadimitriou, and A. M. Politis, “Apathy, cognitive deficits and functional impairment in schizophrenia,” Schizophrenia research, vol. 133, no. 1-3, pp. 193–198, 2011.
  44. S. Gluhm, J. Goldstein, K. Loc, A. Colt, C. Van Liew, and J. Corey-Bloom, “Cognitive performance on the mini-mental state examination and the montreal cognitive assessment across the healthy adult lifespan,” Cognitive and Behavioral Neurology, vol. 26, no. 1, pp. 1–5, 2013.
  45. I. Arevalo-Rodriguez, N. Smailagic, M. R. i Figuls, A. Ciapponi, E. Sanchez-Perez, A. Giannakou, O. L. Pedraza, X. B. Cosp, and S. Cullum, “Mini‐mental state examination (mmse) for the detection of alzheimer’s disease and other dementias in people with mild cognitive impairment (mci),” Cochrane Database of Systematic Reviews, vol. 3, 2015. [Online]. Available: https://doi.org//10.1002/14651858.CD010783.pub2
  46. J. S. Shiroky, H. M. Schipper, H. Bergman, and H. Chertkow, “Can you have dementia with an mmse score of 30?” American Journal of Alzheimer’s Disease & Other Dementias®, vol. 22, no. 5, pp. 406–415, 2007.
  47. P. T. Trzepacz, H. Hochstetler, S. Wang, B. Walker, A. J. Saykin, and A. D. N. Initiative, “Relationship between the montreal cognitive assessment and mini-mental state examination for assessment of mild cognitive impairment in older adults,” BMC geriatrics, vol. 15, pp. 1–9, 2015.
  48. S. Hoops, S. Nazem, A. Siderowf, J. Duda, S. Xie, M. Stern, and D. Weintraub, “Validity of the moca and mmse in the detection of mci and dementia in parkinson disease,” Neurology, vol. 73, no. 21, pp. 1738–1745, 2009.
  49. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” 2023.
  50. D. Moher, A. Liberati, J. Tetzlaff, D. G. Altman, P. Group et al., “Preferred reporting items for systematic reviews and meta-analyses: the prisma statement,” International journal of surgery, vol. 8, no. 5, pp. 336–341, 2010.
  51. A. M. Lanzi, A. K. Saylor, D. Fromm, H. Liu, B. MacWhinney, and M. L. Cohen, “Dementiabank: Theoretical rationale, protocol, and illustrative analyses,” American Journal of Speech-Language Pathology, vol. 32, no. 2, pp. 426–438, 2023.
  52. J. T. Becker, F. Boiler, O. L. Lopez, J. Saxton, and K. L. McGonigle, “The natural history of alzheimer’s disease: description of study cohort and accuracy of diagnosis,” Archives of neurology, vol. 51, no. 6, pp. 585–594, 1994.
  53. K. P. Elaine Giles and J. R. Hodges, “Performance on the boston cookie theft picture description task in patients with early dementia of the alzheimer’s type: Missing information,” Aphasiology, vol. 10, no. 4, pp. 395–408, 1996. [Online]. Available: https://doi.org/10.1080/02687039608248419
  54. S. Luz, F. Haider, S. de la Fuente, D. Fromm, and B. MacWhinney, “Alzheimer’s dementia recognition through spontaneous speech: The adress challenge,” 2020.
  55. ——, “Detecting cognitive decline using speech only: The adresso challenge,” 2021.
  56. S. Luz, F. Haider, D. Fromm, I. Lazarou, I. Kompatsiaris, and B. MacWhinney, “Multilingual alzheimer’s dementia recognition through spontaneous speech: a signal processing grand challenge,” 2023.
  57. S. D. L. F. Garcia, F. Haider, D. Fromm, B. MacWhinney, A. Lanzi, Y.-N. Chang, C.-J. Chou, Y.-C. Liu et al., “Connected speech-based cognitive assessment in chinese and english,” arXiv preprint arXiv:2406.10272, 2024.
  58. B. MacWhinney, D. Fromm, M. Forbes, and A. Holland, “Aphasiabank: Methods for studying discourse,” Aphasiology, vol. 25, no. 11, pp. 1286–1307, 2011.
  59. C. Pope and B. H. Davis, “Finding a balance: The carolinas conversation collection,” Corpus Linguistics and Linguistic Theory, vol. 7, no. 1, pp. 143–161, 2011. [Online]. Available: https://doi.org/10.1515/cllt.2011.007
  60. R. A. Li, I. Hajjar, F. Goldstein, and J. D. Choi, “Analysis of hierarchical multi-content text classification model on B-SHARP dataset for early detection of Alzheimer’s disease,” in Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, K.-F. Wong, K. Knight, and H. Wu, Eds.   Suzhou, China: Association for Computational Linguistics, Dec. 2020, pp. 358–365. [Online]. Available: https://aclanthology.org/2020.aacl-main.38
  61. D. Carr, “How to successfully navigate a revise-and-resubmit decision and handle rejections,” Innovation in Aging, vol. 3, no. Suppl 1, p. S224, 2019.
  62. K. Yu, K. Wild, K. Potempa, B. M. Hampstead, P. A. Lichtenberg, L. M. Struble, P. Pruitt, E. L. Alfaro, J. Lindsley, M. MacDonald, J. A. Kaye, L. C. Silbert, and H. H. Dodge, “The internet-based conversational engagement clinical trial (i-conect) in socially isolated adults 75+ years old: Randomized controlled trial protocol and covid-19 related study modifications,” Frontiers in Digital Health, vol. 3, 2021. [Online]. Available: https://www.scopus.com/inward/record.uri?eid=2-s2.0-85121367382&doi=10.3389%2ffdgth.2021.714813&partnerID=40&md5=dad167760b9a0f0c30c4a9e08937c379
  63. C.-Y. Wu, N. Mattek, K. Wild, L. M. Miller, J. A. Kaye, L. C. Silbert, and H. H. Dodge, “Can changes in social contact (frequency and mode) mitigate low mood before and during the covid-19 pandemic? the i-conect project,” Journal of the American Geriatrics Society, vol. 70, no. 3, pp. 669–676, 2022.
  64. J. R. Orozco-Arroyave, J. D. Arias-Londoño, J. F. Vargas-Bonilla, M. C. González-Rátiva, and E. Nöth, “New Spanish speech corpus database for the analysis of people suffering from Parkinson’s disease,” in Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), N. Calzolari, K. Choukri, T. Declerck, H. Loftsson, B. Maegaard, J. Mariani, A. Moreno, J. Odijk, and S. Piperidis, Eds.   Reykjavik, Iceland: European Language Resources Association (ELRA), May 2014, pp. 342–347. [Online]. Available: http://www.lrec-conf.org/proceedings/lrec2014/pdf/7_Paper.pdf
  65. A. Karakostas, A. Briassouli, K. Avgerinakis, I. Kompatsiaris, and M. Tsolaki, “The dem@care experiments and datasets: a technical report,” 2016. [Online]. Available: https://arxiv.org/abs/1701.01142
  66. F. Negin, P. Rodriguez, M. Koperski, A. Kerboua, J. Gonzàlez, J. Bourgeois, E. Chapoulie, P. Robert, and F. Bremond, “Praxis: Towards automatic cognitive assessment using gesture recognition,” Expert Systems with Applications, 2018.
  67. J. V. Egas-López, R. Balogh, N. Imre, I. Hoffmann, M. K. Szabó, L. Tóth, M. Pákáski, J. Kálmán, and G. Gosztolya, “Automatic screening of mild cognitive impairment and alzheimer’s disease by means of posterior-thresholding hesitation representation,” Computer Speech & Language, vol. 75, p. 101377, 2022.
  68. C. Themistocleous, M. Eckerström, and D. Kokkinakis, “Voice quality and speech fluency distinguish individuals with mild cognitive impairment from healthy controls,” Plos one, vol. 15, no. 7, p. e0236009, 2020.
  69. B. J. Abbaschian, D. Sierra-Sosa, and A. Elmaghraby, “Deep learning techniques for speech emotion recognition, from databases to models,” Sensors, vol. 21, no. 4, p. 1249, 2021.
  70. S. G. Koolagudi and K. S. Rao, “Emotion recognition from speech: a review,” International journal of speech technology, vol. 15, pp. 99–117, 2012.
  71. T. M. Wani, T. S. Gunawan, S. A. A. Qadri, M. Kartiwi, and E. Ambikairajah, “A comprehensive review of speech emotion recognition systems,” IEEE access, vol. 9, pp. 47 795–47 814, 2021.
  72. D. Ortiz-Perez, P. Ruiz-Ponce, J. Rodríguez-Juan, D. Tomás, J. Garcia-Rodriguez, and G. J. Nalepa, “Deep learning-based emotion detection in aphasia patients,” in International Conference on Soft Computing Models in Industrial and Environmental Applications.   Springer, 2023, pp. 195–204.
  73. C. Code, G. Hemsley, and M. Herrmann, “The emotional impact of aphasia,” in Seminars in speech and language, vol. 20.   © 1999 by Thieme Medical Publishers, Inc., 1999, pp. 19–31.
  74. P. Priyadarshinee, C. J. Clarke, J. Melechovsky, C. M. Y. Lin, B. B. T., and J.-M. Chen, “Alzheimer’s dementia speech (audio vs. text): Multi-modal machine learning at high vs. low resolution,” Applied Sciences, vol. 13, no. 7, 2023. [Online]. Available: https://www.mdpi.com/2076-3417/13/7/4244
  75. Z. Cui, W. Wu, W.-Q. Zhang, J. Wu, and C. Zhang, “Transferring speech-generic and depression-specific knowledge for alzheimer’s disease detection,” in 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU).   IEEE, Dec. 2023. [Online]. Available: http://dx.doi.org/10.1109/ASRU57964.2023.10389785
  76. N. Wang, Y. Cao, S. Hao, Z. Shao, and K. Subbalakshmi, “Modular multi-modal attention network for alzheimer’s disease detection using patient audio and language data.” in Interspeech, 2021, pp. 3835–3839.
  77. Y. Ying, T. Yang, and H. Zhou, “Multimodal fusion for alzheimer’s disease recognition,” Applied Intelligence, vol. 53, no. 12, pp. 16 029–16 040, 2023.
  78. J.-U. Bang, S.-H. Han, and B.-O. Kang, “Alzheimer’s disease recognition from spontaneous speech using large language models,” ETRI Journal, 2024.
  79. M. Rohanian, J. Hough, and M. Purver, “Alzheimer’s dementia recognition using acoustic, lexical, disfluency and speech pause features robust to noisy inputs,” CoRR, vol. abs/2106.15684, 2021. [Online]. Available: https://arxiv.org/abs/2106.15684
  80. S. B. Shah, A. Bhandari, and P. G. Shambharkar, “Leveraging multimodal information in speech data for the non-invasive detection of alzheimer’s disease,” in 2023 14th International Conference on Computing Communication and Networking Technologies (ICCCNT), 2023, pp. 1–6.
  81. A. Hlédiková, D. Woszczyk, A. Akman, S. Demetriou, and B. Schuller, “Data augmentation for dementia detection in spoken language,” 2022.
  82. J. Koo, J. Lee, J. Pyo, Y. Jo, and K. Lee, “Exploiting multi-modal features from pre-trained networks for alzheimer’s dementia recognition,” in Interspeech, 10 2020, pp. 2217–2221.
  83. A. Pompili, T. Rolland, and A. Abad, “The inesc-id multi-modal system for the adress 2020 challenge,” arXiv preprint arXiv:2005.14646, 2020.
  84. N. Cummins, Y. Pan, Z. Ren, J. Fritsch, V. S. Nallanthighal, H. Christensen, D. Blackburn, B. W. Schuller, M. Magimai-Doss, H. Strik et al., “A comparison of acoustic and linguistics methodologies for alzheimer’s dementia recognition,” in Interspeech 2020.   ISCA-International Speech Communication Association, 2020, pp. 2182–2186.
  85. P. Mahajan and V. Baths, “Acoustic and language based deep learning approaches for alzheimer’s dementia detection from spontaneous speech,” Frontiers in Aging Neuroscience, vol. 13, p. 623607, 2021.
  86. Y. Zhu, X. Liang, J. A. Batsis, and R. M. Roth, “Exploring deep transfer learning techniques for alzheimer’s dementia detection,” Frontiers in Computer Science, vol. 3, 2021. [Online]. Available: https://www.frontiersin.org/articles/10.3389/fcomp.2021.624683
  87. L. Ilias, D. Askounis, and J. Psarras, “Detecting dementia from speech and transcripts using transformers,” Computer Speech & Language, vol. 79, p. 101485, 2023. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0885230823000049
  88. A. Meghanani, C. Anoop, and A. Ramakrishnan, “An exploration of log-mel spectrogram and mfcc features for alzheimer’s dementia recognition from spontaneous speech,” in 2021 IEEE spoken language technology workshop (SLT).   IEEE, 2021, pp. 670–677.
  89. Z. Liu, Z. Guo, Z. Ling, and Y. Li, “Detecting alzheimer’s disease from speech using neural networks with bottleneck features and data augmentation,” in ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2021, pp. 7323–7327.
  90. D. Ortiz-Perez, P. Ruiz-Ponce, D. Tomás, J. Garcia-Rodriguez, M. F. Vizcaya-Moreno, and M. Leo, “A deep learning-based multimodal architecture to predict signs of dementia,” Neurocomputing, vol. 548, p. 126413, 2023. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0925231223005362
  91. I. Krstev, M. Pavikjevikj, M. Toshevska, and S. Gievska, “Multimodal data fusion for automatic detection of alzheimer’s disease,” in International Conference on Human-Computer Interaction.   Springer, 2022, pp. 79–94.
  92. D. Escobar-Grisales, C. D. Ríos-Urrego, and J. R. Orozco-Arroyave, “Deep learning and artificial intelligence applied to model speech and language in parkinson’s disease,” Diagnostics, vol. 13, no. 13, p. 2163, 2023.
  93. D. Ortiz-Perez, J. Garcia-Rodriguez, and D. Tomás, “Cognitive insights across languages: Enhancing multimodal interview analysis,” in Interspeech 2024, 2024, pp. 952–956.
  94. F. F. Poor, H. H. Dodge, and M. H. Mahoor, “A multimodal cross-transformer-based model to predict mild cognitive impairment using speech, language and vision,” Computers in Biology and Medicine, vol. 182, p. 109199, 2024. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0010482524012848
  95. Z. Sheng, Z. Guo, X. Li, Y. Li, and Z. Ling, “Dementia detection by fusing speech and eye-tracking representation,” in ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022, pp. 6457–6461.
  96. N. Narendra, B. Schuller, and P. Alku, “The detection of parkinson’s disease from speech using voice source information,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 1925–1936, 2021.
  97. Y. Ge, T. Wang, J. Cao, and S. Xu, “A novel multi-task learning based automatic speech impairment assessment algorithm,” in 2022 China Automation Congress (CAC), 2022, pp. 887–892.
  98. S. Allamy and A. L. Koerich, “1d cnn architectures for music genre classification,” in 2021 IEEE Symposium Series on Computational Intelligence (SSCI), 2021, pp. 01–07.
  99. S. Abdoli, P. Cardinal, and A. Lameiras Koerich, “End-to-end environmental sound classification using a 1d convolutional neural network,” Expert Systems with Applications, vol. 136, pp. 252–263, 2019. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0957417419304403
  100. K. Zaman, M. Sah, C. Direkoglu, and M. Unoki, “A survey of audio classification using deep learning,” IEEE Access, 2023.
  101. Y. M. Costa, L. S. Oliveira, and C. N. Silla Jr, “An evaluation of convolutional neural networks for music classification using spectrograms,” Applied soft computing, vol. 52, pp. 28–38, 2017.
  102. A. Satt, S. Rozenberg, R. Hoory et al., “Efficient emotion recognition from speech using deep learning on spectrograms.” in Interspeech, 2017, pp. 1089–1093.
  103. Y. Zeng, H. Mao, D. Peng, and Z. Yi, “Spectrogram based multi-task audio classification,” Multimedia Tools and Applications, vol. 78, pp. 3705–3722, 2019.
  104. S. S. Stevens, J. Volkmann, and E. B. Newman, “A Scale for the Measurement of the Psychological Magnitude Pitch,” The Journal of the Acoustical Society of America, vol. 8, no. 3, pp. 185–190, 01 1937. [Online]. Available: https://doi.org/10.1121/1.1915893
  105. T. Arias-Vergara, P. Klumpp, J. C. Vasquez-Correa, E. Nöth, J. R. Orozco-Arroyave, and M. Schuster, “Multi-channel spectrograms for speech processing applications using deep learning methods,” Pattern Analysis and Applications, vol. 24, pp. 423–431, 2021.
  106. H. Purwins, B. Li, T. Virtanen, J. Schlüter, S.-Y. Chang, and T. Sainath, “Deep learning for audio signal processing,” IEEE Journal of Selected Topics in Signal Processing, vol. 13, no. 2, pp. 206–219, 2019.
  107. L. Jiao and J. Zhao, “A survey on the new generation of deep learning in image processing,” Ieee Access, vol. 7, pp. 172 231–172 263, 2019.
  108. S. Minaee, Y. Boykov, F. Porikli, A. Plaza, N. Kehtarnavaz, and D. Terzopoulos, “Image segmentation using deep learning: A survey,” IEEE transactions on pattern analysis and machine intelligence, vol. 44, no. 7, pp. 3523–3542, 2021.
  109. J. Gu, Z. Wang, J. Kuen, L. Ma, A. Shahroudy, B. Shuai, T. Liu, X. Wang, G. Wang, J. Cai et al., “Recent advances in convolutional neural networks,” Pattern recognition, vol. 77, pp. 354–377, 2018.
  110. K. O’shea and R. Nash, “An introduction to convolutional neural networks,” arXiv preprint arXiv:1511.08458, 2015.
  111. Z. Li, F. Liu, W. Yang, S. Peng, and J. Zhou, “A survey of convolutional neural networks: analysis, applications, and prospects,” IEEE transactions on neural networks and learning systems, vol. 33, no. 12, pp. 6999–7019, 2021.
  112. J. Devlin, M. Chang, K. Lee, and K. Toutanova, “BERT: pre-training of deep bidirectional transformers for language understanding,” CoRR, vol. abs/1810.04805, 2018. [Online]. Available: http://arxiv.org/abs/1810.04805
  113. V. Sanh, L. Debut, J. Chaumond, and T. Wolf, “Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter,” CoRR, vol. abs/1910.01108, 2019. [Online]. Available: http://arxiv.org/abs/1910.01108
  114. A. Arnab, M. Dehghani, G. Heigold, C. Sun, M. Lucic, and C. Schmid, “Vivit: A video vision transformer,” CoRR, vol. abs/2103.15691, 2021. [Online]. Available: https://arxiv.org/abs/2103.15691
  115. A. Baevski, H. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” CoRR, vol. abs/2006.11477, 2020. [Online]. Available: https://arxiv.org/abs/2006.11477
  116. A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” CoRR, vol. abs/2010.11929, 2020. [Online]. Available: https://arxiv.org/abs/2010.11929
  117. J. Pons, O. Slizovskaia, R. Gong, E. Gómez, and X. Serra, “Timbre analysis of music audio signals with convolutional neural networks,” in 2017 25th European Signal Processing Conference (EUSIPCO).   IEEE, 2017, pp. 2744–2748.
  118. Y. Zhang, B. Li, H. Fang, and Q. Meng, “Spectrogram transformers for audio classification,” in 2022 IEEE International Conference on Imaging Systems and Techniques (IST).   IEEE, 2022, pp. 1–6.
  119. Y. Gong, Y.-A. Chung, and J. Glass, “Ast: Audio spectrogram transformer,” arXiv preprint arXiv:2104.01778, 2021.
  120. A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” 2022.
  121. L. Bertinetto, J. Valmadre, J. F. Henriques, A. Vedaldi, and P. H. Torr, “Fully-convolutional siamese networks for object tracking,” in Computer Vision–ECCV 2016 Workshops: Amsterdam, The Netherlands, October 8-10 and 15-16, 2016, Proceedings, Part II 14.   Springer, 2016, pp. 850–865.
  122. S. Boelders, V. S. Nallanthighal, V. Menkovski, and A. Härmä, “Detection of mild dyspnea from pairs of speech recordings,” in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).   IEEE, 2020, pp. 4102–4106.
  123. Z. Lian, Y. Li, J. Tao, and J. Huang, “Speech emotion recognition via contrastive loss under siamese networks,” in Proceedings of the Joint Workshop of the 4th Workshop on Affective Social Multimedia Computing and First Multi-Modal Affective Computing of Large-Scale Multimedia Data, 2018, pp. 21–26.
  124. J. Wang, Y. Qin, Z. Peng, and T. Lee, “Child speech disorder detection with siamese recurrent network using speech attribute features.” in INTERSPEECH, vol. 2, 2019, pp. 3885–3889.
  125. D. Ortiz-Perez, P. Ruiz-Ponce, D. Tomás, and J. Garcia-Rodriguez, “Deep learning-based dementia prediction using multimodal data,” in 17th International Conference on Soft Computing Models in Industrial and Environmental Applications (SOCO 2022), P. García Bringas, H. Pérez García, F. J. Martinez-de Pison, J. R. Villar Flecha, A. Troncoso Lora, E. A. de la Cal, Á. Herrero, F. Martínez Álvarez, G. Psaila, H. Quintián, and E. S. Corchado Rodriguez, Eds.   Cham: Springer Nature Switzerland, 2023, pp. 260–269.
  126. G. Huang, Z. Liu, and K. Q. Weinberger, “Densely connected convolutional networks,” CoRR, vol. abs/1608.06993, 2016. [Online]. Available: http://arxiv.org/abs/1608.06993
  127. A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam, “Mobilenets: Efficient convolutional neural networks for mobile vision applications,” CoRR, vol. abs/1704.04861, 2017. [Online]. Available: http://arxiv.org/abs/1704.04861
  128. K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” CoRR, vol. abs/1512.03385, 2015. [Online]. Available: http://arxiv.org/abs/1512.03385
  129. F. N. Iandola, S. Han, M. W. Moskewicz, K. Ashraf, W. J. Dally, and K. Keutzer, “Squeezenet: Alexnet-level accuracy with 50x fewer parameters and¡ 0.5 mb model size,” arXiv preprint arXiv:1602.07360, 2016.
  130. K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” 2015.
  131. A. Sherstinsky, “Fundamentals of recurrent neural network (rnn) and long short-term memory (lstm) network,” Physica D: Nonlinear Phenomena, vol. 404, p. 132306, 2020. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0167278919305974
  132. Y. Li, C.-Y. Wu, H. Fan, K. Mangalam, B. Xiong, J. Malik, and C. Feichtenhofer, “Mvitv2: Improved multiscale vision transformers for classification and detection,” 2022.
  133. C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” 2014.
  134. S. Zagoruyko and N. Komodakis, “Wide residual networks,” 2017.
  135. A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in Advances in Neural Information Processing Systems, F. Pereira, C. Burges, L. Bottou, and K. Weinberger, Eds., vol. 25.   Curran Associates, Inc., 2012. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2012/file/c399862d3b9d6b76c8436e924a68c45b-Paper.pdf
  136. M. Sandler, A. G. Howard, M. Zhu, A. Zhmoginov, and L. Chen, “Inverted residuals and linear bottlenecks: Mobile networks for classification, detection and segmentation,” CoRR, vol. abs/1801.04381, 2018. [Online]. Available: http://arxiv.org/abs/1801.04381
  137. M. Tan, B. Chen, R. Pang, V. Vasudevan, and Q. V. Le, “Mnasnet: Platform-aware neural architecture search for mobile,” CoRR, vol. abs/1807.11626, 2018. [Online]. Available: http://arxiv.org/abs/1807.11626
  138. M. Tan and Q. V. Le, “Efficientnet: Rethinking model scaling for convolutional neural networks,” CoRR, vol. abs/1905.11946, 2019. [Online]. Available: http://arxiv.org/abs/1905.11946
  139. Y. Chuang, C. Liu, and H. Lee, “Speechbert: Cross-modal pre-trained language model for end-to-end spoken question answering,” CoRR, vol. abs/1910.11559, 2019. [Online]. Available: http://arxiv.org/abs/1910.11559
  140. S. Gupta, J. Jaafar, W. W. Ahmad, and A. Bansal, “Feature extraction using mfcc,” Signal & Image Processing: An International Journal, vol. 4, no. 4, pp. 101–108, 2013.
  141. L. Muda, M. Begam, and I. Elamvazuthi, “Voice recognition algorithms using mel frequency cepstral coefficient (mfcc) and dynamic time warping (dtw) techniques,” arXiv preprint arXiv:1003.4083, 2010.
  142. V. Tiwari, “Mfcc and its applications in speaker recognition,” International journal on emerging technologies, vol. 1, no. 1, pp. 19–22, 2010.
  143. M. Deng, T. Meng, J. Cao, S. Wang, J. Zhang, and H. Fan, “Heart sound classification based on improved mfcc features and convolutional recurrent neural networks,” Neural Networks, vol. 130, pp. 22–32, 2020.
  144. E. Rejaibi, A. Komaty, F. Meriaudeau, S. Agrebi, and A. Othmani, “Mfcc-based recurrent neural network for automatic clinical depression recognition and assessment from speech,” Biomedical Signal Processing and Control, vol. 71, p. 103107, 2022.
  145. F. Eyben, M. Wöllmer, and B. Schuller, “Opensmile: the munich versatile and fast open-source audio feature extractor,” in Proceedings of the 18th ACM International Conference on Multimedia, ser. MM ’10.   New York, NY, USA: Association for Computing Machinery, 2010, p. 1459–1462. [Online]. Available: https://doi.org/10.1145/1873951.1874246
  146. F. Eyben, F. Weninger, F. Gross, and B. Schuller, “Recent developments in opensmile, the munich open-source multimedia feature extractor,” in Proceedings of the 21st ACM international conference on Multimedia, 2013, pp. 835–838.
  147. G. Degottex, J. Kane, T. Drugman, T. Raitio, and S. Scherer, “Covarep—a collaborative voice analysis repository for speech technologies,” in 2014 ieee international conference on acoustics, speech and signal processing (icassp).   IEEE, 2014, pp. 960–964.
  148. D. Povey, A. Ghoshal, G. Boulianne, L. Burget, O. Glembek, N. Goel, M. Hannemann, P. Motlicek, Y. Qian, P. Schwarz et al., “The kaldi speech recognition toolkit,” in IEEE 2011 workshop on automatic speech recognition and understanding.   IEEE Signal Processing Society, 2011.
  149. P. Nimitsurachat and P. Washington, “Audio-based emotion recognition using self-supervised learning on an engineered feature space,” AI, vol. 5, no. 1, pp. 195–207, 2024.
  150. A. B. Zadeh, P. P. Liang, S. Poria, E. Cambria, and L.-P. Morency, “Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph,” in Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2018, pp. 2236–2246.
  151. K. Qian, Z. Zhang, F. Ringeval, and B. Schuller, “Bird sounds classification by large scale acoustic features and extreme learning machine,” in 2015 IEEE Global Conference on Signal and Information Processing (GlobalSIP).   IEEE, 2015, pp. 1317–1321.
  152. N. Narendra and P. Alku, “Dysarthric speech classification using glottal features computed from non-words, words and sentences.” in Interspeech, 2018, pp. 3403–3407.
  153. B. Schuller, S. Steidl, A. Batliner, F. Burkhardt, L. Devillers, C. Müller, and S. Narayanan, “The interspeech 2010 paralinguistic challenge,” in Proc. INTERSPEECH 2010, Makuhari, Japan, 2010, pp. 2794–2797.
  154. F. Eyben, K. R. Scherer, B. W. Schuller, J. Sundberg, E. André, C. Busso, L. Y. Devillers, J. Epps, P. Laukka, S. S. Narayanan, and K. P. Truong, “The geneva minimalistic acoustic parameter set (gemaps) for voice research and affective computing,” IEEE Transactions on Affective Computing, vol. 7, no. 2, pp. 190–202, 2016.
  155. B. Schuller, S. Steidl, A. Batliner, J. Hirschberg, J. K. Burgoon, A. Baird, A. Elkins, Y. Zhang, E. Coutinho, and K. Evanini, “The INTERSPEECH 2016 Computational Paralinguistics Challenge: Deception, Sincerity & Native Language,” in Proc. Interspeech 2016, 2016, pp. 2001–2005.
  156. S. Hershey, S. Chaudhuri, D. P. W. Ellis, J. F. Gemmeke, A. Jansen, R. C. Moore, M. Plakal, D. Platt, R. A. Saurous, B. Seybold, M. Slaney, R. J. Weiss, and K. W. Wilson, “CNN architectures for large-scale audio classification,” CoRR, vol. abs/1609.09430, 2016. [Online]. Available: http://arxiv.org/abs/1609.09430
  157. J. F. Gemmeke, D. P. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, “Audio set: An ontology and human-labeled dataset for audio events,” in 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP).   IEEE, 2017, pp. 776–780.
  158. N. Dehak, P. J. Kenny, R. Dehak, P. Dumouchel, and P. Ouellet, “Front-end factor analysis for speaker verification,” IEEE Transactions on Audio, Speech, and Language Processing, vol. 19, no. 4, pp. 788–798, 2011.
  159. Y. Hauptman, R. Aloni-Lavi, I. Lapidot, T. Gurevich, Y. Manor, S. Naor, N. Diamant, and I. Opher, “Identifying distinctive acoustic and spectral features in parkinson’s disease.” in Interspeech, 2019, pp. 2498–2502.
  160. I. Laaridh, W. B. Kheder, C. Fredouille, and C. Meunier, “Automatic prediction of speech evaluation metrics for dysarthric speech,” in Interspeech, 2017.
  161. D. Snyder, D. Garcia-Romero, D. Povey, and S. Khudanpur, “Deep neural network embeddings for text-independent speaker verification.” in Interspeech, vol. 2017, 2017, pp. 999–1003.
  162. D. Snyder, D. Garcia-Romero, A. McCree, G. Sell, D. Povey, and S. Khudanpur, “Spoken language recognition using x-vectors.” in Odyssey, vol. 2018, 2018, pp. 105–111.
  163. D. Snyder, D. Garcia-Romero, G. Sell, D. Povey, and S. Khudanpur, “X-vectors: Robust dnn embeddings for speaker recognition,” in 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP).   IEEE, 2018, pp. 5329–5333.
  164. T. Lin, Y. Wang, X. Liu, and X. Qiu, “A survey of transformers,” AI open, vol. 3, pp. 111–132, 2022.
  165. K. Han, Y. Wang, H. Chen, X. Chen, J. Guo, Z. Liu, Y. Tang, A. Xiao, C. Xu, Y. Xu et al., “A survey on vision transformer,” IEEE transactions on pattern analysis and machine intelligence, vol. 45, no. 1, pp. 87–110, 2022.
  166. S. Khan, M. Naseer, M. Hayat, S. W. Zamir, F. S. Khan, and M. Shah, “Transformers in vision: A survey,” ACM computing surveys (CSUR), vol. 54, no. 10s, pp. 1–41, 2022.
  167. S. Schneider, A. Baevski, R. Collobert, and M. Auli, “wav2vec: Unsupervised pre-training for speech recognition,” CoRR, vol. abs/1904.05862, 2019. [Online]. Available: http://arxiv.org/abs/1904.05862
  168. Y. Zhuang, Y. Chen, and J. Zheng, “Music genre classification with transformer classifier,” in Proceedings of the 2020 4th international conference on digital signal processing, 2020, pp. 155–159.
  169. F. Andayani, L. B. Theng, M. T. Tsun, and C. Chua, “Hybrid lstm-transformer model for emotion recognition from speech audio files,” IEEE Access, vol. 10, pp. 36 018–36 027, 2022.
  170. N. Vaessen and D. A. Van Leeuwen, “Fine-tuning wav2vec2 for speaker recognition,” in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).   IEEE, 2022, pp. 7967–7971.
  171. C. G. Lyketsos and H. B. Lee, “Diagnosis and treatment of depression in alzheimer’s disease: a practical update for the clinician,” Dementia and geriatric cognitive disorders, vol. 17, no. 1-2, pp. 55–64, 2003.
  172. R. C. Green, L. A. Cupples, A. Kurz, S. Auerbach, R. Go, D. Sadovnick, R. Duara, W. A. Kukull, H. Chui, T. Edeki et al., “Depression as a risk factor for alzheimer disease: the mirage study,” Archives of neurology, vol. 60, no. 5, pp. 753–759, 2003.
  173. R. L. Ownby, E. Crocco, A. Acevedo, V. John, and D. Loewenstein, “Depression and risk for alzheimer disease: systematic review, meta-analysis, and metaregression analysis,” Archives of general psychiatry, vol. 63, no. 5, pp. 530–538, 2006.
  174. J. Gratch, R. Artstein, G. M. Lucas, G. Stratou, S. Scherer, A. Nazarian, R. Wood, J. Boberg, D. DeVault, S. Marsella et al., “The distress analysis interview corpus of human and computer interviews.” in LREC.   Reykjavik, 2014, pp. 3123–3128.
  175. S. Chen, C. Wang, Z. Chen, Y. Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiao, J. Wu, L. Zhou, S. Ren, Y. Qian, Y. Qian, J. Wu, M. Zeng, and F. Wei, “Wavlm: Large-scale self-supervised pre-training for full stack speech processing,” CoRR, vol. abs/2110.13900, 2021. [Online]. Available: https://arxiv.org/abs/2110.13900
  176. W. Hsu, B. Bolte, Y. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, “Hubert: Self-supervised speech representation learning by masked prediction of hidden units,” CoRR, vol. abs/2106.07447, 2021. [Online]. Available: https://arxiv.org/abs/2106.07447
  177. H. Kheddar, Y. Himeur, S. Al-Maadeed, A. Amira, and F. Bensaali, “Deep transfer learning for automatic speech recognition: Towards better generalization,” Knowledge-Based Systems, vol. 277, p. 110851, 2023. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0950705123006019
  178. N. Jamal, S. Shanta, F. Mahmud, and M. Sha’abani, “Automatic speech recognition (asr) based approach for speech therapy of aphasic patients: A review,” in AIP Conference Proceedings, vol. 1883.   AIP Publishing, 2017.
  179. I. G. Torre, M. Romero, and A. Álvarez, “Improving aphasic speech recognition by using novel semi-supervised learning methods on aphasiabank for english and spanish,” Applied Sciences, vol. 11, no. 19, p. 8872, 2021.
  180. J. Weiner, M. Engelbart, and T. Schultz, “Manual and automatic transcriptions in dementia detection from speech.” in Interspeech, 2017, pp. 3117–3121.
  181. Y. Zhu, A. Obyat, X. Liang, J. A. Batsis, and R. M. Roth, “WavBERT: Exploiting Semantic and Non-Semantic Speech Using Wav2vec and BERT for Dementia Detection,” in Proc. Interspeech 2021, 2021, pp. 3790–3794.
  182. H. Zhang, H. Song, S. Li, M. Zhou, and D. Song, “A survey of controllable text generation using transformer-based pre-trained language models,” ACM Computing Surveys, vol. 56, no. 3, pp. 1–37, 2023.
  183. J. Li, T. Tang, W. X. Zhao, J.-Y. Nie, and J.-R. Wen, “Pre-trained language models for text generation: A survey,” ACM Computing Surveys, vol. 56, no. 9, pp. 1–39, 2024.
  184. A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever et al., “Language models are unsupervised multitask learners,” OpenAI blog, vol. 1, no. 8, p. 9, 2019.
  185. C. Li, D. Knopman, W. Xu, T. Cohen, and S. Pakhomov, “GPT-D: Inducing dementia-related linguistic anomalies by deliberate degradation of artificial neural language models,” in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), S. Muresan, P. Nakov, and A. Villavicencio, Eds.   Dublin, Ireland: Association for Computational Linguistics, May 2022, pp. 1866–1877. [Online]. Available: https://aclanthology.org/2022.acl-long.131
  186. J. Yuan, Y. Bian, X. Cai, J. Huang, Z. Ye, and K. Church, “Disfluencies and fine-tuning pre-trained language models for detection of alzheimer’s disease.” in Interspeech, vol. 2020, 2020, pp. 2162–6.
  187. L. Ilias and D. Askounis, “Explainable identification of dementia from transcripts using transformer networks,” IEEE Journal of Biomedical and Health Informatics, vol. 26, no. 8, pp. 4153–4164, 2022.
  188. A. Balagopalan, B. Eyre, F. Rudzicz, and J. Novikova, “To bert or not to bert: Comparing speech and language-based approaches for alzheimer’s disease detection,” in Interspeech, 2020. [Online]. Available: https://api.semanticscholar.org/CorpusID:220961484
  189. A. S. Nambiar, K. Likhita, K. S. Pujya, D. Gupta, S. Vekkot, and S. Lalitha, “Comparative study of deep classifiers for early dementia detection using speech transcripts,” in 2022 IEEE 19th India Council International Conference (INDICON).   IEEE, 2022, pp. 1–6.
  190. N. Liu and L. Wang, “An approach for assisting diagnosis of alzheimer’s disease based on natural language processing,” Frontiers in Aging Neuroscience, vol. 15, 2023.
  191. A. Roshanzamir, H. Aghajan, and M. Soleymani Baghshah, “Transformer-based deep neural network language models for alzheimer’s disease risk assessment from targeted speech,” BMC Medical Informatics and Decision Making, vol. 21, pp. 1–14, 2021.
  192. M. Bouazizi, C. Zheng, S. Yang, and T. Ohtsuki, “Dementia detection from speech: What if language models are not the answer?” Information, vol. 15, no. 1, 2024. [Online]. Available: https://www.mdpi.com/2078-2489/15/1/2
  193. C. Zheng, M. Bouazizi, and T. Ohtsuki, “An evaluation on information composition in dementia detection based on speech,” IEEE Access, vol. 10, pp. 92 294–92 306, 2022.
  194. Y. Pan, B. Mirheidari, M. Reuber, A. Venneri, D. Blackburn, and H. Christensen, “Automatic hierarchical attention neural network for detecting ad,” in Proceedings of Interspeech 2019.   International Speech Communication Association (ISCA), 2019, pp. 4105–4109.
  195. D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learning internal representations by error propagation, parallel distributed processing, explorations in the microstructure of cognition, ed. de rumelhart and j. mcclelland. vol. 1. 1986,” Biometrika, vol. 71, no. 599-607, p. 6, 1986.
  196. S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural computation, vol. 9, no. 8, pp. 1735–1780, 1997.
  197. G. Van Houdt, C. Mosquera, and G. Nápoles, “A review on the long short-term memory model,” Artificial Intelligence Review, vol. 53, no. 8, pp. 5929–5955, 2020.
  198. K. Cho, B. Van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio, “Learning phrase representations using rnn encoder-decoder for statistical machine translation,” arXiv preprint arXiv:1406.1078, 2014.
  199. J. Chung, C. Gulcehre, K. Cho, and Y. Bengio, “Empirical evaluation of gated recurrent neural networks on sequence modeling,” arXiv preprint arXiv:1412.3555, 2014.
  200. G. Liu and J. Guo, “Bidirectional lstm with attention mechanism and convolutional layer for text classification,” Neurocomputing, vol. 337, pp. 325–338, 2019.
  201. C. Zhou, C. Sun, Z. Liu, and F. Lau, “A c-lstm neural network for text classification,” arXiv preprint arXiv:1511.08630, 2015.
  202. M. E. Basiri, S. Nemati, M. Abdar, E. Cambria, and U. R. Acharya, “Abcdm: An attention-based bidirectional cnn-rnn deep model for sentiment analysis,” Future Generation Computer Systems, vol. 115, pp. 279–294, 2021.
  203. Y. Kim, “Convolutional neural networks for sentence classification,” CoRR, vol. abs/1408.5882, 2014. [Online]. Available: http://arxiv.org/abs/1408.5882
  204. Y. Zhang and B. Wallace, “A sensitivity analysis of (and practitioners’ guide to) convolutional neural networks for sentence classification,” arXiv preprint arXiv:1510.03820, 2015.
  205. Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “Roberta: A robustly optimized bert pretraining approach,” 2019. [Online]. Available: https://arxiv.org/abs/1907.11692
  206. J. Fritsch, S. Wankerl, and E. Nöth, “Automatic diagnosis of alzheimer’s disease using neural network language models,” in ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).   IEEE, 2019, pp. 5841–5845.
  207. T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei, “Language models are few-shot learners,” CoRR, vol. abs/2005.14165, 2020. [Online]. Available: https://arxiv.org/abs/2005.14165
  208. J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al., “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774, 2023.
  209. Y. Liu and M. Lapata, “Text summarization with pretrained encoders,” arXiv preprint arXiv:1908.08345, 2019.
  210. M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer, “BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension,” CoRR, vol. abs/1910.13461, 2019. [Online]. Available: http://arxiv.org/abs/1910.13461
  211. C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” CoRR, vol. abs/1910.10683, 2019. [Online]. Available: http://arxiv.org/abs/1910.10683
  212. L. Xue, N. Constant, A. Roberts, M. Kale, R. Al-Rfou, A. Siddhant, A. Barua, and C. Raffel, “mt5: A massively multilingual pre-trained text-to-text transformer,” CoRR, vol. abs/2010.11934, 2020. [Online]. Available: https://arxiv.org/abs/2010.11934
  213. Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut, “ALBERT: A lite BERT for self-supervised learning of language representations,” CoRR, vol. abs/1909.11942, 2019. [Online]. Available: http://arxiv.org/abs/1909.11942
  214. Z. Yang, Z. Dai, Y. Yang, J. G. Carbonell, R. Salakhutdinov, and Q. V. Le, “Xlnet: Generalized autoregressive pretraining for language understanding,” CoRR, vol. abs/1906.08237, 2019. [Online]. Available: http://arxiv.org/abs/1906.08237
  215. I. Beltagy, M. E. Peters, and A. Cohan, “Longformer: The long-document transformer,” CoRR, vol. abs/2004.05150, 2020. [Online]. Available: https://arxiv.org/abs/2004.05150
  216. J. Lee, W. Yoon, S. Kim, D. Kim, S. Kim, C. H. So, and J. Kang, “Biobert: a pre-trained biomedical language representation model for biomedical text mining,” CoRR, vol. abs/1901.08746, 2019. [Online]. Available: http://arxiv.org/abs/1901.08746
  217. E. Alsentzer, J. R. Murphy, W. Boag, W. Weng, D. Jin, T. Naumann, and M. B. A. McDermott, “Publicly available clinical BERT embeddings,” CoRR, vol. abs/1904.03323, 2019. [Online]. Available: http://arxiv.org/abs/1904.03323
  218. Z. Jiang, W. Yu, D. Zhou, Y. Chen, J. Feng, and S. Yan, “Convbert: Improving BERT with span-based dynamic convolution,” CoRR, vol. abs/2008.02496, 2020. [Online]. Available: https://arxiv.org/abs/2008.02496
  219. M. Yancheva, K. C. Fraser, and F. Rudzicz, “Using linguistic features longitudinally to predict clinical scores for alzheimer’s disease and related dementias,” in SLPAT@Interspeech, 2015. [Online]. Available: https://api.semanticscholar.org/CorpusID:891184
  220. Z. Zhu, J. Novikova, and F. Rudzicz, “Detecting cognitive impairments by agreeing on interpretations of linguistic features,” ArXiv, vol. abs/1808.06570, 2018. [Online]. Available: https://api.semanticscholar.org/CorpusID:51938927
  221. N. B. Mota, N. Vasconcelos, N. Lemos, A. C. de Souza Pieretti, O. Kinouchi, G. A. Cecchi, M. Copelli, and S. Ribeiro, “Speech graphs provide a quantitative measure of thought disorder in psychosis,” PLoS ONE, vol. 7, 2012. [Online]. Available: https://api.semanticscholar.org/CorpusID:9506186
  222. A. B. Warriner, V. Kuperman, and M. Brysbaert, “Norms of valence, arousal, and dominance for 13,915 english lemmas,” Behavior Research Methods, vol. 45, pp. 1191 – 1207, 2013. [Online]. Available: https://api.semanticscholar.org/CorpusID:16918336
  223. H. Ai and X. Lu, “A web-based system for automatic measurement of lexical complexity,” in 27th Annual Symposium of the Computer-Assisted Language Consortium (CALICO-10). Amherst, MA. June, 2010, pp. 8–12.
  224. B. Croisile, B. Ska, M.-J. Brabant, A. Duchêne, Y. Lepage, G. Aimard, and M. Trillet, “Comparative study of oral and written picture description in patients with alzheimer’s disease,” Brain and Language, vol. 53, pp. 1–19, 1996. [Online]. Available: https://api.semanticscholar.org/CorpusID:36544389
  225. J. Pennington, R. Socher, and C. Manning, “GloVe: Global vectors for word representation,” in Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), A. Moschitti, B. Pang, and W. Daelemans, Eds.   Doha, Qatar: Association for Computational Linguistics, Oct. 2014, pp. 1532–1543. [Online]. Available: https://aclanthology.org/D14-1162
  226. Z. Dai, Z. Yang, Y. Yang, J. G. Carbonell, Q. V. Le, and R. Salakhutdinov, “Transformer-xl: Attentive language models beyond a fixed-length context,” CoRR, vol. abs/1901.02860, 2019. [Online]. Available: http://arxiv.org/abs/1901.02860
  227. A. Radford and K. Narasimhan, “Improving language understanding by generative pre-training,” 2018. [Online]. Available: https://api.semanticscholar.org/CorpusID:49313245
  228. K. C. Fraser, J. A. Meltzer, and F. Rudzicz, “Linguistic features identify alzheimer’s disease in narrative speech,” Journal of Alzheimer’s Disease, vol. 49, no. 2, pp. 407–422, 2016.
  229. F. Di Palo and N. Parde, “Enriching neural models with targeted features for dementia detection,” in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop, F. Alva-Manchego, E. Choi, and D. Khashabi, Eds.   Florence, Italy: Association for Computational Linguistics, Jul. 2019, pp. 302–308. [Online]. Available: https://aclanthology.org/P19-2042
  230. Y. Sun, S. Wang, Y. Li, S. Feng, H. Tian, H. Wu, and H. Wang, “ERNIE 2.0: A continual pre-training framework for language understanding,” CoRR, vol. abs/1907.12412, 2019. [Online]. Available: http://arxiv.org/abs/1907.12412
  231. P. Garrard, L. M. Maloney, J. R. Hodges, and K. Patterson, “The effects of very early alzheimer’s disease on the characteristics of writing by a renowned author,” Brain, vol. 128, no. 2, pp. 250–260, 2005.
  232. V. Berisha, S. Wang, A. LaCross, and J. Liss, “Tracking discourse complexity preceding alzheimer’s disease diagnosis: A case study comparing the press conferences of presidents ronald reagan and george herbert walker bush,” Journal of Alzheimer’s Disease, vol. 45, no. 3, pp. 959–963, 2015.
  233. R. S. Bucks, S. Singh, J. M. Cuerden, and G. K. Wilcock, “Analysis of spontaneous, conversational speech in dementia of alzheimer type: Evaluation of an objective technique for analysing lexical performance,” Aphasiology, vol. 14, no. 1, pp. 71–91, 2000.
  234. T. Mikolov, K. Chen, G. Corrado, and J. Dean, “Efficient estimation of word representations in vector space,” arXiv preprint arXiv:1301.3781, 2013.
  235. J. Cañete, G. Chaperon, R. Fuentes, J.-H. Ho, H. Kang, and J. Pérez, “Spanish pre-trained bert model and evaluation data,” 2023. [Online]. Available: https://arxiv.org/abs/2308.02976
  236. A. J. Thirunavukarasu, D. S. J. Ting, K. Elangovan, L. Gutierrez, T. F. Tan, and D. S. W. Ting, “Large language models in medicine,” Nature medicine, vol. 29, no. 8, pp. 1930–1940, 2023.
  237. W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y. Hou, Y. Min, B. Zhang, J. Zhang, Z. Dong et al., “A survey of large language models,” arXiv preprint arXiv:2303.18223, 2023.
  238. Y. Chang, X. Wang, J. Wang, Y. Wu, L. Yang, K. Zhu, H. Chen, X. Yi, C. Wang, Y. Wang et al., “A survey on evaluation of large language models,” ACM Transactions on Intelligent Systems and Technology, vol. 15, no. 3, pp. 1–45, 2024.
  239. OpenAI, “Chatgpt: Openai’s gpt-4 language model,” 2024. [Online]. Available: https://www.openai.com/chatgpt
  240. Q. V. Le and T. Mikolov, “Distributed representations of sentences and documents,” CoRR, vol. abs/1405.4053, 2014. [Online]. Available: http://arxiv.org/abs/1405.4053
  241. N. Reimers and I. Gurevych, “Sentence-bert: Sentence embeddings using siamese bert-networks,” arXiv preprint arXiv:1908.10084, 2019.
  242. A. Joulin, E. Grave, P. Bojanowski, and T. Mikolov, “Bag of tricks for efficient text classification,” arXiv preprint arXiv:1607.01759, 2016.
  243. A. Akbik, D. Blythe, and R. Vollgraf, “Contextual string embeddings for sequence labeling,” in Proceedings of the 27th international conference on computational linguistics, 2018, pp. 1638–1649.
  244. N. D. Cilia, T. D’Alessandro, C. De Stefano, F. Fontanella, and M. Molinara, “From online handwriting to synthetic images for alzheimer’s disease detection using a deep transfer learning approach,” IEEE Journal of Biomedical and Health Informatics, vol. 25, no. 12, pp. 4243–4254, 2021.
  245. J. Sun, H. H. Dodge, and M. H. Mahoor, “Mc-vivit: Multi-branch classifier-vivit to detect mild cognitive impairment in older adults using facial videos,” Expert Systems with Applications, vol. 238, p. 121929, 2024. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0957417423024314
  246. X. Liang, A. Angelopoulou, E. Kapetanios, B. Woll, R. Al Batat, and T. Woolfe, “A multi-modal machine learning approach and toolkit to automate recognition of early stages of dementia among british sign language users,” in Computer Vision – ECCV 2020 Workshops, A. Bartoli and A. Fusiello, Eds.   Cham: Springer International Publishing, 2020, pp. 278–293.
  247. K. Hu, Z. Wang, W. Wang, K. A. E. Martens, L. Wang, T. Tan, S. J. Lewis, and D. D. Feng, “Graph sequence recurrent neural network for vision-based freezing of gait detection,” IEEE Transactions on Image Processing, vol. 29, pp. 1890–1901, 2019.
  248. A. Das, X. Niu, A. Dantcheva, S. L. Happy, H. Han, R. Zeghari, P. Robert, S. Shan, F. Bremond, and X. Chen, “A spatio-temporal approach for apathy classification,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 32, no. 5, pp. 2561–2573, 2022.
  249. G. Vessio, “Dynamic handwriting analysis for neurodegenerative disease assessment: a literary review,” Applied Sciences, vol. 9, no. 21, p. 4666, 2019.
  250. E. Onofri, M. Mercuri, M. Salesi, S. Ferrara, G. M. Troili, C. Simeone, M. R. Ricciardi, S. Ricci, and T. Archer, “Dysgraphia in relation to cognitive performance in patients with alzheimer’s disease,” Journal of Intellectual Disability-Diagnosis and Treatment, vol. 1, no. 2, pp. 113–124, 2013.
  251. S. Müller, O. Preische, P. Heymann, U. Elbing, and C. Laske, “Diagnostic value of a tablet-based drawing task for discrimination of patients in the early course of alzheimer’s disease from healthy individuals,” Journal of Alzheimer’s Disease, vol. 55, no. 4, pp. 1463–1469, 2017.
  252. C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the inception architecture for computer vision,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 2818–2826.
  253. U. Nam, K. Lee, H. Ko, J.-Y. Lee, and E. C. Lee, “Analyzing facial and eye movements to screen for alzheimer’s disease,” Sensors, vol. 20, no. 18, p. 5349, 2020.
  254. M. C. N. Dourado, B. Torres Mendonça de Melo Fádel, J. P. Simões Neto, G. Alves, and C. Alves, “Facial expression recognition patterns in mild and moderate alzheimer’s disease,” Journal of Alzheimer’s Disease, vol. 69, no. 2, pp. 539–549, 2019.
  255. B. Jin, Y. Qu, L. Zhang, and Z. Gao, “Diagnosing parkinson disease through facial expression recognition: video analysis,” Journal of medical Internet research, vol. 22, no. 7, p. e18697, 2020.
  256. H. Tanaka, H. Adachi, H. Kazui, M. Ikeda, T. Kudo, and S. Nakamura, “Detecting dementia from face in human-agent interaction,” in Adjunct of the 2019 international conference on multimodal interaction, 2019, pp. 1–6.
  257. Z. Liu, J. Ning, Y. Cao, Y. Wei, Z. Zhang, S. Lin, and H. Hu, “Video swin transformer,” CoRR, vol. abs/2106.13230, 2021. [Online]. Available: https://arxiv.org/abs/2106.13230
  258. M. A. Hely, W. G. Reid, M. A. Adena, G. M. Halliday, and J. G. Morris, “The sydney multicenter study of parkinson’s disease: the inevitability of dementia at 20 years,” Movement disorders, vol. 23, no. 6, pp. 837–844, 2008.
  259. M. Macht, Y. Kaussner, J. C. Möller, K. Stiasny-Kolster, K. M. Eggert, H.-P. Krüger, and H. Ellgring, “Predictors of freezing in parkinson’s disease: a survey of 6,620 patients,” Movement disorders, vol. 22, no. 7, pp. 953–956, 2007.
  260. B. R. Bloem, J. M. Hausdorff, J. E. Visser, and N. Giladi, “Falls and freezing of gait in parkinson’s disease: a review of two interconnected, episodic phenomena,” Movement disorders: official journal of the Movement Disorder Society, vol. 19, no. 8, pp. 871–884, 2004.
  261. S. J. Lewis and R. A. Barker, “A pathophysiological model of freezing of gait in parkinson’s disease,” Parkinsonism & related disorders, vol. 15, no. 5, pp. 333–338, 2009.
  262. T. Khan, J. Westin, and M. Dougherty, “Motion cue analysis for parkinsonian gait recognition,” The open biomedical engineering journal, vol. 7, p. 1, 2013.
  263. M. Nieto-Hidalgo, F. J. Ferrández-Pastor, R. J. Valdivieso-Sarabia, J. Mora-Pascual, and J. M. García-Chamizo, “A vision based proposal for classification of normal and abnormal gait using rgb camera,” Journal of biomedical informatics, vol. 63, pp. 82–89, 2016.
  264. R. J. Molitor, P. C. Ko, and B. A. Ally, “Eye movements in alzheimer’s disease,” Journal of Alzheimer’s disease, vol. 44, no. 1, pp. 1–12, 2015.
  265. A. Oyama, S. Takeda, Y. Ito, T. Nakajima, Y. Takami, Y. Takeya, K. Yamamoto, K. Sugimoto, H. Shimizu, M. Shimamura et al., “Novel method for rapid assessment of cognitive impairment using high-performance eye-tracking technology,” Scientific reports, vol. 9, no. 1, p. 12932, 2019.
  266. D. Lagun, C. Manzanares, S. M. Zola, E. A. Buffalo, and E. Agichtein, “Detecting cognitive impairment by eye movement analysis using automatic classification algorithms,” Journal of neuroscience methods, vol. 201, no. 1, pp. 196–203, 2011.
  267. K. Mengoudi, D. Ravi, K. X. Yong, S. Primativo, I. M. Pavisic, E. Brotherhood, K. Lu, J. M. Schott, S. J. Crutch, and D. C. Alexander, “Augmenting dementia cognitive assessment with instruction-less eye-tracking tests,” IEEE journal of biomedical and health informatics, vol. 24, no. 11, pp. 3066–3075, 2020.
  268. A. A. Lazarus et al., “Multimodal behavior therapy: I.” 1976.
  269. P. Xu, X. Zhu, and D. A. Clifton, “Multimodal learning with transformers: A survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 10, pp. 12 113–12 132, 2023.
  270. W. Guo, J. Wang, and S. Wang, “Deep multimodal representation learning: A survey,” Ieee Access, vol. 7, pp. 63 373–63 394, 2019.
  271. T. Baltrušaitis, C. Ahuja, and L.-P. Morency, “Multimodal machine learning: A survey and taxonomy,” IEEE transactions on pattern analysis and machine intelligence, vol. 41, no. 2, pp. 423–443, 2018.
  272. D. Altinok, “Explainable multimodal fusion for dementia detection from text and speech,” in International Conference on Text, Speech, and Dialogue.   Springer, 2024, pp. 236–251.
  273. L. Ilias and D. Askounis, “Multimodal deep learning models for detecting dementia from speech and transcripts,” Frontiers in Aging Neuroscience, vol. 14, 2022. [Online]. Available: https://www.frontiersin.org/articles/10.3389/fnagi.2022.830943
  274. L. Ilias, D. Askounis, and J. Psarras, “A multimodal approach for dementia detection from spontaneous speech with tensor fusion layer,” in 2022 IEEE-EMBS International Conference on Biomedical and Health Informatics (BHI), 2022, pp. 1–5.
  275. A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning transferable visual models from natural language supervision,” CoRR, vol. abs/2103.00020, 2021. [Online]. Available: https://arxiv.org/abs/2103.00020
  276. H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample, “Llama: Open and efficient foundation language models,” 2023. [Online]. Available: https://arxiv.org/abs/2302.13971
  277. J. Li, D. Li, S. Savarese, and S. Hoi, “Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,” 2023. [Online]. Available: https://arxiv.org/abs/2301.12597
  278. C. Sun, A. Myers, C. Vondrick, K. Murphy, and C. Schmid, “Videobert: A joint model for video and language representation learning,” 2019. [Online]. Available: https://arxiv.org/abs/1904.01766
  279. L. H. Li, M. Yatskar, D. Yin, C.-J. Hsieh, and K.-W. Chang, “Visualbert: A simple and performant baseline for vision and language,” arXiv preprint arXiv:1908.03557, 2019.
  280. J. Lu, D. Batra, D. Parikh, and S. Lee, “Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks,” CoRR, vol. abs/1908.02265, 2019. [Online]. Available: http://arxiv.org/abs/1908.02265
  281. R. Zheng, J. Chen, M. Ma, and L. Huang, “Fused acoustic and text encoding for multimodal bilingual pretraining and speech translation,” in International Conference on Machine Learning.   PMLR, 2021, pp. 12 736–12 746.
  282. B. Shi, W.-N. Hsu, K. Lakhotia, and A. Mohamed, “Learning audio-visual speech representation by masked multimodal cluster prediction,” arXiv preprint arXiv:2201.02184, 2022.
  283. J. Lin, A. Yang, Y. Zhang, J. Liu, J. Zhou, and H. Yang, “Interbert: Vision-and-language interaction for multi-modal pretraining,” 2021. [Online]. Available: https://arxiv.org/abs/2003.13198
  284. Y.-H. H. Tsai, S. Bai, P. P. Liang, J. Z. Kolter, L.-P. Morency, and R. Salakhutdinov, “Multimodal transformer for unaligned multimodal language sequences,” 2019. [Online]. Available: https://arxiv.org/abs/1906.00295
  285. X. Xu, T. Wang, Y. Yang, L. Zuo, F. Shen, and H. T. Shen, “Cross-modal attention with semantic consistence for image–text matching,” IEEE transactions on neural networks and learning systems, vol. 31, no. 12, pp. 5412–5425, 2020.
  286. A. Khare, S. Parthasarathy, and S. Sundaram, “Self-supervised learning with cross-modal transformers for emotion recognition,” in 2021 IEEE Spoken Language Technology Workshop (SLT), 2021, pp. 381–388.
  287. V. Murahari, D. Batra, D. Parikh, and A. Das, “Large-scale pretraining for visual dialog: A simple state-of-the-art baseline,” 2020. [Online]. Available: https://arxiv.org/abs/1912.02379
  288. J. Arevalo, T. Solorio, M. Montes-y Gomez, and F. A. González, “Gated multimodal networks,” Neural Computing and Applications, vol. 32, pp. 10 209–10 228, 2020.
  289. Y. Zhang, D. Sidibé, O. Morel, and F. Mériaudeau, “Deep multimodal fusion for semantic image segmentation: A survey,” Image and Vision Computing, vol. 105, p. 104042, 2021. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0262885620301748
  290. S. R. Stahlschmidt, B. Ulfenborg, and J. Synnergren, “Multimodal deep learning for biomedical data fusion: a review,” Briefings in Bioinformatics, vol. 23, no. 2, p. bbab569, 01 2022. [Online]. Available: https://doi.org/10.1093/bib/bbab569
  291. M. Wang, J. Xing, and Y. Liu, “Actionclip: A new paradigm for video action recognition,” 2021. [Online]. Available: https://arxiv.org/abs/2109.08472
  292. W. Wu, X. Wang, H. Luo, J. Wang, Y. Yang, and W. Ouyang, “Bidirectional cross-modal knowledge exploration for video recognition with pre-trained vision-language models,” 2023. [Online]. Available: https://arxiv.org/abs/2301.00182
  293. A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning transferable visual models from natural language supervision,” 2021. [Online]. Available: https://arxiv.org/abs/2103.00020
  294. J. Górriz et al., “Computational approaches to explainable artificial intelligence: Advances in theory, applications and trends,” Information Fusion, vol. 100, p. 101945, 2023. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S1566253523002610
  295. A. Holzinger, G. Langs, H. Denk, K. Zatloukal, and H. Müller, “Causability and explainability of artificial intelligence in medicine,” Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, vol. 9, no. 4, p. e1312, 2019.
  296. G. Vilone and L. Longo, “Notions of explainability and evaluation approaches for explainable artificial intelligence,” Information Fusion, vol. 76, pp. 89–106, 2021.
  297. F. Tao, H. Zhang, A. Liu, and A. Y. Nee, “Digital twin in industry: State-of-the-art,” IEEE Transactions on industrial informatics, vol. 15, no. 4, pp. 2405–2415, 2018.
  298. K. Bruynseels, F. Santoni de Sio, and J. Van den Hoven, “Digital twins in health care: ethical implications of an emerging engineering paradigm,” Frontiers in genetics, vol. 9, p. 31, 2018.

Summary

No one has generated a summary of this paper yet.

Paper to Video (Beta)

No one has generated a video about this paper yet.

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Continue Learning

We haven't generated follow-up questions for this paper yet.

Collections

Sign up for free to add this paper to one or more collections.