Deep Insights into Cognitive Decline: A Survey of Leveraging Non-Intrusive Modalities with Deep Learning Techniques
Abstract: Cognitive decline is a natural part of aging, often resulting in reduced cognitive abilities. In some cases, however, this decline is more pronounced, typically due to disorders such as Alzheimer's disease. Early detection of anomalous cognitive decline is crucial, as it can facilitate timely professional intervention. While medical data can help in this detection, it often involves invasive procedures. An alternative approach is to employ non-intrusive techniques such as speech or handwriting analysis, which do not necessarily affect daily activities. This survey reviews the most relevant methodologies that use deep learning techniques to automate the cognitive decline estimation task, including audio, text, and visual processing. We discuss the key features and advantages of each modality and methodology, including state-of-the-art approaches like Transformer architecture and foundation models. In addition, we present works that integrate different modalities to develop multimodal models. We also highlight the most significant datasets and the quantitative results from studies using these resources. From this review, several conclusions emerge. In most cases, the textual modality achieves the best results and is the most relevant for detecting cognitive decline. Moreover, combining various approaches from individual modalities into a multimodal model consistently enhances performance across nearly all scenarios.
- D. Kiely, “Cognitive function in encyclopaedia of quality of life and well-being research (ed. michalos, ac) 974–978,” 2014.
- I. J. Deary, J. Corley, A. J. Gow, S. E. Harris, L. M. Houlihan, R. E. Marioni, L. Penke, S. B. Rafnsson, and J. M. Starr, “Age-associated cognitive decline,” British Medical Bulletin, vol. 92, no. 1, pp. 135–152, 09 2009. [Online]. Available: https://doi.org/10.1093/bmb/ldp033
- S. Duong, T. Patel, and F. Chang, “Dementia: What pharmacists need to know,” Canadian Pharmacists Journal / Revue des Pharmaciens du Canada, vol. 150, no. 2, pp. 118–129, 2017. [Online]. Available: https://doi.org/10.1177/1715163517690745
- F. Portet, P. J. Ousset, P. J. Visser, G. B. Frisoni, F. Nobili, P. Scheltens, B. Vellas, J. Touchon, and the MCI Working Group of the European Consortium on Alzheimer’s Disease (EADC), “Mild cognitive impairment (mci) in medical practice: a critical review of the concept and new diagnostic procedure. report of the mci working group of the european consortium on alzheimer’s disease,” Journal of Neurology, Neurosurgery & Psychiatry, vol. 77, no. 6, pp. 714–718, 2006. [Online]. Available: https://jnnp.bmj.com/content/77/6/714
- P. Celsis, “Age-related cognitive decline, mild cognitive impairment or preclinical alzheimer’s disease?” Annals of Medicine, vol. 32, no. 1, pp. 6–14, 2000, pMID: 10711572. [Online]. Available: https://doi.org/10.3109/07853890008995904
- World Health Organization, “Mental health action plan 2013-2020,” WHO Library Cataloguing-in-Publication DataLibrary Cataloguing-in-Publication Data, pp. 1–44, 2023.
- A. R. Damasio, “Aphasia,” New England Journal of Medicine, vol. 326, no. 8, pp. 531–539, 1992.
- C. Cabello-Collado, J. Rodriguez-Juan, D. Ortiz-Perez, J. Garcia-Rodriguez, D. Tomás, and M. F. Vizcaya-Moreno, “Automated generation of clinical reports using sensing technologies with deep learning techniques,” Sensors, vol. 24, no. 9, 2024. [Online]. Available: https://www.mdpi.com/1424-8220/24/9/2751
- R. Zhao, R. Yan, Z. Chen, K. Mao, P. Wang, and R. X. Gao, “Deep learning and its applications to machine health monitoring,” Mechanical Systems and Signal Processing, vol. 115, pp. 213–237, 2019.
- J. Venugopalan, L. Tong, H. R. Hassanzadeh, and M. D. Wang, “Multimodal deep learning models for early detection of alzheimer’s disease stage,” Scientific reports, vol. 11, no. 1, p. 3254, 2021.
- S. Liu, S. Liu, W. Cai, S. Pujol, R. Kikinis, and D. Feng, “Early diagnosis of alzheimer’s disease with deep learning,” in 2014 IEEE 11th international symposium on biomedical imaging (ISBI). IEEE, 2014, pp. 1015–1018.
- M. Di Luca, D. Nutt, W. Oertel, P. Boyer, J. Jaarsma, F. Destrebecq, G. Esposito, and V. Quoidbach, “Towards earlier diagnosis and treatment of disorders of the brain,” Bulletin of the World Health Organization, vol. 96, no. 5, p. 298, 2018.
- D. Wen, Z. Wei, Y. Zhou, G. Li, X. Zhang, and W. Han, “Deep learning methods to process fmri data and their application in the diagnosis of cognitive impairment: a brief overview and our opinion,” Frontiers in neuroinformatics, vol. 12, p. 23, 2018.
- H. Taheri Gorji and N. Kaabouch, “A deep learning approach for diagnosis of mild cognitive impairment based on mri images,” Brain sciences, vol. 9, no. 9, p. 217, 2019.
- L. Kang, J. Jiang, J. Huang, and T. Zhang, “Identifying early mild cognitive impairment by multi-modality mri-based deep learning,” Frontiers in aging neuroscience, vol. 12, p. 206, 2020.
- X. Feng, Z. C. Lipton, J. Yang, S. A. Small, F. A. Provenzano, A. D. N. Initiative, F. L. D. N. Initiative et al., “Estimating brain age based on a uniform healthy population with deep learning and structural magnetic resonance imaging,” Neurobiology of aging, vol. 91, pp. 15–25, 2020.
- S. Chauhan, L. Vig, M. De Filippo De Grazia, M. Corbetta, S. Ahmad, and M. Zorzi, “A comparison of shallow and deep learning methods for predicting cognitive performance of stroke patients from mri lesion images,” Frontiers in neuroinformatics, vol. 13, p. 53, 2019.
- G. Zhu, B. Jiang, L. Tong, Y. Xie, G. Zaharchuk, and M. Wintermark, “Applications of deep learning to neuro-imaging techniques,” Frontiers in neurology, vol. 10, p. 869, 2019.
- T. Jo, K. Nho, and A. J. Saykin, “Deep learning in alzheimer’s disease: diagnostic classification and prognostic prediction using neuroimaging data,” Frontiers in aging neuroscience, vol. 11, p. 220, 2019.
- E. Pellegrini, L. Ballerini, M. d. C. V. Hernandez, F. M. Chappell, V. González-Castro, D. Anblagan, S. Danso, S. Muñoz-Maniega, D. Job, C. Pernet et al., “Machine learning of neuroimaging for assisted diagnosis of cognitive impairment and dementia: a systematic review,” Alzheimer’s & Dementia: Diagnosis, Assessment & Disease Monitoring, vol. 10, pp. 519–535, 2018.
- S. A. Graham, E. E. Lee, D. V. Jeste, R. Van Patten, E. W. Twamley, C. Nebeker, Y. Yamada, H.-C. Kim, and C. A. Depp, “Artificial intelligence approaches to predicting and detecting cognitive decline in older adults: A conceptual review,” Psychiatry research, vol. 284, p. 112732, 2020.
- H. Choi, K. H. Jin, A. D. N. Initiative et al., “Predicting cognitive decline with deep learning of brain metabolism and amyloid imaging,” Behavioural brain research, vol. 344, pp. 103–109, 2018.
- S. Grueso and R. Viejo-Sobera, “Machine learning methods for predicting progression from mild cognitive impairment to alzheimer’s disease dementia: a systematic review,” Alzheimer’s research & therapy, vol. 13, pp. 1–29, 2021.
- S. L. Warren and A. A. Moustafa, “Functional magnetic resonance imaging, deep learning, and alzheimer’s disease: A systematic review,” Journal of Neuroimaging, vol. 33, no. 1, pp. 5–18, 2023.
- M. Ansart, S. Epelbaum, G. Bassignana, A. Bône, S. Bottani, T. Cattai, R. Couronné, J. Faouzi, I. Koval, M. Louis et al., “Predicting the progression of mild cognitive impairment using machine learning: a systematic, quantitative and critical review,” Medical Image Analysis, vol. 67, p. 101848, 2021.
- A. Alberdi, A. Aztiria, and A. Basarab, “On the early diagnosis of alzheimer’s disease from multimodal signals: A survey,” Artificial Intelligence in Medicine, vol. 71, pp. 1–29, 2016. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0933365716300732
- V. Skaramagkas, A. Pentari, Z. Kefalopoulou, and M. Tsiknakis, “Multi-modal deep learning diagnosis of parkinson’s disease—a systematic review,” IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 31, pp. 2399–2423, 2023.
- Q. Yang, X. Li, X. Ding, F. Xu, and Z. Ling, “Deep learning-based speech analysis for alzheimer’s disease detection: a literature review,” Alzheimer’s Research & Therapy, vol. 14, no. 1, p. 186, 2022.
- X. Qi, Q. Zhou, J. Dong, and W. Bao, “Noninvasive automatic detection of alzheimer’s disease from spontaneous speech: a review,” Frontiers in Aging Neuroscience, vol. 15, 2023. [Online]. Available: https://www.frontiersin.org/journals/aging-neuroscience/articles/10.3389/fnagi.2023.1224723
- S. Gauthier, B. Reisberg, M. Zaudig, R. C. Petersen, K. Ritchie, K. Broich, S. Belleville, H. Brodaty, D. Bennett, H. Chertkow et al., “Mild cognitive impairment,” The lancet, vol. 367, no. 9518, pp. 1262–1270, 2006.
- R. C. Petersen, “Mild cognitive impairment,” CONTINUUM: lifelong Learning in Neurology, vol. 22, no. 2, pp. 404–418, 2016.
- P. B. Rosenberg, M. M. Mielke, B. Appleby, E. Oh, J.-M. Leoutsakos, and C. G. Lyketsos, “Neuropsychiatric symptoms in mci subtypes: the importance of executive dysfunction,” International journal of geriatric psychiatry, vol. 26, no. 4, pp. 364–372, 2011.
- Z. Arvanitakis, R. C. Shah, and D. A. Bennett, “Diagnosis and management of dementia,” Jama, vol. 322, no. 16, pp. 1589–1599, 2019.
- D. S. Geldmacher and P. J. Whitehouse, “Evaluation of dementia,” New England Journal of Medicine, vol. 335, no. 5, pp. 330–336, 1996.
- S. H. Ferris and M. Farlow, “Language impairment in alzheimer’s disease and benefits of acetylcholinesterase inhibitors,” Clinical interventions in aging, pp. 1007–1014, 2013.
- H. S. Kirshner and S. M. Wilson, “Aphasia and aphasic syndromes,” Bradley’s Neurology in Clinical Practice E-Book, vol. 133, 2021.
- M. P. Alexander and A. E. Hillis, “Aphasia,” Handbook of clinical neurology, vol. 88, pp. 287–309, 2008.
- L. V. Kalia and A. E. Lang, “Parkinson’s disease,” The Lancet, vol. 386, no. 9996, pp. 896–912, 2015.
- D. Aarsland, K. Andersen, J. P. Larsen, R. Perry, T. Wentzel-Larsen, A. Lolk, and P. Kragh-Sørensen, “The rate of cognitive decline in parkinson disease,” Archives of neurology, vol. 61, no. 12, pp. 1906–1911, 2004.
- D. Aarsland, B. Creese, M. Politis, K. R. Chaudhuri, D. H. Ffytche, D. Weintraub, and C. Ballard, “Cognitive decline in parkinson disease,” Nature Reviews Neurology, vol. 13, no. 4, pp. 217–231, 2017.
- R. Van Reekum, D. T. Stuss, and L. Ostrander, “Apathy: why care?” The Journal of neuropsychiatry and clinical neurosciences, vol. 17, no. 1, pp. 7–19, 2005.
- G. Montoya-Murillo, N. Ibarretxe-Bilbao, J. Peña, and N. Ojeda, “The impact of apathy on cognitive performance in the elderly,” International Journal of Geriatric Psychiatry, vol. 34, no. 5, pp. 657–665, 2019.
- G. Konstantakopoulos, D. Ploumpidis, P. Oulis, P. Patrikelis, A. Soumani, G. N. Papadimitriou, and A. M. Politis, “Apathy, cognitive deficits and functional impairment in schizophrenia,” Schizophrenia research, vol. 133, no. 1-3, pp. 193–198, 2011.
- S. Gluhm, J. Goldstein, K. Loc, A. Colt, C. Van Liew, and J. Corey-Bloom, “Cognitive performance on the mini-mental state examination and the montreal cognitive assessment across the healthy adult lifespan,” Cognitive and Behavioral Neurology, vol. 26, no. 1, pp. 1–5, 2013.
- I. Arevalo-Rodriguez, N. Smailagic, M. R. i Figuls, A. Ciapponi, E. Sanchez-Perez, A. Giannakou, O. L. Pedraza, X. B. Cosp, and S. Cullum, “Mini‐mental state examination (mmse) for the detection of alzheimer’s disease and other dementias in people with mild cognitive impairment (mci),” Cochrane Database of Systematic Reviews, vol. 3, 2015. [Online]. Available: https://doi.org//10.1002/14651858.CD010783.pub2
- J. S. Shiroky, H. M. Schipper, H. Bergman, and H. Chertkow, “Can you have dementia with an mmse score of 30?” American Journal of Alzheimer’s Disease & Other Dementias®, vol. 22, no. 5, pp. 406–415, 2007.
- P. T. Trzepacz, H. Hochstetler, S. Wang, B. Walker, A. J. Saykin, and A. D. N. Initiative, “Relationship between the montreal cognitive assessment and mini-mental state examination for assessment of mild cognitive impairment in older adults,” BMC geriatrics, vol. 15, pp. 1–9, 2015.
- S. Hoops, S. Nazem, A. Siderowf, J. Duda, S. Xie, M. Stern, and D. Weintraub, “Validity of the moca and mmse in the detection of mci and dementia in parkinson disease,” Neurology, vol. 73, no. 21, pp. 1738–1745, 2009.
- A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” 2023.
- D. Moher, A. Liberati, J. Tetzlaff, D. G. Altman, P. Group et al., “Preferred reporting items for systematic reviews and meta-analyses: the prisma statement,” International journal of surgery, vol. 8, no. 5, pp. 336–341, 2010.
- A. M. Lanzi, A. K. Saylor, D. Fromm, H. Liu, B. MacWhinney, and M. L. Cohen, “Dementiabank: Theoretical rationale, protocol, and illustrative analyses,” American Journal of Speech-Language Pathology, vol. 32, no. 2, pp. 426–438, 2023.
- J. T. Becker, F. Boiler, O. L. Lopez, J. Saxton, and K. L. McGonigle, “The natural history of alzheimer’s disease: description of study cohort and accuracy of diagnosis,” Archives of neurology, vol. 51, no. 6, pp. 585–594, 1994.
- K. P. Elaine Giles and J. R. Hodges, “Performance on the boston cookie theft picture description task in patients with early dementia of the alzheimer’s type: Missing information,” Aphasiology, vol. 10, no. 4, pp. 395–408, 1996. [Online]. Available: https://doi.org/10.1080/02687039608248419
- S. Luz, F. Haider, S. de la Fuente, D. Fromm, and B. MacWhinney, “Alzheimer’s dementia recognition through spontaneous speech: The adress challenge,” 2020.
- ——, “Detecting cognitive decline using speech only: The adresso challenge,” 2021.
- S. Luz, F. Haider, D. Fromm, I. Lazarou, I. Kompatsiaris, and B. MacWhinney, “Multilingual alzheimer’s dementia recognition through spontaneous speech: a signal processing grand challenge,” 2023.
- S. D. L. F. Garcia, F. Haider, D. Fromm, B. MacWhinney, A. Lanzi, Y.-N. Chang, C.-J. Chou, Y.-C. Liu et al., “Connected speech-based cognitive assessment in chinese and english,” arXiv preprint arXiv:2406.10272, 2024.
- B. MacWhinney, D. Fromm, M. Forbes, and A. Holland, “Aphasiabank: Methods for studying discourse,” Aphasiology, vol. 25, no. 11, pp. 1286–1307, 2011.
- C. Pope and B. H. Davis, “Finding a balance: The carolinas conversation collection,” Corpus Linguistics and Linguistic Theory, vol. 7, no. 1, pp. 143–161, 2011. [Online]. Available: https://doi.org/10.1515/cllt.2011.007
- R. A. Li, I. Hajjar, F. Goldstein, and J. D. Choi, “Analysis of hierarchical multi-content text classification model on B-SHARP dataset for early detection of Alzheimer’s disease,” in Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, K.-F. Wong, K. Knight, and H. Wu, Eds. Suzhou, China: Association for Computational Linguistics, Dec. 2020, pp. 358–365. [Online]. Available: https://aclanthology.org/2020.aacl-main.38
- D. Carr, “How to successfully navigate a revise-and-resubmit decision and handle rejections,” Innovation in Aging, vol. 3, no. Suppl 1, p. S224, 2019.
- K. Yu, K. Wild, K. Potempa, B. M. Hampstead, P. A. Lichtenberg, L. M. Struble, P. Pruitt, E. L. Alfaro, J. Lindsley, M. MacDonald, J. A. Kaye, L. C. Silbert, and H. H. Dodge, “The internet-based conversational engagement clinical trial (i-conect) in socially isolated adults 75+ years old: Randomized controlled trial protocol and covid-19 related study modifications,” Frontiers in Digital Health, vol. 3, 2021. [Online]. Available: https://www.scopus.com/inward/record.uri?eid=2-s2.0-85121367382&doi=10.3389%2ffdgth.2021.714813&partnerID=40&md5=dad167760b9a0f0c30c4a9e08937c379
- C.-Y. Wu, N. Mattek, K. Wild, L. M. Miller, J. A. Kaye, L. C. Silbert, and H. H. Dodge, “Can changes in social contact (frequency and mode) mitigate low mood before and during the covid-19 pandemic? the i-conect project,” Journal of the American Geriatrics Society, vol. 70, no. 3, pp. 669–676, 2022.
- J. R. Orozco-Arroyave, J. D. Arias-Londoño, J. F. Vargas-Bonilla, M. C. González-Rátiva, and E. Nöth, “New Spanish speech corpus database for the analysis of people suffering from Parkinson’s disease,” in Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), N. Calzolari, K. Choukri, T. Declerck, H. Loftsson, B. Maegaard, J. Mariani, A. Moreno, J. Odijk, and S. Piperidis, Eds. Reykjavik, Iceland: European Language Resources Association (ELRA), May 2014, pp. 342–347. [Online]. Available: http://www.lrec-conf.org/proceedings/lrec2014/pdf/7_Paper.pdf
- A. Karakostas, A. Briassouli, K. Avgerinakis, I. Kompatsiaris, and M. Tsolaki, “The dem@care experiments and datasets: a technical report,” 2016. [Online]. Available: https://arxiv.org/abs/1701.01142
- F. Negin, P. Rodriguez, M. Koperski, A. Kerboua, J. Gonzàlez, J. Bourgeois, E. Chapoulie, P. Robert, and F. Bremond, “Praxis: Towards automatic cognitive assessment using gesture recognition,” Expert Systems with Applications, 2018.
- J. V. Egas-López, R. Balogh, N. Imre, I. Hoffmann, M. K. Szabó, L. Tóth, M. Pákáski, J. Kálmán, and G. Gosztolya, “Automatic screening of mild cognitive impairment and alzheimer’s disease by means of posterior-thresholding hesitation representation,” Computer Speech & Language, vol. 75, p. 101377, 2022.
- C. Themistocleous, M. Eckerström, and D. Kokkinakis, “Voice quality and speech fluency distinguish individuals with mild cognitive impairment from healthy controls,” Plos one, vol. 15, no. 7, p. e0236009, 2020.
- B. J. Abbaschian, D. Sierra-Sosa, and A. Elmaghraby, “Deep learning techniques for speech emotion recognition, from databases to models,” Sensors, vol. 21, no. 4, p. 1249, 2021.
- S. G. Koolagudi and K. S. Rao, “Emotion recognition from speech: a review,” International journal of speech technology, vol. 15, pp. 99–117, 2012.
- T. M. Wani, T. S. Gunawan, S. A. A. Qadri, M. Kartiwi, and E. Ambikairajah, “A comprehensive review of speech emotion recognition systems,” IEEE access, vol. 9, pp. 47 795–47 814, 2021.
- D. Ortiz-Perez, P. Ruiz-Ponce, J. Rodríguez-Juan, D. Tomás, J. Garcia-Rodriguez, and G. J. Nalepa, “Deep learning-based emotion detection in aphasia patients,” in International Conference on Soft Computing Models in Industrial and Environmental Applications. Springer, 2023, pp. 195–204.
- C. Code, G. Hemsley, and M. Herrmann, “The emotional impact of aphasia,” in Seminars in speech and language, vol. 20. © 1999 by Thieme Medical Publishers, Inc., 1999, pp. 19–31.
- P. Priyadarshinee, C. J. Clarke, J. Melechovsky, C. M. Y. Lin, B. B. T., and J.-M. Chen, “Alzheimer’s dementia speech (audio vs. text): Multi-modal machine learning at high vs. low resolution,” Applied Sciences, vol. 13, no. 7, 2023. [Online]. Available: https://www.mdpi.com/2076-3417/13/7/4244
- Z. Cui, W. Wu, W.-Q. Zhang, J. Wu, and C. Zhang, “Transferring speech-generic and depression-specific knowledge for alzheimer’s disease detection,” in 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, Dec. 2023. [Online]. Available: http://dx.doi.org/10.1109/ASRU57964.2023.10389785
- N. Wang, Y. Cao, S. Hao, Z. Shao, and K. Subbalakshmi, “Modular multi-modal attention network for alzheimer’s disease detection using patient audio and language data.” in Interspeech, 2021, pp. 3835–3839.
- Y. Ying, T. Yang, and H. Zhou, “Multimodal fusion for alzheimer’s disease recognition,” Applied Intelligence, vol. 53, no. 12, pp. 16 029–16 040, 2023.
- J.-U. Bang, S.-H. Han, and B.-O. Kang, “Alzheimer’s disease recognition from spontaneous speech using large language models,” ETRI Journal, 2024.
- M. Rohanian, J. Hough, and M. Purver, “Alzheimer’s dementia recognition using acoustic, lexical, disfluency and speech pause features robust to noisy inputs,” CoRR, vol. abs/2106.15684, 2021. [Online]. Available: https://arxiv.org/abs/2106.15684
- S. B. Shah, A. Bhandari, and P. G. Shambharkar, “Leveraging multimodal information in speech data for the non-invasive detection of alzheimer’s disease,” in 2023 14th International Conference on Computing Communication and Networking Technologies (ICCCNT), 2023, pp. 1–6.
- A. Hlédiková, D. Woszczyk, A. Akman, S. Demetriou, and B. Schuller, “Data augmentation for dementia detection in spoken language,” 2022.
- J. Koo, J. Lee, J. Pyo, Y. Jo, and K. Lee, “Exploiting multi-modal features from pre-trained networks for alzheimer’s dementia recognition,” in Interspeech, 10 2020, pp. 2217–2221.
- A. Pompili, T. Rolland, and A. Abad, “The inesc-id multi-modal system for the adress 2020 challenge,” arXiv preprint arXiv:2005.14646, 2020.
- N. Cummins, Y. Pan, Z. Ren, J. Fritsch, V. S. Nallanthighal, H. Christensen, D. Blackburn, B. W. Schuller, M. Magimai-Doss, H. Strik et al., “A comparison of acoustic and linguistics methodologies for alzheimer’s dementia recognition,” in Interspeech 2020. ISCA-International Speech Communication Association, 2020, pp. 2182–2186.
- P. Mahajan and V. Baths, “Acoustic and language based deep learning approaches for alzheimer’s dementia detection from spontaneous speech,” Frontiers in Aging Neuroscience, vol. 13, p. 623607, 2021.
- Y. Zhu, X. Liang, J. A. Batsis, and R. M. Roth, “Exploring deep transfer learning techniques for alzheimer’s dementia detection,” Frontiers in Computer Science, vol. 3, 2021. [Online]. Available: https://www.frontiersin.org/articles/10.3389/fcomp.2021.624683
- L. Ilias, D. Askounis, and J. Psarras, “Detecting dementia from speech and transcripts using transformers,” Computer Speech & Language, vol. 79, p. 101485, 2023. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0885230823000049
- A. Meghanani, C. Anoop, and A. Ramakrishnan, “An exploration of log-mel spectrogram and mfcc features for alzheimer’s dementia recognition from spontaneous speech,” in 2021 IEEE spoken language technology workshop (SLT). IEEE, 2021, pp. 670–677.
- Z. Liu, Z. Guo, Z. Ling, and Y. Li, “Detecting alzheimer’s disease from speech using neural networks with bottleneck features and data augmentation,” in ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2021, pp. 7323–7327.
- D. Ortiz-Perez, P. Ruiz-Ponce, D. Tomás, J. Garcia-Rodriguez, M. F. Vizcaya-Moreno, and M. Leo, “A deep learning-based multimodal architecture to predict signs of dementia,” Neurocomputing, vol. 548, p. 126413, 2023. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0925231223005362
- I. Krstev, M. Pavikjevikj, M. Toshevska, and S. Gievska, “Multimodal data fusion for automatic detection of alzheimer’s disease,” in International Conference on Human-Computer Interaction. Springer, 2022, pp. 79–94.
- D. Escobar-Grisales, C. D. Ríos-Urrego, and J. R. Orozco-Arroyave, “Deep learning and artificial intelligence applied to model speech and language in parkinson’s disease,” Diagnostics, vol. 13, no. 13, p. 2163, 2023.
- D. Ortiz-Perez, J. Garcia-Rodriguez, and D. Tomás, “Cognitive insights across languages: Enhancing multimodal interview analysis,” in Interspeech 2024, 2024, pp. 952–956.
- F. F. Poor, H. H. Dodge, and M. H. Mahoor, “A multimodal cross-transformer-based model to predict mild cognitive impairment using speech, language and vision,” Computers in Biology and Medicine, vol. 182, p. 109199, 2024. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0010482524012848
- Z. Sheng, Z. Guo, X. Li, Y. Li, and Z. Ling, “Dementia detection by fusing speech and eye-tracking representation,” in ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022, pp. 6457–6461.
- N. Narendra, B. Schuller, and P. Alku, “The detection of parkinson’s disease from speech using voice source information,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 1925–1936, 2021.
- Y. Ge, T. Wang, J. Cao, and S. Xu, “A novel multi-task learning based automatic speech impairment assessment algorithm,” in 2022 China Automation Congress (CAC), 2022, pp. 887–892.
- S. Allamy and A. L. Koerich, “1d cnn architectures for music genre classification,” in 2021 IEEE Symposium Series on Computational Intelligence (SSCI), 2021, pp. 01–07.
- S. Abdoli, P. Cardinal, and A. Lameiras Koerich, “End-to-end environmental sound classification using a 1d convolutional neural network,” Expert Systems with Applications, vol. 136, pp. 252–263, 2019. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0957417419304403
- K. Zaman, M. Sah, C. Direkoglu, and M. Unoki, “A survey of audio classification using deep learning,” IEEE Access, 2023.
- Y. M. Costa, L. S. Oliveira, and C. N. Silla Jr, “An evaluation of convolutional neural networks for music classification using spectrograms,” Applied soft computing, vol. 52, pp. 28–38, 2017.
- A. Satt, S. Rozenberg, R. Hoory et al., “Efficient emotion recognition from speech using deep learning on spectrograms.” in Interspeech, 2017, pp. 1089–1093.
- Y. Zeng, H. Mao, D. Peng, and Z. Yi, “Spectrogram based multi-task audio classification,” Multimedia Tools and Applications, vol. 78, pp. 3705–3722, 2019.
- S. S. Stevens, J. Volkmann, and E. B. Newman, “A Scale for the Measurement of the Psychological Magnitude Pitch,” The Journal of the Acoustical Society of America, vol. 8, no. 3, pp. 185–190, 01 1937. [Online]. Available: https://doi.org/10.1121/1.1915893
- T. Arias-Vergara, P. Klumpp, J. C. Vasquez-Correa, E. Nöth, J. R. Orozco-Arroyave, and M. Schuster, “Multi-channel spectrograms for speech processing applications using deep learning methods,” Pattern Analysis and Applications, vol. 24, pp. 423–431, 2021.
- H. Purwins, B. Li, T. Virtanen, J. Schlüter, S.-Y. Chang, and T. Sainath, “Deep learning for audio signal processing,” IEEE Journal of Selected Topics in Signal Processing, vol. 13, no. 2, pp. 206–219, 2019.
- L. Jiao and J. Zhao, “A survey on the new generation of deep learning in image processing,” Ieee Access, vol. 7, pp. 172 231–172 263, 2019.
- S. Minaee, Y. Boykov, F. Porikli, A. Plaza, N. Kehtarnavaz, and D. Terzopoulos, “Image segmentation using deep learning: A survey,” IEEE transactions on pattern analysis and machine intelligence, vol. 44, no. 7, pp. 3523–3542, 2021.
- J. Gu, Z. Wang, J. Kuen, L. Ma, A. Shahroudy, B. Shuai, T. Liu, X. Wang, G. Wang, J. Cai et al., “Recent advances in convolutional neural networks,” Pattern recognition, vol. 77, pp. 354–377, 2018.
- K. O’shea and R. Nash, “An introduction to convolutional neural networks,” arXiv preprint arXiv:1511.08458, 2015.
- Z. Li, F. Liu, W. Yang, S. Peng, and J. Zhou, “A survey of convolutional neural networks: analysis, applications, and prospects,” IEEE transactions on neural networks and learning systems, vol. 33, no. 12, pp. 6999–7019, 2021.
- J. Devlin, M. Chang, K. Lee, and K. Toutanova, “BERT: pre-training of deep bidirectional transformers for language understanding,” CoRR, vol. abs/1810.04805, 2018. [Online]. Available: http://arxiv.org/abs/1810.04805
- V. Sanh, L. Debut, J. Chaumond, and T. Wolf, “Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter,” CoRR, vol. abs/1910.01108, 2019. [Online]. Available: http://arxiv.org/abs/1910.01108
- A. Arnab, M. Dehghani, G. Heigold, C. Sun, M. Lucic, and C. Schmid, “Vivit: A video vision transformer,” CoRR, vol. abs/2103.15691, 2021. [Online]. Available: https://arxiv.org/abs/2103.15691
- A. Baevski, H. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” CoRR, vol. abs/2006.11477, 2020. [Online]. Available: https://arxiv.org/abs/2006.11477
- A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” CoRR, vol. abs/2010.11929, 2020. [Online]. Available: https://arxiv.org/abs/2010.11929
- J. Pons, O. Slizovskaia, R. Gong, E. Gómez, and X. Serra, “Timbre analysis of music audio signals with convolutional neural networks,” in 2017 25th European Signal Processing Conference (EUSIPCO). IEEE, 2017, pp. 2744–2748.
- Y. Zhang, B. Li, H. Fang, and Q. Meng, “Spectrogram transformers for audio classification,” in 2022 IEEE International Conference on Imaging Systems and Techniques (IST). IEEE, 2022, pp. 1–6.
- Y. Gong, Y.-A. Chung, and J. Glass, “Ast: Audio spectrogram transformer,” arXiv preprint arXiv:2104.01778, 2021.
- A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” 2022.
- L. Bertinetto, J. Valmadre, J. F. Henriques, A. Vedaldi, and P. H. Torr, “Fully-convolutional siamese networks for object tracking,” in Computer Vision–ECCV 2016 Workshops: Amsterdam, The Netherlands, October 8-10 and 15-16, 2016, Proceedings, Part II 14. Springer, 2016, pp. 850–865.
- S. Boelders, V. S. Nallanthighal, V. Menkovski, and A. Härmä, “Detection of mild dyspnea from pairs of speech recordings,” in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp. 4102–4106.
- Z. Lian, Y. Li, J. Tao, and J. Huang, “Speech emotion recognition via contrastive loss under siamese networks,” in Proceedings of the Joint Workshop of the 4th Workshop on Affective Social Multimedia Computing and First Multi-Modal Affective Computing of Large-Scale Multimedia Data, 2018, pp. 21–26.
- J. Wang, Y. Qin, Z. Peng, and T. Lee, “Child speech disorder detection with siamese recurrent network using speech attribute features.” in INTERSPEECH, vol. 2, 2019, pp. 3885–3889.
- D. Ortiz-Perez, P. Ruiz-Ponce, D. Tomás, and J. Garcia-Rodriguez, “Deep learning-based dementia prediction using multimodal data,” in 17th International Conference on Soft Computing Models in Industrial and Environmental Applications (SOCO 2022), P. García Bringas, H. Pérez García, F. J. Martinez-de Pison, J. R. Villar Flecha, A. Troncoso Lora, E. A. de la Cal, Á. Herrero, F. Martínez Álvarez, G. Psaila, H. Quintián, and E. S. Corchado Rodriguez, Eds. Cham: Springer Nature Switzerland, 2023, pp. 260–269.
- G. Huang, Z. Liu, and K. Q. Weinberger, “Densely connected convolutional networks,” CoRR, vol. abs/1608.06993, 2016. [Online]. Available: http://arxiv.org/abs/1608.06993
- A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam, “Mobilenets: Efficient convolutional neural networks for mobile vision applications,” CoRR, vol. abs/1704.04861, 2017. [Online]. Available: http://arxiv.org/abs/1704.04861
- K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” CoRR, vol. abs/1512.03385, 2015. [Online]. Available: http://arxiv.org/abs/1512.03385
- F. N. Iandola, S. Han, M. W. Moskewicz, K. Ashraf, W. J. Dally, and K. Keutzer, “Squeezenet: Alexnet-level accuracy with 50x fewer parameters and¡ 0.5 mb model size,” arXiv preprint arXiv:1602.07360, 2016.
- K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” 2015.
- A. Sherstinsky, “Fundamentals of recurrent neural network (rnn) and long short-term memory (lstm) network,” Physica D: Nonlinear Phenomena, vol. 404, p. 132306, 2020. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0167278919305974
- Y. Li, C.-Y. Wu, H. Fan, K. Mangalam, B. Xiong, J. Malik, and C. Feichtenhofer, “Mvitv2: Improved multiscale vision transformers for classification and detection,” 2022.
- C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” 2014.
- S. Zagoruyko and N. Komodakis, “Wide residual networks,” 2017.
- A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in Advances in Neural Information Processing Systems, F. Pereira, C. Burges, L. Bottou, and K. Weinberger, Eds., vol. 25. Curran Associates, Inc., 2012. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2012/file/c399862d3b9d6b76c8436e924a68c45b-Paper.pdf
- M. Sandler, A. G. Howard, M. Zhu, A. Zhmoginov, and L. Chen, “Inverted residuals and linear bottlenecks: Mobile networks for classification, detection and segmentation,” CoRR, vol. abs/1801.04381, 2018. [Online]. Available: http://arxiv.org/abs/1801.04381
- M. Tan, B. Chen, R. Pang, V. Vasudevan, and Q. V. Le, “Mnasnet: Platform-aware neural architecture search for mobile,” CoRR, vol. abs/1807.11626, 2018. [Online]. Available: http://arxiv.org/abs/1807.11626
- M. Tan and Q. V. Le, “Efficientnet: Rethinking model scaling for convolutional neural networks,” CoRR, vol. abs/1905.11946, 2019. [Online]. Available: http://arxiv.org/abs/1905.11946
- Y. Chuang, C. Liu, and H. Lee, “Speechbert: Cross-modal pre-trained language model for end-to-end spoken question answering,” CoRR, vol. abs/1910.11559, 2019. [Online]. Available: http://arxiv.org/abs/1910.11559
- S. Gupta, J. Jaafar, W. W. Ahmad, and A. Bansal, “Feature extraction using mfcc,” Signal & Image Processing: An International Journal, vol. 4, no. 4, pp. 101–108, 2013.
- L. Muda, M. Begam, and I. Elamvazuthi, “Voice recognition algorithms using mel frequency cepstral coefficient (mfcc) and dynamic time warping (dtw) techniques,” arXiv preprint arXiv:1003.4083, 2010.
- V. Tiwari, “Mfcc and its applications in speaker recognition,” International journal on emerging technologies, vol. 1, no. 1, pp. 19–22, 2010.
- M. Deng, T. Meng, J. Cao, S. Wang, J. Zhang, and H. Fan, “Heart sound classification based on improved mfcc features and convolutional recurrent neural networks,” Neural Networks, vol. 130, pp. 22–32, 2020.
- E. Rejaibi, A. Komaty, F. Meriaudeau, S. Agrebi, and A. Othmani, “Mfcc-based recurrent neural network for automatic clinical depression recognition and assessment from speech,” Biomedical Signal Processing and Control, vol. 71, p. 103107, 2022.
- F. Eyben, M. Wöllmer, and B. Schuller, “Opensmile: the munich versatile and fast open-source audio feature extractor,” in Proceedings of the 18th ACM International Conference on Multimedia, ser. MM ’10. New York, NY, USA: Association for Computing Machinery, 2010, p. 1459–1462. [Online]. Available: https://doi.org/10.1145/1873951.1874246
- F. Eyben, F. Weninger, F. Gross, and B. Schuller, “Recent developments in opensmile, the munich open-source multimedia feature extractor,” in Proceedings of the 21st ACM international conference on Multimedia, 2013, pp. 835–838.
- G. Degottex, J. Kane, T. Drugman, T. Raitio, and S. Scherer, “Covarep—a collaborative voice analysis repository for speech technologies,” in 2014 ieee international conference on acoustics, speech and signal processing (icassp). IEEE, 2014, pp. 960–964.
- D. Povey, A. Ghoshal, G. Boulianne, L. Burget, O. Glembek, N. Goel, M. Hannemann, P. Motlicek, Y. Qian, P. Schwarz et al., “The kaldi speech recognition toolkit,” in IEEE 2011 workshop on automatic speech recognition and understanding. IEEE Signal Processing Society, 2011.
- P. Nimitsurachat and P. Washington, “Audio-based emotion recognition using self-supervised learning on an engineered feature space,” AI, vol. 5, no. 1, pp. 195–207, 2024.
- A. B. Zadeh, P. P. Liang, S. Poria, E. Cambria, and L.-P. Morency, “Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph,” in Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2018, pp. 2236–2246.
- K. Qian, Z. Zhang, F. Ringeval, and B. Schuller, “Bird sounds classification by large scale acoustic features and extreme learning machine,” in 2015 IEEE Global Conference on Signal and Information Processing (GlobalSIP). IEEE, 2015, pp. 1317–1321.
- N. Narendra and P. Alku, “Dysarthric speech classification using glottal features computed from non-words, words and sentences.” in Interspeech, 2018, pp. 3403–3407.
- B. Schuller, S. Steidl, A. Batliner, F. Burkhardt, L. Devillers, C. Müller, and S. Narayanan, “The interspeech 2010 paralinguistic challenge,” in Proc. INTERSPEECH 2010, Makuhari, Japan, 2010, pp. 2794–2797.
- F. Eyben, K. R. Scherer, B. W. Schuller, J. Sundberg, E. André, C. Busso, L. Y. Devillers, J. Epps, P. Laukka, S. S. Narayanan, and K. P. Truong, “The geneva minimalistic acoustic parameter set (gemaps) for voice research and affective computing,” IEEE Transactions on Affective Computing, vol. 7, no. 2, pp. 190–202, 2016.
- B. Schuller, S. Steidl, A. Batliner, J. Hirschberg, J. K. Burgoon, A. Baird, A. Elkins, Y. Zhang, E. Coutinho, and K. Evanini, “The INTERSPEECH 2016 Computational Paralinguistics Challenge: Deception, Sincerity & Native Language,” in Proc. Interspeech 2016, 2016, pp. 2001–2005.
- S. Hershey, S. Chaudhuri, D. P. W. Ellis, J. F. Gemmeke, A. Jansen, R. C. Moore, M. Plakal, D. Platt, R. A. Saurous, B. Seybold, M. Slaney, R. J. Weiss, and K. W. Wilson, “CNN architectures for large-scale audio classification,” CoRR, vol. abs/1609.09430, 2016. [Online]. Available: http://arxiv.org/abs/1609.09430
- J. F. Gemmeke, D. P. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, “Audio set: An ontology and human-labeled dataset for audio events,” in 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2017, pp. 776–780.
- N. Dehak, P. J. Kenny, R. Dehak, P. Dumouchel, and P. Ouellet, “Front-end factor analysis for speaker verification,” IEEE Transactions on Audio, Speech, and Language Processing, vol. 19, no. 4, pp. 788–798, 2011.
- Y. Hauptman, R. Aloni-Lavi, I. Lapidot, T. Gurevich, Y. Manor, S. Naor, N. Diamant, and I. Opher, “Identifying distinctive acoustic and spectral features in parkinson’s disease.” in Interspeech, 2019, pp. 2498–2502.
- I. Laaridh, W. B. Kheder, C. Fredouille, and C. Meunier, “Automatic prediction of speech evaluation metrics for dysarthric speech,” in Interspeech, 2017.
- D. Snyder, D. Garcia-Romero, D. Povey, and S. Khudanpur, “Deep neural network embeddings for text-independent speaker verification.” in Interspeech, vol. 2017, 2017, pp. 999–1003.
- D. Snyder, D. Garcia-Romero, A. McCree, G. Sell, D. Povey, and S. Khudanpur, “Spoken language recognition using x-vectors.” in Odyssey, vol. 2018, 2018, pp. 105–111.
- D. Snyder, D. Garcia-Romero, G. Sell, D. Povey, and S. Khudanpur, “X-vectors: Robust dnn embeddings for speaker recognition,” in 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2018, pp. 5329–5333.
- T. Lin, Y. Wang, X. Liu, and X. Qiu, “A survey of transformers,” AI open, vol. 3, pp. 111–132, 2022.
- K. Han, Y. Wang, H. Chen, X. Chen, J. Guo, Z. Liu, Y. Tang, A. Xiao, C. Xu, Y. Xu et al., “A survey on vision transformer,” IEEE transactions on pattern analysis and machine intelligence, vol. 45, no. 1, pp. 87–110, 2022.
- S. Khan, M. Naseer, M. Hayat, S. W. Zamir, F. S. Khan, and M. Shah, “Transformers in vision: A survey,” ACM computing surveys (CSUR), vol. 54, no. 10s, pp. 1–41, 2022.
- S. Schneider, A. Baevski, R. Collobert, and M. Auli, “wav2vec: Unsupervised pre-training for speech recognition,” CoRR, vol. abs/1904.05862, 2019. [Online]. Available: http://arxiv.org/abs/1904.05862
- Y. Zhuang, Y. Chen, and J. Zheng, “Music genre classification with transformer classifier,” in Proceedings of the 2020 4th international conference on digital signal processing, 2020, pp. 155–159.
- F. Andayani, L. B. Theng, M. T. Tsun, and C. Chua, “Hybrid lstm-transformer model for emotion recognition from speech audio files,” IEEE Access, vol. 10, pp. 36 018–36 027, 2022.
- N. Vaessen and D. A. Van Leeuwen, “Fine-tuning wav2vec2 for speaker recognition,” in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022, pp. 7967–7971.
- C. G. Lyketsos and H. B. Lee, “Diagnosis and treatment of depression in alzheimer’s disease: a practical update for the clinician,” Dementia and geriatric cognitive disorders, vol. 17, no. 1-2, pp. 55–64, 2003.
- R. C. Green, L. A. Cupples, A. Kurz, S. Auerbach, R. Go, D. Sadovnick, R. Duara, W. A. Kukull, H. Chui, T. Edeki et al., “Depression as a risk factor for alzheimer disease: the mirage study,” Archives of neurology, vol. 60, no. 5, pp. 753–759, 2003.
- R. L. Ownby, E. Crocco, A. Acevedo, V. John, and D. Loewenstein, “Depression and risk for alzheimer disease: systematic review, meta-analysis, and metaregression analysis,” Archives of general psychiatry, vol. 63, no. 5, pp. 530–538, 2006.
- J. Gratch, R. Artstein, G. M. Lucas, G. Stratou, S. Scherer, A. Nazarian, R. Wood, J. Boberg, D. DeVault, S. Marsella et al., “The distress analysis interview corpus of human and computer interviews.” in LREC. Reykjavik, 2014, pp. 3123–3128.
- S. Chen, C. Wang, Z. Chen, Y. Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiao, J. Wu, L. Zhou, S. Ren, Y. Qian, Y. Qian, J. Wu, M. Zeng, and F. Wei, “Wavlm: Large-scale self-supervised pre-training for full stack speech processing,” CoRR, vol. abs/2110.13900, 2021. [Online]. Available: https://arxiv.org/abs/2110.13900
- W. Hsu, B. Bolte, Y. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, “Hubert: Self-supervised speech representation learning by masked prediction of hidden units,” CoRR, vol. abs/2106.07447, 2021. [Online]. Available: https://arxiv.org/abs/2106.07447
- H. Kheddar, Y. Himeur, S. Al-Maadeed, A. Amira, and F. Bensaali, “Deep transfer learning for automatic speech recognition: Towards better generalization,” Knowledge-Based Systems, vol. 277, p. 110851, 2023. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0950705123006019
- N. Jamal, S. Shanta, F. Mahmud, and M. Sha’abani, “Automatic speech recognition (asr) based approach for speech therapy of aphasic patients: A review,” in AIP Conference Proceedings, vol. 1883. AIP Publishing, 2017.
- I. G. Torre, M. Romero, and A. Álvarez, “Improving aphasic speech recognition by using novel semi-supervised learning methods on aphasiabank for english and spanish,” Applied Sciences, vol. 11, no. 19, p. 8872, 2021.
- J. Weiner, M. Engelbart, and T. Schultz, “Manual and automatic transcriptions in dementia detection from speech.” in Interspeech, 2017, pp. 3117–3121.
- Y. Zhu, A. Obyat, X. Liang, J. A. Batsis, and R. M. Roth, “WavBERT: Exploiting Semantic and Non-Semantic Speech Using Wav2vec and BERT for Dementia Detection,” in Proc. Interspeech 2021, 2021, pp. 3790–3794.
- H. Zhang, H. Song, S. Li, M. Zhou, and D. Song, “A survey of controllable text generation using transformer-based pre-trained language models,” ACM Computing Surveys, vol. 56, no. 3, pp. 1–37, 2023.
- J. Li, T. Tang, W. X. Zhao, J.-Y. Nie, and J.-R. Wen, “Pre-trained language models for text generation: A survey,” ACM Computing Surveys, vol. 56, no. 9, pp. 1–39, 2024.
- A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever et al., “Language models are unsupervised multitask learners,” OpenAI blog, vol. 1, no. 8, p. 9, 2019.
- C. Li, D. Knopman, W. Xu, T. Cohen, and S. Pakhomov, “GPT-D: Inducing dementia-related linguistic anomalies by deliberate degradation of artificial neural language models,” in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), S. Muresan, P. Nakov, and A. Villavicencio, Eds. Dublin, Ireland: Association for Computational Linguistics, May 2022, pp. 1866–1877. [Online]. Available: https://aclanthology.org/2022.acl-long.131
- J. Yuan, Y. Bian, X. Cai, J. Huang, Z. Ye, and K. Church, “Disfluencies and fine-tuning pre-trained language models for detection of alzheimer’s disease.” in Interspeech, vol. 2020, 2020, pp. 2162–6.
- L. Ilias and D. Askounis, “Explainable identification of dementia from transcripts using transformer networks,” IEEE Journal of Biomedical and Health Informatics, vol. 26, no. 8, pp. 4153–4164, 2022.
- A. Balagopalan, B. Eyre, F. Rudzicz, and J. Novikova, “To bert or not to bert: Comparing speech and language-based approaches for alzheimer’s disease detection,” in Interspeech, 2020. [Online]. Available: https://api.semanticscholar.org/CorpusID:220961484
- A. S. Nambiar, K. Likhita, K. S. Pujya, D. Gupta, S. Vekkot, and S. Lalitha, “Comparative study of deep classifiers for early dementia detection using speech transcripts,” in 2022 IEEE 19th India Council International Conference (INDICON). IEEE, 2022, pp. 1–6.
- N. Liu and L. Wang, “An approach for assisting diagnosis of alzheimer’s disease based on natural language processing,” Frontiers in Aging Neuroscience, vol. 15, 2023.
- A. Roshanzamir, H. Aghajan, and M. Soleymani Baghshah, “Transformer-based deep neural network language models for alzheimer’s disease risk assessment from targeted speech,” BMC Medical Informatics and Decision Making, vol. 21, pp. 1–14, 2021.
- M. Bouazizi, C. Zheng, S. Yang, and T. Ohtsuki, “Dementia detection from speech: What if language models are not the answer?” Information, vol. 15, no. 1, 2024. [Online]. Available: https://www.mdpi.com/2078-2489/15/1/2
- C. Zheng, M. Bouazizi, and T. Ohtsuki, “An evaluation on information composition in dementia detection based on speech,” IEEE Access, vol. 10, pp. 92 294–92 306, 2022.
- Y. Pan, B. Mirheidari, M. Reuber, A. Venneri, D. Blackburn, and H. Christensen, “Automatic hierarchical attention neural network for detecting ad,” in Proceedings of Interspeech 2019. International Speech Communication Association (ISCA), 2019, pp. 4105–4109.
- D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learning internal representations by error propagation, parallel distributed processing, explorations in the microstructure of cognition, ed. de rumelhart and j. mcclelland. vol. 1. 1986,” Biometrika, vol. 71, no. 599-607, p. 6, 1986.
- S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural computation, vol. 9, no. 8, pp. 1735–1780, 1997.
- G. Van Houdt, C. Mosquera, and G. Nápoles, “A review on the long short-term memory model,” Artificial Intelligence Review, vol. 53, no. 8, pp. 5929–5955, 2020.
- K. Cho, B. Van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio, “Learning phrase representations using rnn encoder-decoder for statistical machine translation,” arXiv preprint arXiv:1406.1078, 2014.
- J. Chung, C. Gulcehre, K. Cho, and Y. Bengio, “Empirical evaluation of gated recurrent neural networks on sequence modeling,” arXiv preprint arXiv:1412.3555, 2014.
- G. Liu and J. Guo, “Bidirectional lstm with attention mechanism and convolutional layer for text classification,” Neurocomputing, vol. 337, pp. 325–338, 2019.
- C. Zhou, C. Sun, Z. Liu, and F. Lau, “A c-lstm neural network for text classification,” arXiv preprint arXiv:1511.08630, 2015.
- M. E. Basiri, S. Nemati, M. Abdar, E. Cambria, and U. R. Acharya, “Abcdm: An attention-based bidirectional cnn-rnn deep model for sentiment analysis,” Future Generation Computer Systems, vol. 115, pp. 279–294, 2021.
- Y. Kim, “Convolutional neural networks for sentence classification,” CoRR, vol. abs/1408.5882, 2014. [Online]. Available: http://arxiv.org/abs/1408.5882
- Y. Zhang and B. Wallace, “A sensitivity analysis of (and practitioners’ guide to) convolutional neural networks for sentence classification,” arXiv preprint arXiv:1510.03820, 2015.
- Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “Roberta: A robustly optimized bert pretraining approach,” 2019. [Online]. Available: https://arxiv.org/abs/1907.11692
- J. Fritsch, S. Wankerl, and E. Nöth, “Automatic diagnosis of alzheimer’s disease using neural network language models,” in ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2019, pp. 5841–5845.
- T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei, “Language models are few-shot learners,” CoRR, vol. abs/2005.14165, 2020. [Online]. Available: https://arxiv.org/abs/2005.14165
- J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al., “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774, 2023.
- Y. Liu and M. Lapata, “Text summarization with pretrained encoders,” arXiv preprint arXiv:1908.08345, 2019.
- M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer, “BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension,” CoRR, vol. abs/1910.13461, 2019. [Online]. Available: http://arxiv.org/abs/1910.13461
- C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” CoRR, vol. abs/1910.10683, 2019. [Online]. Available: http://arxiv.org/abs/1910.10683
- L. Xue, N. Constant, A. Roberts, M. Kale, R. Al-Rfou, A. Siddhant, A. Barua, and C. Raffel, “mt5: A massively multilingual pre-trained text-to-text transformer,” CoRR, vol. abs/2010.11934, 2020. [Online]. Available: https://arxiv.org/abs/2010.11934
- Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut, “ALBERT: A lite BERT for self-supervised learning of language representations,” CoRR, vol. abs/1909.11942, 2019. [Online]. Available: http://arxiv.org/abs/1909.11942
- Z. Yang, Z. Dai, Y. Yang, J. G. Carbonell, R. Salakhutdinov, and Q. V. Le, “Xlnet: Generalized autoregressive pretraining for language understanding,” CoRR, vol. abs/1906.08237, 2019. [Online]. Available: http://arxiv.org/abs/1906.08237
- I. Beltagy, M. E. Peters, and A. Cohan, “Longformer: The long-document transformer,” CoRR, vol. abs/2004.05150, 2020. [Online]. Available: https://arxiv.org/abs/2004.05150
- J. Lee, W. Yoon, S. Kim, D. Kim, S. Kim, C. H. So, and J. Kang, “Biobert: a pre-trained biomedical language representation model for biomedical text mining,” CoRR, vol. abs/1901.08746, 2019. [Online]. Available: http://arxiv.org/abs/1901.08746
- E. Alsentzer, J. R. Murphy, W. Boag, W. Weng, D. Jin, T. Naumann, and M. B. A. McDermott, “Publicly available clinical BERT embeddings,” CoRR, vol. abs/1904.03323, 2019. [Online]. Available: http://arxiv.org/abs/1904.03323
- Z. Jiang, W. Yu, D. Zhou, Y. Chen, J. Feng, and S. Yan, “Convbert: Improving BERT with span-based dynamic convolution,” CoRR, vol. abs/2008.02496, 2020. [Online]. Available: https://arxiv.org/abs/2008.02496
- M. Yancheva, K. C. Fraser, and F. Rudzicz, “Using linguistic features longitudinally to predict clinical scores for alzheimer’s disease and related dementias,” in SLPAT@Interspeech, 2015. [Online]. Available: https://api.semanticscholar.org/CorpusID:891184
- Z. Zhu, J. Novikova, and F. Rudzicz, “Detecting cognitive impairments by agreeing on interpretations of linguistic features,” ArXiv, vol. abs/1808.06570, 2018. [Online]. Available: https://api.semanticscholar.org/CorpusID:51938927
- N. B. Mota, N. Vasconcelos, N. Lemos, A. C. de Souza Pieretti, O. Kinouchi, G. A. Cecchi, M. Copelli, and S. Ribeiro, “Speech graphs provide a quantitative measure of thought disorder in psychosis,” PLoS ONE, vol. 7, 2012. [Online]. Available: https://api.semanticscholar.org/CorpusID:9506186
- A. B. Warriner, V. Kuperman, and M. Brysbaert, “Norms of valence, arousal, and dominance for 13,915 english lemmas,” Behavior Research Methods, vol. 45, pp. 1191 – 1207, 2013. [Online]. Available: https://api.semanticscholar.org/CorpusID:16918336
- H. Ai and X. Lu, “A web-based system for automatic measurement of lexical complexity,” in 27th Annual Symposium of the Computer-Assisted Language Consortium (CALICO-10). Amherst, MA. June, 2010, pp. 8–12.
- B. Croisile, B. Ska, M.-J. Brabant, A. Duchêne, Y. Lepage, G. Aimard, and M. Trillet, “Comparative study of oral and written picture description in patients with alzheimer’s disease,” Brain and Language, vol. 53, pp. 1–19, 1996. [Online]. Available: https://api.semanticscholar.org/CorpusID:36544389
- J. Pennington, R. Socher, and C. Manning, “GloVe: Global vectors for word representation,” in Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), A. Moschitti, B. Pang, and W. Daelemans, Eds. Doha, Qatar: Association for Computational Linguistics, Oct. 2014, pp. 1532–1543. [Online]. Available: https://aclanthology.org/D14-1162
- Z. Dai, Z. Yang, Y. Yang, J. G. Carbonell, Q. V. Le, and R. Salakhutdinov, “Transformer-xl: Attentive language models beyond a fixed-length context,” CoRR, vol. abs/1901.02860, 2019. [Online]. Available: http://arxiv.org/abs/1901.02860
- A. Radford and K. Narasimhan, “Improving language understanding by generative pre-training,” 2018. [Online]. Available: https://api.semanticscholar.org/CorpusID:49313245
- K. C. Fraser, J. A. Meltzer, and F. Rudzicz, “Linguistic features identify alzheimer’s disease in narrative speech,” Journal of Alzheimer’s Disease, vol. 49, no. 2, pp. 407–422, 2016.
- F. Di Palo and N. Parde, “Enriching neural models with targeted features for dementia detection,” in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop, F. Alva-Manchego, E. Choi, and D. Khashabi, Eds. Florence, Italy: Association for Computational Linguistics, Jul. 2019, pp. 302–308. [Online]. Available: https://aclanthology.org/P19-2042
- Y. Sun, S. Wang, Y. Li, S. Feng, H. Tian, H. Wu, and H. Wang, “ERNIE 2.0: A continual pre-training framework for language understanding,” CoRR, vol. abs/1907.12412, 2019. [Online]. Available: http://arxiv.org/abs/1907.12412
- P. Garrard, L. M. Maloney, J. R. Hodges, and K. Patterson, “The effects of very early alzheimer’s disease on the characteristics of writing by a renowned author,” Brain, vol. 128, no. 2, pp. 250–260, 2005.
- V. Berisha, S. Wang, A. LaCross, and J. Liss, “Tracking discourse complexity preceding alzheimer’s disease diagnosis: A case study comparing the press conferences of presidents ronald reagan and george herbert walker bush,” Journal of Alzheimer’s Disease, vol. 45, no. 3, pp. 959–963, 2015.
- R. S. Bucks, S. Singh, J. M. Cuerden, and G. K. Wilcock, “Analysis of spontaneous, conversational speech in dementia of alzheimer type: Evaluation of an objective technique for analysing lexical performance,” Aphasiology, vol. 14, no. 1, pp. 71–91, 2000.
- T. Mikolov, K. Chen, G. Corrado, and J. Dean, “Efficient estimation of word representations in vector space,” arXiv preprint arXiv:1301.3781, 2013.
- J. Cañete, G. Chaperon, R. Fuentes, J.-H. Ho, H. Kang, and J. Pérez, “Spanish pre-trained bert model and evaluation data,” 2023. [Online]. Available: https://arxiv.org/abs/2308.02976
- A. J. Thirunavukarasu, D. S. J. Ting, K. Elangovan, L. Gutierrez, T. F. Tan, and D. S. W. Ting, “Large language models in medicine,” Nature medicine, vol. 29, no. 8, pp. 1930–1940, 2023.
- W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y. Hou, Y. Min, B. Zhang, J. Zhang, Z. Dong et al., “A survey of large language models,” arXiv preprint arXiv:2303.18223, 2023.
- Y. Chang, X. Wang, J. Wang, Y. Wu, L. Yang, K. Zhu, H. Chen, X. Yi, C. Wang, Y. Wang et al., “A survey on evaluation of large language models,” ACM Transactions on Intelligent Systems and Technology, vol. 15, no. 3, pp. 1–45, 2024.
- OpenAI, “Chatgpt: Openai’s gpt-4 language model,” 2024. [Online]. Available: https://www.openai.com/chatgpt
- Q. V. Le and T. Mikolov, “Distributed representations of sentences and documents,” CoRR, vol. abs/1405.4053, 2014. [Online]. Available: http://arxiv.org/abs/1405.4053
- N. Reimers and I. Gurevych, “Sentence-bert: Sentence embeddings using siamese bert-networks,” arXiv preprint arXiv:1908.10084, 2019.
- A. Joulin, E. Grave, P. Bojanowski, and T. Mikolov, “Bag of tricks for efficient text classification,” arXiv preprint arXiv:1607.01759, 2016.
- A. Akbik, D. Blythe, and R. Vollgraf, “Contextual string embeddings for sequence labeling,” in Proceedings of the 27th international conference on computational linguistics, 2018, pp. 1638–1649.
- N. D. Cilia, T. D’Alessandro, C. De Stefano, F. Fontanella, and M. Molinara, “From online handwriting to synthetic images for alzheimer’s disease detection using a deep transfer learning approach,” IEEE Journal of Biomedical and Health Informatics, vol. 25, no. 12, pp. 4243–4254, 2021.
- J. Sun, H. H. Dodge, and M. H. Mahoor, “Mc-vivit: Multi-branch classifier-vivit to detect mild cognitive impairment in older adults using facial videos,” Expert Systems with Applications, vol. 238, p. 121929, 2024. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0957417423024314
- X. Liang, A. Angelopoulou, E. Kapetanios, B. Woll, R. Al Batat, and T. Woolfe, “A multi-modal machine learning approach and toolkit to automate recognition of early stages of dementia among british sign language users,” in Computer Vision – ECCV 2020 Workshops, A. Bartoli and A. Fusiello, Eds. Cham: Springer International Publishing, 2020, pp. 278–293.
- K. Hu, Z. Wang, W. Wang, K. A. E. Martens, L. Wang, T. Tan, S. J. Lewis, and D. D. Feng, “Graph sequence recurrent neural network for vision-based freezing of gait detection,” IEEE Transactions on Image Processing, vol. 29, pp. 1890–1901, 2019.
- A. Das, X. Niu, A. Dantcheva, S. L. Happy, H. Han, R. Zeghari, P. Robert, S. Shan, F. Bremond, and X. Chen, “A spatio-temporal approach for apathy classification,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 32, no. 5, pp. 2561–2573, 2022.
- G. Vessio, “Dynamic handwriting analysis for neurodegenerative disease assessment: a literary review,” Applied Sciences, vol. 9, no. 21, p. 4666, 2019.
- E. Onofri, M. Mercuri, M. Salesi, S. Ferrara, G. M. Troili, C. Simeone, M. R. Ricciardi, S. Ricci, and T. Archer, “Dysgraphia in relation to cognitive performance in patients with alzheimer’s disease,” Journal of Intellectual Disability-Diagnosis and Treatment, vol. 1, no. 2, pp. 113–124, 2013.
- S. Müller, O. Preische, P. Heymann, U. Elbing, and C. Laske, “Diagnostic value of a tablet-based drawing task for discrimination of patients in the early course of alzheimer’s disease from healthy individuals,” Journal of Alzheimer’s Disease, vol. 55, no. 4, pp. 1463–1469, 2017.
- C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the inception architecture for computer vision,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 2818–2826.
- U. Nam, K. Lee, H. Ko, J.-Y. Lee, and E. C. Lee, “Analyzing facial and eye movements to screen for alzheimer’s disease,” Sensors, vol. 20, no. 18, p. 5349, 2020.
- M. C. N. Dourado, B. Torres Mendonça de Melo Fádel, J. P. Simões Neto, G. Alves, and C. Alves, “Facial expression recognition patterns in mild and moderate alzheimer’s disease,” Journal of Alzheimer’s Disease, vol. 69, no. 2, pp. 539–549, 2019.
- B. Jin, Y. Qu, L. Zhang, and Z. Gao, “Diagnosing parkinson disease through facial expression recognition: video analysis,” Journal of medical Internet research, vol. 22, no. 7, p. e18697, 2020.
- H. Tanaka, H. Adachi, H. Kazui, M. Ikeda, T. Kudo, and S. Nakamura, “Detecting dementia from face in human-agent interaction,” in Adjunct of the 2019 international conference on multimodal interaction, 2019, pp. 1–6.
- Z. Liu, J. Ning, Y. Cao, Y. Wei, Z. Zhang, S. Lin, and H. Hu, “Video swin transformer,” CoRR, vol. abs/2106.13230, 2021. [Online]. Available: https://arxiv.org/abs/2106.13230
- M. A. Hely, W. G. Reid, M. A. Adena, G. M. Halliday, and J. G. Morris, “The sydney multicenter study of parkinson’s disease: the inevitability of dementia at 20 years,” Movement disorders, vol. 23, no. 6, pp. 837–844, 2008.
- M. Macht, Y. Kaussner, J. C. Möller, K. Stiasny-Kolster, K. M. Eggert, H.-P. Krüger, and H. Ellgring, “Predictors of freezing in parkinson’s disease: a survey of 6,620 patients,” Movement disorders, vol. 22, no. 7, pp. 953–956, 2007.
- B. R. Bloem, J. M. Hausdorff, J. E. Visser, and N. Giladi, “Falls and freezing of gait in parkinson’s disease: a review of two interconnected, episodic phenomena,” Movement disorders: official journal of the Movement Disorder Society, vol. 19, no. 8, pp. 871–884, 2004.
- S. J. Lewis and R. A. Barker, “A pathophysiological model of freezing of gait in parkinson’s disease,” Parkinsonism & related disorders, vol. 15, no. 5, pp. 333–338, 2009.
- T. Khan, J. Westin, and M. Dougherty, “Motion cue analysis for parkinsonian gait recognition,” The open biomedical engineering journal, vol. 7, p. 1, 2013.
- M. Nieto-Hidalgo, F. J. Ferrández-Pastor, R. J. Valdivieso-Sarabia, J. Mora-Pascual, and J. M. García-Chamizo, “A vision based proposal for classification of normal and abnormal gait using rgb camera,” Journal of biomedical informatics, vol. 63, pp. 82–89, 2016.
- R. J. Molitor, P. C. Ko, and B. A. Ally, “Eye movements in alzheimer’s disease,” Journal of Alzheimer’s disease, vol. 44, no. 1, pp. 1–12, 2015.
- A. Oyama, S. Takeda, Y. Ito, T. Nakajima, Y. Takami, Y. Takeya, K. Yamamoto, K. Sugimoto, H. Shimizu, M. Shimamura et al., “Novel method for rapid assessment of cognitive impairment using high-performance eye-tracking technology,” Scientific reports, vol. 9, no. 1, p. 12932, 2019.
- D. Lagun, C. Manzanares, S. M. Zola, E. A. Buffalo, and E. Agichtein, “Detecting cognitive impairment by eye movement analysis using automatic classification algorithms,” Journal of neuroscience methods, vol. 201, no. 1, pp. 196–203, 2011.
- K. Mengoudi, D. Ravi, K. X. Yong, S. Primativo, I. M. Pavisic, E. Brotherhood, K. Lu, J. M. Schott, S. J. Crutch, and D. C. Alexander, “Augmenting dementia cognitive assessment with instruction-less eye-tracking tests,” IEEE journal of biomedical and health informatics, vol. 24, no. 11, pp. 3066–3075, 2020.
- A. A. Lazarus et al., “Multimodal behavior therapy: I.” 1976.
- P. Xu, X. Zhu, and D. A. Clifton, “Multimodal learning with transformers: A survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 10, pp. 12 113–12 132, 2023.
- W. Guo, J. Wang, and S. Wang, “Deep multimodal representation learning: A survey,” Ieee Access, vol. 7, pp. 63 373–63 394, 2019.
- T. Baltrušaitis, C. Ahuja, and L.-P. Morency, “Multimodal machine learning: A survey and taxonomy,” IEEE transactions on pattern analysis and machine intelligence, vol. 41, no. 2, pp. 423–443, 2018.
- D. Altinok, “Explainable multimodal fusion for dementia detection from text and speech,” in International Conference on Text, Speech, and Dialogue. Springer, 2024, pp. 236–251.
- L. Ilias and D. Askounis, “Multimodal deep learning models for detecting dementia from speech and transcripts,” Frontiers in Aging Neuroscience, vol. 14, 2022. [Online]. Available: https://www.frontiersin.org/articles/10.3389/fnagi.2022.830943
- L. Ilias, D. Askounis, and J. Psarras, “A multimodal approach for dementia detection from spontaneous speech with tensor fusion layer,” in 2022 IEEE-EMBS International Conference on Biomedical and Health Informatics (BHI), 2022, pp. 1–5.
- A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning transferable visual models from natural language supervision,” CoRR, vol. abs/2103.00020, 2021. [Online]. Available: https://arxiv.org/abs/2103.00020
- H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample, “Llama: Open and efficient foundation language models,” 2023. [Online]. Available: https://arxiv.org/abs/2302.13971
- J. Li, D. Li, S. Savarese, and S. Hoi, “Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,” 2023. [Online]. Available: https://arxiv.org/abs/2301.12597
- C. Sun, A. Myers, C. Vondrick, K. Murphy, and C. Schmid, “Videobert: A joint model for video and language representation learning,” 2019. [Online]. Available: https://arxiv.org/abs/1904.01766
- L. H. Li, M. Yatskar, D. Yin, C.-J. Hsieh, and K.-W. Chang, “Visualbert: A simple and performant baseline for vision and language,” arXiv preprint arXiv:1908.03557, 2019.
- J. Lu, D. Batra, D. Parikh, and S. Lee, “Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks,” CoRR, vol. abs/1908.02265, 2019. [Online]. Available: http://arxiv.org/abs/1908.02265
- R. Zheng, J. Chen, M. Ma, and L. Huang, “Fused acoustic and text encoding for multimodal bilingual pretraining and speech translation,” in International Conference on Machine Learning. PMLR, 2021, pp. 12 736–12 746.
- B. Shi, W.-N. Hsu, K. Lakhotia, and A. Mohamed, “Learning audio-visual speech representation by masked multimodal cluster prediction,” arXiv preprint arXiv:2201.02184, 2022.
- J. Lin, A. Yang, Y. Zhang, J. Liu, J. Zhou, and H. Yang, “Interbert: Vision-and-language interaction for multi-modal pretraining,” 2021. [Online]. Available: https://arxiv.org/abs/2003.13198
- Y.-H. H. Tsai, S. Bai, P. P. Liang, J. Z. Kolter, L.-P. Morency, and R. Salakhutdinov, “Multimodal transformer for unaligned multimodal language sequences,” 2019. [Online]. Available: https://arxiv.org/abs/1906.00295
- X. Xu, T. Wang, Y. Yang, L. Zuo, F. Shen, and H. T. Shen, “Cross-modal attention with semantic consistence for image–text matching,” IEEE transactions on neural networks and learning systems, vol. 31, no. 12, pp. 5412–5425, 2020.
- A. Khare, S. Parthasarathy, and S. Sundaram, “Self-supervised learning with cross-modal transformers for emotion recognition,” in 2021 IEEE Spoken Language Technology Workshop (SLT), 2021, pp. 381–388.
- V. Murahari, D. Batra, D. Parikh, and A. Das, “Large-scale pretraining for visual dialog: A simple state-of-the-art baseline,” 2020. [Online]. Available: https://arxiv.org/abs/1912.02379
- J. Arevalo, T. Solorio, M. Montes-y Gomez, and F. A. González, “Gated multimodal networks,” Neural Computing and Applications, vol. 32, pp. 10 209–10 228, 2020.
- Y. Zhang, D. Sidibé, O. Morel, and F. Mériaudeau, “Deep multimodal fusion for semantic image segmentation: A survey,” Image and Vision Computing, vol. 105, p. 104042, 2021. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0262885620301748
- S. R. Stahlschmidt, B. Ulfenborg, and J. Synnergren, “Multimodal deep learning for biomedical data fusion: a review,” Briefings in Bioinformatics, vol. 23, no. 2, p. bbab569, 01 2022. [Online]. Available: https://doi.org/10.1093/bib/bbab569
- M. Wang, J. Xing, and Y. Liu, “Actionclip: A new paradigm for video action recognition,” 2021. [Online]. Available: https://arxiv.org/abs/2109.08472
- W. Wu, X. Wang, H. Luo, J. Wang, Y. Yang, and W. Ouyang, “Bidirectional cross-modal knowledge exploration for video recognition with pre-trained vision-language models,” 2023. [Online]. Available: https://arxiv.org/abs/2301.00182
- A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning transferable visual models from natural language supervision,” 2021. [Online]. Available: https://arxiv.org/abs/2103.00020
- J. Górriz et al., “Computational approaches to explainable artificial intelligence: Advances in theory, applications and trends,” Information Fusion, vol. 100, p. 101945, 2023. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S1566253523002610
- A. Holzinger, G. Langs, H. Denk, K. Zatloukal, and H. Müller, “Causability and explainability of artificial intelligence in medicine,” Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, vol. 9, no. 4, p. e1312, 2019.
- G. Vilone and L. Longo, “Notions of explainability and evaluation approaches for explainable artificial intelligence,” Information Fusion, vol. 76, pp. 89–106, 2021.
- F. Tao, H. Zhang, A. Liu, and A. Y. Nee, “Digital twin in industry: State-of-the-art,” IEEE Transactions on industrial informatics, vol. 15, no. 4, pp. 2405–2415, 2018.
- K. Bruynseels, F. Santoni de Sio, and J. Van den Hoven, “Digital twins in health care: ethical implications of an emerging engineering paradigm,” Frontiers in genetics, vol. 9, p. 31, 2018.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.