BrainMass: Advancing Brain Network Analysis for Diagnosis with Large-scale Self-Supervised Learning (2403.01433v1)
Abstract: Foundation models pretrained on large-scale datasets via self-supervised learning demonstrate exceptional versatility across various tasks. Due to the heterogeneity and hard-to-collect medical data, this approach is especially beneficial for medical image analysis and neuroscience research, as it streamlines broad downstream tasks without the need for numerous costly annotations. However, there has been limited investigation into brain network foundation models, limiting their adaptability and generalizability for broad neuroscience studies. In this study, we aim to bridge this gap. In particular, (1) we curated a comprehensive dataset by collating images from 30 datasets, which comprises 70,781 samples of 46,686 participants. Moreover, we introduce pseudo-functional connectivity (pFC) to further generates millions of augmented brain networks by randomly dropping certain timepoints of the BOLD signal. (2) We propose the BrainMass framework for brain network self-supervised learning via mask modeling and feature alignment. BrainMass employs Mask-ROI Modeling (MRM) to bolster intra-network dependencies and regional specificity. Furthermore, Latent Representation Alignment (LRA) module is utilized to regularize augmented brain networks of the same participant with similar topological properties to yield similar latent representations by aligning their latent embeddings. Extensive experiments on eight internal tasks and seven external brain disorder diagnosis tasks show BrainMass's superior performance, highlighting its significant generalizability and adaptability. Nonetheless, BrainMass demonstrates powerful few/zero-shot learning abilities and exhibits meaningful interpretation to various diseases, showcasing its potential use for clinical applications.
- N. K. Logothetis, “What we can do and what we cannot do with fmri,” Nature, vol. 453, no. 7197, pp. 869–878, 2008.
- D. J. Heeger and D. Ress, “What does fmri tell us about neuronal activity?” Nature reviews neuroscience, vol. 3, no. 2, pp. 142–151, 2002.
- N. K. Logothetis, J. Pauls, M. Augath, T. Trinath, and A. Oeltermann, “Neurophysiological investigation of the basis of the fmri signal,” nature, vol. 412, no. 6843, pp. 150–157, 2001.
- A. A. Fingelkurts, A. A. Fingelkurts, and S. Kähkönen, “Functional connectivity in the brain—is it an elusive concept?” Neuroscience & Biobehavioral Reviews, vol. 28, no. 8, pp. 827–836, 2005.
- Y. Yang, C. Ye, J. Sun, L. Liang, H. Lv, L. Gao, J. Fang, T. Ma, and T. Wu, “Alteration of brain structural connectivity in progression of parkinson’s disease: a connectome-wide network analysis,” NeuroImage: Clinical, vol. 31, p. 102715, 2021.
- A. M. Bastos and J.-M. Schoffelen, “A tutorial review of functional connectivity analysis methods and their interpretational pitfalls,” Frontiers in systems neuroscience, vol. 9, p. 175, 2016.
- J. Kawahara, C. J. Brown, S. P. Miller, B. G. Booth, V. Chau, R. E. Grunau, J. G. Zwicker, and G. Hamarneh, “Brainnetcnn: Convolutional neural networks for brain networks; towards predicting neurodevelopment,” NeuroImage, vol. 146, pp. 1038–1049, 2017.
- H. Huang, X. Hu, Y. Zhao, M. Makkie, Q. Dong, S. Zhao, L. Guo, and T. Liu, “Modeling task fmri data via deep convolutional autoencoder,” IEEE transactions on medical imaging, vol. 37, no. 7, pp. 1551–1561, 2017.
- Z.-A. Huang, Z. Zhu, C. H. Yau, and K. C. Tan, “Identifying autism spectrum disorder from resting-state fmri using deep belief network,” IEEE Transactions on neural networks and learning systems, vol. 32, no. 7, pp. 2847–2861, 2020.
- K. Zhao, B. Duka, H. Xie, D. J. Oathes, V. Calhoun, and Y. Zhang, “A dynamic graph convolutional neural network framework reveals new insights into connectome dysfunctions in adhd,” Neuroimage, vol. 246, p. 118774, 2022.
- X. Li, Y. Zhou, N. Dvornek, M. Zhang, S. Gao, J. Zhuang, D. Scheinost, L. H. Staib, P. Ventola, and J. S. Duncan, “Braingnn: Interpretable brain graph neural network for fmri analysis,” Medical Image Analysis, vol. 74, p. 102233, 2021.
- Y. Yang, C. Ye, X. Guo, T. Wu, Y. Xiang, and T. Ma, “Mapping multi-modal brain connectome for brain disorder diagnosis via cross-modal mutual learning,” IEEE Transactions on Medical Imaging, 2023.
- X. Kan, W. Dai, H. Cui, Z. Zhang, Y. Guo, and C. Yang, “Brain network transformer,” Advances in Neural Information Processing Systems, vol. 35, pp. 25 586–25 599, 2022.
- Q. Zhu, H. Wang, B. Xu, Z. Zhang, W. Shao, and D. Zhang, “Multimodal triplet attention network for brain disease diagnosis,” IEEE Transactions on Medical Imaging, vol. 41, no. 12, pp. 3884–3894, 2022.
- G. Wen, P. Cao, L. Liu, J. Yang, X. Zhang, F. Wang, and O. R. Zaiane, “Graph self-supervised learning with application to brain networks analysis,” IEEE Journal of Biomedical and Health Informatics, 2023.
- W. Jung, D.-W. Heo, E. Jeon, J. Lee, and H.-I. Suk, “Inter-regional high-level relation learning from functional connectivity via self-supervision,” in Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part II 24. Springer, 2021, pp. 284–293.
- A. Segal, L. Parkes, K. Aquino, S. M. Kia, T. Wolfers, B. Franke, M. Hoogman, C. F. Beckmann, L. T. Westlye, O. A. Andreassen et al., “Regional, circuit and network heterogeneity of brain abnormalities in psychiatric disorders,” Nature Neuroscience, vol. 26, no. 9, pp. 1613–1629, 2023.
- Y. Zhang, L. Zhan, W. Cai, P. Thompson, and H. Huang, “Integrating heterogeneous brain networks for predicting brain disease conditions,” in Medical Image Computing and Computer Assisted Intervention–MICCAI 2019: 22nd International Conference, Shenzhen, China, October 13–17, 2019, Proceedings, Part IV 22. Springer, 2019, pp. 214–222.
- K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick, “Momentum contrast for unsupervised visual representation learning,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 9729–9738.
- X. Chen, H. Fan, R. Girshick, and K. He, “Improved baselines with momentum contrastive learning,” arXiv preprint arXiv:2003.04297, 2020.
- J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805, 2018.
- A. Conneau and G. Lample, “Cross-lingual language model pretraining,” Advances in neural information processing systems, vol. 32, 2019.
- S. Zhang and D. Metaxas, “On the challenges and perspectives of foundation models for medical image analysis,” arXiv preprint arXiv:2306.05705, 2023.
- J. Hu, Y. Huang, N. Wang, and S. Dong, “Brainnpt: Pre-training of transformer networks for brain network classification,” arXiv preprint arXiv:2305.01666, 2023.
- B. Jie, M. Liu, C. Lian, F. Shi, and D. Shen, “Designing weighted correlation kernels in convolutional neural networks for functional connectivity based brain disease diagnosis,” Medical image analysis, vol. 63, p. 101709, 2020.
- S. I. Ktena, S. Parisot, E. Ferrante, M. Rajchl, M. Lee, B. Glocker, and D. Rueckert, “Metric learning with spectral graph convolutions on brain connectivity networks,” NeuroImage, vol. 169, pp. 431–442, 2018.
- N. S. Dsouza, M. B. Nebel, D. Crocetti, J. Robinson, S. Mostofsky, and A. Venkataraman, “M-gcn: A multimodal graph convolutional network to integrate functional and structural connectomics data to predict multidimensional phenotypic characterizations,” in Medical Imaging with Deep Learning. PMLR, 2021, pp. 119–130.
- S. Parisot, S. I. Ktena, E. Ferrante, M. Lee, R. Guerrero, B. Glocker, and D. Rueckert, “Disease prediction using graph convolutional networks: application to autism spectrum disorder and alzheimer’s disease,” Medical image analysis, vol. 48, pp. 117–130, 2018.
- X. Song, F. Zhou, A. F. Frangi, J. Cao, X. Xiao, Y. Lei, T. Wang, and B. Lei, “Graph convolution network with similarity awareness and adaptive calibration for disease-induced deterioration prediction,” Medical Image Analysis, vol. 69, p. 101947, 2021.
- Y. Yang, C. Ye, and T. Ma, “A deep connectome learning network using graph convolution for connectome-disease association study,” Neural Networks, 2023.
- C. Ying, T. Cai, S. Luo, S. Zheng, G. Ke, D. He, Y. Shen, and T.-Y. Liu, “Do transformers really perform badly for graph representation?” Advances in Neural Information Processing Systems, vol. 34, pp. 28 877–28 888, 2021.
- D. Kreuzer, D. Beaini, W. Hamilton, V. Létourneau, and P. Tossou, “Rethinking graph transformers with spectral attention,” Advances in Neural Information Processing Systems, vol. 34, pp. 21 618–21 629, 2021.
- D. Chen, Y. Lin, W. Li, P. Li, J. Zhou, and X. Sun, “Measuring and relieving the over-smoothing problem for graph neural networks from the topological view,” in Proceedings of the AAAI conference on artificial intelligence, vol. 34, no. 04, 2020, pp. 3438–3445.
- A. v. d. Oord, Y. Li, and O. Vinyals, “Representation learning with contrastive predictive coding,” arXiv preprint arXiv:1807.03748, 2018.
- C. Sudlow, J. Gallacher, N. Allen, V. Beral, P. Burton, J. Danesh, P. Downey, P. Elliott, J. Green, M. Landray et al., “Uk biobank: an open access resource for identifying the causes of a wide range of complex diseases of middle and old age,” Plos med, vol. 12, no. 3, p. e1001779, 2015.
- L. M. Alexander, J. Escalera, L. Ai, C. Andreotti, K. Febre, A. Mangone, N. Vega-Potler, N. Langer, A. Alexander, M. Kovacs et al., “An open resource for transdiagnostic research in pediatric mental health and learning disorders,” Scientific data, vol. 4, no. 1, pp. 1–26, 2017.
- A. J. Holmes, M. O. Hollinshead, T. M. O’keefe, V. I. Petrov, G. R. Fariello, L. L. Wald, B. Fischl, B. R. Rosen, R. W. Mair, J. L. Roffman et al., “Brain genomics superstruct project initial data release with structural, functional, and behavioral measures,” Scientific data, vol. 2, no. 1, pp. 1–16, 2015.
- X.-N. Zuo, J. S. Anderson, P. Bellec, R. M. Birn, B. B. Biswal, J. Blautzik, J. Breitner, R. L. Buckner, V. D. Calhoun, F. X. Castellanos et al., “An open science resource for establishing reliability and reproducibility in functional connectomics,” Scientific data, vol. 1, no. 1, pp. 1–13, 2014.
- R. H. Tobe, A. MacKay-Brandt, R. Lim, M. Kramer, M. M. Breland, L. Tu, Y. Tian, K. D. Trautman, C. Hu, R. Sangoi et al., “A longitudinal resource for studying connectome development and its psychiatric associations during childhood,” Scientific Data, vol. 9, no. 1, p. 300, 2022.
- D. C. Van Essen, S. M. Smith, D. M. Barch, T. E. Behrens, E. Yacoub, K. Ugurbil, W.-M. H. Consortium et al., “The wu-minn human connectome project: an overview,” Neuroimage, vol. 80, pp. 62–79, 2013.
- B. Sinclair, N. K. Hansell, G. A. Blokland, N. G. Martin, P. M. Thompson, M. Breakspear, G. I. de Zubicaray, M. J. Wright, and K. L. McMahon, “Heritability of the network architecture of intrinsic brain functional connectivity,” Neuroimage, vol. 121, pp. 243–252, 2015.
- D. Wei, K. Zhuang, L. Ai, Q. Chen, W. Yang, W. Liu, K. Wang, J. Sun, and J. Qiu, “Structural and functional brain scans from the cross-sectional southwest university adult lifespan dataset,” Scientific data, vol. 5, no. 1, pp. 1–10, 2018.
- X.-N. Zuo and C. Consortium, “Chinese Color Nest Project (CCNP),” Feb. 2023.
- J. R. Taylor, N. Williams, R. Cusack, T. Auer, M. A. Shafto, M. Dixon, L. K. Tyler, R. N. Henson et al., “The cambridge centre for ageing and neuroscience (cam-can) data repository: Structural and functional mri, meg, and cognitive data from a cross-sectional adult lifespan sample,” neuroimage, vol. 144, pp. 262–269, 2017.
- L. T. Strike, N. K. Hansell, K.-H. Chuang, J. L. Miller, G. I. de Zubicaray, P. M. Thompson, K. L. McMahon, and M. J. Wright, “The queensland twin adolescent brain project, a longitudinal study of adolescent brain development,” Scientific Data, vol. 10, no. 1, p. 195, 2023.
- R. N. Spreng, R. Setton, U. Alter, B. N. Cassidy, B. Darboh, E. DuPre, K. Kantarovich, A. W. Lockrow, L. Mwilambwe-Tshilobo, W.-M. Luh et al., “Neurocognitive aging data release with behavioral, structural and multi-echo functional mri measures,” Scientific Data, vol. 9, no. 1, p. 119, 2022.
- P. Gao, H.-M. Dong, Y.-S. Wang, C.-S. Yu, and X.-N. Zuo, “Imaging Chinese Young Brains (I See Your Brain),” Aug. 2021.
- J. W. Kable, M. K. Caulfield, M. Falcone, M. McConnell, L. Bernardo, T. Parthasarathi, N. Cooper, R. Ashare, J. Audrain-McGovern, R. Hornik et al., “No effect of commercial cognitive training on brain activity, choice behavior, or cognitive performance,” Journal of Neuroscience, vol. 37, no. 31, pp. 7390–7402, 2017.
- D. Kliemann, R. Adolphs, T. Armstrong, P. Galdi, D. A. Kahn, T. Rusch, A. Z. Enkavi, D. Liang, S. Lograsso, W. Zhu et al., “Caltech conte center, a multimodal data resource for exploring social cognition and decision-making,” Scientific Data, vol. 9, no. 1, p. 138, 2022.
- C. Racey, C. Kampoureli, O. Bowen-Hill, M. Bauer, I. Simpson, C. Rae, M. Del Rio, J. Simner, and J. Ward, “An open science mri database of over 100 synaesthetic brains and accompanying deep phenotypic information,” Scientific Data, vol. 10, no. 1, p. 766, 2023.
- X. Chen, B. Lu, H.-X. Li, X.-Y. Li, Y.-W. Wang, F. X. Castellanos, L.-P. Cao, N.-X. Chen, W. Chen, Y.-Q. Cheng et al., “The direct consortium and the rest-meta-mdd project: towards neuroimaging biomarkers of major depressive disorder,” Psychoradiology, vol. 2, no. 1, pp. 32–42, 2022.
- P. J. LaMontagne, T. L. Benzinger, J. C. Morris, S. Keefe, R. Hornbeck, C. Xiong, E. Grant, J. Hassenstab, K. Moulder, A. G. Vlassenko et al., “Oasis-3: longitudinal neuroimaging, clinical, and cognitive dataset for normal aging and alzheimer disease,” MedRxiv, pp. 2019–12, 2019.
- C. R. Jack Jr, M. A. Bernstein, N. C. Fox, P. Thompson, G. Alexander, D. Harvey, B. Borowski, P. J. Britson, J. L. Whitwell, C. Ward et al., “The alzheimer’s disease neuroimaging initiative (adni): Mri methods,” Journal of Magnetic Resonance Imaging: An Official Journal of the International Society for Magnetic Resonance in Medicine, vol. 27, no. 4, pp. 685–691, 2008.
- K. Marek, D. Jennings, S. Lasch, A. Siderowf, C. Tanner, T. Simuni, C. Coffey, K. Kieburtz, E. Flagg, S. Chowdhury et al., “The parkinson progression marker initiative (ppmi),” Progress in neurobiology, vol. 95, no. 4, pp. 629–635, 2011.
- R. A. Poldrack, E. Congdon, W. Triplett, K. Gorgolewski, K. Karlsgodt, J. Mumford, F. Sabb, N. Freimer, E. London, T. Cannon et al., “A phenome-wide examination of neural and cognitive function,” Scientific data, vol. 3, no. 1, pp. 1–12, 2016.
- L. Wang, K. I. Alpert, V. D. Calhoun, D. J. Cobia, D. B. Keator, M. D. King, A. Kogan, D. Landis, M. Tallis, M. D. Turner et al., “Schizconnect: Mediating neuroimaging databases on schizophrenia and related disorders for large-scale integration,” Neuroimage, vol. 124, pp. 1155–1167, 2016.
- J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, M. Gheshlaghi Azar et al., “Bootstrap your own latent-a new approach to self-supervised learning,” Advances in neural information processing systems, vol. 33, pp. 21 271–21 284, 2020.
- A. Schaefer, R. Kong, E. M. Gordon, T. O. Laumann, X.-N. Zuo, A. J. Holmes, S. B. Eickhoff, and B. T. Yeo, “Local-global parcellation of the human cerebral cortex from intrinsic functional connectivity mri,” Cerebral cortex, vol. 28, no. 9, pp. 3095–3114, 2018.
- J. Jiang, Y. Wei, Y. Feng, J. Cao, and Y. Gao, “Dynamic hypergraph neural networks.” in IJCAI, 2019, pp. 2635–2641.
- Y. Yang, X. Guo, C. Ye, Y. Xiang, and T. Ma, “Creg-kd: Model refinement via confidence regularized knowledge distillation for brain imaging,” Medical Image Analysis, vol. 89, p. 102916, 2023.
- U. Mahmood, Z. Fu, S. Ghosh, V. Calhoun, and S. Plis, “Through the looking glass: Deep interpretable dynamic directed connectivity in resting fmri,” NeuroImage, vol. 264, p. 119737, 2022.
- A. Mitra and M. E. Raichle, “Principles of cross-network communication in human resting state fmri,” Scandinavian Journal of Psychology, vol. 59, no. 1, pp. 83–90, 2018.
- A. I. Luppi, P. A. Mediano, F. E. Rosas, N. Holland, T. D. Fryer, J. T. O’Brien, J. B. Rowe, D. K. Menon, D. Bor, and E. A. Stamatakis, “A synergistic core for human brain evolution and cognition,” Nature Neuroscience, vol. 25, no. 6, pp. 771–782, 2022.
- T. Ito, K. R. Kulkarni, D. H. Schultz, R. D. Mill, R. H. Chen, L. I. Solomyak, and M. W. Cole, “Cognitive task information is transferred between brain regions via resting-state network topology,” Nature communications, vol. 8, no. 1, p. 1027, 2017.
- Z. Shehzad, C. Kelly, P. T. Reiss, R. C. Craddock, J. W. Emerson, K. McMahon, D. A. Copland, F. X. Castellanos, and M. P. Milham, “A multivariate distance-based analytic framework for connectome-wide association studies,” Neuroimage, vol. 93, pp. 74–94, 2014.
- M. Wei, J. Qin, R. Yan, K. Bi, C. Liu, Z. Yao, and Q. Lu, “Association of resting-state network dysfunction with their dynamics of inter-network interactions in depression,” Journal of affective disorders, vol. 174, pp. 527–534, 2015.
- J. Caspers, C. Rubbert, S. B. Eickhoff, F. Hoffstaedter, M. Südmeyer, C. J. Hartmann, B. Sigl, N. Teichert, J. Aissa, B. Turowski et al., “Within-and across-network alterations of the sensorimotor network in parkinson’s disease,” Neuroradiology, vol. 63, no. 12, pp. 2073–2085, 2021.
- Z. Qi, Y. An, M. Zhang, H.-J. Li, and J. Lu, “Altered cerebro-cerebellar limbic network in ad spectrum: a resting-state fmri study,” Frontiers in Neural Circuits, vol. 13, p. 72, 2019.