Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
102 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Large-scale Long-tailed Disease Diagnosis on Radiology Images (2312.16151v3)

Published 26 Dec 2023 in cs.CV

Abstract: Developing a generalist radiology diagnosis system can greatly enhance clinical diagnostics. In this paper, we introduce RadDiag, a foundational model supporting 2D and 3D inputs across various modalities and anatomies, using a transformer-based fusion module for comprehensive disease diagnosis. Due to patient privacy concerns and the lack of large-scale radiology diagnosis datasets, we utilize high-quality, clinician-reviewed radiological images available online with diagnosis labels. Our dataset, RP3D-DiagDS, contains 40,936 cases with 195,010 scans covering 5,568 disorders (930 unique ICD-10-CM codes). Experimentally, our RadDiag achieves 95.14% AUC on internal evaluation with the knowledge-enhancement strategy. Additionally, RadDiag can be zero-shot applied or fine-tuned to external diagnosis datasets sourced from various hospitals, demonstrating state-of-the-art results. In conclusion, we show that publicly shared medical data on the Internet is a tremendous and valuable resource that can potentially support building a generalist AI for healthcare.

PDF Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (87)
  1. ICD10. https://www.icd10data.com/ICD10CM/Codes.
  2. Kaggle: Brain mri scans for brain tumor classification. https://www.kaggle.com/datasets/shreyag1103/brain-mri-scans-for-brain-tumor-classification.
  3. Kaggle: Brain tumor mri dataset. https://www.kaggle.com/dsv/2645886.
  4. Kaggle: Brain tumor mri images 17 classes. https://www.kaggle.com/datasets/fernando2rad/brain-tumor-mri-images-17-classes.
  5. Radiopaedia. https://radiopaedia.org.
  6. Dataset of breast ultrasound images. Data In Brief, 28:104863, 2020.
  7. The lung image database consortium (lidc) and image database resource initiative (idri): a completed reference database of lung nodules on ct scans. Medical Physics, 38(2):915–931, 2011.
  8. Aucreshaping: improved sensitivity at high-specificity. Scientific Reports, 13(1):21097, 2023.
  9. Deep-learning-assisted diagnosis for knee magnetic resonance imaging: Development and retrospective validation of mrnet. PLoS Medicine, 15(11), 2018.
  10. POCOVID-Net: automatic detection of COVID-19 from a new lung ultrasound imaging dataset (POCUS). arXiv preprint arXiv:2004.12084, 2020.
  11. Andrew P Bradley. The use of the area under the roc curve in the evaluation of machine learning algorithms. Pattern Recognition, 30(7):1145–1159, 1997.
  12. Padchest: A large chest x-ray image dataset with multi-label annotated reports. Medical Image Analysis, 66:101797, 2020.
  13. Deep learning system to screen coronavirus disease 2019 pneumonia. Applied Intelligence (Dordrecht, Netherlands), page 1, 2020.
  14. Breast microcalcification diagnosis using deep convolutional neural network from digital mammograms. Computational and Mathematical Methods in Medicine, 2019, 2019.
  15. Kai Cao et al. Large-scale pancreatic cancer detection via non-contrast ct and deep learning. Nature Medicine, 2023.
  16. Monai: An open-source framework for deep learning in healthcare. arXiv preprint arXiv:2211.02701, 2022.
  17. Enhanced performance of brain tumor classification via tumor region augmentation and partition. PloS One, 10(10):e0140381, 2015.
  18. The advantages of the matthews correlation coefficient (mcc) over f1 score and accuracy in binary classification evaluation. BMC Genomics, 21(1):1–13, 2020.
  19. Can ai help in screening viral and covid-19 pneumonia? IEEE Access, 8:132665–132676, 2020.
  20. Transmed: Transformers advance multi-modal medical image classification. Diagnostics, 11(8):1384, 2021.
  21. Brain tumor classification using deep cnn features via transfer learning. Computers in Biology and Medicine, 111:103345, 2019.
  22. Preparing a collection of radiology examinations for distribution and retrieval. Journal of the American Medical Informatics Association, 23(2):304–310, 2016.
  23. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
  24. A multiple resampling method for learning from imbalanced data sets. Computational Intelligence, 20(1):18–36, 2004.
  25. Crowdsourcing pneumothorax annotations using machine learning annotations on the nih chest x-ray dataset. Journal of Digital Imaging, 33:490–496, 2020.
  26. COVID-VIT: Classification of COVID-19 from CT chest images based on vision transformer models. arXiv preprint arXiv:2107.01682, 2021.
  27. Vision transformers for classification of breast ultrasound images. In 2022 44th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), pages 480–483. IEEE, 2022.
  28. Lvis: A dataset for large vocabulary instance segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
  29. A hybrid cnn-glcm classifier for detection and grade classification of brain tumor. Brain Imaging and Behavior, 16(3):1410–1427, 2022.
  30. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2015.
  31. JF Healthcare. Object-CXR - Automatic detection of foreign objects on chest X-rays. 2020.
  32. The digital database for screening mammography. In Proceedings of the Fifth International Workshop on Digital Mammography, pages 212–218. Medical Physics Publishing, 2001.
  33. Intracranial hemorrhage segmentation using a deep convolutional model. Data, 5(1), 2020.
  34. CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 590–597, 2019.
  35. The alzheimer’s disease neuroimaging initiative (adni): Mri methods. Journal of Magnetic Resonance Imaging: An Official Journal of the International Society for Magnetic Resonance in Medicine, 27(4):685–691, 2008.
  36. Two public chest x-ray datasets for computer-aided screening of pulmonary diseases. Quantitative Imaging in Medicine and Surgery, 4(6):475, 2014.
  37. Medcpt: Contrastive pre-trained transformers with large-scale pubmed search logs for zero-shot biomedical information retrieval. Bioinformatics, 39(11):btad651, 2023.
  38. Mimic-cxr, a de-identified publicly available database of chest radiographs with free-text reports. Scientific Data, 6(1):317, 2019.
  39. Select atrophied regions in alzheimer disease (sara): An improved volumetric model for identifying alzheimer disease dementia. NeuroImage: Clinical, 26:102248, 2020.
  40. Mia-cov19d: Covid-19 detection through 3-d chest ct image analysis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 537–544, 2021.
  41. Residual and plain convolutional neural networks for 3D brain MRI classification. In Proceedings-International Symposium on Biomedical Imaging, pages 835–838, 2017.
  42. Michał Koziarski. Radial-based undersampling for imbalanced data classification. Pattern Recognition, 102:107262, 2020.
  43. Oasis-3: Longitudinal neuroimaging, clinical, and cognitive dataset for normal aging and alzheimer disease. medRxiv, 2019.
  44. A curated mammography data set for use in computer-aided detection and diagnosis research. Scientific Data, 4(1):1–9, 2017.
  45. Equalized focal loss for dense long-tailed object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6990–6999, 2022.
  46. Artificial intelligence distinguishes COVID-19 from community acquired pneumonia on chest CT. Radiology, 2020.
  47. Focal loss for dense object detection. IEEE International Conference on Computer Vision (ICCV), pages 2999–3007, 2017.
  48. Automatic diagnosis of covid-19 using a tailored transformer-like network. In Journal of Physics: Conference Series, volume 2010, page 012175. IOP Publishing, 2021.
  49. Exploratory undersampling for class-imbalance learning. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 39(2):539–550, 2008.
  50. Anna Majkowska et al. Chest radiograph interpretation with deep learning models: assessment with radiologist-adjudicated reference standards and population-adjusted evaluation. Radiology, 294(2):421–431, 2020.
  51. Open Access Series of Imaging Studies (OASIS): Cross-sectional MRI Data in Young, Middle Aged, Nondemented, and Demented Older Adults. Journal of Cognitive Neuroscience, 19(9):1498–1507, 09 2007.
  52. Is it time to replace cnns with transformers for medical images? arXiv preprint arXiv:2108.09038, 2021.
  53. Radimagenet: an open radiologic deep learning research dataset for effective transfer learning. Radiology: Artificial Intelligence, 4(5):e210315, 2022.
  54. Low-dose ct image and projection dataset. Medical Physics, 48(2):902–911, 2021.
  55. xViTCOS: explainable vision transformer based COVID-19 screening using radiography. IEEE Journal of Translational Engineering in Health and Medicine, 10:1–10, 2021.
  56. Med-Flamingo: A Multimodal Medical Few-shot Learner. arXiv preprint arXiv:2307.15189, July 2023.
  57. Mosmeddata: Chest ct scans with covid-19 related findings dataset. arXiv preprint arXiv:2005.06465, 2020.
  58. Sayyed Mostafa Mostafavi. COVID19-CT-Dataset: An Open-Access Chest CT Image Repository of 1000+ Patients with Confirmed COVID-19 Diagnosis. 2021.
  59. VinDr-CXR: An open dataset of chest X-rays with radiologist’s annotations. Scientific Data, 9(1):429, 2022.
  60. VinDr-Mammo: A large-scale benchmark dataset for computer-aided diagnosis in full-field digital mammography. Scientific Data, 10(1):277, 2023.
  61. VinDr-Mammo: A large-scale benchmark dataset for computer-aided diagnosis in full-field digital mammography. PhysioNet, 2022.
  62. VinDr-SpineXR: A deep learning framework for spinal lesions detection and classification from radiographs. In Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part V 24, pages 291–301. Springer, 2021.
  63. VinDr-PCXR: An open, large-scale chest radiograph dataset for interpretation of common thoracic diseases in children. PhysioNet, 2022.
  64. OpenAI. GPT-4 Technical Report. arXiv preprint arXiv:2303.08774, 2023.
  65. Vision transformer for covid-19 cxr diagnosis using chest x-ray feature corpus. arXiv preprint arXiv:2103.07055, 2021.
  66. Maya Pavlova et al. Covid-net cxr-2: An enhanced deep convolutional neural network design for detection of covid-19 cases from chest x-ray images. Frontiers in Medicine, 9, 2022.
  67. Pocformer: A lightweight transformer architecture for detection of covid-19 using point of care ultrasound. In 2021 IEEE international conference on image processing (ICIP), pages 195–199. IEEE, 2021.
  68. Asymmetric loss for multi-label classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 82–91, 2021.
  69. Multi-grade brain tumor classification using deep cnn with extensive data augmentation. Journal of Computational Science, 30:174–182, 2019.
  70. TransMIL: Transformer based Correlated Multiple Instance Learning for Whole Slide Image Classification. Advances in Neural Information Processing Systems, 34:2136–2147, 2021.
  71. Augmenting the national institutes of health chest radiograph dataset with expert annotations of possible pneumonia. Radiology: Artificial Intelligence, 1(1):e180041, 2019.
  72. Covid-transformer: Interpretable covid-19 detection using vision transformer for healthcare. International Journal of Environmental Research and Public Health, 18(21):11086, 2021.
  73. Brain tumor classification for mr images using transfer learning and fine-tuning. Computerized Medical Imaging and Graphics, 75:34–46, 2019.
  74. One click lesion recist measurement and segmentation on ct scans. In Medical Image Computing and Computer Assisted Intervention (MICCAI), 2020.
  75. Towards generalist biomedical ai. arXiv preprint arXiv:2307.14334, 2023.
  76. Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2097–2106, 2017.
  77. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171, 2022.
  78. Integrating features from lymph node stations for metastatic lymph node detection. Computerized medical imaging and graphics : the official journal of the Computerized Medical Imaging Society, 101:102108, 2022.
  79. K-diag: Knowledge-enhanced disease diagnosis in radiographic imaging. Medical Image Computing and Computer Assisted Intervention – MICCAI Workshop, 2023.
  80. MedKLIP: Medical Knowledge Enhanced Language-Image Pre-Training. arXiv preprint arXiv:2301.02228, 2023.
  81. Towards generalist foundation model for radiology by leveraging web-scale 2d&3d medical data. IEEE International Conference on Computer Vision (ICCV), 2023.
  82. Deeplesion: Automated deep mining, categorization and detection of significant radiology image findings using large-scale clinical lesion annotations. arXiv preprint arXiv:1710.01766, 2017.
  83. Mil-vt: Multiple instance learning enhanced vision transformer for fundus image classification. In Medical Image Computing and Computer Assisted Intervention(MICCAI), 2021.
  84. Knowledge-enhanced pre-training for auto-diagnosis of chest radiology images. Nature Communications, 2023.
  85. Pmc-vqa: Visual instruction tuning for medical visual question answering. arXiv preprint arXiv:2305.10415, 2023.
  86. mmformer: Multimodal medical transformer for incomplete multimodal learning of brain tumor segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 107–117. Springer, 2022.
  87. A graph-transformer for whole slide image classification. IEEE Transactions on Medical Imaging, 41(11):3003–3015, 2022.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (10)
  1. Qiaoyu Zheng (4 papers)
  2. Weike Zhao (6 papers)
  3. Chaoyi Wu (24 papers)
  4. Xiaoman Zhang (31 papers)
  5. Ya Zhang (222 papers)
  6. Yanfeng Wang (211 papers)
  7. Weidi Xie (132 papers)
  8. Lisong Dai (7 papers)
  9. Hengyu Guan (1 paper)
  10. Yuehua Li (6 papers)
Citations (4)
View on Semantic Scholar