Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Artificial Intelligence for Digital and Computational Pathology (2401.06148v1)

Published 13 Dec 2023 in eess.IV, cs.AI, cs.CV, and q-bio.QM

Abstract: Advances in digitizing tissue slides and the fast-paced progress in artificial intelligence, including deep learning, have boosted the field of computational pathology. This field holds tremendous potential to automate clinical diagnosis, predict patient prognosis and response to therapy, and discover new morphological biomarkers from tissue images. Some of these artificial intelligence-based systems are now getting approved to assist clinical diagnosis; however, technical barriers remain for their widespread clinical adoption and integration as a research tool. This Review consolidates recent methodological advances in computational pathology for predicting clinical end points in whole-slide images and highlights how these developments enable the automation of clinical practice and the discovery of new biomarkers. We then provide future perspectives as the field expands into a broader range of clinical and research tasks with increasingly diverse modalities of clinical data.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (226)
  1. Abels, E. et al. Computational pathology definitions, best practices, and recommendations for regulatory guidance: a white paper from the digital pathology association. The Journal of pathology 249, 286–294 (2019).
  2. Lu, M. Y. et al. AI-based pathology predicts origins for cancers of unknown primary. Nature 594, 106–110 (2021).
  3. Bulten, W. et al. Artificial intelligence for diagnosis and gleason grading of prostate cancer: the panda challenge. Nature Medicine 28, 154–163 (2022).
  4. Skrede, O.-J. et al. Deep learning for prediction of colorectal cancer outcome: a discovery and validation study. The Lancet 395, 350–360 (2020).
  5. Ehteshami Bejnordi, B. et al. Diagnostic Assessment of Deep Learning Algorithms for Detection of Lymph Node Metastases in Women With Breast Cancer. JAMA 318, 2199–2210 (2017).
  6. Deep learning. Nature 521, 436–444 (2015).
  7. Instrumentation for automatically prescreening cytological smears. Proceedings of the IRE 47, 1895–1900 (1959).
  8. The analysis of cell images. Annals of the New York Academy of Sciences 128, 1035–1053 (1966).
  9. Computational pathology analysis of tissue microarrays predicts survival of renal clear cell carcinoma patients. In International Conference on Medical Image Computing and Computer-Assisted Intervention (Springer, 2008).
  10. Beck, A. H. et al. Systematic analysis of breast cancer morphology uncovers stromal features associated with survival. Science translational medicine 3 (2011).
  11. Image analysis and machine learning in digital pathology: Challenges and opportunities. Medical image analysis 33, 170–175 (2016).
  12. The evolving paradigm of biomarker actionability: histology-agnosticism as a spectrum, rather than a binary quality. Cancer Treatment Reviews 94, 102169 (2021).
  13. Lee, Y. et al. Derivation of prognostic contextual histopathological features from whole-slide images of tumours via graph deep learning. Nature Biomedical Engineering (2022).
  14. Saltz, J. et al. Spatial organization and molecular correlation of tumor-infiltrating lymphocytes using deep learning on pathology images. Cell reports 23, 181–193 (2018).
  15. Intratumor heterogeneity: the rosetta stone of therapy resistance. Cancer cell 37, 471–484 (2020).
  16. Vamathevan, J. et al. Applications of machine learning in drug discovery and development. Nature Reviews Drug discovery 18, 463–477 (2019).
  17. Dosovitskiy, A. et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (2021).
  18. Self-supervised learning in medicine and healthcare. Nature Biomedical Engineering (2022).
  19. Jiang, P. et al. Big data in basic and translational cancer research. Nature Reviews Cancer (2022).
  20. Multiplexed imaging in oncology. Nature Biomedical Engineering 6, 527–540 (2022).
  21. Marx, V. Method of the year: spatially resolved transcriptomics. Nature Methods 18, 9–14 (2021).
  22. Liu, J. T. et al. Harnessing non-destructive 3d pathology. Nature Biomedical Engineering 5, 203–218 (2021).
  23. Bandi, P. et al. From detection of individual metastases to classification of lymph node status at the patient level: the camelyon17 challenge. IEEE transactions on medical imaging 38, 550–560 (2018).
  24. Deng, J. et al. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, 248–255 (2009).
  25. Coudray, N. et al. Classification and mutation prediction from non–small cell lung cancer histopathology images using deep learning. Nature Medicine 24, 1559–1567 (2018).
  26. Intra-tumour heterogeneity: a looking glass for cancer? Nature Reviews Cancer 12, 323–334 (2012).
  27. Intratumoral heterogeneity in cancer progression and response to immunotherapy. Nature Medicine 27, 212–224 (2021).
  28. Kather, J. N. et al. Pan-cancer image-based detection of clinically actionable genetic alterations. Nature cancer 1, 789–799 (2020).
  29. Attention-based deep multiple instance learning. In Proceedings of the 35th International Conference on Machine Learning (2018).
  30. Solving the multiple instance problem with axis-parallel rectangles. Artificial intelligence 89, 31–71 (1997).
  31. Dundar, M. M. et al. A multiple instance learning approach toward optimal classification of pathology slides. In 2010 20th International Conference on Pattern Recognition, 2732–2735 (IEEE, 2010).
  32. Lu, Z. et al. Deep-learning–based characterization of tumor-infiltrating lymphocytes in breast cancers from histopathology images and multiomics data. JCO clinical cancer informatics 4, 480–490 (2020).
  33. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, 770–778 (2016).
  34. Shaban, M. et al. Context-aware convolutional neural network for grading of colorectal cancer histology images. IEEE Transactions on Medical Imaging 39, 2395 – 2405 (2020).
  35. Tellez, D. et al. Whole-slide mitosis detection in H&E breast histology using PHH3 as a reference to train distilled stain-invariant convolutional networks. IEEE Transactions on Medical Imaging 37, 2126–2136 (2018).
  36. Campanella, G. et al. Clinical-grade computational pathology using weakly supervised deep learning on whole slide images. Nature Medicine (2019).
  37. Wang, X. et al. Transformer-based unsupervised contrastive learning for histopathological image classification. Medical Image Analysis 81 (2022).
  38. Self supervised contrastive learning for digital histopathology. Machine Learning with Applications 7 (2022).
  39. Chen, R. J. et al. Scaling vision transformers to gigapixel images via hierarchical self-supervised learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2022).
  40. Chen, C.-L. et al. An annotation-free whole-slide training approach to pathological classification of lung cancer types using deep learning. Nature communications 12, 1–13 (2021).
  41. Streaming convolutional neural networks for end-to-end learning with multi-megapixel images. IEEE transactions on pattern analysis and machine intelligence (2020).
  42. Huang, S.-C. et al. Deep neural network trained on gigapixel images improves lymph node metastasis detection in clinical settings. Nature Communications 13, 1–14 (2022).
  43. Wulczyn, E. et al. Deep learning-based survival prediction for multiple cancer types using histopathology images. PloS one 15, e0233678 (2020).
  44. Wulczyn, E. et al. Interpretable survival prediction for colorectal cancer using deep learning. NPJ digital medicine 4, 1–13 (2021).
  45. Laleh, N. G. et al. Benchmarking weakly-supervised deep learning pipelines for whole slide classification in computational pathology. Medical image analysis 79 (2022).
  46. Integrating context for superior cancer prognosis. Nat. Biomed. Eng (2022).
  47. Taube, J. M. et al. Implications of the tumor immune microenvironment for staging and therapeutics. Modern Pathology 31, 214–234 (2018).
  48. Pati, P. et al. Hierarchical graph representations in digital pathology. Medical image analysis 75, 102264 (2022).
  49. Chen, R. J. et al. Whole slide images are 2d point clouds: Context-aware survival prediction using patch-based graph convolutional networks. In International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI) (2021).
  50. Adnan, M. et al. Representation learning of histopathology images using graph neural networks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops (2020).
  51. The graph neural network model. IEEE transactions on neural networks 20, 61–80 (2008).
  52. The cell graphs of cancer. Bioinformatics 20, i145–i151 (2004).
  53. Zhou, Y. et al. Cgc-net: Cell graph convolutional network for grading of colorectal cancer histology images. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, 0–0 (2019).
  54. Chen, R. J. et al. Pathomic fusion: an integrated framework for fusing histopathology and genomic features for cancer diagnosis and prognosis. IEEE Transactions on Medical Imaging (2020).
  55. Ahmedt, D. et al. A survey on graph-based deep learning for computational histopathology. Computerized Medical Imaging and Graphics (2021).
  56. Vaswani, A. et al. Attention Is All You Need. In Neural Information Processing Systems (NeurIPS) (2017).
  57. Shao, Z. et al. Transmil: Transformer based correlated multiple instance learning for whole slide image classification. Advances in Neural Information Processing Systems 34, 2136–2147 (2021).
  58. Flowformer: Linearizing transformers with conservation flows. In International Conference on Machine Learning (2022).
  59. Iizuka, O. et al. Deep learning models for histopathological classification of gastric and colonic epithelial tumours. Scientific Reports 10 (2020).
  60. Learning permutation invariant representations using memory networks. In European Conference on Computer Vision, 677–693 (Springer, 2020).
  61. Lipkova, J. et al. Deep learning-enabled assessment of cardiac allograft rejection from endomyocardial biopsies. Nature Medicine (2022).
  62. Sirinukunwattana, K. et al. Improving whole slide segmentation through visual context - a systematic study. In Medical Image Computing and Computer Assisted Intervention (MICCAI) (2018).
  63. Thandiackal, K. et al. Differentiable zooming for multiple instance learning on whole-slide images. In European Conference on Computer Vision (ECCV) (2022).
  64. Processing megapixel images with deep attention-sampling models. In International Conference on Machine Learning (ICML), vol. 36 (2019).
  65. Efficient classification of very large images with tiny objects. Computer Vision and Pattern Recognition (CVPR) (2021).
  66. Bulten, W. et al. Epithelium segmentation using deep learning in h&e-stained prostate specimens with immunohistochemistry as reference standard. Nature Scientific Report 9 (2019).
  67. Anklin, V. et al. Learning whole-slide segmentation from inexact and incomplete labels using tissue graphs. In Medical Image Computing and Computer Assisted Intervention (MICCAI) (2021).
  68. Histosegnet: Semantic segmentation of histological tissue type in whole slide images. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 10662–10671 (2019).
  69. Graham, S. et al. Hover-net: Simultaneous segmentation and classification of nuclei in multi-tissue histology images. Medical Image Analysis 58 (2019).
  70. Sirinukunwattana, K. et al. Gland segmentation in colon histology images: The glas challenge contest. Medical Image Analysis 35, 489–502 (2017).
  71. Deepmitosis: Mitosis detection via deep detection, verification and segmentation networks. Medical image analysis 45, 121–133 (2018).
  72. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, 3431–3440 (2015).
  73. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention – MICCAI (2015).
  74. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, 2961–2969 (2017).
  75. Kumar, N. et al. A dataset and a technique for generalized nuclear segmentation for computational pathology. In IEEE Transactions on Medical Imaging, vol. 36, 1550–1560 (2017).
  76. Nuclick: A deep learning framework for interactive segmentation of microscopic images. Medical Image Analysis 65 (2020).
  77. Greenwald, N. F. et al. Whole-cell segmentation of tissue images with human-level performance using large-scale data annotation and deep learning. Nature biotechnology 40, 555–565 (2022).
  78. ATHENA: analysis of tumor heterogeneity from spatial omics measurements. Bioinformatics 38, 3151–3153 (2022).
  79. Cell segmentation for immunofluorescence multiplexed images using two-stage domain adaptation and weakly labeled data for pre-training. Scientific Reports 12, 1–14 (2022).
  80. Tellez, D. et al. H and e stain augmentation improves generalization of convolutional networks for histopathological mitosis detection. In Medical Imaging (2018).
  81. Stain normalization of histopathology images using generative adversarial networks. In 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), 573–577 (2018).
  82. Macenko, M. et al. A method for normalizing histology slides for quantitative analysis. In IEEE International Symposium on Biomedical Imaging (ISBI) (2009).
  83. Vahadane, A. et al. Structure-preserving color normalization and sparse stain separation for histological images. IEEE Transactions on Medical Imaging 35, 1962–1971 (2016).
  84. Neural stain-style transfer learning using gan for histopathological images. arXiv preprint arXiv:1710.08543 (2017).
  85. Enhanced cycle-consistent generative adversarial network for color normalization of h&e stained images. In International Conference on Medical Image Computing and Computer-Assisted Intervention, 694–702 (Springer, 2019).
  86. Kang, H. et al. Stainnet: a fast and robust stain normalization network. Frontiers in Medicine 8 (2021).
  87. He, B. et al. AI-enabled in silico immunohistochemical characterization for alzheimer’s disease. Cell Reports Methods 2, 100191 (2022).
  88. Ghahremani, P. et al. Deep learning-inferred multiplex immunofluorescence for immunohistochemical image quantification. Nature Machine Intelligence 4, 401–412 (2022).
  89. Cao, R. et al. Label-free intraoperative histology of bone tissue via deep-learning-assisted ultraviolet photoacoustic microscopy. Nature Biomedical Engineering 1–11 (2022).
  90. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, 2223–2232 (2017).
  91. Contrastive learning for unpaired image-to-image translation. In European Conference on Computer Vision, 319–345 (2020).
  92. Cyclegan for virtual stain transfer: Is seeing really believing? Artificial Intelligence in Medicine 133, 102420 (2022).
  93. Holzinger, A. et al. Towards the Augmented Pathologist: Challenges of Explainable-AI in Digital Pathology. In arXiv:1712.06657 (2017).
  94. Selvaraju, R. et al. Grad-CAM : Visual Explanations from Deep Networks. In International Conference on Computer Vision, 618–626 (2017).
  95. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning, 3319–3328 (2017).
  96. Lu, M. Y. et al. Data-efficient and weakly supervised computational pathology on whole-slide images. Nature Biomedical Engineering 5, 555–570 (2021).
  97. Chen, R. J. et al. Pan-cancer integrative histology-genomic analysis via multimodal deep learning. Cancer Cell 40, 865–878 (2022).
  98. Javed, S. A. et al. Additive mil: Intrinsically interpretable multiple instance learning for pathology. In Advances in Neural Information Processing Systems (NeurIPS) (2022).
  99. Diao, J. A. et al. Human-interpretable image features derived from densely mapped cancer pathology slides predict diverse molecular phenotypes. Nature Communications 12 (2021).
  100. Jaume, G. et al. Quantifying explainers of graph neural networks in computational pathology. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2021).
  101. Ludovic, R. et al. Mitosis detection in breast cancer histological images An ICPR 2012 contest. Journal of pathology informatics 4 (2013).
  102. Veta, M. et al. Predicting breast tumor proliferation from whole-slide images: The tupac16 challenge. Medical Image Analysis 54, 111–121 (2019).
  103. Aubreville, M. et al. MItosis DOmain Generalization Challenge 2022. In 25th International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI 2022) (2022).
  104. Aresta, G. et al. ICIAR 2018 grand challenge on breast cancer histology images. In International Conference on Image Analysis and Recognition (ICIAR) (2018).
  105. Ruchika, G. et al. Multi-organ nuclei segmentation and classification challenge 2020. IEEE Transactions on Medical Imaging (2020).
  106. IMI—Bigpicture: A Central Repository for Digital Pathology. Toxicologic Pathology 49, 711–713 (2021).
  107. Jennings, C. N. et al. Bridging the gap with the UK Genomics Pathology Imaging Collection. Nature Medicine 28, 1107–1108 (2022).
  108. Wagner, S. J. et al. Make deep learning algorithms in computational pathology more reproducible and reusable. Nature Medicine (2022).
  109. Gilbert, B. et al. Openslide (2020). URL https://github.com/openslide/openslide-python/.
  110. Bankhead, P. et al. Qupath (2021). URL https://qupath.github.io/.
  111. Beezley, J. et al. Histomicstk (2021). URL https://github.com/DigitalSlideArchive/HistomicsTK.
  112. Jaume, G. et al. Histocartography: A toolkit for graph analytics in digital pathology. In International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI) Workshop on Computational Pathology, 117–128 (2021).
  113. Rosenthal, J. et al. Building tools for machine learning and artificial intelligence in cancer research: best practices and a case study with the pathml toolkit for computational pathology. Molecular Cancer Research 20, 202–206 (2022).
  114. Pocock, J. et al. TIAToolbox as an end-to-end library for advanced tissue image analytics. Communications Medicine 2, 120 (2022). URL https://www.nature.com/articles/s43856-022-00186-5.
  115. Mitosis detection in breast cancer histology images with deep neural networks. In International conference on medical image computing and computer-assisted intervention, 411–418 (Springer, 2013).
  116. Kapil, A. et al. Deep semi supervised generative learning for automated tumor proportion scoring on nsclc tissue needle biopsies. Scientific reports 8, 1–10 (2018).
  117. Swiderska-Chadaj, Z. et al. Learning to detect lymphocytes in immunohistochemistry with deep learning. Medical Image Analysis 58, 101547 (2019).
  118. Fassler, D. J. et al. Deep learning-based image analysis methods for brightfield-acquired multiplex immunohistochemistry images. Diagnostic pathology 15, 1–11 (2020).
  119. Naylor, P. et al. Segmentation of nuclei in histopathology images by deep regression of the distance map. In IEEE Trans Med Imaging (2019).
  120. Widmaier, M. et al. Comparison of continuous measures across diagnostic pd-l1 assays in non-small cell lung cancer using automated image analysis. Modern Pathology 33, 380–390 (2020).
  121. Human-level recognition of blast cells in acute myeloid leukaemia with convolutional neural networks. Nature Machine Intelligence 1, 538–544 (2019).
  122. Chan, L. et al. HistoSegNet: Semantic segmentation of histological tissue type in whole slide images. In IEEE ICCV (2019).
  123. Binder, T. et al. Multi-organ gland segmentation using deep learning. In Frontiers in Medicine (2019).
  124. Graham, S. et al. A dataset and a technique for generalized nuclear segmentation for computational pathology. In Proceedings of the IEEE/CVF International Conference on Computer Vision (2021).
  125. Fraz, M. et al. Fabnet: feature attention-based network for simultaneous segmentation of microvessels and nerves in routine histology images of oral cancer. Neural Comput and Applic 32 (2020).
  126. Ing, N. et al. Semantic segmentation for prostate cancer grading by convolutional neural networks. In Tomaszewski, J. E. & Gurcan, M. N. (eds.) Medical Imaging 2018: Digital Pathology (International Society for Optics and Photonics, 2018).
  127. Geessink, O. G. et al. Computer aided quantification of intratumoral stroma yields an independent prognosticator in rectal cancer. Cellular Oncology 42, 331–341 (2019).
  128. Amgad, M. et al. Report on computational assessment of tumor infiltrating lymphocytes from the international immuno-oncology biomarker working group. NPJ Breast Cancer 6, 1–13 (2020).
  129. Taylor-Weiner, A. et al. A machine learning approach enables quantitative measurement of liver histology and disease monitoring in nash. Hepatology 74, 133–147 (2021).
  130. Heinemann, F. et al. Deep learning-based quantification of nafld/nash progression in human liver biopsies. Scientific Reports 12, 1–11 (2022).
  131. Ström, P. et al. Artificial intelligence for diagnosis and grading of prostate cancer in biopsies: a population-based, diagnostic study. The Lancet Oncology 21, 222–232 (2020).
  132. Bulten, W. et al. Automated deep-learning system for gleason grading of prostate cancer using biopsies: a diagnostic study. The Lancet Oncology 21, 233–241 (2020).
  133. Automated grading of gliomas using deep learning in digital pathology images: a modular approach with ensemble of convolutional neural networks. In AMIA annual symposium proceedings (2015).
  134. Couture, H. D. et al. Image analysis with deep learning to predict breast cancer grade, er status, histologic subtype, and intrinsic subtype. NPJ Breast Cancer 4 (2018).
  135. Kers, J. et al. Deep learning-based classification of kidney transplant pathology: a retrospective, multicentre, proof-of-concept study. The Lancet Digital Health 4, 18–26 (2022).
  136. Korbar, B. et al. Deep learning for classification of colorectal polyps on whole-slide images. Journal of Pathology Informatics 8, 30 (2017).
  137. Ianni, J. D. et al. Tailored for real-world: A whole slide image classification system validated on uncurated multi-site data emulating the prospective pathology workload. Scientific Reports 10 (2020).
  138. Kiani, A. et al. Impact of a deep learning assistant on the histopathologic classification of liver cancer. NPJ digital medicine 3, 1–8 (2020).
  139. Zhao, Y. et al. Predicting lymph node metastasis using histopathological images based on multiple instance learning with deep graph convolution. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 4837–4846 (2020).
  140. Ozyoruk, K. et al. Deep learning-based frozen section to FFPE translation. Nature Biomedical Engineering (2021).
  141. Hollon, T. C. et al. Near real-time intraoperative brain tumor diagnosis using stimulated raman histology and deep neural networks. Nature medicine 26, 52–58 (2020).
  142. Chen, P.-H. C. et al. An augmented reality microscope with real-time artificial intelligence integration for cancer diagnosis. Nature medicine 25, 1453–1457 (2019).
  143. Gehrung, M. et al. Triage-driven diagnosis of barrett’s esophagus for early detection of esophageal adenocarcinoma using deep learning. Nature medicine 27, 833–841 (2021).
  144. Kather, J. N. et al. Deep learning can predict microsatellite instability directly from histology in gastrointestinal cancer. Nature Medicine 25, 1054–1056 (2019).
  145. Bychkov, D. et al. Deep learning based tissue analysis predicts outcome in colorectal cancer. Scientific Reports 8 (2018).
  146. Kather, J. N. et al. Predicting survival from colorectal cancer histology slides using deep learning: A retrospective multicenter study. PLoS Medicine 16 (2019).
  147. Leo, P. et al. Computer extracted gland features from h&e predicts prostate cancer recurrence comparably to a genomic companion diagnostic test: a large multi-site study. NPJ Precision Oncology 5 (2021).
  148. Wang, X. et al. Prediction of recurrence in early stage non-small cell lung cancer using computer extracted nuclear features from digital h&e images. Scientific Reports 7 (2017).
  149. Shaban, M. et al. A novel digital score for abundance of tumour infiltrating lymphocytes predicts disease free survival in oral squamous cell carcinoma. Scientific Reports 9 (2019).
  150. Kulkarni, P. M. et al. Deep learning based on standard h&e images of primary melanoma tumors identifies patients at risk for visceral recurrence and deathdeep learning–based prognostic biomarker for melanoma. Clinical Cancer Research 26, 1126–1134 (2020).
  151. Yang, J. et al. Prediction of HER2-positive breast cancer recurrence and metastasis risk from histopathological images and clinical information via multimodal deep learning. Computational and Structural Biotechnology Journal 20, 333–342 (2022).
  152. Klimov, S. et al. Predicting metastasis risk in pancreatic neuroendocrine tumors using deep learning image analysis. Frontiers in Oncology 10, 593211 (2021).
  153. Kleppe, A. et al. A clinical decision support system optimising adjuvant chemotherapy for colorectal cancers by integrating deep learning and pathological staging markers: a development and validation study. The Lancet Oncology (2022).
  154. Courtiol, P. et al. Deep learning-based classification of mesothelioma improves prediction of patient outcome. Nature Medicine 25, 1519–1525 (2019).
  155. Saillard, C. et al. Predicting survival after hepatocellular carcinoma resection using deep learning on histological slides. Hepatology 72, 2000–2013 (2020).
  156. Integrating context for superior cancer prognosis. Nature Biomedical Engineering (2022).
  157. Zhao, Y. et al. Predicting lymph node metastasis using histopathological images based on multiple instance learning with deep graph convolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2020).
  158. Harnessing multimodal data integration to advance precision oncology. Nature Reviews Cancer 22, 114–126 (2022).
  159. Multimodal predictors for precision immunotherapy. Immuno-Oncology and Technology (2022).
  160. Boehm, K. M. et al. Multimodal data integration using machine learning improves risk stratification of high-grade serous ovarian cancer. Nature cancer 3, 723–733 (2022).
  161. Deep learning with multimodal representation for pancancer prognosis prediction. Bioinformatics 35, 446–454 (2019).
  162. Foersch, S. et al. Multistain deep learning for prediction of prognosis and therapy response in colorectal cancer. Nature Medicine 29, 1–10 (2023).
  163. Echle, A. et al. Deep learning in cancer pathology: a new generation of clinical biomarkers. British Journal of Cancer 124, 686–696 (2021).
  164. Fu, Y. et al. Pan-cancer computational histopathology reveals mutations, tumor composition and prognosis. Nature Cancer 1, 800–810 (2020).
  165. Wang, S. et al. Predicting egfr mutation status in lung adenocarcinoma on computed tomography image using deep learning. European Respiratory Journal 53 (2019).
  166. Loeffler, C. M. L. et al. Artificial intelligence–based detection of fgfr3 mutational status directly from routine histology in bladder cancer: A possible preselection for molecular testing? European Urology Focus 8, 472–479 (2022).
  167. Schmauch, B. et al. A deep learning model to predict rna-seq expression of tumours from whole slide images. Nature Communications 11 (2020).
  168. Predicting endometrial cancer subtypes and molecular features from histopathology images using multi-resolution deep learning models. Cell Reports Medicine 2, 100400 (2021).
  169. Shamai, G. et al. Deep learning-based image analysis predicts pd-l1 status from h&e-stained histopathology images in breast cancer. Nature Communications 13, 1–13 (2022).
  170. Naik, N. et al. Deep learning-enabled breast cancer hormonal receptor status determination from base-level h&e stains. Nature Communications 11 (2020).
  171. He, B. et al. Integrating spatial gene expression and breast tumour morphology via deep learning. Nature Biomedical Engineering 4, 827–834 (2020).
  172. Acosta, P. et al. Intratumoral resolution of driver gene mutation heterogeneity in renal cancer using deep learning. Cancer Research (2022).
  173. Investigating morphologic correlates of driver gene mutation heterogeneity via deep learning. Cancer Research 82, 2672–2673 (2022).
  174. Harder, N. et al. Automatic discovery of image-based signatures for ipilimumab response prediction in malignant melanoma. Scientific Reports 9 (2019).
  175. Hu, J. et al. Using deep learning to predict anti-PD-1 response in melanoma and lung cancer patients from histopathology images. Translational Oncology 14 (2021).
  176. Berry, S. et al. Analysis of multispectral imaging with the astropath platform informs efficacy of PD-1 blockade. Science 372 (2021).
  177. Farahmand, S. et al. Deep learning trained on hematoxylin and eosin tumor region of interest predicts her2 status and trastuzumab treatment response in her2+ breast cancer. Modern Pathology 35, 44–51 (2022).
  178. Bychkov, D. et al. Deep learning identifies morphological features in breast cancer predictive of cancer ERBB2 status and trastuzumab treatment efficacy. Scientific Reports 11 (2021).
  179. Li, F. et al. Deep learning-based predictive biomarker of pathological complete response to neoadjuvant chemotherapy from histological images in breast cancer. Journal of Translational Medicine 19 (2021).
  180. Sammut, S.-J. et al. Multi-omic machine learning predictor of breast cancer therapy response. Nature 601, 623–629 (2022).
  181. Vanguri, R. S. et al. Multimodal integration of radiology, pathology and genomics for prediction of response to pd-(l) 1 blockade in patients with non-small cell lung cancer. Nature Cancer (2022).
  182. Liu, R. et al. Systematic pan-cancer analysis of mutation–treatment interactions using large real-world clinicogenomics data. Nature Medicine (2022).
  183. Artificial intelligence in digital pathology—new tools for diagnosis and precision oncology. Nature reviews Clinical oncology 16, 703–715 (2019).
  184. Digital pathology and artificial intelligence. The lancet oncology 20, e253–e261 (2019).
  185. Kleppe, A. et al. Designing deep learning studies in cancer diagnostics. Nature Reviews Cancer 21, 199–211 (2021).
  186. Deep learning in histopathology: the path to the clinic. Nature Medicine 27, 775–784 (2021).
  187. Digital pathology and artificial intelligence in translational medicine and clinical practice. Modern Pathology 35, 23–32 (2022).
  188. Kalra, S. et al. Yottixel–an image search engine for large archives of histopathology whole slide images. Medical Image Analysis 65, 101757 (2020).
  189. Chen, C. et al. Fast and scalable search of whole-slide images via self-supervised deep learning. Nature Biomedical Engineering 1–15 (2022).
  190. Chen, R. J. et al. Multimodal co-attention transformer for survival prediction in gigapixel whole slide images. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 4015–4025 (2021).
  191. Singer, D. S. A new phase of the cancer moonshot to end cancer as we know it. Nature Medicine (2022).
  192. Kiemen, A. L. et al. Coda: quantitative 3d reconstruction of large tissues at cellular resolution. Nature Methods 1–10 (2022).
  193. Glaser, A. K. et al. Light-sheet microscopy for slide-free non-destructive pathology of large clinical specimens. Nature Biomedical Engineering 1 (2017).
  194. Fereidouni, F. et al. Microscopy with ultraviolet surface excitation for rapid slide-free histology. Nature Biomedical Engineering 1, 957–966 (2017).
  195. Katsamenis, O. L. et al. X-ray micro-computed tomography for nondestructive three-dimensional (3d) x-ray histology. The American Journal of Pathology 189, 1608–1620 (2019).
  196. Xie, W. et al. Prostate cancer risk stratification via non-destructive 3d pathology with deep learning-assisted gland analysis. Cancer Research 82 (2022).
  197. Multiplex bioimaging of single-cell spatial profiles for precision cancer diagnostics and therapeutics. NPJ precision oncology 4, 1–14 (2020).
  198. Wu, Z. et al. Graph deep learning for the characterization of tumour microenvironments from spatial protein profiles in tissue specimens. Nature Biomedical Engineering 1–14 (2022).
  199. Multiplex computational pathology for treatment response prediction. Cancer Cell 39, 1053–1055 (2021).
  200. Privacy in the age of medical big data. Nature Medicine 25, 37–43 (2019).
  201. Artificial intelligence in histopathology: enhancing cancer research and clinical oncology. Nature Cancer 3, 1026–1038 (2022).
  202. Shifting machine learning for healthcare from development to deployment and from models to data. Nature Biomedical Engineering (2022).
  203. Lu, M. Y. et al. Federated learning for computational pathology on gigapixel whole slide images. Medical image analysis 76, 102298 (2022).
  204. Federated learning and differential privacy for medical image analysis. Scientific Reports 12, 1–10 (2022).
  205. Continual lifelong learning with neural networks: A review. Neural Networks 113, 54–71 (2019).
  206. Saldanha, O. L. et al. Swarm learning for decentralized artificial intelligence in cancer histopathology. Nature Medicine 1–8 (2022).
  207. Dissecting racial bias in an algorithm used to manage the health of populations. Science 366, 447–453 (2019).
  208. munderdiagnosis bias of artificial intelligence algorithms applied to chest radiographs in under-served patient populations. Nature Medicine 27, 2176–2182 (2021).
  209. Addressing fairness in artificial intelligence for medical imaging. nature communications 13, 1–6 (2022).
  210. Synthetic data in machine learning for medicine and healthcare. Nature Biomedical Engineering 5, 493–497 (2021).
  211. Adaptive sensitive reweighting to mitigate bias in fairness-aware classification. In Proceedings of the 2018 world wide web conference, 853–862 (2018).
  212. Identifying and correcting label bias in machine learning. In International Conference on Artificial Intelligence and Statistics, 702–712 (2020).
  213. Dual-stream multiple instance learning network for whole slide image classification with self-supervised contrastive learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 14318–14328 (2021).
  214. Huang, Z. et al. Integration of patch features through self-supervised learning and transformer for survival analysis on whole slide images. In International Conference on Medical Image Computing and Computer-Assisted Intervention, 561–570 (2021).
  215. He, K. et al. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 16000–16009 (2022).
  216. Divide-and-rule: self-supervised learning for survival analysis in colorectal cancer. In International Conference on Medical Image Computing and Computer-Assisted Intervention, 480–489 (Springer, 2020).
  217. Self-supervised vision transformers learn visual concepts in histopathology. In Advances in Neural Information Processing Systems (NeurIPS) workshop (2022).
  218. Tellez, D. et al. Quantifying the effects of data augmentation and stain color normalization in convolutional neural networks for computational pathology. In Medical Image Analysis, vol. 58 (2019).
  219. Howard, F. et al. The impact of site-specific digital histology signatures on deep learning model accuracy and bias. Nature Communications 12 (2021).
  220. Deep ensembles: A loss landscape perspective. In Advances in Neural Information Processing Systems (NeurIPS) (2019).
  221. On calibration of modern neural networks. In International Conference on Machine Learning (ICML), 1321–1330 (2019).
  222. Generalisation effects of predictive uncertainty estimation in deep learning for digital pathology. Scientific Reports 12, 1–15 (2022).
  223. Dolezal, J. et al. Uncertainty-informed deep learning models enable high-confidence predictions for digital histopathology. Nature Communications 13 (2022).
  224. Predictive uncertainty estimation for out-of-distribution detection in digital pathology. Medical Image Analysis 83, 102655 (2023).
  225. Causality matters in medical imaging. Nature Communications 11 (2020).
  226. A unifying force for the realization of medical ai. npj Digital Medicine 5, 1–3 (2022).
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (7)
  1. Andrew H. Song (25 papers)
  2. Guillaume Jaume (29 papers)
  3. Drew F. K. Williamson (24 papers)
  4. Ming Y. Lu (23 papers)
  5. Anurag Vaidya (10 papers)
  6. Tiffany R. Miller (1 paper)
  7. Faisal Mahmood (53 papers)
Citations (86)
View on Semantic Scholar

Summary

Artificial Intelligence for Digital and Computational Pathology: An Overview

The reviewed paper presents a comprehensive examination of the integration of AI into digital and computational pathology (CPath). The authors focus on the emerging role of AI in improving diagnostic capabilities, prognosis predictions, and the discovery of novel biomarkers within whole-slide images (WSIs). By leveraging deep learning methodologies, CPath is advancing towards achieving clinical-grade performance and holds the potential for significant impacts on both clinical practice and biomedical research.

Methodological Advances in Computational Pathology

The paper categorizes AI methodologies in CPath into two primary frameworks: predicting clinical endpoints and developing assistive tools for pathologists. Importantly, the paper elaborates on multiple instance learning (MIL), increasingly adopted to predict disease-related outcomes from WSIs. In MIL, WSIs are divided into patches, and a global WSI prediction is made by aggregating these patches. This paradigm is especially beneficial given the high dimensionality of WSIs, as it allows models to leverage information across various magnification levels.

Moreover, the paper discusses context-aware approaches, emphasizing graph neural networks (GNNs) and transformers, which integrate context within patches to capture holistic tissue information. These novel methodological advances aim to overcome limitations posed by patch-level learning and are anticipated to be pivotal in tasks like prognosis prediction.

AI-based Assistive Tools and Interpretability

Within the field of assistive tools, the paper highlights advancements in segmentation and virtual staining. AI-driven segmentation algorithms are employed for tasks like nucleus and gland segregation, enabling quantitative morphological analyses critical for diagnosis. Meanwhile, virtual staining techniques enhance the standardization of tissue imaging across different institutions, addressing variability concerns.

The paper stresses the importance of interpretability in CPath systems, recognizing the need for pathologists to trust and understand AI decisions. Feature attribution methods and attention-based MIL offer insights into the relevant regions contributing to predictions. However, combining these qualitative insights with quantitative morphological characterizations remains crucial to ensuring the practicality and transparency of AI solutions.

Data Considerations and Methodological Challenges

The paper underscores the necessity of large, diverse, and multimodal datasets to refine AI models further. Initiatives such as The Cancer Genome Atlas (TCGA) provide valuable resources, but challenges related to data availability, quality, and integration persist. The authors point out that methodological improvements, including self-supervised learning and decentralized learning, are instrumental in building more robust and generalizable models that transcend institutional domain shifts.

Implications and Future Directions

The incorporation of AI into CPath carries significant implications for both clinical practice and research. Clinically, AI aims to automate routine pathology tasks, reducing interobserver variability and the manual workload. In research, AI facilitates biomarker discovery and builds more accurate models for predicting patient outcomes, thereby contributing to the precision medicine paradigm.

Future developments in CPath are anticipated to focus on further enhancing representation learning, leveraging 3D pathology, integrating next-generation imaging techniques, and ensuring fairness across diverse patient populations. These advancements promise to elevate CPath's role in clinical decision-making and biomedical discoveries.

Conclusion

The paper provides a thorough review of the current state and future directions of AI in computational pathology. It highlights the field's potential to transform diagnostic and prognostic processes and advance research through innovative AI methodologies. While challenges remain, ongoing efforts to develop better datasets and refine deep learning models offer promising avenues for integrating CPath into routine clinical workflows and biomedical research, paving the way for impactful, data-driven healthcare solutions.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown