Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
184 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

SI-MIL: Taming Deep MIL for Self-Interpretability in Gigapixel Histopathology (2312.15010v2)

Published 22 Dec 2023 in cs.CV

Abstract: Introducing interpretability and reasoning into Multiple Instance Learning (MIL) methods for Whole Slide Image (WSI) analysis is challenging, given the complexity of gigapixel slides. Traditionally, MIL interpretability is limited to identifying salient regions deemed pertinent for downstream tasks, offering little insight to the end-user (pathologist) regarding the rationale behind these selections. To address this, we propose Self-Interpretable MIL (SI-MIL), a method intrinsically designed for interpretability from the very outset. SI-MIL employs a deep MIL framework to guide an interpretable branch grounded on handcrafted pathological features, facilitating linear predictions. Beyond identifying salient regions, SI-MIL uniquely provides feature-level interpretations rooted in pathological insights for WSIs. Notably, SI-MIL, with its linear prediction constraints, challenges the prevalent myth of an inevitable trade-off between model interpretability and performance, demonstrating competitive results compared to state-of-the-art methods on WSI-level prediction tasks across three cancer types. In addition, we thoroughly benchmark the local and global-interpretability of SI-MIL in terms of statistical analysis, a domain expert study, and desiderata of interpretability, namely, user-friendliness and faithfulness.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (70)
  1. The cancer genome atlas (tcga) research network. https://www.cancer.gov/tcga.
  2. Vl-interpret: An interactive visualization tool for interpreting vision-language transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21406–21415, 2022.
  3. Radiology data from the cancer genome atlas lung adenocarcinoma [tcga-luad] collection. The Cancer Imaging Archive, 2016.
  4. Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information fusion, 58:82–115, 2020.
  5. Interpretable neural-symbolic concept reasoning. arXiv preprint arXiv:2304.14068, 2023.
  6. Development and validation of a weakly supervised deep learning framework to predict the status of molecular pathways and key mutations in colorectal cancer from routine histology images: a retrospective study. The Lancet Digital Health, 3(12):e763–e772, 2021.
  7. From what to why, the growing need for a focus shift toward explainability of ai in digital pathology. Frontiers in Physiology, 12:821217, 2022.
  8. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650–9660, 2021.
  9. Scaling vision transformers to gigapixel images via hierarchical self-supervised learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16144–16155, 2022a.
  10. Pan-cancer integrative histology-genomic analysis via multimodal deep learning. Cancer Cell, 40(8):865–878, 2022b.
  11. Differentiable patch selection for image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2351–2360, 2021.
  12. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  13. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
  14. Concept embedding models: Beyond the accuracy-explainability trade-off. Advances in Neural Information Processing Systems, 35:21400–21413, 2022.
  15. Interpretable deep learning model to predict the molecular classification of endometrial cancer from haematoxylin and eosin-stained whole-slide images: a combined analysis of the portec randomised trials and clinical cohorts. The Lancet Digital Health, 5(2):e71–e82, 2023.
  16. Pannuke: an open pan-cancer histology dataset for nuclei instance segmentation and classification. In Digital Pathology: 15th European Congress, ECDP 2019, Warwick, UK, April 10–13, 2019, Proceedings 15, pages 11–19. Springer, 2019.
  17. Hover-net: Simultaneous segmentation and classification of nuclei in multi-tissue histology images. Medical image analysis, 58:101563, 2019.
  18. A global taxonomy of interpretable ai: unifying the terminology for the technical and social sciences. Artificial intelligence review, 56(4):3473–3504, 2023.
  19. Resolving challenges in deep learning-based analyses of histopathological images using explanation methods. Scientific reports, 10(1):6423, 2020.
  20. A visual–language foundation model for pathology image analysis using medical twitter. Nature medicine, pages 1–10, 2023.
  21. Attention-based deep multiple instance learning. In International conference on machine learning, pages 2127–2136. PMLR, 2018.
  22. Towards explainable graph representations in digital pathology. arXiv preprint arXiv:2007.00311, 2020.
  23. Quantifying explainers of graph neural networks in computational pathology. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8106–8116, 2021.
  24. Additive mil: intrinsically interpretable multiple instance learning for pathology. Advances in Neural Information Processing Systems, 35:20689–20702, 2022.
  25. Crowds cure cancer: Data collected at the rsna 2017 annual meeting. The Cancer Imaging Archive. DOI, 10:K9.
  26. Attention de-sparsification matters: Inducing diversity in digital pathology representation learning. arXiv preprint arXiv:2309.06439, 2023.
  27. Kenji Kawaguchi. Deep learning without poor local minima. Advances in neural information processing systems, 29, 2016.
  28. Radiology data from the cancer genome atlas lung squamous cell carcinoma [tcga-lusc] collection. The Cancer Imaging Archive, 2016.
  29. Concept bottleneck models. In International conference on machine learning, pages 5338–5348. PMLR, 2020.
  30. Deep linear networks with arbitrary loss: All local minima are global. In International conference on machine learning, pages 2902–2907. PMLR, 2018.
  31. Dual-stream multiple instance learning network for whole slide image classification with self-supervised contrastive learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14318–14328, 2021.
  32. Llava-med: Training a large language-and-vision assistant for biomedicine in one day. arXiv preprint arXiv:2306.00890, 2023.
  33. Radiology data from the cancer genome atlas breast invasive carcinoma [tcga-brca] collection. The Cancer Imaging Archive, 10:K9, 2016.
  34. Comparative molecular analysis of gastrointestinal adenocarcinomas. Cancer cell, 33(4):721–735, 2018.
  35. Feature-driven local cell graph (flock): new computational pathology-based descriptors for prognosis of lung cancer and hpv status of oropharyngeal cancers. Medical image analysis, 68:101903, 2021a.
  36. Data-efficient and weakly supervised computational pathology on whole-slide images. Nature biomedical engineering, 5(6):555–570, 2021b.
  37. Towards a visual-language foundation model for computational pathology. arXiv preprint arXiv:2307.12914, 2023a.
  38. Visual language pretrained multiple instance zero-shot transfer for histopathology images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19764–19775, 2023b.
  39. Image analysis and machine learning in digital pathology: Challenges and opportunities. Medical image analysis, 33:170–175, 2016.
  40. Promises and pitfalls of black-box concept learning models. arXiv preprint arXiv:2106.13314, 2021.
  41. Do concept bottleneck models learn as intended? arXiv preprint arXiv:2105.04289, 2021.
  42. Athena: analysis of tumor heterogeneity from spatial omics measurements. Bioinformatics, 38(11):3151–3153, 2022.
  43. Interpretable deep learning of myelin histopathology in age-related cognitive impairment. Acta Neuropathologica Communications, 10(1):131, 2022.
  44. Cancer Genome Atlas Network et al. Comprehensive molecular characterization of human colon and rectal cancer. Nature, 487(7407):330, 2012.
  45. The loss surface of deep and wide neural networks. In International conference on machine learning, pages 2603–2612. PMLR, 2017.
  46. Label-free concept bottleneck models. arXiv preprint arXiv:2304.06129, 2023.
  47. Weakly supervised joint whole-slide segmentation and classification in prostate cancer. Medical Image Analysis, 89:102915, 2023.
  48. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  49. Abtin Riasatian. Kimianet: Training a deep network for histopathology using high-cellularity. Master’s thesis, University of Waterloo, 2020.
  50. Peter J Rousseeuw. Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. Journal of computational and applied mathematics, 20:53–65, 1987.
  51. Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature machine intelligence, 1(5):206–215, 2019.
  52. Protomil: Multiple instance learning with prototypical parts for whole-slide image classification. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 421–436, 2022.
  53. Pixel-level explanation of multiple instance learning models in biomedical single cell images. In International Conference on Information Processing in Medical Imaging, pages 170–182. Springer, 2023.
  54. Transparency of deep neural networks for medical image analysis: A review of interpretability methods. Computers in biology and medicine, 140:105111, 2022.
  55. Tumor-infiltrating lymphocytes maps from tcga h&e whole slide pathology images [data set]. Cancer Imaging Arch, 2018.
  56. Transmil: Transformer based correlated multiple instance learning for whole slide image classification. Advances in neural information processing systems, 34:2136–2147, 2021.
  57. Visualization for histopathology images using graph convolutional neural networks. In 2020 IEEE 20th international conference on bioinformatics and bioengineering (BIBE), pages 331–335. IEEE, 2020.
  58. Differentiable zooming for multiple instance learning on whole-slide images. In European Conference on Computer Vision, pages 699–715. Springer, 2022.
  59. Mlp-mixer: An all-mlp architecture for vision. Advances in neural information processing systems, 34:24261–24272, 2021.
  60. Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
  61. Virchow: A million-slide digital pathology foundation model. arXiv preprint arXiv:2309.07778, 2023.
  62. Transformer-based unsupervised contrastive learning for histopathological image classification. Medical image analysis, 81:102559, 2022.
  63. Retccl: clustering-guided contrastive learning for whole-slide image retrieval. Medical image analysis, 83:102645, 2023.
  64. Language in a bottle: Language model guided concept bottlenecks for interpretable image classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19187–19197, 2023.
  65. Post-hoc concept bottleneck models. arXiv preprint arXiv:2205.15480, 2022.
  66. Cells are actors: Social network analysis with classical ml for sota histology image classification. In Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part VIII 24, pages 288–298. Springer, 2021.
  67. A joint spatial and magnification based attention framework for large scale histopathology classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3776–3784, 2021.
  68. Prompt-mil: Boosting multi-instance learning schemes via task-specific prompt tuning. arXiv preprint arXiv:2303.12214, 2023.
  69. Mdnet: A semantically and visually interpretable medical image diagnosis network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6428–6436, 2017.
  70. A graph-transformer for whole slide image classification. IEEE transactions on medical imaging, 41(11):3003–3015, 2022.
Citations (4)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets