Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
169 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Rethinking Multiple Instance Learning for Whole Slide Image Classification: A Good Instance Classifier is All You Need (2307.02249v2)

Published 5 Jul 2023 in cs.CV

Abstract: Weakly supervised whole slide image classification is usually formulated as a multiple instance learning (MIL) problem, where each slide is treated as a bag, and the patches cut out of it are treated as instances. Existing methods either train an instance classifier through pseudo-labeling or aggregate instance features into a bag feature through attention mechanisms and then train a bag classifier, where the attention scores can be used for instance-level classification. However, the pseudo instance labels constructed by the former usually contain a lot of noise, and the attention scores constructed by the latter are not accurate enough, both of which affect their performance. In this paper, we propose an instance-level MIL framework based on contrastive learning and prototype learning to effectively accomplish both instance classification and bag classification tasks. To this end, we propose an instance-level weakly supervised contrastive learning algorithm for the first time under the MIL setting to effectively learn instance feature representation. We also propose an accurate pseudo label generation method through prototype learning. We then develop a joint training strategy for weakly supervised contrastive learning, prototype learning, and instance classifier training. Extensive experiments and visualizations on four datasets demonstrate the powerful performance of our method. Codes are available at https://github.com/miccaiif/INS.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (55)
  1. L. Qu, S. Liu, X. Liu, M. Wang, and Z. Song, “Towards label-efficient automatic diagnosis and analysis: a comprehensive survey of advanced deep learning-based weakly-supervised, semi-supervised and self-supervised techniques in histopathological image analysis,” Physics in Medicine & Biology, 2022.
  2. C. L. Srinidhi, O. Ciga, and A. L. Martel, “Deep neural network models for computational histopathology: A survey,” Medical Image Analysis, vol. 67, p. 101813, 2021.
  3. M. Y. Lu, T. Y. Chen, D. F. Williamson, M. Zhao, M. Shady, J. Lipkova, and F. Mahmood, “Ai-based pathology predicts origins for cancers of unknown primary,” Nature, vol. 594, no. 7861, pp. 106–110, 2021.
  4. F. Mahmood, D. Borders, R. J. Chen, G. N. McKay, K. J. Salimian, A. Baras, and N. J. Durr, “Deep adversarial training for multi-organ nuclei segmentation in histopathology images,” IEEE Transactions on Medical Imaging, vol. 39, no. 11, pp. 3257–3267, 2019.
  5. M. Y. Lu, R. J. Chen, D. Kong, J. Lipkova, R. Singh, D. F. Williamson, T. Y. Chen, and F. Mahmood, “Federated learning for computational pathology on gigapixel whole slide images,” Medical Image Analysis, vol. 76, p. 102298, 2022.
  6. X. Wang, D. Cai, S. Yang, Y. Cui, J. Zhu, K. Wang, and J. Zhao, “Sac-net: Enhancing spatiotemporal aggregation in cervical histological image classification via label-efficient weakly supervised learning,” IEEE Transactions on Circuits and Systems for Video Technology, 2023.
  7. L. Qu, K. Fu, M. Wang, Z. Song et al., “The rise of ai language pathologists: Exploring two-level prompt learning for few-shot weakly-supervised whole slide image classification,” Advances in Neural Information Processing Systems, vol. 36, 2024.
  8. J. Rony, S. Belharbi, J. Dolz, I. B. Ayed, L. McCaffrey, and E. Granger, “Deep weakly-supervised learning methods for classification and localization in histology images: a survey,” arXiv preprint arXiv:1909.03354, 2019.
  9. V. Cheplygina, M. de Bruijne, and J. P. Pluim, “Not-so-supervised: a survey of semi-supervised, multi-instance, and transfer learning in medical image analysis,” Medical Image Analysis, vol. 54, pp. 280–296, 2019.
  10. R. J. Chen, M. Y. Lu, J. Wang, D. F. Williamson, S. J. Rodig, N. I. Lindeman, and F. Mahmood, “Pathomic fusion: an integrated framework for fusing histopathology and genomic features for cancer diagnosis and prognosis,” IEEE Transactions on Medical Imaging, vol. 41, no. 4, pp. 757–770, 2020.
  11. M. Y. Lu, R. J. Chen, J. Wang, D. Dillon, and F. Mahmood, “Semi-supervised histology classification using deep multiple instance learning and contrastive predictive coding,” arXiv preprint arXiv:1910.10825, 2019.
  12. M. Y. Lu, D. F. Williamson, T. Y. Chen, R. J. Chen, M. Barbieri, and F. Mahmood, “Data-efficient and weakly supervised computational pathology on whole-slide images,” Nature Biomedical Engineering, vol. 5, no. 6, pp. 555–570, 2021.
  13. L. Qu, X. Luo, S. Liu, M. Wang, and Z. Song, “Dgmil: Distribution guided multiple instance learning for whole slide image classification,” in Medical Image Computing and Computer Assisted Intervention (MICCAI).   Springer, 2022, pp. 24–34.
  14. L. Qu, X. Luo, M. Wang, and Z. Song, “Bi-directional weakly supervised knowledge distillation for whole slide image classification,” Advances in Neural Information Processing Systems (NeurIPS), vol. 35, pp. 15 368–15 381, 2022.
  15. G. Campanella, M. G. Hanna, L. Geneslaw, A. Miraflor, V. Werneck Krauss Silva, K. J. Busam, E. Brogi, V. E. Reuter, D. S. Klimstra, and T. J. Fuchs, “Clinical-grade computational pathology using weakly supervised deep learning on whole slide images,” Nature Medicine, vol. 25, no. 8, pp. 1301–1309, 2019.
  16. P. Chikontwe, M. Kim, S. J. Nam, H. Go, and S. H. Park, “Multiple instance learning with center embeddings for histopathology classification,” in Medical Image Computing and Computer Assisted Intervention (MICCAI).   Springer, 2020, pp. 519–528.
  17. F. Kanavati, G. Toyokawa, S. Momosaki, M. Rambeau, Y. Kozuma, F. Shoji, K. Yamazaki, S. Takeo, O. Iizuka, and M. Tsuneki, “Weakly-supervised learning for lung carcinoma classification using deep learning,” Scientific Reports, vol. 10, no. 1, p. 9297, 2020.
  18. M. Ilse, J. Tomczak, and M. Welling, “Attention-based deep multiple instance learning,” in International Conference on Machine Learning (ICML).   PMLR, 2018, pp. 2127–2136.
  19. N. Hashimoto, D. Fukushima, R. Koga, Y. Takagi, K. Ko, K. Kohno, M. Nakaguro, S. Nakamura, H. Hontani, and I. Takeuchi, “Multi-scale domain-adversarial multiple-instance cnn for cancer subtype classification with unannotated histopathological images,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 3852–3861.
  20. J. Yao, X. Zhu, J. Jonnagaddala, N. Hawkins, and J. Huang, “Whole slide images based cancer survival prediction using attention guided deep multiple instance learning networks,” Medical Image Analysis, vol. 65, p. 101789, 2020.
  21. X. Shi, F. Xing, Y. Xie, Z. Zhang, L. Cui, and L. Yang, “Loss-based attention for deep multiple instance learning,” in Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), vol. 34, no. 04, 2020, pp. 5742–5749.
  22. B. Li, Y. Li, and K. W. Eliceiri, “Dual-stream multiple instance learning network for whole slide image classification with self-supervised contrastive learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 14 318–14 328.
  23. H. Zhang, Y. Meng, Y. Zhao, Y. Qiao, X. Yang, S. E. Coupland, and Y. Zheng, “Dtfd-mil: Double-tier feature distillation multiple instance learning for histopathology whole slide image classification,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 18 802–18 812.
  24. Z. Shao, H. Bian, Y. Chen, Y. Wang, J. Zhang, X. Ji et al., “Transmil: Transformer based correlated multiple instance learning for whole slide image classification,” Advances in Neural Information Processing Systems (NeurIPS), vol. 34, pp. 2136–2147, 2021.
  25. R. J. Chen, M. Y. Lu, W.-H. Weng, T. Y. Chen, D. F. Williamson, T. Manz, M. Shady, and F. Mahmood, “Multimodal co-attention transformer for survival prediction in gigapixel whole slide images,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021, pp. 4015–4025.
  26. A. Myronenko, Z. Xu, D. Yang, H. R. Roth, and D. Xu, “Accounting for dependencies in deep learning based multiple instance learning for whole slide imaging,” in Medical Image Computing and Computer Assisted Intervention (MICCAI).   Springer, 2021, pp. 329–338.
  27. X. Wang, J. Xiang, J. Zhang, S. Yang, Z. Yang, M.-H. Wang, J. Zhang, W. Yang, J. Huang, and X. Han, “Scl-wc: Cross-slide contrastive learning for weakly-supervised whole-slide image classification,” Advances in Neural Information Processing Systems (NeurIPS), vol. 35, pp. 18 009–18 021, 2022.
  28. Y. Yan, X. Wang, X. Guo, J. Fang, W. Liu, and J. Huang, “Deep multi-instance learning with dynamic pooling,” in Asian Conference on Machine Learning (ACML).   PMLR, 2018, pp. 662–677.
  29. X. Wang, Y. Yan, P. Tang, X. Bai, and W. Liu, “Revisiting multiple instance neural networks,” Pattern Recognition, vol. 74, pp. 15–24, 2018.
  30. O. Z. Kraus, J. L. Ba, and B. J. Frey, “Classifying and segmenting microscopy images with deep multiple instance learning,” Bioinformatics, vol. 32, no. 12, pp. i52–i59, 2016.
  31. O. Maron and T. Lozano-Pérez, “A framework for multiple-instance learning,” Advances in Neural Information Processing Systems (NeurIPS), vol. 10, 1997.
  32. Q. Guo, L. Qu, J. Zhu, H. Li, Y. Wu, S. Wang, M. Yu, J. Wu, H. Wen, X. Ju et al., “Predicting lymph node metastasis from primary cervical squamous cell carcinoma based on deep learning in histopathologic images,” Modern Pathology, vol. 36, no. 12, p. 100316, 2023.
  33. Z. Zhu, L. Yu, W. Wu, R. Yu, D. Zhang, and L. Wang, “Murcl: Multi-instance reinforcement contrastive learning for whole slide image classification,” IEEE Transactions on Medical Imaging, 2022.
  34. Y. Zheng, R. H. Gindra, E. J. Green, E. J. Burks, M. Betke, J. E. Beane, and V. B. Kolachalama, “A graph-transformer for whole slide image classification,” IEEE Transactions on Medical Imaging, vol. 41, no. 11, pp. 3003–3015, 2022.
  35. W. Hou, C. Lin, L. Yu, J. Qin, R. Yu, and L. Wang, “Hybrid graph convolutional network with online masked autoencoder for robust multimodal cancer survival prediction,” IEEE Transactions on Medical Imaging, 2023.
  36. J.-G. Yu, Z. Wu, Y. Ming, S. Deng, Q. Wu, Z. Xiong, T. Yu, G.-S. Xia, Q. Jiang, and Y. Li, “Bayesian collaborative learning for whole-slide image classification,” IEEE Transactions on Medical Imaging, 2023.
  37. R. J. Chen, C. Chen, Y. Li, T. Y. Chen, A. D. Trister, R. G. Krishnan, and F. Mahmood, “Scaling vision transformers to gigapixel images via hierarchical self-supervised learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 16 144–16 155.
  38. R. J. Chen and R. G. Krishnan, “Self-supervised vision transformers learn visual concepts in histopathology,” arXiv preprint arXiv:2203.00585, 2022.
  39. H. Cai, X. Feng, R. Yin, Y. Zhao, L. Guo, X. Fan, and J. Liao, “Mist: multiple instance learning network based on swin transformer for whole slide image classification of colorectal adenomas,” The Journal of Pathology, vol. 259, no. 2, pp. 125–135, 2023.
  40. T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” in International Conference on Machine Learning (ICML).   PMLR, 2020, pp. 1597–1607.
  41. K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick, “Momentum contrast for unsupervised visual representation learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 9729–9738.
  42. M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin, “Emerging properties in self-supervised vision transformers,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021, pp. 9650–9660.
  43. Z. Chen, K.-Y. Lin, and W.-S. Zheng, “Consistent intra-video contrastive learning with asynchronous long-term memory bank,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 33, no. 3, pp. 1168–1180, 2022.
  44. Y. Wei, L. Yang, Y. Han, and Q. Hu, “Multi-source collaborative contrastive learning for decentralized domain adaptation,” IEEE Transactions on Circuits and Systems for Video Technology, 2022.
  45. Y. Zhu, H. Shuai, G. Liu, and Q. Liu, “Self-supervised video representation learning using improved instance-wise contrastive learning and deep clustering,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 32, no. 10, pp. 6741–6752, 2022.
  46. L. Tao, X. Wang, and T. Yamasaki, “An improved inter-intra contrastive learning framework on self-supervised video representation,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 32, no. 8, pp. 5266–5280, 2022.
  47. X. Ding, B. Li, Y. Li, W. Guo, Y. Liu, W. Xiong, and W. Hu, “Web objectionable video recognition based on deep multi-instance learning with representative prototypes selection,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 31, no. 3, pp. 1222–1233, 2020.
  48. F. Zhou, S. Huang, B. Liu, and D. Yang, “Multi-label image classification via category prototype compositional learning,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 32, no. 7, pp. 4513–4525, 2021.
  49. H. Yang, B. Sun, B. Li, C. Yang, Z. Wang, J. Chen, L. Wang, and H. Li, “Iterative class prototype calibration for transductive zero-shot learning,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 33, no. 3, pp. 1236–1246, 2022.
  50. Y. Su, X. Xu, and K. Jia, “Weakly supervised 3d point cloud segmentation via multi-prototype learning,” IEEE Transactions on Circuits and Systems for Video Technology, 2023.
  51. D. Rymarczyk, A. Pardyl, J. Kraus, A. Kaczyńska, M. Skomorowski, and B. Zieliński, “Protomil: multiple instance learning with prototypical parts for whole-slide image classification,” in Joint European Conference on Machine Learning and Knowledge Discovery in Databases.   Springer, 2022, pp. 421–436.
  52. J.-G. Yu, Z. Wu, Y. Ming, S. Deng, Y. Li, C. Ou, C. He, B. Wang, P. Zhang, and Y. Wang, “Prototypical multiple instance learning for predicting lymph node metastasis of breast cancer from whole-slide pathological images,” Medical Image Analysis, p. 102748, 2023.
  53. L. Yang, D. Mehta, S. Liu, D. Mahapatra, A. Di Ieva, and Z. Ge, “Tpmil: Trainable prototype enhanced multiple instance learning for whole slide image classification,” arXiv:2305.00696, 2023.
  54. H. Wang, R. Xiao, Y. Li, L. Feng, G. Niu, G. Chen, and J. Zhao, “Pico: Contrastive label disambiguation for partial label learning,” arXiv preprint arXiv:2201.08984, 2022.
  55. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).   IEEE, 2009, pp. 248–255.
Citations (11)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now