Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
97 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
5 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Attention to detail: inter-resolution knowledge distillation (2401.06010v1)

Published 11 Jan 2024 in cs.CV

Abstract: The development of computer vision solutions for gigapixel images in digital pathology is hampered by significant computational limitations due to the large size of whole slide images. In particular, digitizing biopsies at high resolutions is a time-consuming process, which is necessary due to the worsening results from the decrease in image detail. To alleviate this issue, recent literature has proposed using knowledge distillation to enhance the model performance at reduced image resolutions. In particular, soft labels and features extracted at the highest magnification level are distilled into a model that takes lower-magnification images as input. However, this approach fails to transfer knowledge about the most discriminative image regions in the classification process, which may be lost when the resolution is decreased. In this work, we propose to distill this information by incorporating attention maps during training. In particular, our formulation leverages saliency maps of the target class via grad-CAMs, which guides the lower-resolution Student model to match the Teacher distribution by minimizing the l2 distance between them. Comprehensive experiments on prostate histology image grading demonstrate that the proposed approach substantially improves the model performance across different image resolutions compared to previous literature.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (21)
  1. G. Campanella, M. G. Hanna, L. Geneslaw, A. Miraflor, V. W. K. Silva, K. J. Busam, E. Brogi, V. E. Reuter, D. S. Klimstra, and T. J. Fuchs, “Clinical-grade computational pathology using weakly supervised deep learning on whole slide images,” Nature Medicine, vol. 25, pp. 1301–1309, 8 2019.
  2. J. Silva-Rodríguez, A. Colomer, and V. Naranjo, “Weglenet: A weakly-supervised convolutional neural network for the semantic segmentation of gleason grades in prostate histology images,” Computerized Medical Imaging and Graphics, vol. 88, pp. 1–10, 3 2021.
  3. D. C. Ciresan, A. Giusti, L. M. Gambardella, and J. Schmidhuber, “Mitosis detection in breast cancer histology images with deep neural networks,” Medical Image Computing and Computer Assisted Intervention (MICCAI), pp. 411–418, 2013.
  4. J. DiPalma, A. A. Suriawinata, L. J. Tafe, L. Torresani, and S. Hassanpour, “Resolution-based distillation for efficient histology image classification,” Artificial Intelligence in Medicine, vol. 119, 9 2021.
  5. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, “Attention is all you need,” 2017.
  6. R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, “Grad-cam: Visual explanations from deep networks via gradient-based localization,” Proceedings of the IEEE International Conference on Computer Vision (ICCV), vol. 128, 10 2017.
  7. D. Pathak, P. Krähenbühl, and T. Darrell, “Constrained convolutional neural networks for weakly supervised segmentation,” Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 1–12, 6 2015.
  8. J. Silva-Rodrguez, A. Schmidt, M. A. Sales, R. Molina, and V. Naranjo, “Proportion constrained weakly supervised histopathology image classification,” Computers in Biology and Medicine, vol. 147, 2022.
  9. R. del Amor, P. Meseguer, T. L. Parigi, V. Villanacci, A. Colomer, L. Launet, A. Bazarova, G. E. Tontini, R. Bisschops, G. de Hertogh, J. G. Ferraz, M. Götz, X. Gui, B. H. Hayee, M. Lazarev, R. Panaccione, A. Parra-Blanco, P. Bhandari, L. Pastorelli, T. Rath, E. S. Røyset, M. Vieth, D. Zardo, E. Grisan, S. Ghosh, M. Iacucci, and V. Naranjo, “Constrained multiple instance learning for ulcerative colitis prediction using histological images,” Computer Methods and Programs in Biomedicine, vol. 224, 9 2022.
  10. J. Silva-Rodríguez and J. Dolz, “Looking at the whole picture: constrained unsupervised anomaly segmentation,” British Machine Vision Conference (BMVC), 2021.
  11. J. Silva-Rodríguez, V. Naranjo, and J. Dolz, “Constrained unsupervised anomaly segmentation,” Medical Image Analysis, vol. 80, 2022.
  12. S. Venkataramanan, K. C. Peng, R. V. Singh, and A. Mahalanobis, “Attention guided anomaly localization in images,” Proceedings of the European Conference on Computer Vision (ECCV), 2020.
  13. R. R. Selvaraju, K. Desai, J. Johnson, and N. Naik, “Casting your model: Learning to localize improves self-supervised representations,” Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  14. M. Sun, Y. Yuan, F. Zhou, and E. Ding, “Multi-attention multi-class constraint for fine-grained image recognition,” Proceedings of the European COnference on Computer Vision (ECCV), 6 2018.
  15. G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” Advances in Neural Information Processing Systems (NeurIPS), 3 2015.
  16. A. Romero, N. Ballas, S. E. Kahou, A. Chassang, C. Gatta, and Y. Bengio, “Fitnets: Hints for thin deep nets,” International Conference on Learning Representations (ICLR), 12 2015.
  17. J. Yang, B. Marinez, A. Bulat, and G. Tzimiropoulos, “Knowledge distillation via softmax regression representation learning,” International Conference on Learning Representations (ICLR), 2021.
  18. S. Zagoruyko and N. Komodakis, “Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer,” International Conference on Learning Representations (ICLR), 12 2017.
  19. Q. Xie, M.-T. Luong, E. Hovy, and Q. V. Le, “Self-training with noisy student improves imagenet classification,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 11 2020.
  20. M. Hu, M. Maillard, Y. Zhang, T. Ciceri, G. Barbera, I. Bloch, P. Gori, and G. L. Barbera, “Knowledge distillation from multi-modal to mono-modal segmentation networks,” Medical Image Computing and Computer Assisted Intervention (MICCAI), 2020.
  21. J. Silva-Rodríguez, A. Colomer, M. Sales, R. Molina, and V. Naranjo, “Going deeper through the gleason scoring scale: An automatic end-to-end system for histology prostate grading and cribriform pattern detection,” Computer Methods and Programs in Biomedicine, vol. 195, 2020.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (4)
  1. Rocío del Amor (12 papers)
  2. Adrián Colomer (20 papers)
  3. Valery Naranjo (32 papers)
  4. Julio Silva-Rodríguez (20 papers)

Summary

We haven't generated a summary for this paper yet.