Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
102 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

ToNNO: Tomographic Reconstruction of a Neural Network's Output for Weakly Supervised Segmentation of 3D Medical Images (2404.13103v1)

Published 19 Apr 2024 in eess.IV, cs.CV, and cs.LG

Abstract: Annotating lots of 3D medical images for training segmentation models is time-consuming. The goal of weakly supervised semantic segmentation is to train segmentation models without using any ground truth segmentation masks. Our work addresses the case where only image-level categorical labels, indicating the presence or absence of a particular region of interest (such as tumours or lesions), are available. Most existing methods rely on class activation mapping (CAM). We propose a novel approach, ToNNO, which is based on the Tomographic reconstruction of a Neural Network's Output. Our technique extracts stacks of slices with different angles from the input 3D volume, feeds these slices to a 2D encoder, and applies the inverse Radon transform in order to reconstruct a 3D heatmap of the encoder's predictions. This generic method allows to perform dense prediction tasks on 3D volumes using any 2D image encoder. We apply it to weakly supervised medical image segmentation by training the 2D encoder to output high values for slices containing the regions of interest. We test it on four large scale medical image datasets and outperform 2D CAM methods. We then extend ToNNO by combining tomographic reconstruction with CAM methods, proposing Averaged CAM and Tomographic CAM, which obtain even better results.

View on arXiv
Definition Search Book Streamline Icon: https://streamlinehq.com
References (65)
  1. Learning pixel-level semantic affinity with image-level supervision for weakly supervised semantic segmentation. In CVPR, pages 4981–4990. Computer Vision Foundation / IEEE Computer Society, 2018.
  2. What’s the point: Semantic segmentation with point supervision. In ECCV (7), pages 549–565. Springer, 2016.
  3. Interactive graph cuts for optimal boundary and region segmentation of objects in N-D images. In ICCV, pages 105–112. IEEE Computer Society, 2001.
  4. Orthogonal annotation benefits barely-supervised medical image segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pages 3302–3311, 2023.
  5. Grad-cam++: Generalized gradient-based visual explanations for deep convolutional networks. In WACV, pages 839–847. IEEE Computer Society, 2018.
  6. AME-CAM: attentive multiple-exit CAM for weakly supervised segmentation on MRI brain tumor. In MICCAI (1), pages 173–182. Springer, 2023.
  7. C-CAM: causal CAM for weakly supervised semantic segmentation on medical image. In CVPR, pages 11666–11675. IEEE, 2022.
  8. Box2mask: Weakly supervised 3d semantic instance segmentation using bounding boxes. In ECCV (31), pages 681–699. Springer, 2022.
  9. Pulmonary vessel segmentation based on orthogonal fused u-net++ of chest CT images. In Medical Image Computing and Computer Assisted Intervention - MICCAI 2019 - 22nd International Conference, Shenzhen, China, October 13-17, 2019, Proceedings, Part VI, pages 293–300. Springer, 2019.
  10. Boxsup: Exploiting bounding boxes to supervise convolutional networks for semantic segmentation. In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pages 1635–1643, 2015.
  11. Imagenet: A large-scale hierarchical image database. 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255, 2009.
  12. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
  13. J. Frank. Electron Tomography: Methods for Three-Dimensional Visualization of Structures in the Cell. Springer New York, 2008.
  14. Weakly supervised segmentation of tumor lesions in PET-CT hybrid imaging. J. Med. Imaging (Bellingham), 8(5):054003, 2021.
  15. A whole-body FDG-PET/CT dataset with manually annotated tumor lesions, 2022.
  16. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  17. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9729–9738, 2020.
  18. Sigurdur Helgason. The Radon Transform. Springer US, 1999.
  19. Olivier J. Hénaff. Data-efficient image recognition with contrastive predictive coding. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, pages 4182–4192. PMLR, 2020.
  20. Fastsurfer - A fast and accurate deep learning based neuroimaging pipeline. NeuroImage, 219:117012, 2020.
  21. G.T. Herman. Fundamentals of Computerized Tomography: Image Reconstruction from Projections. Springer London, 2009.
  22. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 448–456. PMLR, 2015.
  23. Layercam: Exploring hierarchical class activation maps for localization. IEEE Transactions on Image Processing, 30:5875–5888, 2021.
  24. Simple does it: Weakly supervised instance and semantic segmentation. In CVPR, pages 1665–1674. IEEE Computer Society, 2017.
  25. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  26. VesselNet: A deep convolutional neural network with multi pathways for robust hepatic vessel segmentation. Computerized Medical Imaging and Graphics, 75:74–83, 2019.
  27. Seed, expand and constrain: Three principles for weakly-supervised image segmentation. In ECCV (4), pages 695–711. Springer, 2016.
  28. Scribblesup: Scribble-supervised convolutional networks for semantic segmentation. In CVPR, pages 3159–3167. IEEE Computer Society, 2016.
  29. Swin transformer: Hierarchical vision transformer using shifted windows. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pages 9992–10002. IEEE, 2021.
  30. A convnet for the 2020s. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 11966–11976. IEEE, 2022.
  31. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983, 2016.
  32. V-net: Fully convolutional neural networks for volumetric medical image segmentation. In 3DV, pages 565–571. IEEE Computer Society, 2016.
  33. Mosmeddata: Chest ct scans with covid-19 related findings dataset. arXiv preprint arXiv:2005.06465, 2020.
  34. Eigen-cam: Class activation map using principal components. In IJCNN, pages 1–7. IEEE, 2020.
  35. Heatmap regression for lesion detection using pointwise annotations. In MILLanD@MICCAI, pages 3–12. Springer, 2022.
  36. Weakly-and semi-supervised learning of a deep convolutional network for semantic image segmentation. In ICCV, pages 1742–1750. IEEE Computer Society, 2015.
  37. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 8024–8035, 2019.
  38. Pedro H. O. Pinheiro and Ronan Collobert. From image-level to pixel-level labeling with convolutional networks. In CVPR, pages 1713–1721. IEEE Computer Society, 2015.
  39. J. Radon. Über die Bestimmung von Funktionen durch ihre Integralwerte längs gewisser Mannigfaltigkeiten. Berichte über die Verhandlungen der Sächsische Akademie der Wissenschaften, 69:262–277, 1917. 00000.
  40. icobrain ms 5.1: Combining unsupervised and supervised approaches for improving the detection of multiple sclerosis lesions. NeuroImage: Clinical, 31:102707, 2021.
  41. Towards automated polyp segmentation using weakly- and semi-supervised learning and deformable transformers. In CVPR Workshops, pages 4355–4364. IEEE, 2023.
  42. "grabcut": interactive foreground extraction using iterated graph cuts. ACM Trans. Graph., 23(3):309–314, 2004.
  43. Quicknat: A fully convolutional network for quick and accurate segmentation of neuroanatomy. NeuroImage, 186:713–727, 2019.
  44. A machine learning approach to radiogenomics of breast cancer: a study of 922 subjects and 529 DCE-MRI features. British Journal of Cancer, 119(4):508–516, 2018.
  45. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision, pages 618–626, 2017.
  46. 3d guided weakly supervised semantic segmentation. In ACCV (1), pages 585–602. Springer, 2020.
  47. Weakly supervised semantic segmentation for MRI: exploring the advantages and disadvantages of class activation maps for biological image segmentation with soft boundaries. Scientific Reports, 13(1), 2023.
  48. A review of deep learning CT reconstruction: Concepts, limitations, and promise in clinical practice. Current Radiology Reports, 10(9):101–115, 2022.
  49. Efficientnet: Rethinking model scaling for convolutional neural networks. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, pages 6105–6114. PMLR, 2019.
  50. Boxinst: High-performance instance segmentation with box annotations. In CVPR, pages 5443–5452. Computer Vision Foundation / IEEE, 2021.
  51. Learning random-walk label propagation for weakly-supervised semantic segmentation. In CVPR, pages 2953–2961. IEEE Computer Society, 2017.
  52. Weakly-supervised segmentation for disease localization in chest x-ray images. In Artificial Intelligence in Medicine, pages 249–259. Springer International Publishing, 2020.
  53. Automatic brain tumor segmentation using cascaded anisotropic convolutional neural networks. In Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries - Third International Workshop, BrainLes 2017, Held in Conjunction with MICCAI 2017, Quebec City, QC, Canada, September 14, 2017, Revised Selected Papers, pages 178–190. Springer, 2017.
  54. Machine Learning for Tomographic Imaging. IOP Publishing, 2019.
  55. Score-cam: Score-weighted visual explanations for convolutional neural networks. In CVPR Workshops, pages 111–119. Computer Vision Foundation / IEEE, 2020.
  56. Resnet strikes back: An improved training procedure in timm. arXiv preprint arXiv:2110.00476, 2021.
  57. Pytorch image models: A pytorch hub of pre-trained image classification models, 2022.
  58. Group normalization. In Computer Vision - ECCV 2018 - 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part XIII, pages 3–19. Springer, 2018.
  59. Bridging the gap between 2d and 3d organ segmentation with volumetric fusion net. In Medical Image Computing and Computer Assisted Intervention - MICCAI 2018 - 21st International Conference, Granada, Spain, September 16-20, 2018, Proceedings, Part IV, pages 445–453. Springer, 2018.
  60. Improvement of fully automated airway segmentation on volumetric computed tomographic images using a 2.5 dimensional convolutional neural net. Medical Image Anal., 51:13–20, 2019.
  61. Weakly supervised segmentation with point annotations for histopathology images via contrast-based variational model. In CVPR, pages 15630–15640. IEEE, 2023.
  62. Cyclemix: A holistic strategy for medical image segmentation from scribble supervision. In CVPR, pages 11646–11655. IEEE, 2022.
  63. Trimix: A general framework for medical image segmentation from limited supervision. In ACCV (6), pages 185–202. Springer, 2022.
  64. Learning deep features for discriminative localization. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 2921–2929. IEEE Computer Society, 2016.
  65. Weakly supervised 3d semantic segmentation using cross-image consensus and inter-voxel affinity relations. In ICCV, pages 2814–2824. IEEE, 2021.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
  1. Marius Schmidt-Mengin (2 papers)
  2. Alexis Benichoux (2 papers)
  3. Shibeshih Belachew (1 paper)
  4. Nikos Komodakis (37 papers)
  5. Nikos Paragios (34 papers)

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now