Papers
Topics
Authors
Recent
Detailed Answer
Quick Answer
Concise responses based on abstracts only
Detailed Answer
Well-researched responses based on abstracts and relevant paper content.
Custom Instructions Pro
Preferences or requirements that you'd like Emergent Mind to consider when generating responses
Gemini 2.5 Flash
Gemini 2.5 Flash 88 tok/s
Gemini 2.5 Pro 52 tok/s Pro
GPT-5 Medium 17 tok/s Pro
GPT-5 High 17 tok/s Pro
GPT-4o 73 tok/s Pro
GPT OSS 120B 464 tok/s Pro
Kimi K2 190 tok/s Pro
2000 character limit reached

PnPNet: Pull-and-Push Networks for Volumetric Segmentation with Boundary Confusion (2312.08323v1)

Published 13 Dec 2023 in cs.CV

Abstract: Precise boundary segmentation of volumetric images is a critical task for image-guided diagnosis and computer-assisted intervention, especially for boundary confusion in clinical practice. However, U-shape networks cannot effectively resolve this challenge due to the lack of boundary shape constraints. Besides, existing methods of refining boundaries overemphasize the slender structure, which results in the overfitting phenomenon due to networks' limited abilities to model tiny objects. In this paper, we reconceptualize the mechanism of boundary generation by encompassing the interaction dynamics with adjacent regions. Moreover, we propose a unified network termed PnPNet to model shape characteristics of the confused boundary region. Core ingredients of PnPNet contain the pushing and pulling branches. Specifically, based on diffusion theory, we devise the semantic difference module (SDM) from the pushing branch to squeeze the boundary region. Explicit and implicit differential information inside SDM significantly boost representation abilities for inter-class boundaries. Additionally, motivated by the K-means algorithm, the class clustering module (CCM) from the pulling branch is introduced to stretch the intersected boundary region. Thus, pushing and pulling branches will shrink and enlarge the boundary uncertainty respectively. They furnish two adversarial forces to promote models to output a more precise delineation of boundaries. We carry out experiments on three challenging public datasets and one in-house dataset, containing three types of boundary confusion in model predictions. Experimental results demonstrate the superiority of PnPNet over other segmentation networks, especially on evaluation metrics of HD and ASSD. Besides, pushing and pulling branches can serve as plug-and-play modules to enhance classic U-shape baseline models. Codes are available.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (88)
  1. Fabian Isensee et al. nnu-net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods, 18(2):203–211, 2021.
  2. Hugo JWL Aerts et al. Decoding tumour phenotype by noninvasive imaging using a quantitative radiomics approach. Nature communications, 5(1):4006, 2014.
  3. Olivier Bernard et al. Deep learning techniques for automatic mri cardiac multi-structures segmentation and diagnosis: is the problem solved? TMI, 37(11):2514–2525, 2018.
  4. Jeffrey De Fauw et al. Clinically applicable deep learning for diagnosis and referral in retinal disease. Nature medicine, 24(9):1342–1350, 2018.
  5. Stanislav Nikolov et al. Deep learning to achieve clinically applicable segmentation of head and neck anatomy for radiotherapy. arXiv preprint arXiv:1809.04430, 2018.
  6. Todd C Hollon et al. Near real-time intraoperative brain tumor diagnosis using stimulated raman histology and deep neural networks. Nature medicine, 26(1):52–58, 2020.
  7. Philipp Kickingereder et al. Automated quantitative tumour response assessment of mri in neuro-oncology with artificial neural networks: a multicentre, retrospective study. The Lancet Oncology, 20(5):728–740, 2019.
  8. Reza Azad et al. Medical image segmentation review: The success of u-net. arXiv preprint arXiv:2211.14830, 2022.
  9. Dzung L Pham et al. Current methods in medical image segmentation. Annual review of biomedical engineering, 2(1):315–337, 2000.
  10. Mohammad Hesam Hesamian et al. Deep learning techniques for medical image segmentation: achievements and challenges. Journal of digital imaging, 32:582–596, 2019.
  11. Fahad Shamshad et al. Transformers in medical imaging: A survey. Medical Image Analysis, page 102802, 2023.
  12. Olaf Ronneberger et al. U-net: Convolutional networks for biomedical image segmentation. In MICCAI, pages 234–241. Springer, 2015.
  13. Ruxin Wang et al. Boundary-aware context neural network for medical image segmentation. Medical Image Analysis, 78:102395, 2022.
  14. Jongha Park et al. Fully automated lung lobe segmentation in volumetric chest ct with 3d u-net: validation with intra-and extra-datasets. Journal of digital imaging, 33:221–230, 2020.
  15. Weiyi Xie et al. Relational modeling for robust and efficient pulmonary lobe segmentation in ct scans. IEEE transactions on medical imaging, 39(8):2664–2675, 2020.
  16. Yan Wang et al. Segmentation of lumen and outer wall of abdominal aortic aneurysms from 3d black-blood mri with a registration based geodesic active contour model. Medical image analysis, 40:1–10, 2017.
  17. Yefeng Zheng et al. Multi-part modeling and segmentation of left atrium in c-arm ct for image-guided ablation of atrial fibrillation. IEEE TMI, 33(2):318–331, 2013.
  18. Hwanjun Song et al. Learning from noisy labels with deep neural networks: A survey. TNNLS, 2022.
  19. Xin You et al. Verteformer: A single-staged transformer network for vertebrae segmentation from ct images with arbitrary field of views. Medical Physics, 2023.
  20. Anjany Sekuboyina et al. Verse: A vertebrae labelling and segmentation benchmark for multi-detector ct images. Medical image analysis, 73:102166, 2021.
  21. Saumya Gupta et al. Learning topological interactions for multi-class medical image segmentation. In ECCV, pages 701–718. Springer, 2022.
  22. Ali Hatamizadeh et al. End-to-end boundary aware networks for medical image segmentation. In MLMI 2019. Springer, 2019.
  23. Hoel Kervadec et al. Boundary loss for highly unbalanced segmentation. Medical image analysis, 67:101851, 2021.
  24. Reducing the hausdorff distance in medical image segmentation with convolutional neural networks. TMI, 39(2):499–513, 2019.
  25. Jun Ma et al. Loss odyssey in medical image segmentation. Medical Image Analysis, 71:102035, 2021.
  26. Ali Akbari et al. How does loss function affect generalization performance of deep learning? application to human age estimation. In ICLR, pages 141–151. PMLR, 2021.
  27. Chiyuan Zhang et al. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107–115, 2021.
  28. Yuhui Yuan et al. Segfix: Model-agnostic boundary refinement for segmentation. In ECCV, pages 489–506. Springer, 2020.
  29. Alexander Kirillov et al. Pointrend: Image segmentation as rendering. In CVPR, pages 9799–9808, 2020.
  30. Peng Zhou et al. Deepstrip: High-resolution boundary refinement. In CVPR, pages 10558–10567, 2020.
  31. Chufeng Tang et al. Look closer to segment better: Boundary patch refinement for instance segmentation. In CVPR, pages 13926–13935, 2021.
  32. Guilin Zhu et al. An adaptive post-processing network with the global-local aggregation for semantic segmentation. TCSVT, 2023.
  33. Yanting Xie et al. Uncertainty-aware cascade network for ultrasound image segmentation with ambiguous boundary. In MICCAI, pages 268–278. Springer, 2022.
  34. Xin You et al. Semantic difference guidance for the uncertain boundary segmentation of ct left atrial appendage. In MICCAI, pages 121–131. Springer, 2023.
  35. Bowen Cheng et al. Per-pixel classification is not all you need for semantic segmentation. NeurIPS, 34:17864–17875, 2021.
  36. Bowen Cheng et al. Masked-attention mask transformer for universal image segmentation. In CVPR, pages 1290–1299, 2022.
  37. Zhijie Zhang et al. Et-net: A generic edge-attention guidance network for medical image segmentation. In MICCAI, pages 442–450. Springer, 2019.
  38. Xier Chen et al. Supervised edge attention network for accurate image instance segmentation. In ECCV, pages 617–631. Springer, 2020.
  39. Kun Wang et al. Eanet: Iterative edge attention network for medical image segmentation. Pattern Recognition, 127:108636, 2022.
  40. Chi Wang et al. Active boundary loss for semantic segmentation. In AAAI, volume 36, pages 2397–2405, 2022.
  41. Shubhankar Borse et al. Inverseform: A loss function for structured boundary-aware segmentation. In CVPR, pages 5901–5911, 2021.
  42. Chuong Huynh et al. Progressive semantic segmentation. In CVPR, pages 16755–16764, 2021.
  43. Chenming Zhu et al. Sharpcontour: a contour-based boundary refinement approach for efficient and accurate instance segmentation. In CVPR, pages 4392–4401, 2022.
  44. Deng-Ping Fan et al. Pranet: Parallel reverse attention network for polyp segmentation. In MICCAI, pages 263–273. Springer, 2020.
  45. Hong Joo Lee et al. Structure boundary preserving segmentation for medical image with ambiguous boundary. In CVPR, pages 4817–4826, 2020.
  46. Feilong Cao et al. A novel method for image segmentation: two-stage decoding network with boundary attention. International Journal of Machine Learning and Cybernetics, pages 1–13, 2022.
  47. Yi Lin et al. Rethinking boundary detection in deep learning models for medical image segmentation. In International Conference on IPMI, pages 730–742, 2023.
  48. Jiacheng Wang et al. Xbound-former: Toward cross-scale boundary modeling in transformers. IEEE TMI, 2023.
  49. Ept-net: Edge perception transformer for 3d medical image segmentation. IEEE Transactions on Medical Imaging, 2023.
  50. Guillermo Sapiro. Geometric partial differential equations and image analysis. Cambridge university press, 2006.
  51. Haoru Tan et al. Semantic diffusion network for semantic segmentation. NeurIPS, 35:8702–8716, 2022.
  52. Joachim Weickert. Coherence-enhancing diffusion filtering. International journal of computer vision, 31(2-3):111, 1999.
  53. Joachim Weickert et al. Efficient and reliable schemes for nonlinear diffusion filtering. TIP, 7(3):398–410, 1998.
  54. Özgün Çiçek et al. 3d u-net: learning dense volumetric segmentation from sparse annotation. In MICCAI, pages 424–432. Springer, 2016.
  55. Saikat Roy et al. Mednext: Transformer-driven scaling of convnets for medical image segmentation. arXiv preprint arXiv:2303.09975, 2023.
  56. Jieneng Chen et al. Transunet: Transformers make strong encoders for medical image segmentation. arXiv preprint arXiv:2102.04306, 2021.
  57. Yucheng Tang et al. Self-supervised pre-training of swin transformers for 3d medical image analysis. In CVPR, 2022.
  58. Jeya Maria Jose Valanarasu et al. Unext: Mlp-based rapid medical image segmentation network. In MICCAI, pages 23–33. Springer, 2022.
  59. Algorithm as 136: A k-means clustering algorithm. Journal of the royal statistical society. series c (applied statistics), 28(1):100–108, 1979.
  60. Alexey Dosovitskiy et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2020.
  61. How do vision transformers work? In International Conference on Learning Representations, 2021.
  62. Nicolas Carion et al. End-to-end object detection with transformers. In European conference on computer vision, pages 213–229. Springer, 2020.
  63. Qihang Yu et al. Cmt-deeplab: Clustering mask transformers for panoptic segmentation. In CVPR, pages 2560–2570, 2022.
  64. Depu Meng et al. Conditional detr for fast training convergence. In ICCV, pages 3651–3660, 2021.
  65. Shilong Liu et al. Dab-detr: Dynamic anchor boxes are better queries for detr. In ICLR, 2021.
  66. Tianfei Zhou et al. Rethinking semantic segmentation: A prototype view. In CVPR, pages 2582–2593, 2022.
  67. Scribblesup: Scribble-supervised convolutional networks for semantic segmentation. In CVPR, pages 3159–3167, 2016.
  68. Alexander Kirillov et al. Segment anything. arXiv preprint arXiv:2304.02643, 2023.
  69. Yuhao Huang et al. Segment anything model for medical images? arXiv preprint arXiv:2304.14660, 2023.
  70. Maciej A Mazurowski et al. Segment anything model for medical image analysis: an experimental study. Medical Image Analysis, 89:102918, 2023.
  71. Junlong Cheng et al. Sam-med2d. arXiv preprint arXiv:2308.16184, 2023.
  72. Bolei Zhou et al. Learning deep features for discriminative localization. In CVPR, pages 2921–2929, 2016.
  73. Muhammad Muzammal Naseer et al. Intriguing properties of vision transformers. Advances in Neural Information Processing Systems, 34:23296–23308, 2021.
  74. Fausto Milletari et al. V-net: Fully convolutional neural networks for volumetric medical image segmentation. In 2016 fourth international conference on 3D vision (3DV), pages 565–571. Ieee, 2016.
  75. Foivos I Diakogiannis et al. Resunet-a: A deep learning framework for semantic segmentation of remotely sensed data. ISPRS Journal of Photogrammetry and Remote Sensing, 162:94–114, 2020.
  76. Wenxuan Wang et al. Transbts: Multimodal brain tumor segmentation using transformer. In MICCAI, pages 109–119, 2021.
  77. Ho Hin Lee et al. 3d ux-net: A large kernel volumetric convnet modernizing hierarchical transformer for medical image segmentation. In ICLR, 2022.
  78. Hao Tang et al. Automatic pulmonary lobe segmentation using deep learning. In ISBI, pages 1225–1228. IEEE, 2019.
  79. Arnaud Arindra Adiyoso Setio et al. Validation, comparison, and combination of algorithms for automatic detection of pulmonary nodules in computed tomography images: the luna16 challenge. Medical image analysis, 42:1–13, 2017.
  80. Colin Jacobs et al. Automatic detection of subsolid pulmonary nodules in thoracic computed tomography images. Medical image analysis, 18(2):374–384, 2014.
  81. Jun Ma et al. Toward data-efficient learning: A benchmark for covid-19 ct lung and infection segmentation. Medical physics, 48(3):1197–1210, 2021.
  82. Varduhi Yeghiazaryan et al. Family of boundary overlap metrics for the evaluation of medical image segmentation. Journal of Medical Imaging, 5(1):015006–015006, 2018.
  83. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
  84. Hong-Yu Zhou et al. nnformer: Interleaved transformer for volumetric segmentation. arXiv preprint arXiv:2109.03201, 2021.
  85. Yutong Xie et al. Cotr: Efficiently bridging cnn and transformer for 3d medical image segmentation. In MICCAI, pages 171–180. Springer, 2021.
  86. Xin You et al. Eg-trans3dunet: a single-staged transformer-based model for accurate vertebrae segmentation from spinal ct images. In ISBI, pages 1–5. IEEE, 2022.
  87. Rosana El Jurdi et al. High-level prior-based loss functions for medical image segmentation: A survey. Computer Vision and Image Understanding, 210:103248, 2021.
  88. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
List To Do Tasks Checklist Streamline Icon: https://streamlinehq.com

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Ai Generate Text Spark Streamline Icon: https://streamlinehq.com

Paper Prompts

Sign up for free to create and run prompts on this paper using GPT-5.

List Streamline Icon: https://streamlinehq.com Explore 10 Community Prompts
Dice Question Streamline Icon: https://streamlinehq.com

Follow-up Questions

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now