Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash 91 tok/s
Gemini 2.5 Pro 58 tok/s Pro
GPT-5 Medium 29 tok/s
GPT-5 High 29 tok/s Pro
GPT-4o 102 tok/s
GPT OSS 120B 462 tok/s Pro
Kimi K2 181 tok/s Pro
2000 character limit reached

D-TrAttUnet: Toward Hybrid CNN-Transformer Architecture for Generic and Subtle Segmentation in Medical Images (2405.04169v1)

Published 7 May 2024 in eess.IV and cs.CV

Abstract: Over the past two decades, machine analysis of medical imaging has advanced rapidly, opening up significant potential for several important medical applications. As complicated diseases increase and the number of cases rises, the role of machine-based imaging analysis has become indispensable. It serves as both a tool and an assistant to medical experts, providing valuable insights and guidance. A particularly challenging task in this area is lesion segmentation, a task that is challenging even for experienced radiologists. The complexity of this task highlights the urgent need for robust machine learning approaches to support medical staff. In response, we present our novel solution: the D-TrAttUnet architecture. This framework is based on the observation that different diseases often target specific organs. Our architecture includes an encoder-decoder structure with a composite Transformer-CNN encoder and dual decoders. The encoder includes two paths: the Transformer path and the Encoders Fusion Module path. The Dual-Decoder configuration uses two identical decoders, each with attention gates. This allows the model to simultaneously segment lesions and organs and integrate their segmentation losses. To validate our approach, we performed evaluations on the Covid-19 and Bone Metastasis segmentation tasks. We also investigated the adaptability of the model by testing it without the second decoder in the segmentation of glands and nuclei. The results confirmed the superiority of our approach, especially in Covid-19 infections and the segmentation of bone metastases. In addition, the hybrid encoder showed exceptional performance in the segmentation of glands and nuclei, solidifying its role in modern medical image analysis.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (55)
  1. Bm-seg: A new bone metastases segmentation dataset and ensemble of cnn-based segmentation approach. Expert Systems with Applications 228, 120376.
  2. Infectious disease in an era of global change. Nature Reviews Microbiology 20, 193–205.
  3. ILC-Unet++ for Covid-19 Infection Segmentation, in: Mazzeo, P.L., Frontoni, E., Sclaroff, S., Distante, C. (Eds.), Image Analysis and Processing. ICIAP 2022 Workshops, Springer International Publishing, Cham. pp. 461–472. doi:10.1007/978-3-031-13324-4_39.
  4. CNN based facial aesthetics analysis through dynamic robust losses and ensemble regression. Applied Intelligence URL: https://doi.org/10.1007/s10489-022-03943-0, doi:10.1007/s10489-022-03943-0.
  5. Deep learning based face beauty prediction via dynamic robust losses and ensemble regression. Knowledge-Based Systems 242, 108246.
  6. Swin-unet: Unet-like pure transformer for medical image segmentation, in: European conference on computer vision, Springer. pp. 205–218.
  7. Contrastive learning of global and local features for medical image segmentation with limited annotations. Advances in neural information processing systems 33, 12546–12558.
  8. TransMed: Transformers Advance Multi-Modal Medical Image Classification. Diagnostics 11, 1384. URL: https://www.mdpi.com/2075-4418/11/8/1384, doi:10.3390/diagnostics11081384. number: 8 Publisher: Multidisciplinary Digital Publishing Institute.
  9. ImageNet: A large-scale hierarchical image database, in: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255. doi:10.1109/CVPR.2009.5206848. iSSN: 1063-6919.
  10. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 .
  11. Inf-Net: Automatic COVID-19 Lung Infection Segmentation From CT Images. IEEE Transactions on Medical Imaging 39, 2626–2637. doi:10.1109/TMI.2020.2996645. conference Name: IEEE Transactions on Medical Imaging.
  12. Data augmentation for medical imaging: A systematic literature review. Computers in Biology and Medicine 152, 106391.
  13. The rising burden of non-communicable diseases in the americas and the impact of population aging: a secondary analysis of available data. The Lancet Regional Health–Americas 21.
  14. UNETR: Transformers for 3D Medical Image Segmentation, pp. 574–584. URL: https://openaccess.thecvf.com/content/WACV2022/html/Hatamizadeh_UNETR_Transformers_for_3D_Medical_Image_Segmentation_WACV_2022_paper.html.
  15. Medical image segmentation method based on multi-feature interaction and fusion over cloud computing. Simulation Modelling Practice and Theory 126, 102769.
  16. The diagnostic imaging of bone metastases. Deutsches Ärzteblatt International 111, 741.
  17. Squeeze-and-Excitation Networks, pp. 7132–7141. URL: https://openaccess.thecvf.com/content_cvpr_2018/html/Hu_Squeeze-and-Excitation_Networks_CVPR_2018_paper.html.
  18. Missformer: An effective transformer for 2d medical image segmentation. IEEE Transactions on Medical Imaging .
  19. Transformers in vision: A survey. ACM Computing Surveys (CSUR) Publisher: ACM New York, NY.
  20. ImageNet Classification with Deep Convolutional Neural Networks, in: Advances in Neural Information Processing Systems, Curran Associates, Inc. URL: https://proceedings.neurips.cc/paper/2012/hash/c399862d3b9d6b76c8436e924a68c45b-Abstract.html.
  21. A multi-organ nucleus segmentation challenge. IEEE transactions on medical imaging 39, 1380–1391.
  22. LungINFseg: Segmenting COVID-19 Infected Regions in Lung CT Images Based on a Receptive-Field-Aware Deep Learning Framework. Diagnostics 11, 158. doi:10.3390/diagnostics11020158. number: 2 Publisher: Multidisciplinary Digital Publishing Institute.
  23. A weakly supervised consistency-based learning method for covid-19 segmentation in ct images, in: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 2453–2462.
  24. Joint categorical and ordinal learning for cancer grading in pathology images. Medical image analysis 73, 102206.
  25. Semi-supervised medical image segmentation using adversarial consistency learning and dynamic convolution network. IEEE Transactions on Medical Imaging .
  26. COVID-19 lung infection segmentation with a novel two-stage cross-domain transfer learning framework. Medical Image Analysis 74, 102205. doi:10.1016/j.media.2021.102205.
  27. Swin transformer: Hierarchical vision transformer using shifted windows, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10012–10022.
  28. Attention U-Net: Learning Where to Look for the Pancreas. arXiv:1804.03999 [cs] ArXiv: 1804.03999.
  29. Anam-Net: Anamorphic Depth Embedding-Based Lightweight CNN for Segmentation of Anomalies in COVID-19 Chest CT Images. IEEE Transactions on Neural Networks and Learning Systems 32, 932–946. Conference Name: IEEE Transactions on Neural Networks and Learning Systems.
  30. Pytorch: An imperative style, high-performance deep learning library, in: Advances in neural information processing systems, pp. 8026–8037.
  31. U-Net Transformer: Self and Cross Attention for Medical Image Segmentation, in: Lian, C., Cao, X., Rekik, I., Xu, X., Yan, P. (Eds.), Machine Learning in Medical Imaging, Springer International Publishing, Cham. pp. 267–276. doi:10.1007/978-3-030-87589-3_28.
  32. RADIOLOGISTS, 2019. COVID-19 CT-scans segmentation datasets, available at: http://medicalsegmentation.com/covid19/. Last visited: 18-08-2021.
  33. U-Net: Convolutional Networks for Biomedical Image Segmentation, in: Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015, Springer International Publishing, Cham. pp. 234–241.
  34. Transformers in Medical Imaging: A Survey. URL: http://arxiv.org/abs/2201.09873, doi:10.48550/arXiv.2201.09873. arXiv:2201.09873 [cs, eess].
  35. Transformers in medical imaging: A survey. Medical Image Analysis , 102802.
  36. COTR: Convolution in Transformer Network for End to End Polyp Detection, in: 2021 7th International Conference on Computer and Communications (ICCC), pp. 1757–1761. doi:10.1109/ICCC54389.2021.9674267.
  37. Gland segmentation in colon histology images: The glas challenge contest. Medical image analysis 35, 489–502.
  38. Gland segmentation in colon histology images the glas challenge contest. Medical image analysis 35, 489–502.
  39. Locality sensitive deep learning for detection and classification of nuclei in routine colon cancer histology images. IEEE transactions on medical imaging 35, 1196–1206.
  40. A systematic review of chest imaging findings in COVID-19. Quantitative Imaging in Medicine and Surgery 10, 1058–1079. doi:10.21037/qims-20-564.
  41. FANet: A Feedback Attention Network for Improved Biomedical Image Segmentation. IEEE Transactions on Neural Networks and Learning Systems , 1–14doi:10.1109/TNNLS.2022.3159394. conference Name: IEEE Transactions on Neural Networks and Learning Systems.
  42. Training data-efficient image transformers and distillation through attention, in: Meila, M., Zhang, T. (Eds.), Proceedings of the 38th International Conference on Machine Learning, PMLR. pp. 10347–10357. URL: https://proceedings.mlr.press/v139/touvron21a.html.
  43. Attention is all you need. Advances in neural information processing systems 30.
  44. A Noise-Robust Framework for Automatic Segmentation of COVID-19 Pneumonia Lesions From CT Images. IEEE Transactions on Medical Imaging 39, 2653–2663. doi:10.1109/TMI.2020.3000314. conference Name: IEEE Transactions on Medical Imaging.
  45. Uctransnet: rethinking the skip connections in u-net from a channel-wise perspective with transformer, in: Proceedings of the AAAI conference on artificial intelligence, pp. 2441–2449.
  46. Mixed transformer u-net for medical image segmentation, in: ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE. pp. 2390–2394.
  47. Focus, Fusion, and Rectify: Context-Aware Learning for COVID-19 Lung Infection Segmentation. IEEE Transactions on Neural Networks and Learning Systems 33. doi:10.1109/TNNLS.2021.3126305. iEEE Transactions on Neural Networks and Learning Systems.
  48. Transbts: Multimodal brain tumor segmentation using transformer, in: Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part I 24, Springer. pp. 109–119.
  49. TransBTS: Multimodal Brain Tumor Segmentation Using Transformer, in: de Bruijne, M., Cattin, P.C., Cotin, S., Padoy, N., Speidel, S., Zheng, Y., Essert, C. (Eds.), Medical Image Computing and Computer Assisted Intervention – MICCAI 2021, Springer International Publishing, Cham. pp. 109–119. doi:10.1007/978-3-030-87193-2_11.
  50. FAT-Net: Feature adaptive transformers for automated skin lesion segmentation. Medical Image Analysis 76, 102327. URL: https://www.sciencedirect.com/science/article/pii/S1361841521003728, doi:10.1016/j.media.2021.102327.
  51. Road Extraction by Deep Residual U-Net. IEEE Geoscience and Remote Sensing Letters 15, 749–753. doi:10.1109/LGRS.2018.2802944. conference Name: IEEE Geoscience and Remote Sensing Letters.
  52. Scoat-net: A novel network for segmenting covid-19 lung opacification from ct images. Pattern Recognition , 108109.
  53. Evolutionary Compression of Deep Neural Networks for Biomedical Image Segmentation. IEEE Transactions on Neural Networks and Learning Systems 31, 2916–2929. doi:10.1109/TNNLS.2019.2933879. conference Name: IEEE Transactions on Neural Networks and Learning Systems.
  54. UNet++: A Nested U-Net Architecture for Medical Image Segmentation, in: Stoyanov, D., Taylor, Z., Carneiro, G.e.a. (Eds.), Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, Springer International Publishing, Cham. pp. 3–11.
  55. Brain tumor segmentation based on the fusion of deep semantics and edge information in multimodal mri. Information Fusion 91, 376–387.
Citations (3)
View on Semantic Scholar
List To Do Tasks Checklist Streamline Icon: https://streamlinehq.com

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Ai Generate Text Spark Streamline Icon: https://streamlinehq.com

Paper Prompts

Sign up for free to create and run prompts on this paper using GPT-5.

List Streamline Icon: https://streamlinehq.com Explore 10 Community Prompts
Dice Question Streamline Icon: https://streamlinehq.com

Follow-up Questions

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now