Papers
Topics
Authors
Recent
Search
2000 character limit reached

E2ENet: Dynamic Sparse Feature Fusion for Accurate and Efficient 3D Medical Image Segmentation

Published 7 Dec 2023 in cs.CV | (2312.04727v2)

Abstract: Deep neural networks have evolved as the leading approach in 3D medical image segmentation due to their outstanding performance. However, the ever-increasing model size and computation cost of deep neural networks have become the primary barrier to deploying them on real-world resource-limited hardware. In pursuit of improving performance and efficiency, we propose a 3D medical image segmentation model, named Efficient to Efficient Network (E2ENet), incorporating two parametrically and computationally efficient designs. i. Dynamic sparse feature fusion (DSFF) mechanism: it adaptively learns to fuse informative multi-scale features while reducing redundancy. ii. Restricted depth-shift in 3D convolution: it leverages the 3D spatial information while keeping the model and computational complexity as 2D-based methods. We conduct extensive experiments on BTCV, AMOS-CT and Brain Tumor Segmentation Challenge, demonstrating that E2ENet consistently achieves a superior trade-off between accuracy and efficiency than prior arts across various resource constraints. E2ENet achieves comparable accuracy on the large-scale challenge AMOS-CT, while saving over 68\% parameter count and 29\% FLOPs in the inference phase, compared with the previous best-performing method. Our code has been made available at: https://github.com/boqian333/E2ENet-Medical.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (61)
  1. The medical segmentation decathlon. Nature Communications, 13(1):1–13, 2022.
  2. Quick and robust feature selection: the strength of energy-efficient sparse training for autoencoders. Machine Learning, 111(1):377–414, 2022.
  3. Swin-Unet: Unet-like pure transformer for medical image segmentation. arXiv:2105.05537, 2021.
  4. Rethinking atrous convolution for semantic image segmentation. CoRR, abs/1706.05587, 2017.
  5. Dynamic convolution: Attention over convolution kernels. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pp. 11027–11036. Computer Vision Foundation / IEEE, 2020.
  6. 3D U-Net: learning dense volumetric segmentation from sparse annotation. In International conference on medical image computing and computer-assisted intervention, pp.  424–432. Springer, 2016.
  7. CoAtNet: Marrying convolution and attention for all data sizes. Advances in Neural Information Processing Systems, 34:3965–3977, 2021.
  8. ConViT: Improving vision transformers with soft convolutional inductive biases. In International Conference on Machine Learning, pp. 2286–2296. PMLR, 2021.
  9. Neural architecture search: A survey. The Journal of Machine Learning Research, 20(1):1997–2017, 2019.
  10. Rigging the lottery: Making all tickets winners. In International Conference on Machine Learning, pp. 2943–2952. PMLR, 2020.
  11. RubiksNet: Learnable 3D-shift for efficient video action recognition. In European Conference on Computer Vision, pp.  505–521. Springer, 2020.
  12. A data-scalable transformer for medical image segmentation: Architecture, model efficiency, and benchmark. arXiv preprint arXiv:2203.00131, 2022.
  13. UNETR: Transformers for 3D medical image segmentation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp.  574–584, 2022.
  14. DiNTS: Differentiable neural network topology search for 3D medical image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  5841–5850, 2021.
  15. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. J. Mach. Learn. Res., 22(241):1–124, 2021.
  16. Pseudo 3D auto-correlation network for real image denoising. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  16175–16184, 2021.
  17. UNet 3+: A full-scale connected UNet for medical image segmentation. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.  1055–1059. IEEE, 2020.
  18. MultiResUNet: Rethinking the U-Net architecture for multimodal biomedical image segmentation. Neural Networks, 121:74–87, 2020.
  19. nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature Methods, 18(2):203–211, 2021.
  20. Extending nnU-Net is all you need. arXiv:2208.10791, 2022.
  21. Top-KAST: Top-K always sparse training. Advances in Neural Information Processing Systems, 33:20744–20754, 2020.
  22. ResUNet++: An advanced architecture for medical image segmentation. In 2019 IEEE International Symposium on Multimedia (ISM), pp. 225–2255. IEEE, 2019.
  23. AMOS: A large-scale abdominal multi-organ benchmark for versatile medical image segmentation. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
  24. Convolution-free medical image segmentation using transformers. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pp.  78–88. Springer, 2021.
  25. SNIP: Single-shot network pruning based on connection sensitivity. In International Conference on Learning Representations, 2018.
  26. Attention Unet++: A nested attention-aware U-Net for liver ct image segmentation. In 2020 IEEE International Conference on Image Processing (ICIP), pp.  345–349. IEEE, 2020.
  27. AS-MLP: An axial shifted mlp architecture for vision. In International Conference on Learning Representations, 2021.
  28. TSM: Temporal shift module for efficient video understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.  7083–7093, 2019.
  29. Sparse training via boosting pruning plasticity with neuroregeneration. Advances in Neural Information Processing Systems, 34:9908–9922, 2021a.
  30. Sparse evolutionary deep learning with over one million artificial neurons on commodity hardware. Neural Computing and Applications, 33(7):2589–2604, 2021b.
  31. Do we actually need dense over-parameterization? in-time over-parameterization in sparse training. In Proceedings of the 38th International Conference on Machine Learning, volume 139, pp.  6989–7000. PMLR, 2021c.
  32. More ConvNets in the 2020s: Scaling up kernels beyond 51x51 using sparsity. arXiv:2207.03620, 2022.
  33. Rectifier nonlinearities improve neural network acoustic models. In Proceedings of the 30th International Conference on Machine Learning, number 1, pp.  3. Atlanta, Georgia, USA, 2013.
  34. V-Net: Fully convolutional neural networks for volumetric medical image segmentation. In 2016 Fourth International Conference on 3D vision (3DV), pp.  565–571. IEEE, 2016.
  35. A topological insight into restricted boltzmann machines. Machine Learning, 104(2):243–270, 2016.
  36. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature Communications, 9(1):1–12, 2018.
  37. Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization. In International Conference on Machine Learning, pp. 4646–4655. PMLR, 2019.
  38. Attention U-Net: Learning where to look for the pancreas. In Medical Imaging with Deep Learning, 2018.
  39. PyTorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
  40. U-Net: Convolutional networks for biomedical image segmentation. In International Conference on Medical Image Computing and Computer-assisted Intervention, pp.  234–241. Springer, 2015.
  41. A large annotated medical image dataset for the development and evaluation of segmentation algorithms. arXiv:1902.09063, 2019.
  42. Where to pay attention in sparse training for feature selection? In Advances in Neural Information Processing Systems, 2022.
  43. Dynamic group convolution for accelerating convolutional neural networks. In Vedaldi, A., Bischof, H., Brox, T., and Frahm, J. (eds.), Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VI, 2020.
  44. EfficientDet: Scalable and efficient object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  10781–10790, 2020.
  45. Pruning neural networks without any data by iteratively conserving synaptic flow. Advances in Neural Information Processing Systems, 33:6377–6389, 2020.
  46. High-resolution 3D abdominal segmentation with random patch network fusion. Medical Image Analysis, 69:101894, 2021.
  47. Self-supervised pre-training of swin transformers for 3D medical image analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  20730–20740, 2022.
  48. Instance normalization: The missing ingredient for fast stylization. arXiv:1607.08022, 2016.
  49. UNeXt: MLP-based rapid medical image segmentation network. arXiv:2203.04967, 2022.
  50. Picking winning tickets before training by preserving gradient flow. In International Conference on Learning Representations, 2019.
  51. UCTransNet: rethinking the skip connections in U-Net from a channel-wise perspective with transformer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp.  2441–2449, 2022.
  52. TransBTS: Multimodal brain tumor segmentation using transformer. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pp.  109–119. Springer, 2021.
  53. NAS-Unet: Neural architecture search for medical image segmentation. IEEE Access, 7:44247–44257, 2019.
  54. Dynamic sparse network for time series classification: Learning what to “see”. In Advances in Neural Information Processing Systems, 2022.
  55. CoTr: Efficiently bridging CNN and transformer for 3D medical image segmentation. In International Conference on Medical Image Computing and Computer-assisted Intervention, pp.  171–180. Springer, 2021.
  56. C2FNAS: coarse-to-fine neural architecture search for 3d medical image segmentation. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pp. 4125–4134. Computer Vision Foundation / IEEE, 2020a.
  57. C2FNAS: Coarse-to-fine neural architecture search for 3D medical image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  4126–4135, 2020b.
  58. MEST: Accurate and fast memory-economic sparse training framework on the edge. Advances in Neural Information Processing Systems, 34:20838–20850, 2021.
  59. nnFormer: Interleaved transformer for volumetric segmentation. arXiv:2109.03201, 2021.
  60. Prior-aware neural network for partially-supervised multi-organ segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.  10672–10681, 2019.
  61. UNet++: A nested U-Net architecture for medical image segmentation. In Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, pp.  3–11. Springer, 2018.
Citations (1)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Sign Up to Summarize

Paper to Video (Beta)

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up

Tweets

Sign up for free to view the 1 tweet with 29 likes about this paper.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up for Free