Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
167 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Blaze3DM: Marry Triplane Representation with Diffusion for 3D Medical Inverse Problem Solving (2405.15241v1)

Published 24 May 2024 in eess.IV and cs.CV

Abstract: Solving 3D medical inverse problems such as image restoration and reconstruction is crucial in modern medical field. However, the curse of dimensionality in 3D medical data leads mainstream volume-wise methods to suffer from high resource consumption and challenges models to successfully capture the natural distribution, resulting in inevitable volume inconsistency and artifacts. Some recent works attempt to simplify generation in the latent space but lack the capability to efficiently model intricate image details. To address these limitations, we present Blaze3DM, a novel approach that enables fast and high-fidelity generation by integrating compact triplane neural field and powerful diffusion model. In technique, Blaze3DM begins by optimizing data-dependent triplane embeddings and a shared decoder simultaneously, reconstructing each triplane back to the corresponding 3D volume. To further enhance 3D consistency, we introduce a lightweight 3D aware module to model the correlation of three vertical planes. Then, diffusion model is trained on latent triplane embeddings and achieves both unconditional and conditional triplane generation, which is finally decoded to arbitrary size volume. Extensive experiments on zero-shot 3D medical inverse problem solving, including sparse-view CT, limited-angle CT, compressed-sensing MRI, and MRI isotropic super-resolution, demonstrate that Blaze3DM not only achieves state-of-the-art performance but also markedly improves computational efficiency over existing methods (22~40x faster than previous work).

Definition Search Book Streamline Icon: https://streamlinehq.com
References (41)
  1. Solving 3d inverse problems using pre-trained 2d diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22542–22551, 2023a.
  2. Improving 3d imaging with pre-trained perpendicular 2d diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10710–10720, 2023.
  3. Improving diffusion models for inverse problems using manifold constraints. Advances in Neural Information Processing Systems, 35:25683–25696, 2022a.
  4. Solving inverse problems in medical imaging with score-based generative models. arXiv preprint arXiv:2111.08005, 2021.
  5. Come-closer-diffuse-faster: Accelerating conditional diffusion models for inverse problems through stochastic contraction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12413–12422, 2022b.
  6. Score-based diffusion models for accelerated mri. Medical image analysis, 80:102479, 2022.
  7. Medical image synthesis with deep convolutional adversarial networks. IEEE Transactions on Biomedical Engineering, 65(12):2720–2730, 2018.
  8. Ea-gans: edge-aware generative adversarial networks for cross-modality mr image synthesis. IEEE transactions on medical imaging, 38(7):1750–1762, 2019.
  9. Generating realistic brain mris via a conditional diffusion probabilistic model. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 14–24. Springer, 2023.
  10. Three-dimensional medical image synthesis with denoising diffusion probabilistic models. In Medical Imaging with Deep Learning, 2022.
  11. Diffusion deformable model for 4d temporal medical image generation. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 539–548. Springer, 2022.
  12. 3d cgan based cross-modality mr image synthesis for brain tumor segmentation. In 2018 IEEE 15th international symposium on biomedical imaging (ISBI 2018), pages 626–630. IEEE, 2018.
  13. Vox2vox: 3d-gan for brain tumour segmentation. In Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries: 6th International Workshop, BrainLes 2020, Held in Conjunction with MICCAI 2020, Lima, Peru, October 4, 2020, Revised Selected Papers, Part I 6, pages 274–284. Springer, 2021.
  14. Generative adversarial networks. Communications of the ACM, 63(11):139–144, 2020.
  15. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020.
  16. Make-a-volume: Leveraging latent diffusion models for cross-modality 3d brain mri synthesis. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 592–601. Springer, 2023.
  17. Denoising diffusion probabilistic models for 3d medical image generation. Scientific Reports, 13(1):7303, 2023.
  18. Brain imaging generation with latent diffusion models. In MICCAI Workshop on Deep Generative Models, pages 117–126. Springer, 2022.
  19. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022.
  20. Adaptive latent diffusion model for 3d medical image to image translation: Multi-modal magnetic resonance imaging study. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 7604–7613, 2024.
  21. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99–106, 2021.
  22. Irem: high-resolution magnetic resonance image reconstruction via implicit neural representation. In Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part VI 24, pages 65–74. Springer, 2021.
  23. Efficient geometry-aware 3d generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16123–16133, 2022.
  24. Poco: Point convolution for surface reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6302–6314, 2022.
  25. Local deep implicit functions for 3d shape. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4857–4866, 2020.
  26. Learning shape templates with structured implicit functions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7154–7164, 2019.
  27. Acorn: adaptive coordinate networks for neural scene representation. ACM Transactions on Graphics (TOG), 40(4):1–13, 2021.
  28. 3d neural field generation using triplane diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20875–20886, 2023.
  29. Rodin: A generative model for sculpting 3d digital avatars using diffusion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4563–4573, 2023.
  30. Trinerflet: A wavelet based multiscale triplane nerf representation. arXiv preprint arXiv:2401.06191, 2024.
  31. Sin3dm: Learning a diffusion model from a single 3d textured shape. In The Twelfth International Conference on Learning Representations, 2023.
  32. Denoising diffusion implicit models. In International Conference on Learning Representations, 2020.
  33. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780–8794, 2021.
  34. Improved denoising diffusion probabilistic models. In International conference on machine learning, pages 8162–8171. PMLR, 2021.
  35. Model-based iterative reconstruction technique for radiation dose reduction in chest ct: comparison with the adaptive statistical iterative reconstruction technique. European radiology, 22:1613–1623, 2012.
  36. Diffusion posterior sampling for general noisy inverse problems. In The Eleventh International Conference on Learning Representations, ICLR 2023. The International Conference on Learning Representations, 2023b.
  37. On the spectral bias of neural networks. In International conference on machine learning, pages 5301–5310. PMLR, 2019.
  38. Implicit neural representation in medical imaging: A comparative survey. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2381–2391, 2023.
  39. Low-dose ct for the detection and classification of metastatic liver lesions: results of the 2016 low dose ct grand challenge. Medical physics, 44(10):e339–e352, 2017.
  40. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600–612, 2004.
  41. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:1512.03012, 2015.
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets