Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
167 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

CoNeS: Conditional neural fields with shift modulation for multi-sequence MRI translation (2309.03320v3)

Published 6 Sep 2023 in eess.IV and cs.CV

Abstract: Multi-sequence magnetic resonance imaging (MRI) has found wide applications in both modern clinical studies and deep learning research. However, in clinical practice, it frequently occurs that one or more of the MRI sequences are missing due to different image acquisition protocols or contrast agent contraindications of patients, limiting the utilization of deep learning models trained on multi-sequence data. One promising approach is to leverage generative models to synthesize the missing sequences, which can serve as a surrogate acquisition. State-of-the-art methods tackling this problem are based on convolutional neural networks (CNN) which usually suffer from spectral biases, resulting in poor reconstruction of high-frequency fine details. In this paper, we propose Conditional Neural fields with Shift modulation (CoNeS), a model that takes voxel coordinates as input and learns a representation of the target images for multi-sequence MRI translation. The proposed model uses a multi-layer perceptron (MLP) instead of a CNN as the decoder for pixel-to-pixel mapping. Hence, each target image is represented as a neural field that is conditioned on the source image via shift modulation with a learned latent code. Experiments on BraTS 2018 and an in-house clinical dataset of vestibular schwannoma patients showed that the proposed method outperformed state-of-the-art methods for multi-sequence MRI translation both visually and quantitatively. Moreover, we conducted spectral analysis, showing that CoNeS was able to overcome the spectral bias issue common in conventional CNN models. To further evaluate the usage of synthesized images in clinical downstream tasks, we tested a segmentation network using the synthesized images at inference.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (63)
  1. Learning shape reconstruction from sparse measurements with neural implicit functions. In International Conference on Medical Imaging with Deep Learning, pages 22–34. PMLR, 2022.
  2. Pathology synthesis of 3D-Consistent cardiac MR images using 2D VAEs and GANs. Machine Learning for Biomedical Imaging, 2:288–311, 2023.
  3. Image generators with conditionally-independent pixel synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14278–14287, 2021.
  4. MedGAN: Medical image translation using GANs. Computerized Medical Imaging and Graphics, 79:101684, 2020.
  5. Medical image segmentation on MRI images with missing modalities: a review. arXiv preprint arxiv:2203.06217, 2022a.
  6. SMU-Net: Style matching U-Net for brain tumor segmentation with missing modalities. In International Conference on Medical Imaging with Deep Learning, pages 48–62, 2022b.
  7. Brain microstructure by multi-modal MRI: Is the whole greater than the sum of its parts? NeuroImage, 182:117–127, 2018.
  8. Principled weight initialization for hypernetworks. In International Conference on Learning Representations, 2019.
  9. Multimodal MR synthesis via modality-invariant latent representation. IEEE Transactions on Medical Imaging, 37(3):803–814, 2017.
  10. Pre-trained image processing transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12299–12310, 2021a.
  11. Learning continuous image representation with local implicit image function. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8628–8638, 2021b.
  12. Local implicit neural representations for multi-sequence MRI translation. In 2023 IEEE 20th International Symposium on Biomedical Imaging (ISBI), pages 1–5, 2023. .
  13. Importance of multimodal MRI in characterizing brain tissue and its potential application for individual age prediction. IEEE Journal of Biomedical and Health Informatics, 20(5):1232–1239, 2016.
  14. ResViT: Residual vision transformers for multimodal medical image synthesis. IEEE Transactions on Medical Imaging, 41(10):2598–2614, 2022.
  15. Image synthesis in multi-contrast MRI with conditional generative adversarial networks. IEEE Transactions on Medical Imaging, 38(10):2375–2388, 2019.
  16. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
  17. From data to functa: Your data point is a function and you can treat it like one. In International Conference on Machine Learning, pages 5694–5725, 2022.
  18. Watch your up-convolution: CNN based generative deep neural networks are failing to reproduce spectral distributions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7890–7899, 2020.
  19. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12873–12883, 2021.
  20. HyperNetworks. In International Conference on Learning Representations, 2017.
  21. HeMIS: Hetero-modal image segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 469–477, 2016.
  22. GANs trained by a two time-scale update rule converge to a local nash equilibrium. In Advances in neural information processing systems, volume 30, 2017.
  23. Knowledge distillation from multi-modal to mono-modal segmentation networks. In International Conference on Medical Image Computing and Computer Assisted Intervention, pages 772–781, 2020.
  24. Is synthesizing MRI contrast useful for inter-modality analysis? In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 631–638, 2013.
  25. nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature Methods, 18(2):203–211, 2021.
  26. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1125–1134, 2017.
  27. TransGAN: Two pure transformers can make one strong GAN, and that can scale up. In Advances in Neural Information Processing Systems, volume 34, pages 14745–14758, 2021.
  28. Robust multi-modal MR image synthesis. In International Conference on Medical Image Computing and Computer Assisted Intervention, pages 347–355, 2017.
  29. Denoising diffusion restoration models. In Advances in Neural Information Processing Systems, volume 35, pages 23593–23606, 2022.
  30. InstaFormer: Instance-aware image-to-image translation with transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18321–18331, June 2022.
  31. elastix: a toolbox for intensity-based medical image registration. IEEE Transactions on Medical Imaging, 29(1):196–205, 2009.
  32. DiamondGAN: Unified multi-modal generative adversarial networks for MRI sequences synthesis. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 795–803, 2019.
  33. The brain tumor segmentation (BraTS) challenge 2023: Brain MR image synthesis for tumor segmentation (BraSyn). arXiv preprint arXiv:2305.09011, 2023.
  34. Geometric GAN. arXiv preprint arXiv:1705.02894, 2017.
  35. Feature pyramid networks for object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2117–2125, 2017.
  36. One model to synthesize them all: Multi-contrast multi-scale transformer for missing data imputation. IEEE Transactions on Medical Imaging, 2023a.
  37. Cascaded multi-modal mixing transformers for Alzheimer’s disease classification with incomplete data. NeuroImage, page 120267, 2023b.
  38. Least squares generative adversarial networks. In Proceedings of the IEEE international conference on computer vision, pages 2794–2802, 2017.
  39. Single-subject multi-contrast MRI super-resolution via implicit neural representations. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 173–183. Springer, 2023.
  40. The multimodal brain tumor image segmentation benchmark (BRATS). IEEE Transactions on Medical Imaging, 34(10):1993–2024, 2014.
  41. NeRF: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99–106, 2021.
  42. Implicit neural representation in medical imaging: a comparative survey. arXiv preprint arXiv:2307.16142, 2023.
  43. Fully automated 3D vestibular schwannoma segmentation with and without gadolinium-based contrast material: A multicenter, multivendor study. Radiology: Artificial Intelligence, 4(4):e210300, 2022.
  44. Medical image synthesis with deep convolutional adversarial networks. IEEE Transactions on Biomedical Engineering, 65(12):2720–2730, 2018.
  45. DeepSDF: Learning continuous signed distance functions for shape representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 165–174, 2019.
  46. Convolutional occupancy networks. In European Conference on Computer Vision, pages 523–540, 2020.
  47. FiLM: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
  48. On the spectral bias of neural networks. In International Conference on Machine Learning, pages 5301–5310, 2019.
  49. Whole image synthesis using a deep encoder-decoder network. In International Workshop on Simulation and Synthesis in Medical Imaging, pages 127–137, 2016.
  50. Spatially-adaptive pixelwise networks for fast image translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14882–14891, June 2021.
  51. Missing MRI pulse sequence synthesis using multi-modal generative adversarial network. IEEE Transactions on Medical Imaging, 39(4):1170–1183, 2019.
  52. Multi-domain image completion for random missing input data. IEEE Transactions on Medical Imaging, 40(4):1113–1122, 2020.
  53. 4K real time image to image translation network with transformers. IEEE Access, 10:73057–73067, 2022.
  54. Implicit neural representations with periodic activation functions. In Advances in neural information processing systems, volume 33, pages 7462–7473, 2020.
  55. On the effectiveness of GAN generated cardiac MRIs for segmentation. In Medical Imaging with Deep Learning, 2020.
  56. Why does synthesized data improve multi-sequence classification? In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 531–538, 2015.
  57. Spline positional encoding for learning 3D implicit signed distance fields. In International Joint Conference on Artificial Intelligence, 2021.
  58. High-resolution image synthesis and semantic manipulation with conditional GANs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8798–8807, 2018.
  59. Fluid-attenuated inversion recovery MRI synthesis from multisequence MRI using three-dimensional fully convolutional networks for multiple sclerosis. Journal of Medical Imaging, 6(1):014005, 2019.
  60. Implicit neural representations for deformable image registration. In International Conference on Medical Imaging with Deep Learning, pages 1349–1359, 2022.
  61. Neural fields in visual computing and beyond. In Computer Graphics Forum, volume 41, pages 641–676, 2022.
  62. mustGAN: multi-stream generative adversarial networks for MR image synthesis. Medical image analysis, 70:101944, 2021.
  63. Reconstructing continuous distributions of 3D protein structure from cryo-EM images. In International Conference on Learning Representations, 2020.
Citations (3)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now