Papers
Topics
Authors
Recent
AI Research Assistant
AI Research Assistant
Well-researched responses based on relevant abstracts and paper content.
Custom Instructions Pro
Preferences or requirements that you'd like Emergent Mind to consider when generating responses.
Gemini 2.5 Flash
Gemini 2.5 Flash 60 tok/s
Gemini 2.5 Pro 46 tok/s Pro
GPT-5 Medium 14 tok/s Pro
GPT-5 High 15 tok/s Pro
GPT-4o 93 tok/s Pro
Kimi K2 156 tok/s Pro
GPT OSS 120B 441 tok/s Pro
Claude Sonnet 4 37 tok/s Pro
2000 character limit reached

Unified Cross-Modal Image Synthesis with Hierarchical Mixture of Product-of-Experts (2410.19378v1)

Published 25 Oct 2024 in cs.CV, cs.LG, and eess.IV

Abstract: We propose a deep mixture of multimodal hierarchical variational auto-encoders called MMHVAE that synthesizes missing images from observed images in different modalities. MMHVAE's design focuses on tackling four challenges: (i) creating a complex latent representation of multimodal data to generate high-resolution images; (ii) encouraging the variational distributions to estimate the missing information needed for cross-modal image synthesis; (iii) learning to fuse multimodal information in the context of missing data; (iv) leveraging dataset-level information to handle incomplete data sets at training time. Extensive experiments are performed on the challenging problem of pre-operative brain multi-parametric magnetic resonance and intra-operative ultrasound imaging.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (44)
  1. P. Juvekar, R. Dorent, F. Kögl, E. Torio, C. Barr, L. Rigolo, C. Galvin, N. Jowkar, A. Kazi, N. Haouchine, H. Cheema, N. Navab, S. Pieper, W. M. Wells, W. L. Bi, A. Golby, S. Frisken, and T. Kapur, “ReMIND: The Brain Resection Multimodal Imaging Database,” medRxiv, 2023.
  2. H. Shen, X. Li, Q. Cheng, C. Zeng, G. Yang, H. Li, and L. Zhang, “Missing Information Reconstruction of Remote Sensing Data: A Technical Review,” IEEE Geoscience and Remote Sensing Magazine, vol. 3, no. 3, pp. 61–85, 2015.
  3. Z. Wang, Y. Wu, and Q. Niu, “Multi-Sensor Fusion in Automated Driving: A Survey,” IEEE Access, vol. 8, pp. 2847–2868, 2020.
  4. D. B. Rubin, “Multiple Imputation After 18+ Years,” Journal of the American Statistical Association, vol. 91, no. 434, pp. 473–489, 1996.
  5. O. Troyanskaya, M. Cantor, G. Sherlock, P. Brown, T. Hastie, R. Tibshirani, D. Botstein, and R. B. Altman, “Missing value estimation methods for DNA microarrays ,” Bioinformatics, vol. 17, no. 6, pp. 520–525, 06 2001.
  6. Y. Dong, Q.-A. Fu, X. Yang, T. Pang, H. Su, Z. Xiao, and J. Zhu, “Benchmarking adversarial robustness on image classification,” in CVPR, June 2020.
  7. T. Wang, Y. Lei, Y. Fu, J. F. Wynne, W. J. Curran, T. Liu, and X. Yang, “A review on medical imaging synthesis using deep learning and its clinical applications,” Journal of applied clinical medical physics, vol. 22, no. 1, pp. 11–36, 2021.
  8. P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros, “Image-To-Image Translation With Conditional Adversarial Networks,” in CVPR, 2017.
  9. T. Park, M.-Y. Liu, T.-C. Wang, and J.-Y. Zhu, “Semantic Image Synthesis with Spatially-Adaptive Normalization,” in CVPR, 2019.
  10. M. Donnez, F.-X. Carton, F. Le Lann, E. De Schlichting, and M. Chabanas, “Realistic synthesis of brain tumor resection ultrasound images with a generative adversarial network,” in Medical Imaging 2021: Image-Guided Procedures, Robotic Interventions, and Modeling, vol. 11598.   SPIE, 2021, pp. 637–642.
  11. J. Jiao, A. I. L. Namburete, A. T. Papageorghiou, and J. A. Noble, “Self-Supervised Ultrasound to MRI Fetal Brain Image Synthesis,” IEEE Transactions on Medical Imaging, vol. 39, no. 12, 2020.
  12. A. Chartsias, T. Joyce, G. Papanastasiou, S. Semple, M. Williams, D. E. Newby, R. Dharmakumar, and S. A. Tsaftaris, “Disentangled representation learning in cardiac image analysis,” Medical image analysis, vol. 58, p. 101535, 2019.
  13. A. Sharma and G. Hamarneh, “Missing MRI Pulse Sequence Synthesis Using Multi-Modal Generative Adversarial Network,” IEEE Transactions on Medical Imaging, vol. 39, no. 4, pp. 1170–1183, 2020.
  14. O. Dalmaz, M. Yurt, and T. Çukur, “Resvit: Residual vision transformers for multimodal medical image synthesis,” IEEE Transactions on Medical Imaging, vol. 41, no. 10, pp. 2598–2614, 2022.
  15. H. Li, J. C. Paetzold, A. Sekuboyina, F. Kofler, J. Zhang, J. S. Kirschke, B. Wiestler, and B. Menze, “DiamondGAN: Unified Multi-Modal Generative Adversarial Networks for MRI Sequences Synthesis,” in MICCAI 2019, 2019.
  16. D. Lee, J. Kim, W.-J. Moon, and J. C. Ye, “CollaGAN: Collaborative GAN for missing image data imputation,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 2487–2496.
  17. M. Havaei, N. Guizard, N. Chapados, and Y. Bengio, “Hemis: Hetero-modal image segmentation,” in MICCAI 2016.   Cham: Springer, 2016, pp. 469–477.
  18. R. Dorent, T. Booth, W. Li, C. H. Sudre, S. Kafiabadi, J. Cardoso, S. Ourselin, and T. Vercauteren, “Learning joint segmentation of tissues and brain lesions from task-specific hetero-modal domain-shifted datasets,” Medical image analysis, vol. 67, 2021.
  19. A. Chartsias, T. Joyce, M. V. Giuffrida, and S. A. Tsaftaris, “Multimodal mr synthesis via modality-invariant latent representation,” IEEE transactions on medical imaging, vol. 37, no. 3, pp. 803–814, 2017.
  20. T. Zhou, H. Fu, G. Chen, J. Shen, and L. Shao, “Hi-net: hybrid-fusion network for multi-modal mr image synthesis,” IEEE transactions on medical imaging, vol. 39, no. 9, pp. 2772–2781, 2020.
  21. M. Wu and N. Goodman, “Multimodal Generative Models for Scalable Weakly-Supervised Learning,” NeurIPS, vol. 31, 2018.
  22. R. Dorent, S. Joutard, M. Modat, S. Ourselin, and T. Vercauteren, “Hetero-modal variational encoder-decoder for joint modality completion and segmentation,” in MICCAI 2019, 2019.
  23. R. Dorent, N. Haouchine, F. Kogl, S. Joutard, P. Juvekar, E. Torio, A. J. Golby, S. Ourselin, S. Frisken, T. Vercauteren, T. Kapur, and W. M. Wells, “Unified Brain MR-Ultrasound Synthesis Using Multi-modal Hierarchical Representations,” in MICCAI 2023, 2023, pp. 448–458.
  24. D. P. Kingma and M. Welling, “Auto-encoding variational Bayes,” in ICLR, 2014.
  25. A. Vahdat and J. Kautz, “NVAE: A Deep Hierarchical Variational Autoencoder,” NeurIPS, vol. 33, 2020.
  26. R. Ranganath, D. Tran, and D. Blei, “Hierarchical Variational Models,” in ICML, 2016.
  27. L. Maaløe, M. Fraccaro, V. Liévin, and O. Winther, “BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling,” NeurIPS, 2019.
  28. C. K. Sønderby, T. Raiko, L. Maaløe, S. K. Sønderby, and O. Winther, “Ladder Variational Autoencoders,” NeurIPS, 2016.
  29. Y. Shi, B. Paige, P. Torr et al., “Variational mixture-of-experts autoencoders for multi-modal deep generative models,” NeurIPS, vol. 32, 2019.
  30. J. Kittler, M. Hatef, R. P. Duin, and J. Matas, “On combining classifiers,” IEEE transactions on pattern analysis and machine intelligence, vol. 20, no. 3, pp. 226–239, 1998.
  31. M. Modat, G. R. Ridgway, Z. A. Taylor, M. Lehmann, J. Barnes, D. J. Hawkes, N. C. Fox, and S. Ourselin, “Fast free-form deformation using graphics processing units,” Computer Methods and Programs in Biomedicine, vol. 98, 2010.
  32. D. Drobny, T. Vercauteren, S. Ourselin, and M. Modat, “Registration of MRI and iUS data to compensate brain shift using a symmetric block-matching based approach,” in CuRIOUS, 2018.
  33. S. Bakas, C. Sako, H. Akbari, M. Bilello, A. Sotiras, G. Shukla, J. D. Rudie, N. F. Santamaría, A. F. Kazerooni, S. Pati et al., “The University of Pennsylvania glioblastoma (UPenn-GBM) cohort: advanced MRI, clinical, genomics, & radiomics,” Scientific data, vol. 9, no. 1, p. 453, 2022.
  34. S. Bakas and et al, “Identifying the Best Machine Learning Algorithms for Brain Tumor Segmentation, Progression Assessment, and Overall Survival Prediction in the BRATS Challenge,” 2019.
  35. B. Behboodi, F.-X. Carton, M. Chabanas, S. D. Ribaupierre, O. Solheim, B. K. R. Munkvold, H. Rivaz, Y. Xiao, and I. Reinertsen, “RESECT-SEG: Open access annotations of intra-operative brain tumor ultrasound images,” 2022.
  36. M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen, “MobileNetV2: Inverted Residuals and Linear Bottlenecks,” in CVPR, 2018.
  37. J. Hu, L. Shen, and G. Sun, “Squeeze-and-excitation networks,” in CVPR, 2018.
  38. B. Billot, D. N. Greve, O. Puonti, A. Thielscher, K. Van Leemput, B. Fischl, A. V. Dalca, and J. E. Iglesias, “Synthseg: Segmentation of brain MRI scans of any contrast and resolution without retraining,” Medical Image Analysis, vol. 86, p. 102789, 2023.
  39. R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang, “The Unreasonable Effectiveness of Deep Features as a Perceptual Metric,” in CVPR, 2018.
  40. R. Dorent, A. Kujawa, M. Ivory, S. Bakas, N. Rieke et al., “Crossmoda 2021 challenge: Benchmark of cross-modality domain adaptation techniques for vestibular schwannoma and cochlea segmentation,” Medical Image Analysis, vol. 83, 2023.
  41. R. Dorent, E. Torio, N. Haouchine, C. Galvin, S. Frisken, A. Golby, T. Kapur, and W. Wells, “Patient-specific real-time segmentation in trackerless brain ultrasound,” 2024.
  42. F. Isensee, P. F. Jaeger, S. A. Kohl, J. Petersen, and K. H. Maier-Hein, “nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation,” Nature methods, vol. 18, no. 2, pp. 203–211, 2021.
  43. R. Dorent, E. Torio, N. Haouchine, P. Juvekar, S. Frisken, A. J. Golby, T. Kapur, and W. Wells, “223 an artificial intelligence framework for brain tumor delineation in intraoperative ultrasound,” Neurosurgery, vol. 70, no. Supplement_1, p. 60, 2024.
  44. T. C. W. Mok and A. C. S. Chung, “Large deformation diffeomorphic image registration with laplacian pyramid networks,” in MICCAI 2020, 2020, pp. 211–221.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Lightbulb On Streamline Icon: https://streamlinehq.com

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
List To Do Tasks Checklist Streamline Icon: https://streamlinehq.com

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up

Don't miss out on important new AI/ML research

See which papers are being discussed right now on X, Reddit, and more:

Explore Trending Papers

“Emergent Mind helps me see which AI papers have caught fire online.”

Philip

Philip

Creator, AI Explained on YouTube