Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
125 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Towards Generalizable Tumor Synthesis (2402.19470v2)

Published 29 Feb 2024 in eess.IV and cs.CV

Abstract: Tumor synthesis enables the creation of artificial tumors in medical images, facilitating the training of AI models for tumor detection and segmentation. However, success in tumor synthesis hinges on creating visually realistic tumors that are generalizable across multiple organs and, furthermore, the resulting AI models being capable of detecting real tumors in images sourced from different domains (e.g., hospitals). This paper made a progressive stride toward generalizable tumor synthesis by leveraging a critical observation: early-stage tumors (< 2cm) tend to have similar imaging characteristics in computed tomography (CT), whether they originate in the liver, pancreas, or kidneys. We have ascertained that generative AI models, e.g., Diffusion Models, can create realistic tumors generalized to a range of organs even when trained on a limited number of tumor examples from only one organ. Moreover, we have shown that AI models trained on these synthetic tumors can be generalized to detect and segment real tumors from CT volumes, encompassing a broad spectrum of patient demographics, imaging protocols, and healthcare facilities.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (94)
  1. Automatic detection of pancreatic lesions and main pancreatic duct dilatation on portal venous ct scans using deep learning. Investigative Radiology, 2023.
  2. The medical segmentation decathlon. arXiv preprint arXiv:2106.05735, 2021.
  3. ViViT: A video vision transformer. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6836–6846, 2021.
  4. Diagnosis and staging of hepatocellular carcinoma (hcc): current guidelines. European journal of radiology, 101:72–81, 2018.
  5. Is space-time attention all you need for video understanding? In International Conference on Machine Learning, pages 813–824. PMLR, 2021.
  6. The liver tumor segmentation benchmark (lits). arXiv preprint arXiv:1901.04056, 2019.
  7. Synthseg: Segmentation of brain mri scans of any contrast and resolution without retraining. Medical image analysis, 86:102789, 2023.
  8. Harry B Burke. Outcome prediction and the future of the tnm staging system, 2004.
  9. Cancerunit: Towards a single unified model for effective detection, segmentation, and diagnosis of eight major cancers using a large collection of ct scans. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 21327–21338, 2023.
  10. Synthetic data in machine learning for medicine and healthcare. Nature Biomedical Engineering, 5(6):493–497, 2021.
  11. A review of medical image data augmentation techniques for deep learning applications. Journal of Medical Imaging and Radiation Oncology, 65(5):545–563, 2021.
  12. Ct and mr imaging diagnosis and staging of hepatocellular carcinoma: part i. development, growth, and spread: key pathologic and imaging aspects. Radiology, 272(3):635–654, 2014.
  13. Acquiring weak annotations for tumor localization in temporal and volumetric data. Machine Intelligence Research, pages 1–13, 2024.
  14. Diagnosis and detection of pancreatic cancer. The Cancer Journal, 23(6):333–342, 2017.
  15. Utility of ct radiomics features in differentiation of pancreatic ductal adenocarcinoma from normal pancreatic tissue. American Journal of Roentgenology, 213(2):349–357, 2019.
  16. Generative adversarial networks: An overview. IEEE signal processing magazine, 35(1):53–65, 2018.
  17. Boosting dermatoscopic lesion segmentation via diffusion models with visual and textual prompts. arXiv preprint arXiv:2310.02906, 2023.
  18. Implicit generation and modeling with energy based models. Advances in Neural Information Processing Systems, 32, 2019.
  19. N Reed Dunnick. Renal cell carcinoma: staging and surveillance. Abdominal Radiology, 41:1079–1085, 2016.
  20. Imaging diagnosis and staging of pancreatic ductal adenocarcinoma: a comprehensive review. Insights into imaging, 11(1):1–13, 2020.
  21. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873–12883, 2021.
  22. Can segmentation models be trained with fully synthetically generated data? In International Workshop on Simulation and Synthesis in Medical Imaging, pages 79–90. Springer, 2022.
  23. Tnm staging system for renal-cell carcinoma: current status and future perspectives. The lancet oncology, 8(6):554–558, 2007.
  24. Pathologic, molecular, and prognostic radiologic features of hepatocellular carcinoma. Radiographics, 41(6):1611–1631, 2021.
  25. Synthetic data accelerates the development of generalizable learning-based algorithms for x-ray image analysis. Nature Machine Intelligence, 5(3):294–308, 2023.
  26. Pet image denoising based on denoising diffusion probabilistic model. European Journal of Nuclear Medicine and Molecular Imaging, pages 1–11, 2023.
  27. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
  28. Deep learning. MIT press Cambridge, 2016.
  29. Generative adversarial networks. Communications of the ACM, 63(11):139–144, 2020.
  30. Synthesizing diverse lung nodules wherever massively: 3d multi-conditional gan-based ct image augmentation for object detection. In 2019 International Conference on 3D Vision (3DV), pages 729–737. IEEE, 2019.
  31. Swin unetr: Swin transformers for semantic segmentation of brain tumors in mri images. In International MICCAI Brainlesion Workshop, pages 272–284. Springer, 2021.
  32. An international challenge to use artificial intelligence to define the state-of-the-art in kidney and kidney tumor segmentation in ct imaging., 2020.
  33. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840–6851, 2020.
  34. Metgan: Generative tumour inpainting and modality synthesis in light sheet microscopy. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 227–237, 2022.
  35. Synthetic tumors make ai segment tumors better. NeurIPS Workshop on Medical Imaging meets NeurIPS, 2022.
  36. Label-free liver tumor segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7422–7432, 2023a.
  37. Synthetic data as validation, 2023b.
  38. nnu-net: a self-configuring method for deep learning-based biomedical image segmentation. Nature Methods, 18(2):203–211, 2021.
  39. Deflating dataset bias using synthetic data augmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 772–773, 2020.
  40. Free-form tumor synthesis in computed tomography images via richer generative adversarial network. Knowledge-Based Systems, 218:106753, 2021.
  41. Pate-gan: Generating synthetic data with differential privacy guarantees. In International conference on learning representations, 2018.
  42. Label-assemble: Leveraging multiple datasets with partial labels. In 2023 IEEE 20th International Symposium on Biomedical Imaging (ISBI), pages 1–5. IEEE, 2023.
  43. Diffusion adversarial representation learning for self-supervised vessel segmentation. arXiv preprint arXiv:2209.14566, 2022.
  44. Maximum likelihood training of parametrized diffusion model. 2021.
  45. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
  46. Semi-supervised learning with deep generative models. Advances in neural information processing systems, 27, 2014.
  47. An introduction to variational autoencoders. Foundations and Trends® in Machine Learning, 12(4):307–392, 2019.
  48. Normalizing flows: An introduction and review of current methods. IEEE transactions on pattern analysis and machine intelligence, 43(11):3964–3979, 2020.
  49. Combining in vitro diagnostics with in vivo imaging for earlier detection of pancreatic ductal adenocarcinoma: challenges and solutions. Radiology, 277(3):644–661, 2015.
  50. A tutorial on energy-based learning. Predicting structured data, 1(0), 2006.
  51. Imaging renal cell carcinoma with ultrasonography, ct and mri. Nature Reviews Urology, 7(6):311–325, 2010.
  52. Early detection and localization of pancreatic cancer by label-free tumor synthesis. MICCAI Workshop on Big Task Small Data, 1001-AI, 2023.
  53. How well do supervised models transfer to 3d image segmentation? In The Twelfth International Conference on Learning Representations, 2024.
  54. Clip-driven universal model for organ segmentation and tumor detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 21152–21164, 2023.
  55. Pseudo-label guided image synthesis for semi-supervised covid-19 pneumonia infection segmentation. IEEE Transactions on Medical Imaging, 2022a.
  56. Learning from synthetic ct images via test-time training for liver tumor segmentation. IEEE transactions on medical imaging, 41(9):2510–2520, 2022b.
  57. Conversion between ct and mri images using diffusion and score-matching models. arXiv preprint arXiv:2209.12104, 2022.
  58. A novel unified conditional score-based generative framework for multi-modal medical image completion. arXiv preprint arXiv:2207.03430, 2022.
  59. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pages 8162–8171. PMLR, 2021.
  60. Multi-domain adaptation in brain mri through paired consistency and adversarial learning. In Domain Adaptation and Representation Transfer and Medical Image Learning with Less Labels and Imperfect Data, pages 54–62. Springer, 2019.
  61. Unsupervised medical image translation with adversarial diffusion models. IEEE Transactions on Medical Imaging, 2023.
  62. Normalizing flows for probabilistic modeling and inference. The Journal of Machine Learning Research, 22(1):2617–2680, 2021.
  63. Abdomenatlas-8k: Annotating 8,000 abdominal ct volumes for multi-organ segmentation in three weeks. Conference on Neural Information Processing Systems, 2023.
  64. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):3, 2022.
  65. Tnm staging of neoplasms of the endocrine pancreas: results from a large international cohort study. Journal of the National Cancer Institute, 104(10):764–777, 2012.
  66. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022.
  67. Palette: Image-to-image diffusion models. In ACM SIGGRAPH 2022 Conference Proceedings, pages 1–10, 2022.
  68. Abnormal colon polyp image synthesis using conditional adversarial networks for improved detection performance. IEEE Access, 6:56007–56017, 2018.
  69. Learning fixed points in generative adversarial networks: From image-to-image translation to disease detection and localization. In Proceedings of the IEEE International Conference on Computer Vision, pages 191–200, 2019.
  70. Arthur T Skarin. Atlas of Diagnostic Oncology E-Book. Elsevier Health Sciences, 2015.
  71. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, pages 2256–2265. PMLR, 2015.
  72. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020.
  73. Solving inverse problems in medical imaging with score-based generative models. arXiv preprint arXiv:2111.08005, 2021.
  74. Score-based generative modeling in latent space. Advances in Neural Information Processing Systems, 34:11287–11302, 2021.
  75. Computational radiomics system to decode the radiographic phenotype. Cancer research, 77(21):e104–e107, 2017.
  76. Comparison of machine learning methods for classifying mediastinal lymph node metastasis of non-small cell lung cancer from 18f-fdg pet/ct images. EJNMMI research, 7(1):1–11, 2017.
  77. Anomaly segmentation in retinal images with poisson-blending data augmentation. Medical Image Analysis, page 102534, 2022.
  78. CT and MRI of small renal masses. Br J Radiol, 91(1087):20180131, 2018.
  79. Pancreatic image augmentation based on local region texture synthesis for tumor segmentation. In International Conference on Artificial Neural Networks, pages 419–431. Springer, 2022.
  80. Diffusion models for implicit image segmentation ensembles. In International Conference on Medical Imaging with Deep Learning, pages 1336–1348. PMLR, 2022.
  81. Anoddpm: Anomaly detection with denoising diffusion probabilistic models using simplex noise. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 650–656, 2022.
  82. The felix project: Deep networks to detect pancreatic neoplasms. medRxiv, 2022.
  83. Squid: Deep feature in-painting for unsupervised anomaly detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23890–23901, 2023.
  84. Measurement-conditioned denoising diffusion probabilistic model for under-sampled medical image reconstruction. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 655–664. Springer, 2022.
  85. Class-aware adversarial lung nodule synthesis in ct images. In 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019), pages 1348–1352. IEEE, 2019.
  86. Label-free segmentation of covid-19 lesions in lung ct. IEEE Transactions on Medical Imaging, 2021.
  87. Time-series generative adversarial networks. Advances in neural information processing systems, 32, 2019.
  88. Fastflow: Unsupervised anomaly detection and localization via 2d normalizing flows. arXiv preprint arXiv:2111.07677, 2021.
  89. Self-supervised tumor segmentation with sim2real adaptation. IEEE Journal of Biomedical and Health Informatics, 2023a.
  90. Continual learning for abdominal multi-organ and tumor segmentation. In International conference on medical image computing and computer-assisted intervention, pages 35–45. Springer, 2023b.
  91. Unsupervised liver tumor segmentation with pseudo anomaly synthesis. In International Workshop on Simulation and Synthesis in Medical Imaging, pages 86–96. Springer, 2023c.
  92. Energy-based generative adversarial network. arXiv preprint arXiv:1609.03126, 2016.
  93. Zongwei Zhou. Towards Annotation-Efficient Deep Learning for Computer-Aided Diagnosis. PhD thesis, Arizona State University, 2021.
  94. Interpreting medical images. In Intelligent Systems in Medicine and Health, pages 343–371. Springer, 2022.
Citations (22)
View on Semantic Scholar

Summary

  • The paper presents DiffTumor, a generative AI framework that synthesizes realistic tumors across multiple organs using minimal annotated data.
  • It employs a three-stage process with autoencoder, diffusion, and segmentation models, validated through visual Turing tests and radiomics analysis.
  • The approach significantly improves early-stage tumor detection, achieving up to a 10.7% Dice Similarity Coefficient gain and robust generalizability across demographics.

Leveraging Generative AI for Cross-Organ Tumor Synthesis

Introduction

The synthesis of tumors in medical images, particularly using computed tomography (CT), stands at the forefront of enhancing training datasets for AI-driven tumor detection and segmentation models. This paper introduces a novel approach, coined DiffTumor, to generate realistic, synthetic tumors across different organs, such as the liver, pancreas, and kidneys. This initiative stems from the observation that early-stage tumors share similar imaging characteristics across these organs, thus enabling the generation of generalized synthetic tumors from sparse annotated examples, and significantly augmenting the datasets necessary for training robust AI models.

Methodology

Observation and Preliminary Validation

The foundation of DiffTumor is the observed imaging similarity of sub-2cm early-stage tumors across various organs in CT images. This was rigorously validated through a blend of radiologist reader studies, radiomics feature analysis, deep feature analysis, and clinical evidence, establishing their generalized imaging characteristics despite originating from different organs.

The DiffTumor Framework

The approach unfolds in three stages:

  1. Autoencoder Model Training: A novel autoencoder model is trained to compress three-dimensional CT volumes into a lower-dimensional latent space, enhancing the model's generalization capability across different patient demographics and reducing the necessity for large numbers of annotated tumor volumes.
  2. Diffusion Model Training: Leveraging the latent features and tumor masks, a diffusion model generates the necessary latent features to reconstruct CT volumes with synthetic tumors based on arbitrary masks.
  3. Segmentation Model Training: Employing synthetic tumors and their corresponding masks, a segmentation model is trained to detect and segment real tumors within CT volumes, further substantiated by a comprehensive dataset of healthy organs.

Results

Efficacy of DiffTumor

  • Visual Turing Test: Conducted with real and synthetic tumors across three organs, it showcased the synthetic tumors' striking resemblance to real ones, attested by radiologists' challenges in distinguishing between them.
  • Generalization Across Organs: Demonstrated remarkable generalizability of synthetic tumors generated for one organ to others, notably improving early-stage tumor detection with Dice Similarity Coefficient (DSC) improvements up to +10.7% for specific organ models.
  • Demographic Generalizability: Significantly enhanced tumor detection and segmentation across diverse patient demographics, including various ages and genders, marking an average DSC improvement of +6.9%.

Advantages Highlighted

  • The requirement of minimal annotated data for efficient training of the Diffusion Model, deviating from conventional extensive data dependency.
  • The accelerated tumor synthesis process, generating tumors in mere milliseconds, aligns with the demand for rapid data augmentation.
  • A significant leap in the detection and segmentation of early-stage tumors, essential for improving clinical outcomes and survival rates.

Conclusions and Future Directions

DiffTumor has carved a niche in tumor synthesis by facilitating the creation of visually realistic, synthetic tumors across various organs from a minimal set of annotated examples. This breakthrough not only promises to enrich AI model training datasets significantly but also improves model generalizability across patient demographics and healthcare facilities, a pivotal step towards personalized and precise cancer detection and assessment.

The paper's implications extend beyond current achievements, potentially catalyzing further research into generative models' applicability across diverse medical imaging modalities and diseases, thereby broadening the horizons of AI in healthcare.

Acknowledgments

The research benefited from the support of the Lustgarten Foundation for Pancreatic Cancer Research and the Patrick J. McGovern Foundation Award, highlighting the collaborative effort behind this innovative journey.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown