Data-Efficient Unsupervised Interpolation Without Any Intermediate Frame for 4D Medical Images (2404.01464v1)
Abstract: 4D medical images, which represent 3D images with temporal information, are crucial in clinical practice for capturing dynamic changes and monitoring long-term disease progression. However, acquiring 4D medical images poses challenges due to factors such as radiation exposure and imaging duration, necessitating a balance between achieving high temporal resolution and minimizing adverse effects. Given these circumstances, not only is data acquisition challenging, but increasing the frame rate for each dataset also proves difficult. To address this challenge, this paper proposes a simple yet effective Unsupervised Volumetric Interpolation framework, UVI-Net. This framework facilitates temporal interpolation without the need for any intermediate frames, distinguishing it from the majority of other existing unsupervised methods. Experiments on benchmark datasets demonstrate significant improvements across diverse evaluation metrics compared to unsupervised and supervised baselines. Remarkably, our approach achieves this superior performance even when trained with a dataset as small as one, highlighting its exceptional robustness and efficiency in scenarios with sparse supervision. This positions UVI-Net as a compelling alternative for 4D medical imaging, particularly in settings where data availability is limited. The source code is available at https://github.com/jungeun122333/UVI-Net.
- Vnet: An end-to-end fully convolutional neural network for road extraction from high-resolution remote sensing data. IEEE Access, 8:179424–179436, 2020.
- An unsupervised learning model for deformable medical image registration. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9252–9260, 2018.
- Voxelmorph: A learning framework for deformable medical image registration. IEEE Transactions on Medical Imaging, 38(8):1788–1800, 2019.
- Computing large deformation metric mappings via geodesic flows of diffeomorphisms. International journal of computer vision, 61:139–157, 2005.
- Itv versus mid-ventilation for treatment planning in lung sbrt: a comparison of target coverage and ptv adequacy by using in-treatment 4d cone beam ct. Radiation Oncology, 15:1–10, 2020.
- Deep learning techniques for automatic mri cardiac multi-structures segmentation and diagnosis: is the problem solved? IEEE transactions on medical imaging, 37(11):2514–2525, 2018.
- 4dct and vmat for lung patients with irregular breathing. Journal of Applied Clinical Medical Physics, 23(1):e13453, 2022.
- Deep learning based inter-modality image registration supervised by intra-modality similarity. In Machine Learning in Medical Imaging: 9th International Workshop, MLMI 2018, Held in Conjunction with MICCAI 2018, Granada, Spain, September 16, 2018, Proceedings 9, pages 55–63. Springer, 2018.
- Two deterministic half-quadratic regularization algorithms for computed imaging. In Proceedings of 1st international conference on image processing, pages 168–172. IEEE, 1994.
- Transmorph: Transformer for unsupervised medical image registration. Medical image analysis, 82:102615, 2022a.
- Videoinr: Learning video implicit neural representation for continuous space-time super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2047–2057, 2022b.
- Multimodal mri synthesis using unified generative adversarial networks. Medical physics, 47(12):6343–6354, 2020.
- Lee R Dice. Measures of the amount of ecologic association between species. Ecology, 26(3):297–302, 1945.
- Existing and emerging image quality metrics. In Canadian Conference on Electrical and Computer Engineering, 2005., pages 1906–1913, 2005.
- Accuracy of registration algorithms in subtraction ct of the lungs: A digital phantom study. Medical physics, 46(5):2264–2274, 2019.
- A spatiotemporal volumetric interpolation network for 4d dynamic medical image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4726–4735, 2020.
- Unsupervised landmark detection-based spatiotemporal motion estimation for 4-d dynamic medical images. IEEE Transactions on Cybernetics, 2021.
- Unetr: Transformers for 3d medical image segmentation. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 574–584, 2022.
- Timereplayer: Unlocking the potential of event cameras for video interpolation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17804–17813, 2022.
- Learn2reg: comprehensive multi-task medical image registration challenge, dataset and evaluation in the era of deep learning. IEEE Transactions on Medical Imaging, 2022.
- Magnetic resonance derived myocardial strain assessment using feature tracking. JoVE (Journal of Visualized Experiments), (48):e2356, 2011.
- Data from 4d lung imaging of nsclc patients. 2016.
- Myocardial tagging with mr imaging: overview of normal and pathologic findings. Radiographics, 32(5):1381–1398, 2012.
- Fourier-net+: Leveraging band-limited representation for efficient 3d medical image registration. arXiv preprint arXiv:2307.02997, 2023.
- Super slomo: High quality estimation of multiple intermediate frames for video interpolation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9000–9008, 2018.
- A unified pyramid recurrent network for video frame interpolation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1578–1587, 2023.
- R2net: Efficient and flexible diffeomorphic image registration using lipschitz continuous residual networks. Medical Image Analysis, 89:102917, 2023.
- An image interpolation approach for acquisition time reduction in navigator-based 4d mri. Medical image analysis, 54:20–29, 2019.
- Med-vt: Multiscale encoder-decoder video transformer with application to object segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6323–6333, 2023.
- Diffusion deformable model for 4d temporal medical image generation. In Medical Image Computing and Computer Assisted Intervention–MICCAI 2022: 25th International Conference, Singapore, September 18–22, 2022, Proceedings, Part I, pages 539–548. Springer, 2022.
- Cyclemorph: cycle consistent unsupervised deformable image registration. Medical image analysis, 71:102036, 2021.
- Diffusemorph: Unsupervised deformable image registration using diffusion model. In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXI, pages 347–364. Springer, 2022.
- Event-based video frame interpolation with cross-modal asymmetric bidirectional motion fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18032–18042, 2023.
- Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
- Dongyang Kuang. Cycle-consistent training for reducing negative jacobian determinant in deep registration networks. In Simulation and Synthesis in Medical Imaging: 4th International Workshop, SASHIMI 2019, Held in Conjunction with MICCAI 2019, Shenzhen, China, October 13, 2019, Proceedings 4, pages 120–129. Springer, 2019.
- Unsupervised video frame interpolation using online refinement. In Institute of Electronics, Information and Communication Engineers, 2020.
- Enhanced correlation matching based video frame interpolation. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2839–2847, 2022.
- 4d-ct deformable image registration using multiscale unsupervised deep learning. Physics in Medicine & Biology, 65(8):085003, 2020.
- Deep video frame interpolation using cyclic frame generation. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 8794–8802, 2019.
- Devon: Deformable volume network for learning optical flow. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2705–2713, 2020.
- Imaging myocardial strain. IEEE Signal Processing Magazine, 18(6):44–56, 2001.
- Regional myocardial strain measurements from 4dct in patients with normal lv function. Journal of cardiovascular computed tomography, 12(5):372–378, 2018.
- Phasenet for video frame interpolation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 498–507, 2018.
- V-net: Fully convolutional neural networks for volumetric medical image segmentation. In 2016 fourth international conference on 3D vision (3DV), pages 565–571. Ieee, 2016.
- Preoperative evaluation of pleural adhesion in patients with lung tumors using four-dimensional computed tomography performed during natural breathing. Medicine, 100(47), 2021.
- Medical image synthesis with context-aware generative adversarial networks. In Medical Image Computing and Computer Assisted Intervention- MICCAI 2017: 20th International Conference, Quebec City, QC, Canada, September 11-13, 2017, Proceedings, Part III 20, pages 417–425. Springer, 2017.
- Medical image synthesis with deep convolutional adversarial networks. IEEE Transactions on Biomedical Engineering, 65(12):2720–2730, 2018.
- Context-aware synthesis for video frame interpolation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1701–1710, 2018.
- Softmax splatting for video frame interpolation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5437–5446, 2020.
- Video frame interpolation via adaptive separable convolution. In Proceedings of the IEEE international conference on computer vision, pages 261–270, 2017.
- Biformer: Learning bilateral motion estimation via bilateral transformer for 4k video frame interpolation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1568–1577, 2023.
- Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32, pages 8024–8035. Curran Associates, Inc., 2019.
- Im-net for high resolution video frame interpolation. In Proceedings of the IEEE/CVF conference on computer vision and pattern Recognition, pages 2398–2407, 2019.
- Unsupervised video interpolation using cycle consistency. In Proceedings of the IEEE/CVF international conference on computer Vision, pages 892–900, 2019.
- U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234–241. Springer, 2015.
- Reduction of procedure times in routine clinical practice with compressed sense magnetic resonance imaging technique. PLoS One, 14(4):e0214887, 2019.
- Claude E Shannon. A mathematical theory of communication. The Bell system technical journal, 27(3):379–423, 1948.
- Medical image registration based on uncoupled learning and accumulative enhancement. In Medical Image Computing and Computer Assisted Intervention 2021, pages 3–13, Cham, 2021. Springer International Publishing.
- Xvfi: extreme video frame interpolation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 14489–14498, 2021.
- A large annotated medical image dataset for the development and evaluation of segmentation algorithms. arXiv preprint arXiv:1902.09063, 2019.
- Deep animation video interpolation in the wild. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6587–6595, 2021.
- Pierre Soille. Erosion and Dilation, pages 63–103. Springer Berlin Heidelberg, Berlin, Heidelberg, 2004.
- Nonrigid image registration using multi-scale 3d convolutional neural networks. In Medical Image Computing and Computer Assisted Intervention- MICCAI 2017: 20th International Conference, Quebec City, QC, Canada, September 11-13, 2017, Proceedings, Part I 20, pages 232–239. Springer, 2017.
- Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402, 2012.
- Deep video deblurring for hand-held cameras. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1279–1288, 2017.
- Dosimetric comparison of stereotactic body radiotherapy using 4d ct and multiphase ct images for treatment planning of lung cancer: evaluation of the impact on daily dose coverage. Radiotherapy and Oncology, 91(3):314–324, 2009.
- Risks of leukemia, intracranial tumours and lymphomas in childhood and early adulthood after pediatric radiation exposure from computed tomography. CMAJ, 195(16):E575–E583, 2023.
- Mpvf: 4d medical image inpainting by multi-pyramid voxel flows. IEEE Journal of Biomedical and Health Informatics, 2023.
- Implicit neural representations for deformable image registration. In International Conference on Medical Imaging with Deep Learning, pages 1349–1359. PMLR, 2022.
- Defining internal target volume (itv) for hepatocellular carcinoma using four-dimensional ct. Radiotherapy and Oncology, 84(3):272–278, 2007.
- Zooming slow-mo: Fast and accurate one-stage space-time video super-resolution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3370–3379, 2020.
- Video enhancement with task-oriented flow. International Journal of Computer Vision, 127:1106–1125, 2019.
- Unpaired brain mr-to-ct synthesis using a structure-constrained cyclegan. In Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support: 4th International Workshop, DLMIA 2018, and 8th International Workshop, ML-CDS 2018, Held in Conjunction with MICCAI 2018, Granada, Spain, September 20, 2018, Proceedings 4, pages 174–182. Springer, 2018.
- Quicksilver: Fast predictive image registration–a deep learning approach. NeuroImage, 158:378–396, 2017.
- Extracting motion and appearance via inter-frame attention for efficient video frame interpolation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5682–5692, 2023.
- The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586–595, 2018.
- Recursive cascaded networks for unsupervised medical image registration. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019.
- Exploring motion ambiguity and alignment for high-quality video frame interpolation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22169–22179, 2023.
- Wang Zhou. Image quality assessment: from error measurement to structural similarity. IEEE transactions on image processing, 13:600–613, 2004.