Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
156 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

MEMO: Dataset and Methods for Robust Multimodal Retinal Image Registration with Large or Small Vessel Density Differences (2309.14550v2)

Published 25 Sep 2023 in eess.IV and cs.CV

Abstract: The measurement of retinal blood flow (RBF) in capillaries can provide a powerful biomarker for the early diagnosis and treatment of ocular diseases. However, no single modality can determine capillary flowrates with high precision. Combining erythrocyte-mediated angiography (EMA) with optical coherence tomography angiography (OCTA) has the potential to achieve this goal, as EMA can measure the absolute 2D RBF of retinal microvasculature and OCTA can provide the 3D structural images of capillaries. However, multimodal retinal image registration between these two modalities remains largely unexplored. To fill this gap, we establish MEMO, the first public multimodal EMA and OCTA retinal image dataset. A unique challenge in multimodal retinal image registration between these modalities is the relatively large difference in vessel density (VD). To address this challenge, we propose a segmentation-based deep-learning framework (VDD-Reg) and a new evaluation metric (MSD), which provide robust results despite differences in vessel density. VDD-Reg consists of a vessel segmentation module and a registration module. To train the vessel segmentation module, we further designed a two-stage semi-supervised learning framework (LVD-Seg) combining supervised and unsupervised losses. We demonstrate that VDD-Reg outperforms baseline methods quantitatively and qualitatively for cases of both small VD differences (using the CF-FA dataset) and large VD differences (using our MEMO dataset). Moreover, VDD-Reg requires as few as three annotated vessel segmentation masks to maintain its accuracy, demonstrating its feasibility.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (71)
  1. M. T. Nicolela, B. E. Walman, A. R. Buckley, and S. M. Drance, “Ocular hypertension and primary open-angle glaucoma: a comparative study of their retrobulbar blood flow velocity.,” Journal of glaucoma, vol. 5, no. 5, pp. 308–310, 1996.
  2. V. Patel, S. Rassam, R. Newsom, J. Wiek, and E. Kohner, “Retinal blood flow in diabetic retinopathy.,” British Medical Journal, vol. 305, no. 6855, pp. 678–683, 1992.
  3. T. A. Ciulla et al., “Color doppler imaging discloses reduced ocular blood flow velocities in nonexudative age-related macular degeneration,” American journal of ophthalmology, vol. 128, no. 1, pp. 75–80, 1999.
  4. G. T. Feke, B. T. Hyman, R. A. Stern, and L. R. Pasquale, “Retinal blood flow in mild cognitive impairment and alzheimer’s disease,” Alzheimer’s & Dementia: Diagnosis, Assessment & Disease Monitoring, vol. 1, no. 2, pp. 144–151, 2015.
  5. F. Berisha, G. T. Feke, C. L. Trempe, J. W. McMeel, and C. L. Schepens, “Retinal abnormalities in early alzheimer’s disease,” Investigative ophthalmology & visual science, vol. 48, no. 5, pp. 2285–2289, 2007.
  6. C. E. Riva, M. Geiser, B. L. Petrig, and O. B. F. R. Association, “Ocular blood flow assessment using continuous laser doppler flowmetry,” Acta ophthalmologica, vol. 88, no. 6, pp. 622–629, 2010.
  7. B. Lee et al., “En face doppler optical coherence tomography measurement of total retinal blood flow in diabetic retinopathy and diabetic macular edema,” JAMA ophthalmology, vol. 135, no. 3, pp. 244–251, 2017.
  8. Y. Jia et al., “Split-spectrum amplitude-decorrelation angiography with optical coherence tomography,” Optics express, vol. 20, no. 4, pp. 4710–4725, 2012.
  9. W. Goebel, W. E. Lieb, A. Ho, R. C. Sergott, R. Farhoumand, and F. Grehn, “Color doppler imaging: a new technique to assess orbital blood flow in patients with diabetic retinopathy.,” Investigative ophthalmology & visual science, vol. 36, no. 5, pp. 864–870, 1995.
  10. A. Roorda, “Applications of adaptive optics scanning laser ophthalmoscopy,” Optometry and vision science: official publication of the American Academy of Optometry, vol. 87, no. 4, p. 260, 2010.
  11. S. Arichika, A. Uji, M. Hangai, S. Ooto, and N. Yoshimura, “Noninvasive and direct monitoring of erythrocyte aggregates in human retinal microvasculature using adaptive optics scanning laser ophthalmoscopy,” Investigative ophthalmology & visual science, vol. 54, no. 6, pp. 4394–4402, 2013.
  12. M. Pircher and R. J. Zawadzki, “Review of adaptive optics oct (ao-oct): principles and applications for retinal imaging,” Biomedical optics express, vol. 8, no. 5, pp. 2536–2562, 2017.
  13. J. Carroll, D. B. Kay, D. Scoles, A. Dubra, and M. Lombardo, “Adaptive optics retinal imaging–clinical opportunities and challenges,” Current Eye Research, vol. 38, no. 7, pp. 709–721, 2013.
  14. R. Flower, E. Peiretti, M. Magnani, L. Rossi, S. Serafini, Z. Gryczynski, and I. Gryczynski, “Observation of erythrocyte dynamics in the retinal capillaries and choriocapillaris using icg-loaded erythrocyte ghost cells,” Investigative ophthalmology & visual science, vol. 49, no. 12, pp. 5510–5516, 2008.
  15. O. Saeedi et al., “Determination of absolute erythrocyte velocity and flow in the human retinal microvasculature by direct visualization of icg-labelled erythrocytes,” Investigative Ophthalmology & Visual Science, vol. 59, no. 9, pp. 3950–3950, 2018.
  16. B. M. Tracey et al., “Measurement of retinal microvascular blood velocity using erythrocyte mediated velocimetry,” Scientific reports, vol. 9, no. 1, p. 20178, 2019.
  17. S. Asanad, A. Park, J. Pottenburgh, A. Siddiqui, L. Mayo, and O. J. Saeedi, “Erythrocyte-mediated angiography: quantifying absolute episcleral blood flow in humans,” Ophthalmology, vol. 128, no. 5, pp. 799–801, 2021.
  18. D. Wang, A. Haytham, L. Mayo, Y. Tao, and O. Saeedi, “Automated retinal microvascular velocimetry based on erythrocyte mediated angiography,” Biomedical optics express, vol. 10, no. 7, pp. 3681–3697, 2019.
  19. A. H. Kashani et al., “Optical coherence tomography angiography: a comprehensive review of current methods and clinical applications,” Progress in retinal and eye research, vol. 60, pp. 66–100, 2017.
  20. S. S. Gao et al., “Optical coherence tomography angiography,” Investigative ophthalmology & visual science, vol. 57, no. 9, pp. OCT27–OCT36, 2016.
  21. Y. Watanabe, Y. Takahashi, and H. Numazawa, “Graphics processing unit accelerated intensity-based optical coherence tomography angiography using differential frames with real-time motion correction,” Journal of biomedical optics, vol. 19, no. 2, pp. 021105–021105, 2014.
  22. M. Santarossa et al., “Medregnet: unsupervised multimodal retinal-image registration with gans and ranking loss,” in Medical Imaging 2022: Image Processing, vol. 12032, pp. 321–333, SPIE, 2022.
  23. T. De Silva, E. Y. Chew, N. Hotaling, and C. A. Cukras, “Deep-learning based multi-modal retinal image registration for the longitudinal analysis of patients with age-related macular degeneration,” Biomedical Optics Express, vol. 12, no. 1, pp. 619–636, 2021.
  24. G. Luo et al., “Multimodal affine registration for icga and mcsl fundus images of high myopia,” Biomedical Optics Express, vol. 11, no. 8, pp. 4443–4457, 2020.
  25. M. Arikan, A. Sadeghipour, B. Gerendas, R. Told, and U. Schmidt-Erfurt, “Deep learning based multi-modal registration for retinal imaging,” in Interpretability of Machine Intelligence in Medical Image Computing and Multimodal Learning for Clinical Decision Support: Second International Workshop, iMIMIC 2019, and 9th International Workshop, ML-CDS 2019, Held in Conjunction with MICCAI 2019, Shenzhen, China, October 17, 2019, Proceedings 9, pp. 75–82, Springer, 2019.
  26. J. A. Lee, P. Liu, J. Cheng, and H. Fu, “A deep step pattern representation for multimodal retinal image registration,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5077–5086, 2019.
  27. J. Zhang et al., “Two-step registration on multi-modal retinal images via deep neural networks,” IEEE Transactions on Image Processing, vol. 31, pp. 823–838, 2021.
  28. Y. Wang, J. Zhang, M. Cavichini, D.-U. G. Bartsch, W. R. Freeman, T. Q. Nguyen, and C. An, “Robust content-adaptive global registration for multimodal retinal images using weakly supervised deep-learning framework,” IEEE Transactions on Image Processing, vol. 30, pp. 3167–3178, 2021.
  29. A. Sindel, B. Hohberger, A. Maier, and V. Christlein, “Multi-modal retinal image registration using a keypoint-based vessel structure aligning network,” in International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 108–118, Springer, 2022.
  30. S. Hajeb Mohammad Alipour, H. Rabbani, and M. R. Akhlaghi, “Diabetic retinopathy grading by digital curvelet transform,” Computational and mathematical methods in medicine, vol. 2012, 2012.
  31. S.-E. Chen, V. Chen, J. Pottenburgh, and O. Saeedi, “In vivo measurement of plexus-specific retinal erythrocyte velocity and acceleration in human subjects and nhps,” Investigative Ophthalmology & Visual Science, vol. 63, no. 7, pp. 3502–3502, 2022.
  32. R. Rocholz, F. Corvi, J. Weichsel, S. Schmidt, and G. Staurenghi, “Oct angiography (octa) in retinal diagnostics,” High resolution imaging in microscopy and ophthalmology: new frontiers in biomedical optics, pp. 135–160, 2019.
  33. J. Johnson, A. Alahi, and L. Fei-Fei, “Perceptual losses for real-time style transfer and super-resolution,” in Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pp. 694–711, Springer, 2016.
  34. E. Decenciere et al., “Teleophta: Machine learning and image processing methods for teleophthalmology,” Irbm, vol. 34, no. 2, pp. 196–203, 2013.
  35. K. M. Adal, P. G. van Etten, J. P. Martinez, L. J. van Vliet, and K. A. Vermeer, “Accuracy assessment of intra-and intervisit fundus image registration for diabetic retinopathy screening,” Investigative ophthalmology & visual science, vol. 56, no. 3, pp. 1805–1812, 2015.
  36. M. Ortega, M. G. Penedo, J. Rouco, N. Barreira, and M. J. Carreira, “Retinal verification using a feature points-based biometric pattern,” EURASIP Journal on Advances in Signal Processing, vol. 2009, pp. 1–13, 2009.
  37. C. Hernandez-Matas, X. Zabulis, A. Triantafyllou, P. Anyfanti, S. Douma, and A. A. Argyros, “Fire: fundus image registration dataset,” Modeling and Artificial Intelligence in Ophthalmology, vol. 1, no. 4, pp. 16–28, 2017.
  38. L. Ding et al., “Flori21: Fluorescein angiography longitudinal retinal image registration dataset,” IEEE Dataport, 2021.
  39. L. Ding, A. E. Kuriyan, R. S. Ramchandran, C. C. Wykoff, and G. Sharma, “Weakly-supervised vessel detection in ultra-widefield fundus photography via iterative multi-modal registration and learning,” IEEE Transactions on Medical Imaging, vol. 40, no. 10, pp. 2748–2758, 2020.
  40. M. Li et al., “Image projection network: 3d to 2d image segmentation in octa images,” IEEE Transactions on Medical Imaging, vol. 39, no. 11, pp. 3343–3354, 2020.
  41. J. Chen, J. Tian, N. Lee, J. Zheng, R. T. Smith, and A. F. Laine, “A partial intensity invariant feature descriptor for multimodal retinal image registration,” IEEE Transactions on Biomedical Engineering, vol. 57, no. 7, pp. 1707–1718, 2010.
  42. Z. Ghassabi, J. Shanbehzadeh, A. Sedaghat, and E. Fatemizadeh, “An efficient approach for robust multimodal retinal image registration based on ur-sift features and piifd descriptors,” EURASIP Journal on Image and Video Processing, vol. 2013, pp. 1–16, 2013.
  43. G. Wang, Z. Wang, Y. Chen, and W. Zhao, “Robust point matching method for multimodal retinal image registration,” Biomedical Signal Processing and Control, vol. 19, pp. 68–76, 2015.
  44. J. Addison Lee et al., “A low-dimensional step pattern analysis algorithm with application to multimodal retinal image registration,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1046–1053, 2015.
  45. Z. Hossein-Nejad and M. Nasri, “A-ransac: Adaptive random sample consensus method in multimodal retinal image registration,” Biomedical Signal Processing and Control, vol. 45, pp. 325–338, 2018.
  46. M. Hernandez, G. Medioni, Z. Hu, and S. Sadda, “Multimodal registration of multiple retinal images based on line structures,” in 2015 IEEE Winter Conference on Applications of Computer Vision, pp. 907–914, IEEE, 2015.
  47. Á. S. Hervella, J. Rouco, J. Novo, and M. Ortega, “Multimodal registration of retinal images using domain-specific landmarks and vessel enhancement,” Procedia Computer Science, vol. 126, pp. 97–104, 2018.
  48. D. Motta, W. Casaca, and A. Paiva, “Vessel optimal transport for automated alignment of retinal fundus images,” IEEE Transactions on Image Processing, vol. 28, no. 12, pp. 6154–6168, 2019.
  49. Z. Li et al., “Multi-modal and multi-vendor retina image registration,” Biomedical optics express, vol. 9, no. 2, pp. 410–422, 2018.
  50. K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” arXiv preprint arXiv:1409.1556, 2014.
  51. H. Su, V. Jampani, D. Sun, O. Gallo, E. Learned-Miller, and J. Kautz, “Pixel-adaptive convolutional neural networks,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11166–11175, 2019.
  52. J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, “Unpaired image-to-image translation using cycle-consistent adversarial networks,” in Proceedings of the IEEE international conference on computer vision, pp. 2223–2232, 2017.
  53. J. Pottenburgh et al., “Use of fitc and cfse labeled erythrocytes for in vivo retinal imaging in non-human primates,” Investigative Ophthalmology & Visual Science, vol. 61, no. 7, pp. 897–897, 2020.
  54. K.-K. Maninis, J. Pont-Tuset, P. Arbeláez, and L. Van Gool, “Deep retinal image understanding,” in Medical Image Computing and Computer-Assisted Intervention–MICCAI 2016: 19th International Conference, Athens, Greece, October 17-21, 2016, Proceedings, Part II 19, pp. 140–148, Springer, 2016.
  55. D. DeTone, T. Malisiewicz, and A. Rabinovich, “Superpoint: Self-supervised interest point detection and description,” in Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp. 224–236, 2018.
  56. M. A. Fischler and R. C. Bolles, “Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography,” Communications of the ACM, vol. 24, no. 6, pp. 381–395, 1981.
  57. J. Sun, Z. Shen, Y. Wang, H. Bao, and X. Zhou, “Loftr: Detector-free local feature matching with transformers,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8922–8931, 2021.
  58. P.-E. Sarlin, D. DeTone, T. Malisiewicz, and A. Rabinovich, “Superglue: Learning feature matching with graph neural networks,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4938–4947, 2020.
  59. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems, vol. 30, 2017.
  60. A. Sindel, B. Hohberger, S. F. Dehcordi, C. Mardin, R. Lämmer, A. Maier, and V. Christlein, “A keypoint detection and description network based on the vessel structure for multi-modal retinal image registration,” in Bildverarbeitung für die Medizin 2022: Proceedings, German Workshop on Medical Image Computing, Heidelberg, June 26-28, 2022, pp. 57–62, Springer, 2022.
  61. J. Zhang et al., “Joint vessel segmentation and deformable registration on multi-modal retinal images based on style transfer,” in 2019 IEEE International Conference on Image Processing (ICIP), pp. 839–843, IEEE, 2019.
  62. D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014.
  63. K. J. Zuiderveld, “Contrast limited adaptive histogram equalization,” in Graphics gems, 1994.
  64. A. F. Frangi, W. J. Niessen, K. L. Vincken, and M. A. Viergever, “Multiscale vessel enhancement filtering,” in Medical Image Computing and Computer-Assisted Intervention—MICCAI’98: First International Conference Cambridge, MA, USA, October 11–13, 1998 Proceedings 1, pp. 130–137, Springer, 1998.
  65. V. Balntas, K. Lenc, A. Vedaldi, and K. Mikolajczyk, “Hpatches: A benchmark and evaluation of handcrafted and learned local descriptors,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5173–5182, 2017.
  66. A. Dai, A. X. Chang, M. Savva, M. Halber, T. Funkhouser, and M. Nießner, “Scannet: Richly-annotated 3d reconstructions of indoor scenes,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5828–5839, 2017.
  67. A. Budai, R. Bock, A. Maier, J. Hornegger, G. Michelson, et al., “Robust vessel segmentation in fundus images,” International journal of biomedical imaging, vol. 2013, 2013.
  68. J. Staal, M. D. Abràmoff, M. Niemeijer, M. A. Viergever, and B. Van Ginneken, “Ridge-based vessel segmentation in color images of the retina,” IEEE transactions on medical imaging, vol. 23, no. 4, pp. 501–509, 2004.
  69. H. Narasimha-Iyer, B. Lujan, J. Oakley, S. Meyer, and S. Dastmalchi, “Registration of cirrus hd-oct images with fundus photographs, fluorescein angiographs and fundus autofluorescence images,” Investigative Ophthalmology & Visual Science, vol. 49, no. 13, pp. 1831–1831, 2008.
  70. Z. Liu, J. Tam, O. Saeedi, and D. X. Hammer, “Trans-retinal cellular imaging with multimodal adaptive optics,” Biomedical optics express, vol. 9, no. 9, pp. 4246–4262, 2018.
  71. C. T. Le et al., “Novel application of long short-term memory network for 3d to 2d retinal vessel segmentation in adaptive optics—optical coherence tomography volumes,” Applied Sciences, vol. 11, no. 20, p. 9475, 2021.
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets