Fundus2Video: Cross-Modal Angiography Video Generation from Static Fundus Photography with Clinical Knowledge Guidance (2408.15217v1)

Abstract: Fundus Fluorescein Angiography (FFA) is a critical tool for assessing retinal vascular dynamics and aiding in the diagnosis of eye diseases. However, its invasive nature and less accessibility compared to Color Fundus (CF) images pose significant challenges. Current CF to FFA translation methods are limited to static generation. In this work, we pioneer dynamic FFA video generation from static CF images. We introduce an autoregressive GAN for smooth, memory-saving frame-by-frame FFA synthesis. To enhance the focus on dynamic lesion changes in FFA regions, we design a knowledge mask based on clinical experience. Leveraging this mask, our approach integrates innovative knowledge mask-guided techniques, including knowledge-boosted attention, knowledge-aware discriminators, and mask-enhanced patchNCE loss, aimed at refining generation in critical areas and addressing the pixel misalignment challenge. Our method achieves the best FVD of 1503.21 and PSNR of 11.81 compared to other common video generation approaches. Human assessment by an ophthalmologist confirms its high generation quality. Notably, our knowledge mask surpasses supervised lesion segmentation masks, offering a promising non-invasive alternative to traditional FFA for research and clinical applications. The code is available at https://github.com/Michi-3000/Fundus2Video.

• The paper introduces Fundus2Video, an autoregressive GAN for generating dynamic Fundus Fluorescein Angiography videos from static Color Fundus images.
• A key innovation is the use of an unsupervised clinical knowledge-based mask to guide the network's focus on critical lesion areas without manual annotation.
• Quantitative evaluation shows superior metrics like FVD and PSNR, while qualitative assessment by an ophthalmologist confirms improved vascular and lesion visualization, offering a non-invasive diagnostic alternative.

Cross-Modal Angiography Video Generation: Insights from Fundus2Video

The paper presented here discusses a novel methodology for generating dynamic Fundus Fluorescein Angiography (FFA) videos from static Color Fundus (CF) images using an autoregressive Generative Adversarial Network (GAN). Titled "Fundus2Video," this approach provides a non-invasive alternative to traditional FFA, an invasive procedure often limited by availability and patient comfort. The authors address several challenges inherent in this cross-modal generation, including dynamic lesion representation and pixel misalignment, proposing solutions informed by clinical knowledge.

Core Contributions

1. Autoregressive GAN Architecture: Fundus2Video pioneers a frame-by-frame FFA synthesis methodology derived from CF images. Utilizing pix2pixHD as a foundation, the model is optimized for memory efficiency and smooth output. By leveraging temporal dependencies within an autoregressive framework, the proposed architecture refines the generation process to more accurately reflect the dynamic nature of FFA sequences.
2. Clinical Knowledge-based Masking: A significant advancement in this work is the introduction of a knowledge mask informed by clinical insights. This unsupervised mask enhances the identification of critical lesion areas without necessitating manual annotation, a process which is both time-consuming and labor-intensive. Computed as a differential threshold between initial and late-phase FFA frames, the mask guides the network’s focus during learning, emphasizing regions of significant change.
3. Refinement Techniques: The paper presents multiple innovative techniques including knowledge-boosted attention and knowledge-aware discriminators. These methods prioritize regions with clinically significant features, addressing challenges like pixel misalignment. The authors notably incorporate a mask-enhanced PatchNCE loss, adapting contrastive learning principles to focus network attention on crucial vascular and lesion structures within the FFA series.

Quantitative and Qualitative Evaluation

The methodology’s efficacy is evidenced through quantitative benchmarks, including achieving superior Fréchet Video Distance (FVD) of 1503.21 and a Peak Signal-to-Noise Ratio (PSNR) of 11.81, compared to conventional image-to-video generation models. Additionally, qualitative assessments illustrate the model's capability to generate clearer videos with more distinct lesion depiction than state-of-the-art approaches. Evaluation by an ophthalmologist further corroborates the model's performance, highlighting substantial improvements in vascular and lesion visualization.

Evaluation and Implications

Fundus2Video's contributions have significant implications for both research and clinical practice. The algorithm not only furnishes a non-invasive alternative to FFA but also facilitates dynamic lesion mapping critical for diagnosing diseases such as diabetic retinopathy and macular degeneration. The autoregressive GAN framework and accompanying refinement techniques can be generalized to other cross-modal generation challenges, potentially sparking further innovation in AI-driven medical imaging.

Future developments may explore integrating additional clinical parameters or deep learning techniques to enhance predictive accuracy and functional capacity. Furthermore, deploying this technology within real-world ophthalmic settings can provide valuable insights into refining model training and evaluation frameworks, especially concerning longitudinal assessments of retinal diseases.

In conclusion, this work offers a promising avenue for non-invasive retinal diagnostics, delivering technological solutions that align closely with the needs of clinical ophthalmology. The advancements laid forth by Fundus2Video reflect a crucial step towards more accessible, efficient, and precise imaging in medical contexts, meriting detailed exploration and potential adoption in medical imaging pipelines.