Cross-Modal Fusion of OCT and OCT angiography enface for Improved Diagnostics of Diabetic Retinopathy

Abstract: Diabetic retinopathy (DR) is a leading cause of vision impairment worldwide, highlighting the need for accurate and accessible screening tools. Optical Coherence Tomography (OCT) provides high-resolution structural information of the retina, whereas OCT angiography (OCTA) offers complementary vascular information that is highly relevant for DR diagnosis. In this study, we propose a cross-modal fusion of OCT B-scans with single-channel en face OCTA using a bidirectional cross-modal attention network for automated DR classification. Two independent datasets, OCT500 and UIC, comprising 730 subjects in total, were utilized to evaluate performance under within-dataset, combined-dataset, and cross-dataset generalization settings. A ConvNeXt V2 model trained solely on OCT images served as the unimodal baseline. In addition to ground-truth (GT) OCTA, we explored the use of translated (TR) OCTA generated from OCT scans, eliminating the requirement for dedicated OCTA hardware. Experimental results demonstrate that cross-modal fusion consistently outperforms unimodal OCT classification across all evaluation scenarios. Fusion with GT OCTA improved classification accuracy and discriminative performance, while TR OCTA achieved comparable or superior results in most settings. Furthermore, TR OCTA improved sensitivity and cross-dataset generalization, indicating enhanced robustness to domain shifts. These findings demonstrate that attention-based OCT-OCTA en face fusion provides clinically meaningful improvements for DR detection and suggest that computationally generated OCTA can serve as a practical, low-cost alternative to hardware-acquired OCTA, enabling broader deployment of high-performance retinal screening systems in resource-limited clinical environments.

• The paper introduces a cross-modal fusion framework that leverages bidirectional attention between OCT and OCTA data to enhance diabetic retinopathy classification.
• It demonstrates that synthetic OCTA images generated via a conditional diffusion model can replace costly hardware-based OCTA, improving sensitivity and AUC in cross-dataset evaluations.
• The study shows that integrating structural and vascular features significantly boosts diagnostic accuracy and scalability for DR screening in resource-limited settings.

Cross-Modal Fusion of OCT and OCTA Enface for Enhanced Diabetic Retinopathy Diagnostics

Introduction

Diabetic retinopathy (DR) remains a major cause of preventable vision impairment worldwide, making accurate, scalable screening technology a medical priority. While OCT imaging provides depth-resolved structural data of the retina, OCT angiography (OCTA) supplies functional microvascular information essential for early DR detection. This paper introduces a cross-modal fusion framework that synergistically leverages OCT B-scans and single-channel enface OCTA via a bidirectional cross-modal attention network to improve automated DR classification. Notably, the practical barrier posed by OCTA hardware is addressed through the generation of synthetic, translated (TR) OCTA images—offering an affordable alternative for clinical environments constrained by resources.

Figure 1: Representative B-scan images (OCT) and vascular enface projections (OCTA) highlight complementary structural and vascular content utilized in the fusion framework.

Methods and Model Architecture

Datasets

The benchmarking relies on two large-scale, heterogeneously collected datasets: OCT500 and UIC, summing to 730 subjects with diverse imaging protocols and population demographics. Experiments are stratified into within-dataset, pooled, and rigorous cross-dataset settings to evaluate both performance and generalizability, with strict patient-level separation between splits.

Fusion Architecture

The core of the methodology is the adaptation of the Cross-modal Retinal Disease Diagnosis Network (CRD-Net), integrating the following key components:

• Modality-Specific Feature Extraction: Independent CNN backbones, initialized with ImageNet weights, extract high-level features from OCT B-scans and enface OCTA images, preserving modality-specific nuances.
• Cross-Modal Attention Module: This mechanism enacts bidirectional querying—each modality can interrogate salient features in the other. Multi-head self-attention is first applied within each modality, followed by cross-attention layers that fuse spatial context and disease markers across structural and vascular features.
• Composite Loss: Supervision is enforced simultaneously on multimodal and single-modality outputs, mitigating overfitting to one modality and supporting robust training.

The pipeline supports both ground-truth (GT) OCTA and computationally-generated TR OCTA, produced via a conditional diffusion model trained exclusively on the OCT500 dataset.

Figure 2: Schematic of the cross-modal attention-based fusion network (CRD-Net) highlighting independent modality encoding, bidirectional cross-attention, and joint classification head.

Experimental Results

Extensive experiments reveal several noteworthy outcomes:

1. Unimodal Baseline Performance: ConvNeXt V2, serving as the sole-OCT point of comparison, achieves moderate within-dataset accuracy (up to 0.9621), but collapses in cross-dataset generalization (accuracy drops to 0.5735 and F1 to 0.5396).
2. Cross-Modal Fusion with GT OCTA: Inclusion of the GT OCTA enface modality yields significant enhancements:
   • Within-dataset accuracy improves (OCT500: 0.8788; UIC: near-perfect 0.9971; Combined: 0.9730).
   • AUC improvements are pronounced (e.g., from 0.7994 to 0.9719 on OCT500).
   • Cross-dataset generalization is improved but not uniformly optimal—recall notably drops in some transfer settings, indicating persistent domain-specific features in acquisition.
3. Cross-Modal Fusion with TR OCTA: Employing synthetic OCTA generated from only OCT data yields performances rivaling or exceeding hardware-acquired OCTA:
   • Highest within-dataset metrics (e.g., OCT500: accuracy 0.9697, recall 0.9286, AUC 0.9897).
   • In the cross-dataset setting, TR OCTA markedly boosts sensitivity (recall 0.9013) and AUC (0.9595) relative to both unimodal and GT OCTA fusion.
   • The robustness to domain shift when generalizing across acquisition protocols is notably superior for the TR OCTA variant, attributed to learned structural-vascular correspondences without hardware-induced noise.

Highlighted Claim: The paper asserts, with empirical support, that computationally translated OCTA images not only obviate expensive hardware but in most cases outperform ground-truth OCTA fusion and always surpass unimodal OCT, especially in generalization scenarios.

Discussion and Analysis

Implications

• Clinical Utility: Cross-modal attention fusion directly addresses the diagnostic limitation of single-modality imaging, enabling extraction of both deep-structural and topological-vascular markers critical for early and robust DR classification.
• Scalability: By validating a TR OCTA computational pathway, the barrier of costly hardware for vascular imaging is substantially lowered, broadening access to high-quality screening in resource-limited environments.
• Generalizability: The architecture is rigorously tested in cross-dataset regimes; the TR OCTA approach demonstrates meaningful advances in sensitivity and overall classification robustness, critical for real-world deployment outside tightly curated academic datasets.

Theoretical Considerations

The study substantiates that bidirectional cross-modal attention does not merely average multimodal features but actively learns spatially-aligned, pathology-driven correspondences. This outperforms naive early or late fusion and generic feature concatenation. Moreover, the use of learned synthetic OCTA improves resilience to acquisition noise, scanner variability, and population drift, suggesting a promising avenue for synthetic augmentation in other multimodal medical contexts.

Limitations and Future Directions

• Dataset size and diversity remain constraints: larger, multiclass, and multi-institutional datasets will be needed to generalize to DR severity grading and broader retinopathy phenotypes.
• The current methods focus on binary DR classification; extension to ordinal grading and comorbidity extraction would be immediate clinical next steps.
• Translation networks for OCTA synthesis can be further optimized for cross-population generality and interpretability, potentially integrating uncertainty quantification for deployment in critical clinical workflows.

Conclusion

This work establishes that cross-modal attention-based fusion of OCT and enface OCTA significantly elevates DR classification accuracy, outperforming unimodal approaches across all experimental settings. The demonstration that translated OCTA, generated from readily available OCT data, can substitute for expensive OCTA hardware without performance tradeoffs—indeed sometimes with improvements—has substantial implications for global eye care accessibility. The pipeline validated here is poised to accelerate the deployment of advanced AI-driven retinal diagnostics in varied clinical and screening environments, offering a technically sound and clinically practical solution for improving diabetic retinopathy outcomes.