Dice Question Streamline Icon: https://streamlinehq.com

Integration of multiple retinal imaging modalities into a joint 3D foundation model

Develop computational strategies to jointly integrate fundus autofluorescence (FAF), color fundus photography (CFP), fluorescein angiography (FA), infrared retinal images (IR), and three-dimensional OCT volumes into a single multi-modal 3D foundation model, addressing computational cost and structural mismatches among the modalities.

Information Square Streamline Icon: https://streamlinehq.com

Background

The authors propose extending beyond OCT and IR to additional retinal imaging modalities (FAF, CFP, FA) to more comprehensively characterize the retina. However, modality-specific fields of view, dimensionality (2D vs 3D), sampling densities, and device differences complicate joint modeling.

While the paper presents COIP to align OCT volumes with IR images, broader multi-modal integration across additional retinal modalities remains methodologically unresolved due to computational constraints and mismatched modality structures.

References

However, it remains unclear how to computationally integrate all of them given the computational cost and unmatched modality structures.

— OCTCube-M: A 3D multimodal optical coherence tomography foundation model for retinal and systemic diseases with cross-cohort and cross-device validation (2408.11227 - Liu et al., 20 Aug 2024) in Discussion (Limitations)