DREAM: Dynamic Retinal Enhancement with Adaptive Multi-modal Fusion for Expert Precision Medical Report Generation

Abstract: Automating medical reports for retinal images requires a sophisticated blend of visual pattern recognition and deep clinical knowledge. Current Large Vision-LLMs (LVLMs) often struggle in specialized medical fields where data is scarce, leading to models that overfit and miss subtle but critical pathologies. To address this, we introduce DREAM (Dynamic Retinal Enhancement with Adaptive Multi-modal Fusion), a novel framework for high-fidelity medical report generation that excels even with limited data. DREAM employs a unique two-stage fusion mechanism that intelligently integrates visual data with clinical keywords curated by ophthalmologists. First, the Abstractor module maps image and keyword features into a shared space, enhancing visual data with pathology-relevant insights. Next, the Adaptor performs adaptive multi-modal fusion, dynamically weighting the importance of each modality using learnable parameters to create a unified representation. To ensure the model's outputs are semantically grounded in clinical reality, a Contrastive Alignment module aligns these fused representations with ground-truth medical reports during training. By combining medical expertise with an efficient fusion strategy, DREAM sets a new state-of-the-art on the DeepEyeNet benchmark, achieving a BLEU-4 score of 0.241, and further demonstrates strong generalization to the ROCO dataset.

• The paper presents a novel dynamic retinal enhancement architecture that adaptively fuses visual features and expert-curated keywords for precise report generation.
• It employs Guided Context Attention and bidirectional cross-modal abstractor to enhance subtle pathology detection and reduce hallucinations.
• Empirical results on DeepEyeNet show state-of-the-art performance with improved BLEU, CIDEr, and clinical fidelity, enabling real-time deployment.

Expert Summary of "DREAM: Dynamic Retinal Enhancement with Adaptive Multi-modal Fusion for Expert Precision Medical Report Generation"

Motivation and Context

Automating retinal medical report generation is constrained by the scarcity of annotated datasets and the nuanced expertise required for detecting subtle pathologies. Contemporary LVLMs exhibit limitations in specialized medical contexts: they overfit small datasets and fail to modulate reliance on visual content versus expert clinical priors according to the individual case. DREAM addresses these limitations with an architecture that fuses expert-curated keywords and visual evidence through adaptive, context-sensitive mechanisms.

Architecture Overview

DREAM’s architecture is a structured cascade of Representation Learning, Dynamic Retinal Enhancement (Abstractor), Adaptive Multi-modal Fusion (Adaptor), and a LLM with Contrastive Alignment.

Figure 1: Architecture of DREAM, demonstrating the multi-stage pipeline from retinal image and clinical keywords to expert-level report generation.

The model utilizes a ConvBase (EfficientNetV2B0) enhanced via Guided Context Attention (GCA) for vision, and a Transformer encoder for clinical keywords. The Abstractor performs bidirectional cross-attention, aligning vision and text in a shared space. The Adaptor introduces a learnable modality indicator and decoupled cross-modal attention, allowing per-sample adaptive fusion rather than static concatenation. The output is decoded by a compact LLaMA-X LLM, and a Contrastive Alignment module ensures the latent representation matches the holistic semantics of clinical reports.

Technical Innovations

Guided Context Attention (GCA)

The GCA module processes image feature maps with global-local attention, ensuring sensitivity to both widespread context and fine-grained lesion cues, improving detection of subtle retinal pathologies.

Abstractor: Cross-Modal Enhancement

The Abstractor employs parallel MLP projections of visual and keyword feature sets into a shared latent space. Its bidirectional cross-attention mechanisms ($\mathbf{V2L}$ and $\mathbf{L2V}$) facilitate selective enhancement: visual features are modulated according to the semantic cues in keywords, and irrelevant or unsupported linguistic tokens are suppressed by image evidence.

Adaptor: Adaptive Multi-modal Fusion

The Adaptor flattens visual features and aligns them with the keyword embeddings, then modulates the concatenated representation using a learnable modality indicator. Decoupled cross-modal attention enables the model to dynamically arbitrate which modality should dominate the fused representation on a per-sample basis. This is critical for addressing clinical ambiguity, such as distinguishing rare findings or coping with poor image quality.

Contrastive Alignment

DREAM introduces a global InfoNCE-based contrastive loss to align report and fused vision-language representations, minimizing factual hallucinations and fostering semantic correspondence between latent features and ground-truth report narratives.

Training and Objectives

DREAM is trained end-to-end using a composite objective: standard autoregressive cross-entropy for report generation plus the contrastive alignment objective. The former supervises fine-grained (token-level) accuracy, while the latter imposes holistic semantic alignment and discourages semantically unsupported hallucinations. The Abstractor and Adaptor are parallelized for training efficiency on a single GPU.

Empirical Results

DREAM sets the new state-of-the-art on the DeepEyeNet benchmark, achieving:

• BLEU-4: 0.241 (vs. prior best 0.231)
• CIDEr: 0.576
• ROUGE: 0.491
• BERT-F$_1$: 0.91

These scores correspond to improved grammatical and clinical fidelity in report generation, achieving an absolute gain of $0.03$ in BERT-F$_1$ over leading LVLMs and specialist models.

Ablation Results

Component ablation shows that the Abstractor, Adaptor, and Contrastive Alignment modules each yield incremental improvements, verifying that adaptive fusion and global alignment collectively drive performance.

Cross-Domain Generalization

When evaluated on the heterogeneous ROCOv2 radiology caption corpus, DREAM maintains robust scores (BLEU-4: 0.228, ROUGE: 0.483, CIDEr: 0.562), underscoring the broad applicability of its fusion paradigm.

Qualitative Results

Figure 2: Comparison of reports for a central serous retinopathy case. DREAM aligns more closely with the ground truth, while baselines hallucinate findings or misattribute details.

Qualitative assessment demonstrates that DREAM sharply reduces hallucination rates. Attention maps highlight the model’s focus on the central lesion concordant with clinical expectation, a direct consequence of GCA-driven feature enhancement and keyword-guided spatial priors.

Practical and Theoretical Implications

DREAM operates with a parameter count (0.36B) and computational requirements (5 GFLOPs) that are orders of magnitude lower than general LVLMs, supporting real-time deployment on commodity clinical hardware. Its framework for modality-adaptive fusion transcends ophthalmology, suggesting applicability to any specialized medical imaging domain characterized by small labelled datasets and the ready availability of structured expert knowledge.

Theoretical implications extend to the design of generalizable multi-modal systems: DREAM demonstrates that per-sample dynamic fusion, rather than static combination or naive gating, results in improved factual reliability and semantic robustness, especially in tasks requiring high sensitivity and precision.

Limitations and Future Directions

While DREAM is robust to missing or noisy expert keywords, systematic evaluation under realistic clinical data corruption is pending. Expansion to multi-center datasets is necessary to confirm resilience across clinical sites and devices. Furthermore, current evaluation primarily leverages automated metrics; future work should include human expert review for assessing real-world diagnostic utility.

Conclusion

DREAM exemplifies the integration of expert priors and visual information through adaptive, context-aware fusion and contrastive alignment. Its improvements in clinical fidelity, efficiency, and generalizability signal a significant advance for automatic expert-level report generation and lay the architectural groundwork for next-generation medical LVLMs.