Dual-Attention Semantic Parallel-LSTM Network

• The paper introduces a multimodal framework (DA-SPL) that integrates dual-attention mechanisms and parallel LSTM decoding to generate precise clinical glaucoma reports.
• Methodologically, the model refines cross-modal feature extraction from fundus images and clinical data using head and cosine similarity weightings, ensuring robust diagnostic narrative synthesis.
• Evaluation shows that DA-SPL outperforms previous models in standard metrics such as BLEU, ROUGE, and CIDEr, highlighting its clinical accuracy and consistency in report generation.

The Dual-Attention Semantic Parallel-LSTM Network (DA-SPL) is a multimodal generative framework designed for advanced clinical report generation, primarily within the context of automated glaucoma diagnosis. DA-SPL leverages joint dual-attention mechanisms in its encoder stage to refine cross-modal feature extraction from fundus imaging and supplementary clinical inputs. The architecture also incorporates a parallelized LSTM decoder to enhance temporal and semantic consistency and employs a specialized label enhancement module for precise alignment of generated language with disease-relevant terminology.

1. Architectural Foundations

DA-SPL is structured as an encoder–decoder network tailored for multimodal input integration. The encoder consists of a vision backbone (e.g., ConViT), partitioning fundus images into patches processed by GPSA layers, which maintain spatial structure via positional embeddings. Features extracted from both imaging and clinical data sources are jointly refined via the dual-attention mechanism prior to downstream LSTM-based decoding. This composite structure allows DA-SPL to capture both fine-grained visual details (such as optic disc morphology and retinal nerve fiber layer defects) and abstract semantic relationships crucial for diagnostic narrative synthesis.

2. Joint Dual-Attention Mechanism

The dual-attention mechanism within the encoder operates along two principal weighting axes:

• Head Weighting: Attention head weights $w_a$ are iteratively scaled with a softmax function to emphasize those heads that contribute most significantly to extracting disease-relevant features:

$$w_a^{(i)} = \text{softmax}(w_a^{(i-1)}) \cdot N,\quad att_{wa}^{(i)} = w_a^{(i)} * att_a^{(i)}$$

• Cosine Similarity Weighting: The second dimension computes the cosine similarity between each attention head and a designated base head, with the similarity aggregated across the batch:

$$cos(i)^{(j)} = cos(att_a^{(j)}(i-1), att_a^{(base)}(i-1)),\quad w_{cos(i)}^{(j)} = \frac{1}{N} \sum_k cos(i)_k^{(j)}$$

An overall scaling factor $\beta$ is then defined as the geometric mean over all heads:

$$\beta = \sqrt[n]{\prod_{j=1}^{n} |w_{cos(i)}^{(j)}|},\quad w_{dwa(i)}^{(j)} = \beta - w_{cos(i)}^{(j)}$$

This dual weighting scheme amplifies attention to pathological findings while attenuating normal anatomical variance, thereby producing robustly refined cross-modal feature representations necessary for precise disease description in generated reports.

3. Parallelized LSTM Decoder Architecture

DA-SPL's decoder employs a parallel LSTM configuration with three mutually interacting streams:

• The primary LSTM stream processes the joint attention-weighted feature vector from the encoder and generates hidden states conditioned on previously decoded word embeddings:

$$h_t^1 = \text{LSTM}(T_t^1, w_{dwa(i)}^{(j)} * att_a^{(j)}(i))$$

• Two additional LSTM pathways incorporate both the primary encoded features and further context:

$$h_t^2 = \text{LSTM}(T_t^1, h_t^1, W_e^2 x_t, h_{t-1}^2),\quad h_t^3 = \text{LSTM}(T_t^2, h_t^2, W_e^2 x_t)$$

Each pathway outputs word probabilities via a softmax over the respective fully connected layers:

$$\text{MLP}_{t+1}^1 = \text{softmax}(W_{FC1} h_t^2),\quad \text{MLP}_{t+1}^2 = \text{softmax}(W_{FC2} h_t^3)$$

This parallel configuration ensures simultaneous modeling of both short-term temporal detail and long-range semantic dependencies, supporting generation of coherent and medically detailed diagnostic narratives.

4. Label Enhancement and Semantic Alignment Module

To augment the generated report text with clinically salient terminology, DA-SPL integrates a label enhancement module. This module operates as follows:

• The generated report embedding, $\text{RP}_{emb}$, is input to a dedicated category prediction LSTM.
• The output feature is further processed through a fully connected layer and softmax activation to produce diagnostic label probabilities:

$$\text{Label} = \text{softmax}(W_t * \text{LSTM}(\text{RP}_{emb}) + b_t)$$

A plausible implication is that this mechanism directly realigns generated content with clinical keyword categories (e.g., ISNT rule violations, neuroretinal rim thinning), thereby elevating report precision and ensuring compliance with standard diagnostic nomenclature.

5. Quantitative Evaluation and Ablation Analysis

Extensive quantitative assessments on standard glaucoma datasets confirm the superiority of DA-SPL over prior multimodal report generation systems including CNN-RNN, CNN-LSTM, CaptionNet, AoANet, and JE-TriNet. Notable metric increases are observed in BLEU-1, BLEU-2, BLEU-3, BLEU-4, ROUGE-L, and CIDEr scores. Ablation studies demonstrate that each core component—dual-attention mechanism (DAM), parallelized LSTM network (PLN), and label enhancement module (LEM)—uniquely contributes to aggregate performance, with complete DA-SPL models (especially those tuned via frameworks such as Optuna) outperforming non-tuned baselines by approximately two percentage points in automated evaluations. The improvements are empirically validated and traceable to the design innovations described.

6. Clinical Impact and Functional Significance

In clinical implementation, DA-SPL effectively extracts nuanced pathological indicators (e.g., optic disc cupping, retinal nerve fiber layer defects, visual field abnormalities) from heterogeneous multimodal input streams. The dual-attention mechanism permits selective amplification of features correlated with disease, while the parallelized decoder architecture supports the temporal and semantic consistency critical for diagnostic clarity. The label enhancement module further ensures alignment of report output with expert-selected vocabulary, facilitating more accurate disease management and clinical auditing. This suggests an operational paradigm that closely mirrors expert decision processes, contributing meaningfully to automated report generation in ophthalmology.

7. Relationship to Historical and Related Dual-Attention Architectures

DA-SPL extends foundational concepts in dual-attention from dialogue act classification (Li et al., 2018) and multivariate time series modeling (Farahani et al., 2023). Its unique integration of dual-attention in multimodal clinical domains, coupled with a parallel LSTM decoder and specialized semantic alignment, differentiates it from previous transformer-based or sequential dual-attention systems. A plausible implication is that by combining structured attention weighting, enhanced temporal modeling, and constrained semantic guidance, DA-SPL sets a precedent for future multimodal medical report generation networks.