Divergent Fusion Attention (DiFA) in Medical Imaging

• DiFA is a specialized cross-attention mechanism that integrates non-aligned multi-view imaging data for improved feature refinement.
• The method aggregates complementary cues from different imaging planes without positional encoding, addressing alignment challenges.
• Empirical results in cardiac MRI segmentation show increased Dice scores and reduced Hausdorff distances when using DiFA.

Divergent Fusion Attention (DiFA) is a specialized cross-attention mechanism introduced to address the challenges of integrating unaligned, multi-view medical imaging data by modeling semantic dependencies and long-range correlations across different image views. Developed within the context of the TransFusion architecture for multi-view cardiac MRI segmentation, DiFA enables joint feature refinement by leveraging complementary cues from disparate imaging planes, which lack direct spatial alignment but offer synergistic anatomical insights (Liu et al., 2022).

1. Architectural Context and Motivation

TransFusion is structured as a multi-branch encoder–decoder system, where each branch corresponds to a specific image view (for example, 3D short-axis or 2D long-axis cardiac MRI). Each branch processes its view via a convolutional stem followed by Residual+Transformer layers. DiFA modules are strategically placed at intermediate or high encoder stages where they facilitate cross-view feature fusion. For each view $m$, the DiFA block refines its token map by aggregating information exclusively from the token maps of all other views. After DiFA, the fused representations are routed back into their respective decoders at matching spatial scales, equipping each decoder path with both strong local (view-specific) features and non-local (cross-view) cues.

This approach directly addresses the issue that multi-view medical images, such as those acquired per different cardiac planes, are inherently non-aligned. Traditional alignment or registration-based fusion is suboptimal. DiFA operates without positional encodings, relying purely on content-driven attention.

2. Mathematical Formulation

Given $M$ views, each with a feature token map $f_m \in \mathbb{R}^{C \times N_m}$ (where $N_m$ is the spatial extent either as $H_m \times W_m$ or $D_m \times H_m \times W_m$), the DiFA block performs the following projections:

$$Q_m = W^Q_m f_m \in \mathbb{R}^{d \times N_m} \ K_m = W^K_m f_m \in \mathbb{R}^{d \times N_m} \ V_m = W^V_m f_m \in \mathbb{R}^{d \times N_m}$$

For view $m$, the keys and values from all other views are concatenated:

$$\tilde{K}_m = \operatorname{Concatenate}_{i \neq m}(K_i) \in \mathbb{R}^{d \times \sum_{i \neq m} N_i} \ \tilde{V}_m = \operatorname{Concatenate}_{i \neq m}(V_i) \in \mathbb{R}^{d \times \sum_{i \neq m} N_i}$$

Attention weights are computed and applied as follows:

$$A_m = \operatorname{softmax}(Q_m \tilde{K}_m^T / \sqrt{d}) \in \mathbb{R}^{N_m \times \sum_{i \neq m} N_i}$$

$M$0

In the multi-head setting, the above computation is performed in parallel for each head (of dimension $M$1), and the outputs are concatenated. As the views are not spatially aligned, no positional encoding is used.

3. Processing Workflow and Pseudocode

The DiFA module processes all M views simultaneously. At each application:

$M$5

This module is typically inserted as a residual component within a Transformer block, including skip connections, layer normalization, and a feedforward MLP, consistent with standard Transformer architectures.

4. Comparison with Multi-Head Self-Attention

Standard Multi-Head Self-Attention (MHSA) computes queries, keys, and values from the same feature map $M$2, thereby focusing on intra-view, long-range dependencies. In contrast, DiFA exclusively uses cross-view cues: for each target view $M$3, queries are constructed from $M$4, and keys/values are aggregated from all other views. Thus, DiFA models inter-view semantic relationships, which are critical in scenarios where spatial correspondence is unavailable or unreliable. Furthermore, DiFA omits positional encodings, as anatomical regions in different views have no spatially consistent mapping.

Functionally, DiFA operates as M simultaneous cross-attention blocks, one for each view, in contrast to MHSA’s single-view self-attention. This design directly targets the integration of disparate imaging planes.

5. Integration with Multi-Scale Attention and Segmentation Pipeline

DiFA is integrated into TransFusion in conjunction with Multi-Scale Attention (MSA). The overall segmentation pipeline includes:

• Convolutional stem with 4× downsampling per view.
• Hierarchical stacks of Residual+Transformer blocks yielding multi-scale token features.
• DiFA blocks inserted at selected middle and high encoder stages to enable cross-view fusion.
• In parallel, MSA is applied within each view at the same stages, concatenating multi-scale features from that view, processing them through a Transformer, and distributing (“folding back”) the outputs across all scales.
• Both cross-view (via DiFA) and cross-scale (via MSA) refined features are routed into corresponding decoder stages, which perform upsampling and further fusion to generate the final segmentation maps.

This dual-context strategy ensures that local features are augmented with both cross-view information (from DiFA) and cross-scale consistency (from MSA), enhancing segmentation accuracy for complex anatomical structures.

6. Quantitative Evaluation and Empirical Impact

Ablation experiments demonstrate DiFA’s effectiveness. When incorporated into UTNet, average Dice coefficient for short-axis segmentation improves from 86.10% to 86.42%, and Hausdorff Distance (HD) decreases from 11.84 mm to 11.44 mm; integrating both DiFA and MSA in the full TransFusion pipeline further lifts Dice to 87.58% and reduces HD to 10.82 mm. For long-axis targets, UTNet* with DiFA improves Dice from 85.49% to 86.27% and HD from 9.28 mm to 8.95 mm; the full pipeline reaches 87.70% Dice and 8.02 mm HD.

Evaluations of DiFA placement indicate that applying it at progressively higher encoder levels monotonically increases performance (as measured by Dice), with multi-level stacking yielding diminishing, but still positive, gains. The hybrid 3D+2D mode enabled by DiFA outperforms pure 2D or 2.5D approaches by approximately 1–2% in Dice. Integrating DiFA into disparate architectures (MCTrans, rDLA, InfoTrans) yields consistent improvements in multi-view performance (~0.3–1.0% Dice), demonstrating its generality and practical benefit (Liu et al., 2022).

7. Interpretation and Broader Implications

Divergent Fusion Attention provides a principled approach for modeling long-range, semantic dependencies across non-aligned multi-view medical images. By relying solely on content-based associations, DiFA circumvents the need for explicit spatial alignment or anatomical registration. This mechanism fits naturally within multi-scale, hybrid architectures that combine convolutional and transformer-based encoders and decoders, offering a flexible pathway for robust, multi-modal image integration in challenging clinical segmentation tasks. The consistent improvements across architectures and datasets suggest that use of DiFA is broadly applicable wherever cross-view or cross-modal fusion is required and registration is impractical.