Bidirectional Cross-Modal Fusion Strategy

Updated 12 January 2026

The paper introduces a strategy where each modality informs and refines the other through mutual feedback, enhancing feature representation.
It employs architectural motifs like dual cross-attention and gated fusion to achieve deep, robust integration across diverse data sources.
Empirical evaluations show significant gains in metrics such as DSC and mAP, validating the effectiveness of multi-branch and bidirectional mechanisms.

A bidirectional cross-modal fusion strategy refers to principled architectural and algorithmic mechanisms that explicitly enable two or more data modalities to mutually inform and refine each other's feature representations in both directions within a deep learning system. Unlike unidirectional or late-stage concatenation fusion, bidirectional fusion architectures are designed so that each modality not only influences the fused or cross-modal stream, but also receives feedback and context from it, typically at multiple feature levels or stages. This ensures a richer, more entangled interaction and robust alignment across modalities (e.g., CT/MR, RGB/depth, audio/visual) while retaining the specificity and complementarity of modality-unique cues.

1. Foundations and Theoretical Motivation

The central motivation behind bidirectional cross-modal fusion is to overcome the limitations of unidirectional and simple concatenative strategies, which often fail to exploit the complementary and redundant information distributed across heterogeneous sensor or data streams. In unidirectional fusion, one modality (e.g., depth) "teaches" another (e.g., RGB), but knowledge does not flow reciprocally, causing important cues—such as spatial anchors, noise patterns, or semantic hints—present in the "target" modality to be ignored in subsequent layers (Wang et al., 2021, Zhang et al., 2022). Bidirectional approaches encode pathways where each stream can clean, rectify, or augment the other, improving robustness to noise, distributional shift, and data sparsity (Dong et al., 2024, Meng et al., 22 Jun 2025, Li et al., 22 Oct 2025).

From an information-theoretic standpoint, bidirectional fusion increases the mutual information between resulting representations by forcing both intra- and inter-modal dependencies to be captured at multiple abstraction levels.

2. Key Architectural Patterns

Several core architectural motifs emerge in recent literature:

Parallel multi-branch backbones: Distinct encoder–decoder pairs for each modality, with an additional shared ("cross-modal" or "fusion") branch (e.g., CmFNet (Meng et al., 22 Jun 2025), CMDFusion (Cen et al., 2023)).
Bidirectional fusion modules: At one or several feature levels (e.g., after individual encoding blocks), explicit symmetric operations (elementwise, cross-attention, or gating) let both streams send and receive information (Wang et al., 2021, Dong et al., 2024, Low et al., 30 Sep 2025).
Asymmetric fusion operations: Parameter-free mechanisms such as channel shuffle and pixel shift force bidirectionality to be non-equivalent in each direction—key for learning complementary, not redundant, cues (Wang et al., 2021).
Stacked bidirectional cross-attention blocks: Alternating self-attention and cross-attention in both directions at each depth enables deep, context-aware mutual refinement, as operationalized in hybrid attention layers (Dong et al., 2024, Li et al., 22 Oct 2025, Tian et al., 24 Jun 2025, Low et al., 30 Sep 2025).
Selective feature curation: Intra-modal transformers that select salient source features before using cross-modal attention ensure only the most complementary cues are exchanged (Liu et al., 10 May 2025).

Table 1. Core Bidirectional Fusion Mechanisms

Paper/Framework	Bidirectional Layer	Fusion Mechanism
CmFNet (Meng et al., 22 Jun 2025)	Multi-stage CFF + CFE	Concat + 3D conv, cross-modal
Fusion-Mamba (Dong et al., 2024)	SSCS + DSSF per block	Channel swap + gated cross-state
CDI-DTI (Li et al., 22 Oct 2025)	BCA module (late stage)	Dual cross-attn (Transformer)
CMDFusion (Cen et al., 2023)	Bidirectional Fusion Bl.	MLP residual + attention
Ovi (Low et al., 30 Sep 2025)	At every block	Symmetric cross-attn + RoPE
TUNI (Guo et al., 12 Sep 2025)	Local-global block	Blockwise contextual fusion

3. Mathematical Formulations

Bidirectional cross-modal fusion is often instantiated via paired operations that update both modalities:

Bidirectional convolutional fusion (CmFNet):

$f_\mathrm{mm}^i = \sigma(\operatorname{Conv}_{3\times3\times3}(\operatorname{Concat}[f_\mathrm{ct}^i, f_\mathrm{mr}^i]))$

where fused features flow into a shared decoder branch, and skip connections maintain bidirectionality across all depths (Meng et al., 22 Jun 2025).

Dual cross-attention (CDI-DTI):

$\begin{aligned} d_h &= \mathrm{Softmax}\!\left(\frac{Q_d^{(h)} (K_t^{(h)})^\top}{\sqrt{d_k}}\right) V_t^{(h)} \ t_h &= \mathrm{Softmax}\!\left(\frac{Q_t^{(h)} (K_d^{(h)})^\top}{\sqrt{d_k}}\right) V_d^{(h)} \end{aligned}$

which updates drug and target sequences reciprocally (Li et al., 22 Oct 2025).

Bidirectional gating (Fusion-Mamba):

$y'_{R} = y_{R} \circ z_{R} + z_{R} \circ y_{IR} \qquad y'_{IR} = y_{IR} \circ z_{IR} + z_{IR} \circ y_{R}$

enforcing symmetric hidden-state exchange (Dong et al., 2024).

Asymmetric channel and spatial fusion (Wang et al., 2021):

$F_1(x_1, x_2) = \text{shuffle/shift}(x_1, x_2),\quad F_2(x_2, x_1) = \text{shuffle/shift}(x_2, x_1)$

guaranteeing that neither stream can be reduced to the other by permutation.

4. Training, Losses, and Consistency Objectives

Bidirectional cross-modal fusion strategies typically require hybrid supervision:

Symmetric multi-branch supervision: Losses are computed for each modality-specific decoder, as well as the cross-modal branch, ensuring that both branches learn to leverage and receive cues from the other (Meng et al., 22 Jun 2025, Cen et al., 2023).
Cross-modal consistency losses: Imposing $\ell_2$ , MSE, or contrastive penalties between predictions of the cross and single-modal branches enforces alignment (e.g., $\mathcal L_{imc} = \alpha_1 \|y_{mm}-y_{ct}\|_2^2 + \alpha_2 \|y_{mm}-y_{mr}\|_2^2$ ) (Meng et al., 22 Jun 2025), or knowledge distillation objectives aligning 2D/3D features (Cen et al., 2023).
End-to-end mutual information maximization: In multi-stage settings, e.g., dual-stage contrastive alignment before gated fusion (Yu et al., 13 Mar 2025), matching feature-space statistics across modalities benefits fusion.

A plausible implication is that enforcing not only within-modal regularization but also cross-modal consistency is critical to realizing the full potential of deep bidirectional fusion.

5. Empirical Evaluation and Comparative Benefits

Empirical studies across a range of domains (medical image segmentation (Meng et al., 22 Jun 2025), 3D semantic segmentation (Cen et al., 2023), object detection (Dong et al., 2024), multimodal emotion recognition (Liu et al., 10 May 2025), and RGB-X segmentation (Wang et al., 2021, Zhang et al., 2022, Guo et al., 12 Sep 2025)) consistently report notable gains with bidirectional strategies over both unimodal and unidirectional baselines.

Quantitative benefits include:

CmFNet (CT/MRI segmentation): Bidirectional fusion yields Dice Similarity Coefficient (DSC) of 76.75% vs. 73.76% for single-modal baseline; inter-modal loss/ablation studies show +8% DSC over weakly-supervised only (Meng et al., 22 Jun 2025).
Fusion-Mamba (RGB-IR object detection): Bidirectional channel swapping + deep gating increases mAP $_{50}$ by up to 5.6 percentage points over single-stage or one-way gating on FLIR-Aligned dataset (Dong et al., 2024).
CDI-DTI (drug–target interaction): Full bidirectional cross-attention + redundancy suppression reaches AUROC 0.9104 vs. 0.86–0.88 for unimodal or uni-fusion baselines, with largest improvements in cross-domain transfer and cold-start generalization (Li et al., 22 Oct 2025).
CMDFusion (LIDAR–image segmentation): Bidirectional fusion + cross-modality distillation attains 71.6% mIoU on SemanticKITTI (w/ instance-level augmentation) versus 66.2% for SPVCNN baseline, and outperforms both 2D→3D and 3D→2D only schemes (Cen et al., 2023).
CMX (RGB-X segmentation): Bidirectional channel and spatial rectification achieves 54.1% mIoU on NYU-Depth V2 compared to 52.8% for strongest one-way fusion and 48.0% for RGB-only (Zhang et al., 2022).

A table summarizing some major performance gains is given below.

Domain	Baseline (Uni/no fusion)	Unidirectional Fusion	Bidirectional Fusion
CT-MRI Segmentation	DSC 73.76%	–	DSC 76.75%
RGB-IR Detection	mAP $_{50}$ 78.7%	– (SSCS or DSSF only)	mAP $_{50}$ 84.3%
LIDAR Semantic Seg.	mIoU 66.2%	67.7% / 60.42%	mIoU 71.6%
RGB-X Segmentation	mIoU 48.0%	52.8%	mIoU 54.1%

6. Algorithmic Instantiations and Pseudocode Sketches

State-of-the-art implementations regularly combine multi-scale, block-wise, or layer-adaptive bidirectional exchange with task-specific training objectives:

Blockwise symmetric cross-attention (Ovi): Every DiT block exchanges RoPE-scaled, cross-modally synchronized features both audio→video and video→audio, enabling precise, emergent synchronization in generative pipelines (Low et al., 30 Sep 2025).
Gumbel-softmax adaptive neighborhood selection (TACOcc): For each voxel, a soft, differentiable choice of retrieval scope adapts the degree of bidirectional feature exchange to object scale and context (Lei et al., 19 May 2025).
Stepwise bidirectional alignment (BSAFusion): Forward and backward deformation fields are predicted in $K$ steps each, then composed to yield robust registration for unaligned medical images (Li et al., 2024).

A general pseudocode template found in many works is:

for fusion_stage in stages:
    # Bidirectional fusion for each branch
    x1_fused = Fusion(x1_in, x2_in)   # e.g., channel shuffle/pixel shift, cross-attn
    x2_fused = Fusion(x2_in, x1_in)
    x1_next = NextBlock(x1_fused)
    x2_next = NextBlock(x2_fused)

7. Limitations, Open Challenges, and Extensions

While bidirectional fusion has proven effective across rich application settings, several challenges remain:

Computational cost: Bidirectional blocks, particularly those using multiple attention heads, can double computational throughput compared to unidirectional or concatenative alternatives (Cen et al., 2023).
Feature collapse/redundancy: Naively symmetric fusion can lead to "averaged out" representations; parameter-free asymmetric operations or deep orthogonality regularization alleviate this, but selecting the right balance remains an open question (Wang et al., 2021, Li et al., 22 Oct 2025).
Need for calibration/alignment: Scale and distributional mismatches across modalities (e.g., in field-of-view, granularity, or informativeness) require careful design of projection, gating, or adaptive retrieval mechanisms (Dong et al., 2024, Lei et al., 19 May 2025).
Supervision and generalization: Strong performance in weakly supervised or label-sparse regimes often depends on carefully constructed consistency, distillation, or pseudo-labeling schemes (Meng et al., 22 Jun 2025, Cen et al., 2023).

Extensions currently explored include adaptation to additional modalities (radar, event cameras), task-agnostic plug-and-play designs, and unified, single-stage fusion-registration frameworks for image alignment and fusion (Li et al., 2024).

References:

(Meng et al., 22 Jun 2025, Dong et al., 2024, Cen et al., 2023, Wang et al., 2021, Low et al., 30 Sep 2025, Li et al., 22 Oct 2025, Zhang et al., 2022, Liu et al., 10 May 2025, Guo et al., 12 Sep 2025, Lei et al., 19 May 2025, Li et al., 2024, Yu et al., 13 Mar 2025)