Dual-Branch Deep Feature Fusion Module

• The paper introduces a dual-branch module that preserves complementary feature subspaces before explicit fusion, improving integration across modalities.
• It employs diverse fusion operators such as residual summation, attention mechanisms, and Transformer-based methods to optimize feature interaction.
• The architecture has shown effective applications in semantic segmentation, precision oncology, and ultrasound segmentation with improved performance metrics.

A dual-branch deep feature fusion module is a neural design pattern in which two specialized feature streams are preserved long enough to retain complementary structure and are then integrated by an explicitly defined fusion operator, rather than being mixed only implicitly inside a shared stack. In the literature, the two branches can correspond to RGB and depth/HHA streams in semantic segmentation, base and detail streams in multimodal image fusion, body and boundary streams in ultrasound segmentation, beam-related and position-related representations in beam prediction, or cell-line and drug representations in multimodal prediction pipelines (Su et al., 2021, Zhao et al., 2022, Xu et al., 2024, Jin et al., 10 Feb 2026, Zhao et al., 25 Mar 2026).

1. Conceptual scope

Across the cited literature, “dual-branch” does not denote a single canonical block. It denotes a family of architectures that separate two complementary representational roles and then define how those roles interact. In FSFNet, the duality is explicitly modality-based: RGB and HHA/depth are encoded separately and fused stage by stage through cascaded Symmetric Cross-modality Residual Fusion blocks (Su et al., 2021). In CDDFuse and DAF-Net, the duality is decomposition-based: one branch models shared, low-frequency, global or structural information, while the other preserves high-frequency, local, detail-specific information (Zhao et al., 2022, Xu et al., 2024). In DBF-Net, the two branches are semantic subproblems inside segmentation itself, namely lesion body and lesion boundary (Xu et al., 2024). In DeepDTF, the branches represent different data modalities altogether, with a multi-omics cell-line branch and a graph-based drug branch (Zhao et al., 25 Mar 2026).

This diversity matters because it distinguishes the topic from the narrower idea of a simple two-stream network. Some works use explicit feature-level interaction, some use token-level Transformer fusion, and some use branch-specific supervision without an explicit merge operator. This suggests that the defining property is not merely the existence of two streams, but the deliberate maintenance of complementary feature subspaces and the introduction of a mechanism that makes them jointly useful for the downstream objective (Liu et al., 2023, Gao et al., 31 Mar 2026).

2. Architectural organization

A common pattern is parallel feature extraction followed by repeated interaction. FSFNet is organized as an encoder-decoder with an RGB branch, an HHA/depth branch, and a middle fusion branch; the encoder is effectively three-path during encoding because the two modality-specific streams produce same-stage features while the fusion branch aggregates them at every level (Su et al., 2021). MambaDFuse follows a related principle but separates fusion into two phases: two modality-specific streams first undergo dual-level extraction, then a shallow fusion stage produces an initial fused representation, and a deep fusion stage refines it through Multi-modal Mamba blocks (Li et al., 2024). RL-U$^2$Net likewise maintains CT and MRI streams in parallel and inserts RL-XAlign after each encoder stage, so alignment and fusion are performed repeatedly rather than once at the end (Qu et al., 4 Aug 2025).

A second pattern is a shared trunk with dual task-specific branches. DBF-Net uses a shared encoder and a decoder in which each Feature Fusion and Supervision block splits into a body branch and a boundary branch, supervises them separately, and then fuses them back into a single decoder stream (Xu et al., 2024). This makes clear that “dual-branch” need not imply two independent encoders.

A third pattern is heterogeneous branch construction followed by joint token fusion. DeepDTF uses a CNN-attention-plus-Transformer branch for multi-omics features and a GNN-plus-Transformer branch for molecular graphs, then concatenates the two token sequences and applies a Fusion-Transformer over the combined set (Zhao et al., 25 Mar 2026). The solar-flare model similarly combines multimodal tokenization with two parallel visual scales, then applies cross-modal fusion within each scale and cross-scale interaction across the two visual branches (Zhao et al., 17 May 2026). These cases show that dual-branch fusion can be coupled to branch asymmetry: the two branches may differ not only in input modality but also in inductive bias.

3. Fusion operators and interaction mechanisms

The actual fusion operator varies sharply across implementations. In FSFNet, feature selection is implemented by $1\times1$ convolutions with stride $1$, producing selected complementary features that are added residually across modalities before concatenation with the previous fused state. The resulting mappings are symmetric because RGB injects into HHA and HHA injects into RGB, and the fused representation is propagated from shallow to deep stages (Su et al., 2021). In this formulation, “feature selection” is learned projection rather than an explicit sigmoid or softmax attention gate.

DBF-Net provides one of the cleanest cross-branch residual formulations. Its feature fusion module is

$$F_{\text{body}}^{*}=F_{\text{body}}+Conv(F_{\text{bound}}),$$

$$F_{\text{bound}}^{*}=F_{\text{bound}}+Conv(F_{\text{body}}),$$

followed by

$$F_{\text{out},i}=\lambda F_{\text{body},i}^{*}+F_{\text{bound},i}^{*},$$

where $\lambda$ is a trainable scalar initialized to $1$. This is an explicitly bidirectional exchange followed by a learnable branch balance, not a plain sum or concatenation (Xu et al., 2024).

Other works use attention-weighted fusion. In the solar-flare model, the multimodal cross-attention module uses visual tokens as queries and parameter tokens as keys and values,

$Q=X_vW_q,\quad K=X_tW_k,\quad V=X_tW_v,$

so each magnetogram token is conditioned by the magnetic-parameter embedding. The authors also report that the inverse direction, where parameters query image features, performs worse, which makes the cross-attention direction part of the module definition rather than a minor implementation detail (Zhao et al., 17 May 2026). MBDF-Net uses a different attention pattern: point-aligned image features and point features are concatenated to produce modality-specific attention maps,

$$F_{\text{attention-I}}=\sigma\!\left(Conv_I(F_{\text{image}}\oplus F_{\text{point}})\right),$$

$1\times1$0

and the fused update is

$1\times1$1

This makes the attention explicitly modality-conditioned and preserves a running fused state $1\times1$2 (Tan et al., 2021).

At the token level, DeepDTF uses a simpler but broader interaction rule: $1\times1$3 Here, fusion is not a branch-wise weighted sum but self-attention over the concatenated multimodal token set, so cross-modal alignment emerges through joint token interaction (Zhao et al., 25 Mar 2026). By contrast, the beam-prediction framework names two fusion variants, AutoFusion and GanFusion, both operating on concatenated dual-branch feature vectors rather than on dense feature maps; AutoFusion uses a compact latent reconstruction module, while GanFusion uses adversarial feature alignment (Jin et al., 10 Feb 2026).

4. Supervision and optimization of branch specialization

Dual-branch fusion modules are usually inseparable from branch-specific supervision. In CDDFuse, the decomposition into base and detail branches is enforced statistically rather than by explicit Fourier or wavelet separation. The decomposition loss

$1\times1$4

encourages detail features to become less correlated and base features to become more correlated across modalities. Training is split into Stage I reconstruction with SSIM and decomposition supervision, and Stage II fusion with intensity, gradient, and decomposition losses retained (Zhao et al., 2022).

DBF-Net uses explicit semantic disentanglement through auxiliary masks. It constructs $1\times1$5 and $1\times1$6 from the original segmentation mask and optimizes

$1\times1$7

with each task loss defined as

$1\times1$8

where $1\times1$9 and $1$0. The branch distinction is therefore architectural and supervisory at the same time (Xu et al., 2024).

Other systems make the branches auxiliary tasks. In the beam-prediction framework, one RegNet branch is supervised for beam prediction and the other for coordinate prediction, and the total objective combines $1$1, $1$2, and fusion-specific losses such as $1$3 or $1$4 (Jin et al., 10 Feb 2026). In DeepDTF, the fused representation feeds both a regression head for $1$5 and a sensitivity-classification head, with total loss

$1$6

This suggests that dual-branch fusion is often stabilized by multitask or auxiliary objectives rather than by architecture alone (Zhao et al., 25 Mar 2026).

5. Application domains and empirical evidence

The empirical role of dual-branch fusion is clearest in ablation studies. On NYUDv2, FSFNet reports a baseline of 47.9% mIoU for “Res-101 + SUM”, 50.8% for “Res-101 + SCRF”, 49.6% for “Res-101 + DFP”, and 52.0% for “Res-101 + SCRF + DFP”, indicating that explicit symmetric cross-modality residual fusion contributes more than simple element-wise summation and remains complementary to detail propagation (Su et al., 2021).

In ultrasound segmentation, DBF-Net reports best CNN-comparison Dice Similarity Coefficients of 81.05% on BUSI, 76.41% on UNS, and 87.75% on UHES. Its BUSI ablation also shows that using both branch supervision terms is better than using only $1$7, only $1$8, or only $1$9, and that adding feature fusion inside the FFS block improves over FFS without the internal fusion operator (Xu et al., 2024).

The solar-flare model provides unusually direct evidence about the fusion operator itself. For binary $$F_{\text{body}}^{*}=F_{\text{body}}+Conv(F_{\text{bound}}),$$0-class prediction, the fusion model reports HSS 0.658, TSS 0.661, FAR 0.132, and ACC 0.829, while the ablation comparing fusion mechanisms shows that cross-attention outperforms concatenation, addition, and inverse cross-attention. For example, Cross-Attention yields TSS 0.661, while Concatenation yields 0.653, Addition 0.648, and Inverse Cross-Attention 0.646 (Zhao et al., 17 May 2026).

In multimodal precision oncology, DeepDTF reports with full multi-omics inputs RMSE = 1.248, $$F_{\text{body}}^{*}=F_{\text{body}}+Conv(F_{\text{bound}}),$$1, and AUC = 0.987, while the paper states that classification error $$F_{\text{body}}^{*}=F_{\text{body}}+Conv(F_{\text{bound}}),$$2 is reduced by 9.5%. Its ablation also shows that removing the Fusion-Transformer slightly reduces performance, supporting the claim that explicit cross-modal interaction modeling provides incremental value beyond the two branch encoders themselves (Zhao et al., 25 Mar 2026).

These results span RGB-D segmentation, ultrasound segmentation, solar-flare prediction, and multi-omics drug response. This suggests that the utility of dual-branch fusion is not tied to one sensory pairing or one task family; rather, it recurs whenever a problem contains two feature subspaces that are complementary but not safely mergeable by naive addition or late decision averaging.

6. Limitations, ambiguities, and recurring misconceptions

A recurrent misconception is that “dual-branch” always means “two parallel encoders.” DBF-Net contradicts this directly by using a shared encoder and dual decoder branches for body and boundary features (Xu et al., 2024). A second misconception is that “feature selection” necessarily implies attention masks. In FSFNet, the SCRF selection operators are $$F_{\text{body}}^{*}=F_{\text{body}}+Conv(F_{\text{bound}}),$$3 convolutions, and the paper explicitly distinguishes this from explicit sigmoid or softmax gating (Su et al., 2021).

Another ambiguity concerns what counts as fusion. Some works place the main novelty in feature-space operators, but others shift the emphasis to supervision or optimization. The bi-level framework for multimodal image fusion uses one fusion network plus two supervisory branches—a visual discriminator and a semantic perception network—so its “dual” structure is better described as dual-task supervision than as a canonical two-stream feature extractor (Liu et al., 2023). FedDBP goes further in that direction: its client-side dual-branch projector has a shared branch for prototype fidelity and a decision branch for discriminability, but there is no explicit tensor concatenation or addition between the two projected features; the combination is functional, through shared-backbone joint optimization (Gao et al., 31 Mar 2026).

Published descriptions also remain under-specified surprisingly often. FSFNet reports that exact kernel sizes of some internal convolutions, channel compression after concatenation, normalization layers, and tensor shapes per stage are not fully documented, and the published equations are partly typeset incorrectly (Su et al., 2021). DAF-Net defines branch-specific fusion layers $$F_{\text{body}}^{*}=F_{\text{body}}+Conv(F_{\text{bound}}),$$4 and $$F_{\text{body}}^{*}=F_{\text{body}}+Conv(F_{\text{bound}}),$$5 but does not specify their internal mechanics, and likewise leaves the exact INN detail-branch structure under-described (Xu et al., 2024). The beam-prediction paper states the concatenated feature dimension and the AutoFusion and GanFusion layouts, but omits exact per-branch feature dimensions and the tuned values of $$F_{\text{body}}^{*}=F_{\text{body}}+Conv(F_{\text{bound}}),$$6 and $$F_{\text{body}}^{*}=F_{\text{body}}+Conv(F_{\text{bound}}),$$7 (Jin et al., 10 Feb 2026).

A plausible implication is that “dual-branch deep feature fusion module” should be treated less as a fixed module class, like a residual block or depthwise convolution, and more as an architectural principle. The stable elements are branch specialization, explicit interaction, and nontrivial supervision; the exact instantiation may be residual exchange, attention, decomposition, adversarial alignment, or token-level self-attention.