Hierarchical Fusion Strategy

• Hierarchical Fusion Strategy is a structured design that progressively integrates multi-modal information through ordered stages, preserving modality-specific attributes.
• It is implemented in various domains, such as early–middle–late fusion in UniPTMs and luminance–structure–correlation fusion in VisMoFlow, to optimize multi-scale interactions.
• Empirical ablation studies demonstrate that strategically ordering and weighting modality contributions significantly improves overall performance metrics.

Hierarchical fusion strategy denotes a class of fusion schemes in which information is integrated progressively across ordered stages, levels, or spaces rather than by a single direct merge. In the current literature, this structure appears as early–middle–late fusion in the “Master–Slave” UniPTMs framework for multi-type PTM site prediction, as luminance-space, structure-space, and correlation-space fusion in VisMoFlow for scene flow, and as pairwise-then-trimodal fusion in multimodal sentiment analysis (Lin et al., 5 Jun 2025, Zhou et al., 2024, Majumder et al., 2018). The common premise is that heterogeneous inputs should not be fused only once: they are instead aligned, filtered, or reweighted at the level where their complementarity is most useful.

1. Structural forms of hierarchical fusion

A hierarchical fusion strategy can be organized as a staged pipeline, a semantic hierarchy, or a multi-level estimator. UniPTMs is an explicit staged design: early-stage fusion combines a Bidirectional Gated Cross-Attention module in the master branch with a Low-Dimensional Fusion Network in the slave branch, middle-stage fusion uses the Bidirectional Hierarchical Gated Fusion Network, and late-stage fusion uses Hierarchical Dynamic Weighting Fusion to form a single vector $F^{fusion}$ for classification (Lin et al., 5 Jun 2025). VisMoFlow likewise uses three sequential fusion levels, but each level is defined by a different “homogeneous” feature space: visual luminance fusion, visual structure fusion, and motion correlation fusion (Zhou et al., 2024).

Other papers implement hierarchy through estimator topology rather than deep feature stages. Hierarchical Adaptive Bayesian Data Fusion uses a two-level structure with a bank of Kalman-filter-based experts at Level 1 and a single Kalman filter as fusion center at Level 2; local Mahalanobis-based and voting-based reliabilities modulate the global measurement-noise covariance (Echeverri et al., 2017). In multimodal sentiment analysis, Majumder et al. first fuse modalities two in two and only then fuse all three, optionally inserting GRUs at unimodal, bimodal, and trimodal stages (Majumder et al., 2018).

Paper	Hierarchy	Fusion logic
UniPTMs	Early / middle / late	BGCA + LDFN $\rightarrow$ BHGFN $\rightarrow$ HDWF
VisMoFlow	Luminance / structure / correlation	Event as bridge between RGB and LiDAR
HFusion	Bimodal / trimodal	Pairwise fusion before all-three fusion
HAB-DF	Experts / fusion center	Local statistics + global softened majority voting
HFCORRCA	Spatial dimension / frequency dimension	Nonlinear weighted fusion across coefficients and sub-bands
HPFusion	Pixel / texture–structure / semantic	Image losses plus LLaVA- and CLIP-guided semantic fusion

This suggests that “hierarchical” is not tied to one implementation style. The hierarchy may be temporal in pipeline order, semantic in representation depth, or probabilistic in the organization of estimators.

2. Representation levels, modality roles, and fusion order

Many hierarchical fusion strategies are defined by the level at which each modality contributes. MFRA for vision-and-language navigation organizes fusion into low-level visual cues, mid-level object semantics, high-level instruction semantics, and an auxiliary temporal history stream (Yue et al., 23 Apr 2025). HPFusion for infrared–visible image fusion describes low-level pixel features, mid-level texture/structure features, and high-level semantic priors obtained by a Large Vision–LLM, with semantic guidance injected by cross-attention and enforced by a CLIP-based semantic loss (Yang et al., 2024). In SSVEP frequency recognition, the hierarchy is orthogonal to neural feature depth: spatial-dimension fusion first combines all CORRCA correlation coefficients within each sub-band, and frequency-dimension fusion then combines the sub-band features with a second nonlinear weighting (Zhang et al., 2018).

A recurrent theme is that modalities are assigned distinct functional roles rather than treated as symmetric inputs. In VisMoFlow, RGB contributes high-fidelity absolute brightness, event data contributes relative brightness changes and very high dynamic range, LiDAR contributes accurate 3D shape, and the event modality serves as a bridge between RGB and LiDAR in both visual and motion spaces (Zhou et al., 2024). UniPTMs is also asymmetric: the master branch processes high-dimensional PLM embeddings from ProtT5 and ESM-2, whereas the slave branch handles EMBER2, PseAAC, BLOSUM62, and AAIndex through a lightweight low-dimensional fusion path (Lin et al., 5 Jun 2025).

Fusion order can itself be a learned prior. In the CoSAm study on code-switched autism detection, the hierarchical method that first combines acoustic and linguistic features and then merges paralinguistic features is reported as the best overall configuration when mBERT is used for the linguistic stream, reaching 98.75% accuracy and 97.19% macro F1 (Akhtar et al., 2024). The paper explicitly states that “modality-order matters,” and compares this order against alternatives such as $P+A\rightarrow L$ and $L+P\rightarrow A$. A plausible implication is that hierarchical fusion is often a hypothesis about when a modality should intervene, not merely a mechanism for combining tensors.

3. Fusion operators, gating, and dynamic weighting

The operational core of hierarchical fusion is usually a sequence of gated or attention-based transformations that preserve branch-specific structure while enabling controlled exchange. In UniPTMs, the Bidirectional Hierarchical Gated Fusion Network performs three tiers of gating—attention-level, channel-level, and spatial-level—at each fusion connection between master and slave representations. Its asymmetric hierarchical attention is

$$A_1 = \mathrm{Attention}(Q=H_1W_q,\ K=H_2W_k,\ V=H_2W_v),\qquad A_2 = \mathrm{Attention}(Q=H_2W_q,\ K=H_1,\ V=H_1),$$

with

$$\mathrm{Attention}(Q,K,V)=\mathrm{Softmax}(QK^\top/\sqrt{d_k})V.$$

Attention-level gating then mixes the two directions,

$$g_a=\sigma(W_g\cdot[A_1;A_2]+b_g),\qquad F_a=g_a\odot A_1+(1-g_a)\odot A_2,$$

after which dynamic convolutional enhancement, channel gating, and spatial gating are applied before the residual output

$\mathrm{Output}=\mathrm{LayerNorm}(F_{gs}+H_1).$

The paper characterizes this as bidirectional but master-centric fusion (Lin et al., 5 Jun 2025).

UniPTMs then uses Hierarchical Dynamic Weighting Fusion for late aggregation. The master branch receives channel weighting,

$$a_c=\sigma(W_2\cdot \mathrm{GELU}(W_1\cdot \mathrm{GAP}(M))),\qquad \tilde M=W_p\cdot(a_c\odot M),$$

the slave branch receives spatial weighting,

$\rightarrow$0

and the two are combined by dynamic temperature attention, multi-head aggregation, and residual balancing:

$\rightarrow$1

$\rightarrow$2

$\rightarrow$3

Through $\rightarrow$4, Softmax, and LayerScale, the model learns level-specific weights across shallow, intermediate, and deep representations (Lin et al., 5 Jun 2025).

Comparable design patterns recur in other domains. HPCFNet enforces channel pairing by group convolution and uses Reverse Spatial Attention, where unchanged regions produce large $\rightarrow$5 activations and $\rightarrow$6 therefore emphasizes changed regions (Lei et al., 2020). AHMF for guided depth super-resolution uses a multi-modal attention based fusion module consisting of a Feature Enhancement Block and a Feature Recalibration Block, followed by a Bi-Directional Hierarchical Feature Collaboration module implemented with two-headed GRUs across scales (Zhong et al., 2021). HiPerformer uses a Local-Global Feature Fusion module with Adaptive Channel Interaction, Spatial Perception Enhancement, and an inverted-residual MLP to combine local CNN and global Swin-Transformer branches stage by stage (Tan et al., 24 Sep 2025). Across these examples, hierarchical fusion is not simple concatenation; it is conditioned aggregation with explicit salience or reliability control.

4. Training objectives and consistency constraints

Hierarchical fusion is often accompanied by hierarchy-aware supervision. UniPTMs couples its staged feature pipeline to a Hierarchical Contrastive loss after MACP, BHGFN, and HDWF, producing multi-granularity features $\rightarrow$7, $\rightarrow$8, $\rightarrow$9, and final fused $\rightarrow$0 (Lin et al., 5 Jun 2025). The intra-layer contrastive loss at level $\rightarrow$1 is

$\rightarrow$2

the cross-layer loss between adjacent levels is

$\rightarrow$3

and the total objective is

$\rightarrow$4

The stated purpose is to regularize fusion representations so that they are both discriminative within a level and coherent across levels (Lin et al., 5 Jun 2025).

Other hierarchical systems encode similar alignment pressures with different objectives. VisMoFlow combines photometric loss, adversarial loss on fused RGB, spatiotemporal gradient consistency, pseudo-depth supervision for the structure transformer, and a K–L divergence term in correlation space, with total loss

$\rightarrow$5

(Zhou et al., 2024). Deep surface normal estimation with hierarchical RGB-D fusion uses a hybrid multi-scale loss that applies $\rightarrow$6 at coarse scales and $\rightarrow$7 at fine scales, with weights $\rightarrow$8, to address noisy ground-truth normals (Zeng et al., 2019). TMFUN for multimodal recommendation combines BPR loss, multimodal BPR loss, and a contrastive objective that aligns item ID, visual, and textual representations across early-, mid-, and late-step fusion (Zhou et al., 2023).

This suggests that hierarchical fusion is commonly paired with supervision that mirrors the hierarchy itself: level-wise, cross-level, or view-alignment losses are used to stabilize what the architecture is trying to separate and then recombine.

5. Empirical evidence and ablation patterns

Published ablations repeatedly attribute performance gains to the hierarchical organization rather than to a single fusion block. In UniPTMs, removing BGCA/LDFN drops MCC by $\rightarrow$9, omitting BHGFN incurs an additional $P+A\rightarrow L$0 MCC loss, and skipping HDWF further reduces MCC by $P+A\rightarrow L$1; across five PTM types, the full model attains 3.2–11.4% improvements in MCC and 4.2–14.3% in AP over prior state-of-the-art models (Lin et al., 5 Jun 2025). The paper explicitly frames this as evidence that hierarchical, gated, and dynamically weighted fusion critically enhances cross-modal, multi-scale feature integration.

VisMoFlow presents a direct stage-wise ablation. With no fusion, EPE is 0.151 and ACC is 49.20%; visual fusion only gives EPE = 0.118 and ACC = 54.35%; motion fusion only gives EPE = 0.098 and ACC = 62.87%; full visual–motion fusion gives EPE = 0.084 and ACC = 70.34% (Zhou et al., 2024). The reported interpretation is that motion-space fusion yields the largest gain and visual-space fusion further raises the performance ceiling.

Comparable evidence appears in non-PTM settings. In HFCORRCA for SSVEP-based BCI, standard CORRCA at a 1 s time window gives mean accuracy $P+A\rightarrow L$2 and ITR $P+A\rightarrow L$3 bit/min, whereas HFCORRCA gives mean accuracy $P+A\rightarrow L$4 and ITR $P+A\rightarrow L$5 bit/min; SD-only fusion gives $P+A\rightarrow L$6 percentage points, FD-only fusion gives $P+A\rightarrow L$7 percentage points, and combined hierarchical fusion gives $P+A\rightarrow L$8 percentage points over CORRCA (Zhang et al., 2018). In MH2F-Net for deraining, fusion choice is isolated directly: without HADB + concatenation gives 26.83 dB / 0.8852, with HADB + concatenation gives 27.46 dB / 0.8991, with HADB + addition gives 27.87 dB / 0.9024, and with HADB + RPFF gives 29.63 dB / 0.9225 (Chen et al., 2021). In hierarchical audio-visual-proprioceptive fusion for robotic manipulation, the full H-AVP model gives Pouring 1.86 ± 0.13 and Cabinet 4.50 ± 0.54, compared with B-BFM only at 4.60 / 6.21 and IMM only at 3.38 / 6.70 (Li et al., 14 Feb 2026).

A plausible implication is that hierarchical fusion often works because different stages solve different subproblems: one stage suppresses modality mismatch, another establishes cross-scale or cross-view consistency, and a final stage balances the retained signals.

6. Misconceptions, design tensions, and scope

A common misconception is that hierarchical fusion is equivalent to deep stacking or repeated concatenation. Several papers explicitly reject this. HiPerformer contrasts its modular hierarchical encoder with serial stacking, endpoint concatenation, and pointwise addition, arguing that those strategies struggle with feature inconsistency and are prone to information conflict and loss (Tan et al., 24 Sep 2025). MH2F-Net states that RPFF is introduced to progressively discriminate feature learning and aggregate different features “instead of directly concatenating or adding,” and its ablations report that concatenation leaves residual rain streaks while addition over-smooths edges and blurs details (Chen et al., 2021).

A second design tension concerns symmetry. Not all modalities are assumed to have equal status. UniPTMs preserves the primacy of master features during bidirectional fusion (Lin et al., 5 Jun 2025). H-AVP argues that most multimodal fusion approaches implicitly assume homogeneous roles across modalities, but that this assumption is ill-suited for acoustic signals because they are inherently sparse and contact-driven; its solution is to condition visual and proprioceptive representations on audio first, then model higher-order interactions (Li et al., 14 Feb 2026). In audio-visual source separation, the reported analysis is that middle fusion is better suited for short, transient sounds, while late fusion is more effective for sustained and harmonically rich sounds; the proposed hierarchical strategy therefore integrates both stages (Hu et al., 24 Sep 2025). These examples indicate that branch asymmetry and stage placement are often deliberate modeling choices rather than architectural irregularities.

A third tension concerns generality versus assumptions. HAB-DF is training-free, real-time capable, and generic in the sense that any Bayesian estimator can play the role of expert, but the paper also notes reliance on linear-Gaussian assumptions, the need for at least 3 detectors for majority-vote to detect drift/clusters, and the need to tune parameters such as $P+A\rightarrow L$9, $L+P\rightarrow A$0, $L+P\rightarrow A$1, $L+P\rightarrow A$2, and $L+P\rightarrow A$3 for very different sensor modalities (Echeverri et al., 2017). This suggests that hierarchical fusion is a broad design principle, not a guarantee of robustness independent of modeling assumptions.

Across current examples, hierarchical fusion strategy is best understood as a disciplined allocation of fusion operations across ordered levels: pairwise before joint, local before global, shallow before deep, reliable before uncertain, or coarse before fine. Its technical value lies less in the mere presence of multiple stages than in the explicit assignment of modality roles, the use of level-specific operators and losses, and the empirical observation that static one-shot fusion often discards exactly the structure that hierarchical schemes are designed to preserve.