Hierarchical Fusion Strategy

• Hierarchical Fusion Strategy (HFS) is a design paradigm that integrates heterogeneous information progressively across levels, scales, and modalities.
• It employs staged integration, coarse-to-fine refinement, and branch specialization to address challenges in tasks like image segmentation, speech recognition, and deepfake detection.
• By organizing fusion over multiple levels, HFS improves model performance and efficiency compared to single-step merging, while introducing added architectural complexity.

Hierarchical Fusion Strategy (HFS) is a broad research designation for procedures that combine information progressively across levels rather than by a single flat merge. In the surveyed literature, the term spans recursive assembly in hierarchical tilings, staged multimodal representation learning, cross-scale feature integration, hierarchical supervision, and coarse-to-fine hypothesis refinement. This suggests that HFS is best understood not as one canonical algorithm but as a family of architectures and formalisms in which fusion is organized over a hierarchy of levels, stages, scales, branches, or semantic granularities (Frank, 2013, Majumder et al., 2018, Tan et al., 24 Sep 2025).

1. Terminological scope and conceptual lineage

One formal antecedent appears in the theory of tilings, where “fusion rules” are introduced as a general hierarchical construction extending substitution or inflate-and-subdivide systems. In that framework, $0$-supertiles are prototiles, $1$-supertiles are patches of prototiles, $2$-supertiles are patches of $1$-supertiles, and, in general, $n$-supertiles are finite patches made by fusing $(n-1)$-supertiles. A fusion rule is the sequence

$\mathcal{R}=\{\mathcal{P}_n:n\in\mathbb N\},$

with transition matrices

$$M_{n,N}(i,j)=\#(P_n(i)\text{ in }P_N(j)), \qquad M_{n,N}=M_{n,m}M_{m,N},$$

together with primitivity, van Hove growth, and frequency formulas for admitted tilings (Frank, 2013). This is not a feature-fusion module in the modern machine-learning sense, but it establishes the core hierarchical idea: larger structures are built by recursively fusing smaller ones.

In contemporary machine-learning usage, the surveyed literature suggests that HFS typically denotes staged combination of heterogeneous information sources. The hierarchy may be over modalities, network depths, graph structures, semantic levels, or candidate hypotheses. It may also be explicit in supervision rather than only in architecture, as in HS3-Fuse, where intermediate segmentation stages are trained with clustered label sets of increasing complexity and later aggregated through OCR blocks (Borse et al., 2021).

Paper	Meaning of HFS	Hierarchy
(Frank, 2013)	Fusion rules for tilings	supertiles by level
(Majumder et al., 2018)	Pairwise then trimodal multimodal fusion	unimodal $\rightarrow$ bimodal $\rightarrow$ trimodal
(Huo et al., 2022)	Hierarchical feature fusion from frozen speech encoder layers	layer depth
(Zhou et al., 2023)	Attention-guided multi-step fusion for recommendation	graph $1$0 interaction $1$1 multimodal item
(Mehta et al., 10 Jan 2025)	ViT–ResNet hierarchical feature fusion	low/mid/high features
(Tan et al., 24 Sep 2025)	Modular local-global fusion in segmentation	stage-wise branch fusion

A terminological complication is that the acronym is not stable across papers. In HFS-TriNet, HFS means Heuristic Frame Sampling, a front-end clip-sampling procedure rather than a fusion operator. That paper explicitly distinguishes HFS from the later “pyramid fusion strategy” used for feature fusion (Lu et al., 24 Apr 2026).

2. Structural principles of hierarchical fusion

The surveyed works suggest several recurring structural principles. The first is progressive integration. Instead of directly concatenating all inputs, information is fused in stages so that earlier fusions can regularize or refine later ones. In multimodal sentiment analysis, the pipeline maps audio, video, and text into a common space, fuses them pairwise to form $1$2, $1$3, and $1$4, and only then fuses the three bimodal representations into a trimodal representation; optional GRUs model context at unimodal, bimodal, and trimodal levels (Majumder et al., 2018). In TMFUN, the sequence is even more explicit: modality-specific item graphs are built first, then fused into an item feature graph, then combined with user-item interaction data through attention, then refined by visual-text contrastive alignment, and finally merged with the collaborative-filtering item embedding (Zhou et al., 2023).

The second principle is coarse-to-fine or cross-scale refinement. In HFMF for deepfake detection, ViT tokens are fused sequentially with low-, mid-, and high-level ResNet features: $1$5 so that global transformer context is progressively updated by local multi-scale evidence (Mehta et al., 10 Jan 2025). In HiPerformer, each stage of the local-global fusion branch takes current local features $1$6, current global features $1$7, and the previous fused state $1$8, making the hierarchy explicitly recurrent across encoder depth (Tan et al., 24 Sep 2025).

A third principle is branch specialization. Many HFS designs preserve separate processing streams before fusion. HiPerformer uses a local CNN branch, a global Swin Transformer branch, and a local-global fusion branch (Tan et al., 24 Sep 2025). UniPTMs uses a “Master-Slave” architecture in which the master path carries dominant high-dimensional pretrained sequence representations, whereas the slave path processes lower-dimensional auxiliary features and supplements the master representation through staged interaction (Lin et al., 5 Jun 2025). In RGB-D surface normal estimation, RGB and depth are encoded separately, and depth is further modulated by a confidence branch before decoder-side fusion (Zeng et al., 2019).

A fourth principle is that the hierarchy need not be over feature tensors alone. In visual place recognition, hierarchical multi-process fusion first reduces the candidate set with an early-tier technique and then refines the surviving hypotheses with later tiers; stacked hierarchical variants concatenate candidate sets within a tier to improve retention of the correct place (Hausler et al., 2020). This suggests that HFS can operate over hypothesis spaces as well as representational spaces.

3. Mathematical operators and optimization mechanisms

Although the specific operators vary, the literature suggests that most HFS formulations rely on a small set of recurring mechanisms: weighted aggregation, attention or similarity operators, residual correction, and hierarchy-aware regularization.

A classical weighted formulation appears in SSVEP frequency recognition. The framework first performs spatial-dimension fusion within each subband,

$1$9

and then frequency-dimension fusion across subbands,

$2$0

The hierarchy is thus explicit in a two-stage weighting scheme: coefficients within band, then bands within frequency decision (Zhang et al., 2018).

Attention-style operators dominate recent multimodal HFS designs. HPFusion generates four hierarchical question-answer descriptions with LLaVA, encodes them with a frozen CLIP text encoder, concatenates the reduced text features into $2$1, and injects them into the fusion backbone through cross-attention: $2$2 The training objective couples an image loss with a hierarchical semantic loss in CLIP space,

$2$3

with $2$4 and $2$5 (Yang et al., 2024). MGHFT likewise injects different text views into different PVT stages and uses Soft-Fusion attention for local tokens,

$2$6

combined with stage-wise alignment losses for global tokens (Chen et al., 25 Jul 2025).

Residual and projection-based corrections are also common. In MH2F-Net for deraining, the Residual Projected Feature Fusion strategy computes

$2$7

thereby using differences between extracted and distilled features as complementary signals rather than concatenating them directly (Chen et al., 2021). In RGB-D normal estimation, hierarchical decoder fusion is defined at scale $2$8 by

$2$9

so unreliable depth features are softly suppressed before concatenation (Zeng et al., 2019).

Some HFS designs fuse not merely features but also constraints across levels. In audio-visual separation, hierarchical fusion combines middle fusion at the U-Net bottleneck with late fusion at mask prediction: $1$0 and adds an alignment term

$1$1

to reduce the modality gap between audio and visual features (Hu et al., 24 Sep 2025). In UniPTMs, the staged modules BGCA, LDFN, BHGFN, and HDWF are coupled to a Hierarchical Contrastive loss

$1$2

with the paper reporting $1$3 as best (Lin et al., 5 Jun 2025).

4. Major application families

In multimodal perception and recognition, HFS is typically used to align semantically different signals before final decision-making. HPFusion for infrared-visible image fusion orders four question sets from overall scene understanding to specific salient targets and detailed regions, then uses LLaVA-generated answers and CLIP-space semantic matching to preserve complementary information in a way intended to match human visual interpretation (Yang et al., 2024). MGHFT for sticker emotion recognition follows a similar stage-wise logic: LLaVA-NeXT generates text for intent, overall style, main roles, and fine-grained character details; these four textual views are injected into the four stages of a PVT in the order $1$4, which the paper reports as the best ordering (Chen et al., 25 Jul 2025). Earlier multimodal sentiment work already instantiated the same structural intuition in a simpler setting by composing pairwise fusion before final trimodal fusion and then adding GRU-based context modeling (Majumder et al., 2018).

In medical imaging and biomedical prediction, HFS often addresses the tension between local detail and global context, or between dominant and auxiliary sources. HiPerformer organizes a local branch, a global branch, and a local-global fusion branch, while replacing skip connections with Progressive Pyramid Aggregation to suppress shallow noise and narrow the semantic gap between encoder and decoder (Tan et al., 24 Sep 2025). HFM-Net for RGB-D normal estimation hierarchically fuses RGB and confidence-weighted depth at four decoder scales to ensure global surface smoothness while preserving salient details (Zeng et al., 2019). UniPTMs generalizes the principle to protein post-translational modification prediction through a three-phase master-slave architecture in which dominant pretrained sequence embeddings are fused early, low-dimensional auxiliary features are fused separately, and cross-branch integration is performed at intermediate and late stages (Lin et al., 5 Jun 2025).

In speech, audio, and neuro-signal processing, HFS frequently exploits information distributed across depth or bands. The speech HFF method freezes a 24-layer conformer encoder, extracts multiple internal layer outputs, and fuses them hierarchically before an RNN-T decoder, motivated by the observation that middle layers perform best and that linear fusion of multiple layers improves over using a single layer (Huo et al., 2022). In code-switched ASD detection, hierarchical feature fusion evaluates several orders over acoustic, linguistic, and paralinguistic features, with the paper emphasizing that fusion order is not commutative (Akhtar et al., 2024). In SSVEP-based BCIs, HFCORRCA formalizes hierarchical fusion as a two-level weighted strategy over spatial coefficients and then frequency subbands (Zhang et al., 2018).

In scene understanding, recommendation, and localization, HFS often reconciles heterogeneous structures. VisMoFlow inserts event data as a bridge between RGB and LiDAR and performs visual luminance fusion, visual structure fusion, and motion correlation fusion in sequence, explicitly moving from appearance-level complementarity to geometry-level complementarity and finally to motion-space complementarity (Zhou et al., 2024). TMFUN for multimodal recommendation combines modality feature graphs, attention-guided interaction fusion, cross-modal contrastive fusion, and final global fusion with collaborative-filtering embeddings (Zhou et al., 2023). Hierarchical multi-process fusion for visual place recognition instead refines candidate hypotheses across tiers, showing that hierarchy can be imposed over retrieval distributions rather than latent feature channels (Hausler et al., 2020).

5. Empirical behavior and reported advantages

Reported gains vary substantially by domain, but the literature consistently associates hierarchical organization with better use of complementary information than one-shot fusion. In speech transfer learning, balanced HFF alone reaches 7.0% WER with 12.3M parameters, 7648 MB memory, and 3655 ex/sec, while HFF-b plus Adapter($1$5) at all layers reaches the same 5.5% WER as full fine-tuning with 97% fewer trainable encoder parameters and 53% faster training speed (Huo et al., 2022). This suggests that depth-wise hierarchical fusion can partly substitute for encoder fine-tuning by exploiting fixed intermediate representations more effectively.

In deepfake detection, Module 1 of HFMF improves over its constituent backbones on WildRF: pre-trained ViT-B/16 reports 84.1% mean, pre-trained ResNet50 87.7% mean, and hierarchical fusion of ViT-B/16 + ResNet50 88.5% mean; the full HFMF system reaches 89.4% mean. The same paper reports ECE reductions after calibration, including WildRF Train 0.0580 $1$6 0.0473 and WildRF Twitter 0.1668 $1$7 0.1399 (Mehta et al., 10 Jan 2025).

In audio-visual separation, the comparison among middle, late, and hierarchical fusion is unusually explicit. On MUSIC, SDR progresses from 5.57 for middle fusion and 5.86 for late fusion to 6.65 for hierarchical fusion and 6.72 for hierarchical fusion plus alignment. On MUSIC-21, the sequence is 7.36, 7.27, 7.72, and 8.03; on VGGSound, hierarchical fusion plus alignment reports 1.97 compared with 1.16 for middle fusion and 0.90 for late fusion (Hu et al., 24 Sep 2025). The same paper notes that hierarchical fusion does not always give the best SIR, indicating that the advantage is metric-dependent rather than universal.

In segmentation and related medical tasks, ablation results also support stage-wise fusion. HiPerformer reports a full-model score of 83.93 DSC on Synapse and 76.39 DSC on BTCV; removing the local branch reduces these to 78.46 and 68.77, while removing the global branch reduces them to 82.23 and 73.94 (Tan et al., 24 Sep 2025). HS3-Fuse reports 81.8 mIoU on Cityscapes validation and 85.7 mIoU plus 71.7 iIoU on the Cityscapes test set, building on the semantically calibrated intermediate representations produced by HS3 (Borse et al., 2021).

In multimodal classification from speech, the CoSAm work reports its headline best result with A + L THEN P (Transformer): 98.75% accuracy, 97.19 macro F1 (Akhtar et al., 2024). In visual place recognition, combined hierarchical recall reaches 58.0% on Nordland training and test-set Recall@1 of 77.2% on Nordland and 41.0% on Berlin (Hausler et al., 2020). The older multimodal sentiment model reports about 1% absolute improvement over concatenation on utterance-level multimodal sentiment analysis and up to 2.4% on multi-utterance video clips (Majumder et al., 2018). These results suggest that the empirical benefit of HFS is often modest in absolute terms but repeatable across architectures and tasks.

6. Conceptual boundaries, misconceptions, and limitations

A common misconception is to treat HFS as a single standardized module. The surveyed papers do not support that reading. Some works use HFS for recursive geometric assembly in mathematical tilings (Frank, 2013); others use it for pairwise-to-trimodal neural fusion (Majumder et al., 2018), hierarchical supervision plus OCR-based aggregation (Borse et al., 2021), or multi-branch master-slave integration (Lin et al., 5 Jun 2025). A further misconception is that “hierarchical” always means “deeper neural network.” The place-recognition literature shows a hierarchy over candidate hypothesis sets rather than over latent feature maps (Hausler et al., 2020).

Another boundary concerns nomenclature. HFS-TriNet explicitly defines HFS as Heuristic Frame Sampling, not feature fusion, and states that the “hierarchical fusion” interpretation is misleading if applied to HFS in that paper. The actual fusion there is performed later by a separate pyramid fusion strategy (Lu et al., 24 Apr 2026). This is an important reminder that acronym-level similarities do not guarantee methodological equivalence.

The literature also makes clear that hierarchy alone is not sufficient. Order matters in some settings: the CoSAm study reports the best performance for acoustic + linguistic fusion followed by paralinguistic integration, while other orders perform differently (Akhtar et al., 2024). In sticker emotion recognition, using all textual views uniformly at every stage performs worse than assigning different views to different stages (Chen et al., 25 Jul 2025). Some methods require auxiliary mechanisms to realize their full gains: speech HFF is strongest when combined with adapters (Huo et al., 2022), and audio-visual separation improves further when hierarchical fusion is paired with representation alignment (Hu et al., 24 Sep 2025).

Complexity is a recurrent limitation. UniPTMs is explicitly described as architecturally heavy enough to motivate a lightweight UniPTMs-mini variant (Lin et al., 5 Jun 2025). HS3-Fuse introduces additional OCR-based aggregation, although the paper also presents a lighter version at matched GMACs (Borse et al., 2021). This suggests a general trade-off: hierarchical fusion can improve information transfer and reduce feature conflict, but it often does so by adding stages, branches, losses, or calibration modules.

Taken together, the literature suggests that HFS is most accurately characterized as a design philosophy: information that is heterogeneous in scale, modality, semantics, or reliability is fused gradually, under explicitly structured intermediate states, rather than collapsed into a single undifferentiated representation. The specific realization, however, is domain-dependent and ranges from supertile constructions and weighted subband aggregation to cross-attention, gated residual coupling, hierarchical supervision, and staged hypothesis refinement (Frank, 2013, Zhang et al., 2018, Tan et al., 24 Sep 2025).