Hierarchical Interaction Fusion Block (HIFB)

• HIFB is a multimodal fusion block that organizes cross-source feature integration in stages rather than via a single merge.
• It preserves modality-specific structures through separate encoders and learnable fusion tokens, enabling fine-grained temporal dependency modeling.
• Empirical results show that HIFB improves performance metrics like FDD and enhances motion-emotion alignment compared to traditional fusion methods.

Searching arXiv for recent and foundational papers relevant to “Hierarchical Interaction Fusion Block” and closely related terminology. {"queries":[{"query":"\"Hierarchical Interaction Fusion Block\" arXiv","limit":5},{"query":"multimodal hierarchical fusion context modeling GRU arXiv (Majumder et al., 2018)","limit":5},{"query":"HiFuse hierarchical feature fusion block arXiv (Huo et al., 2022)","limit":5}]} {"queries":[{"query":"LSF-Animation Hierarchical Interaction Fusion Block arXiv (Lu et al., 23 Oct 2025)","limit":10},{"query":"Hierarchical audio visual proprioceptive fusion robotic manipulation arXiv (Li et al., 14 Feb 2026)","limit":10},{"query":"SFusion self-attention based N-to-One multimodal fusion block arXiv (Liu et al., 2022)","limit":10}]} A Hierarchical Interaction Fusion Block (HIFB) is a multimodal fusion module that organizes cross-source integration in stages rather than by a single flat merge. In the cited literature, the exact term appears explicitly in the speech-driven facial animation framework LSF-Animation, where HIFB combines emotion-related, motion-related, and identity-related cues through dual transformer streams and learnable fusion tokens (Lu et al., 23 Oct 2025). Closely related architectures appear under other names in multimodal sentiment analysis, medical image classification, deraining, EEG decoding, hierarchical classification, whole-slide image analysis, robotic manipulation, and missing-modality fusion. Across these works, the recurring idea is to preserve modality- or scale-specific structure long enough to model interactions progressively, then compose higher-order fused representations for downstream prediction or generation.

1. Terminology and scope

The literature does not use a single name consistently for this family of modules. Several papers describe functionally similar designs under different terms, and the relationship to HIFB is often retrospective rather than official.

Paper	Official module name	Relation to HIFB
(Lu et al., 23 Oct 2025)	Hierarchical Interaction Fusion Block	Exact term
(Majumder et al., 2018)	hierarchical fusion / CHFusion / HFusion	Strong precedent
(Huo et al., 2022)	HFF block	Essentially the same concept in spirit
(Chen et al., 2021)	HADB + RPFF	Closest conceptual match
(Zhang et al., 18 Jan 2026)	CFT Block + hierarchical fusion head	HIFB-equivalent
(Sahoo et al., 2023)	query fusion + FT + CAMP	HIFB-like hierarchy-aware fusion
(Guo et al., 2023)	Bidirectional Interaction block	Direct analogue
(Li et al., 14 Feb 2026)	B-BFM + IMM	Nearest equivalent
(Liu et al., 2022)	SFusion	Non-hierarchical precursor

This naming pattern is important because “hierarchical” and “interaction” are used in more than one sense. In some papers, hierarchy refers to pairwise-to-trimodal composition; in others, it refers to cross-stage or cross-scale propagation; in still others, it refers to coarse-to-fine query refinement or bidirectional exchange between adjacent pyramid levels. A plausible implication is that HIFB is best understood as an architectural principle rather than a single standardized operator.

2. Recurrent design principles

Across the cited works, HIFB-like modules share a staged fusion logic. A first stage usually preserves modality- or branch-specific encoders. A second stage introduces structured interaction, such as pairwise latent-coordinate fusion, branch-conditioned modulation, cross-attention, or residual exchange. A third stage composes these intermediate interactions into a higher-order representation for classification, generation, or control. This pattern is explicit in hierarchical fusion for multimodal sentiment (Majumder et al., 2018), in the HIFB of LSF-Animation (Lu et al., 23 Oct 2025), in HiFuse’s HFF block (Huo et al., 2022), and in the audio-visual-proprioceptive robotic fusion stack (Li et al., 14 Feb 2026).

The interaction mechanism itself varies substantially. In multimodal sentiment analysis, interaction is dimension-wise and low-parameter: scalar pairs of aligned latent features are fused with a learned affine transform followed by $\tanh$ (Majumder et al., 2018). In LSF-Animation, interaction is token-based and iterative: learnable fusion tokens are concatenated to both transformer branches, updated across layers, and refined through cross-attention over dense motion-emotion pair sets (Lu et al., 23 Oct 2025). In HiFuse, interaction is stage-wise and branch-asymmetric: channel attention is applied to global Transformer features, spatial attention to local CNN features, and the previous fused feature is injected through a hierarchical carry-over path before IRMLP-based aggregation (Huo et al., 2022). In robotic manipulation, audio first conditions point-cloud and proprioceptive representations, after which three cross-attention branches model higher-order dependencies (Li et al., 14 Feb 2026).

Several works show that interaction need not be attention-based. HIGT’s Bidirectional Interaction block uses SE reweighting, broadcast addition, mean pooling, and residual fusion between region and patch levels (Guo et al., 2023). MH2F-Net combines attentive hierarchical distillation with residual projected fusion based on feature differencing rather than direct concatenation or addition (Chen et al., 2021). SFusion uses a stack of eight self-attention layers followed by voxel-level modal softmax weighting, but it remains single-stage rather than explicitly hierarchical (Liu et al., 2022). This suggests that the defining property of HIFB is staged dependency modeling, not any particular primitive such as cross-attention, tensor fusion, or gating.

3. Canonical explicit HIFB in label-free speech-driven facial animation

In LSF-Animation, HIFB is the core fusion module inside the Speech-Aware Identity-Emotion Encoder. It receives speech-based emotion features $e_{1:T}$, speech-based motion features $m_{1:T}$, and a neutral-face-based identity feature $z_{\text{id}}$, then outputs the fused motion stream $m_{\text{final}}=\tilde{m}_{1:T}^L$ for downstream latent prediction and animation decoding (Lu et al., 23 Oct 2025). The emotion branch uses pretrained Emotion2vec, producing frame-wise embeddings at 50 Hz with $d_e=768$, while the identity branch maps a 300D neutral FLAME shape parameter through a lightweight MLP.

Identity enters before hierarchical interaction by modulating both speech-derived streams: $$s_{\text{id}} = W_{\text{style}} z_{\text{id}}, \qquad \tilde{m}_{1:T} = W_m m_{1:T} \odot s_{\text{id}}, \qquad \tilde{e}_{1:T} = W_e e_{1:T} \odot s_{\text{id}}.$$ The paper states that in the implementation $d_e=d_m=d=768$. HIFB then introduces learnable fusion tokens $\mathbf{F}_0 \in \mathbb{R}^{n_f \times d}$ and performs layer-wise updates: $$\tilde{m}_{1:T}^l = \text{Transformer}_m^l([\mathbf{F}^{l-1}; \tilde{m}^{l-1}]), \qquad \tilde{e}_{1:T}^l = \text{Transformer}_e^l([\mathbf{F}^{l-1}; \tilde{e}^{l-1}]).$$ After each branch update, the fusion tokens are refined through cross-attention over all pairwise motion-emotion combinations: $e_{1:T}$0 The dense pair set has cardinality $e_{1:T}$1, so the token update explicitly aggregates temporal cross-modal dependencies rather than relying on a single global style vector.

The stated motivation is that prior style-vector fusion discards fine-grained temporal emotional variation and that traditional fusion methods struggle to achieve temporal alignment between emotion and motion sequences. HIFB therefore performs middle fusion: motion and emotion remain separate streams, co-evolve across $e_{1:T}$2 layers, and exchange information through fusion tokens, while identity is injected earlier through multiplicative modulation. The final output is routed into a VQ-VAE motion latent space and decoded to 53D FLAME parameters per frame, comprising 50 expression coefficients and 3 jaw rotations.

The ablation evidence links HIFB particularly strongly to upper-face dynamics. Under the paper’s fusion strategy comparison, gate fusion reports MVE 1.2662, LVE 1.1315, FDD 0.7671, MEE 1.0281, CE 9.3436, and Diversity 0.4216; cross-attention fusion reports MVE 1.1901, LVE 1.1312, FDD 0.9337, MEE 1.0388, CE 0.9247, and Diversity 0.4458; LSF-Animation with HIFB reports MVE 1.2244, LVE 1.0985, FDD 0.4724, MEE 1.0177, CE 0.9225, and Diversity 0.4223. The paper states that, compared with cross-attention fusion, HIFB reduces FDD by 49.4%, and compared with gate fusion it reduces FDD by 38.4%. It also reports improvements over a “Baseline (gt one-hot)” model of 5.3% in MVE, 13.5% in LVE, 2.5% in FDD, 14.1% in MEE, and 13.7% in CE. Within the paper’s own framing, the main significance of HIFB is better temporal emotion-motion alignment without explicit emotion or identity labels.

4. Strong precedent: hierarchical multimodal sentiment fusion with context propagation

A major antecedent to later HIFB formulations is the hierarchical fusion mechanism in multimodal sentiment analysis and emotion recognition (Majumder et al., 2018). The architecture processes text, audio, and visual utterance features, then applies modality-specific GRUs, projection into a shared latent space, pairwise fusion for all modality pairs, context-aware bimodal GRUs, trimodal fusion over the pairwise outputs, and a final GRU before softmax classification.

The unimodal feature extractors are heterogeneous. Text uses transcripts represented as sequences of pretrained 300-dimensional word2vec vectors, truncated or padded to 50 tokens, followed by a text CNN and a 500-unit fully connected layer with ReLU. Audio uses openSMILE with the IS13-ComParE configuration after voice intensity thresholding and z-standardization, yielding 6392 handcrafted acoustic features per utterance. Visual input uses a 3D-CNN with 32 filters of size $e_{1:T}$3, max-pooling of size $e_{1:T}$4, and a dense layer with 300 neurons. These yield unimodal utterance-level feature vectors $e_{1:T}$5.

Context is then injected at multiple levels. For modality $e_{1:T}$6,

$e_{1:T}$7

with projected “abstract features”

$e_{1:T}$8

The shared projection dimension is reported as best-performing at $e_{1:T}$9. Pairwise fusion is then performed dimension-wise rather than by whole-vector concatenation: $m_{1:T}$0 and analogously for $m_{1:T}$1 and $m_{1:T}$2. These pairwise fused sequences are each passed through a GRU, giving context-aware bimodal representations of dimensionality $m_{1:T}$3. Trimodal fusion is then applied to the pairwise outputs: $m_{1:T}$4 followed by another GRU to produce context-aware trimodal features with $m_{1:T}$5.

This architecture is notable because it already embodies the defining HIFB sequence: shared latent alignment, pairwise interactions first, higher-order composition second, and optional recurrent context propagation. The paper explicitly contrasts this with early fusion by raw concatenation. On utterance-level multimodal sentiment classification without context, hierarchical fusion improves the trimodal $m_{1:T}$6 result from 77.0% with early fusion to 77.9% with HFusion; for $m_{1:T}$7, the result improves from 77.1% to 77.8%; for $m_{1:T}$8, from 77.1% to 77.3%; and for $m_{1:T}$9, from 56.5% to 56.8%. With full context-aware CHFusion, trimodal performance on CMU-MOSI reaches 80.0%, compared with 78.7% for the prior state of the art and 77.2% for Tensor Fusion Network, while on IEMOCAP trimodal accuracy is 76.5% with F-score 76.8%, compared to 74.1% and 73.6% baselines depending on method. The paper states gains of 1–2% over the context-aware concatenation baseline on MOSI and 1–2.4% on IEMOCAP, with the abstract summarizing up to 2.4% absolute improvement and almost 10% error-rate reduction.

Conceptually, this work shows that a module can be “hierarchical interaction fusion” even without attention, bilinear tensors, or explicit gating outside recurrent units. The interaction is local in latent dimension, pair-specific in parameters, and sequence-aware through GRUs. That design remains a strong precedent for later HIFB-style formulations.

5. Cross-scale, coarse-to-fine, and bidirectional variants

In medical image classification, HiFuse introduces an adaptive hierarchical feature fusion block named HFF block, which is described as using local CNN features, global Transformer features, and the previous fused representation at each of four stages with resolutions $z_{\text{id}}$0, $z_{\text{id}}$1, $z_{\text{id}}$2, and $z_{\text{id}}$3 (Huo et al., 2022). The local and global branches remain separate until fusion. At stage $z_{\text{id}}$4, HFF consumes $z_{\text{id}}$5, $z_{\text{id}}$6, and $z_{\text{id}}$7, applies channel attention to the global branch,

$z_{\text{id}}$8

applies spatial attention to the local branch,

$z_{\text{id}}$9

mixes current local, global, and transformed previous fused features by

$m_{\text{final}}=\tilde{m}_{1:T}^L$0

and aggregates them with IRMLP and a shortcut: $m_{\text{final}}=\tilde{m}_{1:T}^L$1 The HFF branch uses stage-wise channel expansions $m_{\text{final}}=\tilde{m}_{1:T}^L$2, $m_{\text{final}}=\tilde{m}_{1:T}^L$3, $m_{\text{final}}=\tilde{m}_{1:T}^L$4, and $m_{\text{final}}=\tilde{m}_{1:T}^L$5, confirming an approximately 4x expansion ratio. The ablation on ISIC2018 reports a progression from Acc 77.12 and F1 54.12 for the local path only, to 79.59 and 64.32 after adding the global block, to 80.85 and 66.26 after adding channel + spatial attention, to 81.32 and 69.53 after adding IRMLP, and to 82.99 and 72.99 for final HiFuse-Tiny.

In single-image deraining, MH2F-Net does not define HIFB explicitly, but the combination of Hierarchical Attentive Distillation Block (HADB) and Residual Projected Feature Fusion (RPFF) performs a closely related function (Chen et al., 2021). The pipeline has three stages: feature extraction, feature distillation, and feature aggregation. Original features are extracted as

$m_{\text{final}}=\tilde{m}_{1:T}^L$6

deep features are produced by a Stacked Hourglass Group of $m_{\text{final}}=\tilde{m}_{1:T}^L$7 MHEBs, with $m_{\text{final}}=\tilde{m}_{1:T}^L$8 by default, and hierarchical outputs from intermediate MHEBs are distilled by

$m_{\text{final}}=\tilde{m}_{1:T}^L$9

RPFF then fuses shallow, deep, and distilled features through discrepancy modeling: $d_e=768$0 The paper reports on Rain200H that with HADB plus concatenation the result is 27.46 / 0.8991, with HADB plus addition 27.87 / 0.9024, and with HADB plus RPFF 29.63 / 0.9225. It also reports that with 8 MHEBs, the model using HADB needs only 70% of the parameters of the one without HADB.

In hierarchical fine-grained classification, the hierarchy moves into the query space (Sahoo et al., 2023). Fine-level queries are not independent; they are fused with projected coarse-level query information: $d_e=768$1 This is paired with multi-scale Fusion Transformer blocks and CAMP, a prior-conditioned cross-attention module for reducing error propagation. The full model improves fine accuracy on GroceryStore from 77.02% for the base scalable query fusion model to 81.33%, while coarse accuracy rises from 86.29 to 88.43. This suggests that HIFB-like behavior can be implemented in hierarchical query refinement rather than in feature tensors alone.

In whole-slide image analysis, HIGT introduces a Bidirectional Interaction block between region-level and patch-level representations (Guo et al., 2023). Top-down interaction reweights region features by SE and injects them into patches: $d_e=768$2 Bottom-up interaction mean-pools patch features and feeds them back to region features: $d_e=768$3 Here hierarchy is explicitly bidirectional rather than pairwise-to-trimodal. The ablation on KICA reports that “Ours w/o BI” gives staging AUC 72.42 ± 2.09 and staging ACC 71.34 ± 7.23, whereas the full model gives 78.80 ± 2.10 and 76.80 ± 2.30.

6. EEG, robotics, missing-modality fusion, and conceptual boundaries

In EEG decoding, HCFT provides a two-level HIFB-equivalent composed of per-stage Convolutional Fusion Transformer Blocks and a final multi-stage Transformer fusion head (Zhang et al., 18 Jan 2026). A temporal branch and a spatiotemporal branch produce aligned tokens $d_e=768$4. Inside each CFT Block, the main branch is first enhanced by self-attention,

$d_e=768$5

then fused through a conditional cross-attention stage and FFN integration: $d_e=768$6 After four hierarchical stages, stage outputs are concatenated and fused globally: $d_e=768$7 Ablation results on Dataset I show that the full model achieves $d_e=768$8, while removing self-attention yields $d_e=768$9, removing cross-attention yields $$s_{\text{id}} = W_{\text{style}} z_{\text{id}}, \qquad \tilde{m}_{1:T} = W_m m_{1:T} \odot s_{\text{id}}, \qquad \tilde{e}_{1:T} = W_e e_{1:T} \odot s_{\text{id}}.$$0, and removing stage concatenation yields $$s_{\text{id}} = W_{\text{style}} z_{\text{id}}, \qquad \tilde{m}_{1:T} = W_m m_{1:T} \odot s_{\text{id}}, \qquad \tilde{e}_{1:T} = W_e e_{1:T} \odot s_{\text{id}}.$$1. The paper explicitly states that the most substantial performance drop occurred when cross-attention was disabled.

In robotic manipulation, hierarchical fusion is explicitly asymmetric because audio is treated as sparse and contact-driven rather than homogeneous with vision and proprioception (Li et al., 14 Feb 2026). The first stage, Binary-Branched Fusion Module, conditions point-cloud and proprioceptive representations on audio: $$s_{\text{id}} = W_{\text{style}} z_{\text{id}}, \qquad \tilde{m}_{1:T} = W_m m_{1:T} \odot s_{\text{id}}, \qquad \tilde{e}_{1:T} = W_e e_{1:T} \odot s_{\text{id}}.$$2 Audio-conditioned geometry then FiLM-modulates audio-conditioned proprioception: $$s_{\text{id}} = W_{\text{style}} z_{\text{id}}, \qquad \tilde{m}_{1:T} = W_m m_{1:T} \odot s_{\text{id}}, \qquad \tilde{e}_{1:T} = W_e e_{1:T} \odot s_{\text{id}}.$$3 The second stage, Interaction Modeling Module, uses three cross-attention branches and an audio highway, finally concatenating the resulting representations: $$s_{\text{id}} = W_{\text{style}} z_{\text{id}}, \qquad \tilde{m}_{1:T} = W_m m_{1:T} \odot s_{\text{id}}, \qquad \tilde{e}_{1:T} = W_e e_{1:T} \odot s_{\text{id}}.$$4 On the pouring task, $$s_{\text{id}} = W_{\text{style}} z_{\text{id}}, \qquad \tilde{m}_{1:T} = W_m m_{1:T} \odot s_{\text{id}}, \qquad \tilde{e}_{1:T} = W_e e_{1:T} \odot s_{\text{id}}.$$5 yields $$s_{\text{id}} = W_{\text{style}} z_{\text{id}}, \qquad \tilde{m}_{1:T} = W_m m_{1:T} \odot s_{\text{id}}, \qquad \tilde{e}_{1:T} = W_e e_{1:T} \odot s_{\text{id}}.$$6, flat $$s_{\text{id}} = W_{\text{style}} z_{\text{id}}, \qquad \tilde{m}_{1:T} = W_m m_{1:T} \odot s_{\text{id}}, \qquad \tilde{e}_{1:T} = W_e e_{1:T} \odot s_{\text{id}}.$$7 yields $$s_{\text{id}} = W_{\text{style}} z_{\text{id}}, \qquad \tilde{m}_{1:T} = W_m m_{1:T} \odot s_{\text{id}}, \qquad \tilde{e}_{1:T} = W_e e_{1:T} \odot s_{\text{id}}.$$8, ManiWAV Fusion yields $$s_{\text{id}} = W_{\text{style}} z_{\text{id}}, \qquad \tilde{m}_{1:T} = W_m m_{1:T} \odot s_{\text{id}}, \qquad \tilde{e}_{1:T} = W_e e_{1:T} \odot s_{\text{id}}.$$9, and the hierarchical method yields $d_e=d_m=d=768$0. The ablation further shows that B-BFM only gives $d_e=d_m=d=768$1, IMM only gives $d_e=d_m=d=768$2, and the full model gives $d_e=d_m=d=768$3.

SFusion clarifies a conceptual boundary for HIFB (Liu et al., 2022). It is a self-attention-based N-to-One multimodal fusion block, but it is explicitly not hierarchical. Available modality features $d_e=d_m=d=768$4 are flattened into tokens, concatenated into

$d_e=d_m=d=768$5

processed by eight self-attention layers,

$d_e=d_m=d=768$6

and turned into modality weights by voxel-level softmax: $d_e=d_m=d=768$7 Its main significance is native support for variable modality subsets without synthesizing or zero-padding missing ones. This suggests that a genuine HIFB is not defined merely by attention over modality tokens; it additionally requires some staged, recursive, or cross-level organization.

A common misconception is therefore to equate HIFB with any multimodal attention block. The surveyed literature indicates otherwise. Some HIFB-like modules are token-based and transformer-heavy; others are lightweight, dimension-wise, or pooling-based. Some are hierarchical in modality order, some in spatial scale, some in temporal context, and some in label ontology. Another recurrent issue is implementation ambiguity: multiple papers report notation inconsistencies or likely typos, including the $d_e=d_m=d=768$8 inconsistency in multimodal sentiment fusion (Majumder et al., 2018), the $d_e=d_m=d=768$9 versus $\mathbf{F}_0 \in \mathbb{R}^{n_f \times d}$0 spatial-attention notation in HiFuse (Huo et al., 2022), the printed cross-attention equations in HCFT (Zhang et al., 18 Jan 2026), and underspecified temporal alignment and token dimensions in LSF-Animation (Lu et al., 23 Oct 2025). For that reason, HIFB is best viewed as a family of hierarchical interaction strategies whose exact realization depends on the target modality structure and task constraints.