Multi-Granularity Fusion Module

• Multi-Granularity Fusion Module is a design approach that fuses representations at different temporal, spatial, or semantic resolutions to capture both fine details and global context.
• Architectural patterns include branch specialization, level-wise mutual refinement, and stage-wise feature injection to align complementary features effectively.
• Empirical studies demonstrate consistent performance gains across tasks such as motion deblurring, sensing, and text recognition when employing multi-scale fusion techniques.

A Multi-Granularity Fusion Module denotes a class of architectural components that combine representations defined at different temporal, spatial, structural, or semantic resolutions. In current arXiv usage, the term spans event-based motion deblurring, dense affective understanding, Wi-Fi sensing, question answering, cross-city traffic prediction, scene text recognition, molecular representation learning, and other settings. Across these uses, the recurring premise is that fine-grained signals preserve local detail or short-range dynamics, while coarse-grained signals preserve global context or long-range structure, and fusion is used to construct a representation that is simultaneously more informative and more task-aligned (Lin et al., 2024, Yan et al., 2021, Yu et al., 2021, Chen et al., 2023).

1. Domain scope and meanings of “granularity”

The phrase is not attached to a single modality pair or a single backbone family. In event-based deblurring, granularity refers to temporally coarse voxelized events versus temporally fine point-cloud events; in affective video understanding it refers to frame-level, clips-level, and video-level features; in video moment retrieval it refers to local-, event-, and global-level fusion; in skin-lesion segmentation it refers to pixel, patch, and window scales; and in zero-shot Chinese character recognition it refers to stroke-, radical-, and structure-level representations (Lin et al., 2024, Yan et al., 2021, Lin et al., 6 Aug 2025, Liu et al., 4 Apr 2025, Zhu et al., 30 May 2025).

The same phrase also appears in graph and language settings. In cross-city traffic prediction, multi-granularity is realized by hierarchical node clustering over fused graphs; in aspect-based sentiment analysis it refers to dependency, constituency, attention-semantic, and external knowledge-graph views; in open-domain question answering it is instantiated as passage re-ranking at coarse granularity and sentence classification at fine granularity; and in scene text recognition it appears at the output level, where character, BPE, and WordPiece predictions are fused into a final hypothesis (Chen et al., 2023, Zhao et al., 2024, Choi et al., 2024, Da et al., 2023).

Domain	Granularities	Representative fusion form
Event-based deblurring (Lin et al., 2024)	voxel-based and point-cloud events	AMM + AFDM + decoder fusion
Dense affective understanding (Yan et al., 2021)	frame-, clip-, video-level	modal attention before MOE
Wi-Fi sensing (Yu et al., 2021)	CSI and beam SNR at matched levels	all paired maps with learned weights
Video moment retrieval (Lin et al., 6 Aug 2025)	local, event, global	importance-aware weighted fusion
Scene text recognition (Da et al., 2023)	char, BPE, WordPiece	confidence-based or learnable selection

A plausible implication is that “granularity” in this literature is best understood as a relation between representational resolutions rather than as a synonym for image scale alone.

2. Architectural patterns

A recurring pattern is branch specialization before fusion. MTGNet uses two parallel branches that consume the same input interval $[t_0,t_n]$: a coarse temporal fusion branch built on voxel-binned events and multi-scale blurry images, and a fine-grained point branch built on point-cloud events that preserve microsecond-level timing. After branch-specific processing, the coarse fusion feature $F_{\mathcal C}$ and the diffused point feature $F_{\mathbf D}$ are fused and sent to a shared decoder (Lin et al., 2024).

Other systems implement branch specialization across modalities or hierarchy levels rather than across time. MGN-MA constructs five sub-features—image, audio, action, subtitle, and video—and learns a per-modal weighting before a 3-expert MOE classifier. The Wi-Fi sensing model pairs selected CSI and beam-SNR feature maps from two encoders, projects each pair into a fixed latent dimension, and aggregates them into a single fused vector. IMG first applies text-guided fusion to vision and audio separately, then passes the resulting sequences through local-, event-, and global-level fusion branches conditioned by an audio-importance predictor (Yan et al., 2021, Yu et al., 2021, Lin et al., 6 Aug 2025).

A second pattern is level-wise mutual refinement. In Mg-MRN, each granularity level $g_l$ is refined by adjacent coarser levels through a spatial-channel attention block, yielding enhanced features $u_l$ that are then consumed in parallel by the downstream Visual-Semantic Decoder. In Hi-GITA, the image-side fusion module refines stroke features with upsampled radical features and then refines radicals by down-sampling the stroke-refined map, while the text-side fusion module alternates symmetrical cross-attention and global self-attention between stroke and radical token sequences (Wang et al., 11 Nov 2025, Zhu et al., 30 May 2025).

A third pattern is stage-wise injection into a pyramid backbone. MGHFT injects view-specific text into each of four PVT stages at both local and global granularity, then applies a final Text-Guided Fusion Attention over concatenated fused features. VFFM-UNet places its Multi-Granularity Fusion inside a VFFM block, where three Vision Fastformer modules operate at pixel, patch, and window scales before spatial-granularity and channel-granularity fusion (Chen et al., 25 Jul 2025, Liu et al., 4 Apr 2025).

3. Fusion operators and alignment mechanisms

The operator itself varies substantially. A canonical attention-weighted formulation appears in MGN-MA. Given modality features $v_i$, it scores each modality by

$$e_i = w_i^\top v_i,\qquad \alpha_i = \frac{\exp(e_i)}{\sum_{j=1}^N \exp(e_j)},$$

then forms the fused descriptor by re-weighting and concatenation,

$$\widetilde v_i = \alpha_i v_i,\qquad V = [\widetilde v_1;\dots;\widetilde v_N].$$

This is a single-head scalar attention over modalities rather than token-level cross-attention (Yan et al., 2021).

MTGNet uses a different sequence of operators. In the coarse branch, image and voxel features are fused by

$F_{\mathcal C}=F_{\mathcal I}+F_{\mathcal V}.$

In the fine branch, the Aggregation and Mapping Module groups points in time bins, encodes them, temporally fuses them with an LSTM, rescales centroid coordinates by a down-sampling factor $\beta$, and scatters the resulting features onto a sparse 2D map. The Adaptive Feature Diffusion Module then predicts a diffusion range from globally pooled coarse features and diffuses each sparse point feature with Gaussian weights,

$F_{\mathcal C}$0

yielding a dense feature volume $F_{\mathcal C}$1 before final fusion (Lin et al., 2024).

In reading comprehension, fusion is explicitly residual and heuristic-match driven. The module computes

$F_{\mathcal C}$2

and then mixes it with the original representation through a gate,

$F_{\mathcal C}$3

This formulation makes global features persist even after aggressive attention-based interaction (Wang et al., 2018).

Other papers use algebraically lighter or more factorized fusion. The Wi-Fi sensing model linearly projects every matched CSI/beam-SNR pair into $F_{\mathcal C}$4 and then sums them with learned scalar weights, $F_{\mathcal C}$5. EMGF uses factorized bilinear pooling over projected dependency, constituency, semantic, and knowledge-graph features, followed by normalization. By contrast, Hi-GITA’s image-side module uses only up-sampling, down-sampling, and element-wise addition; no extra gating or attention is used on that side (Yu et al., 2021, Zhao et al., 2024, Zhu et al., 30 May 2025).

4. Supervision strategies

Multi-granularity fusion is rarely supervised by a single task loss alone. MTGNet minimizes a composite objective consisting of an $F_{\mathcal C}$6 reconstruction term, $F_{\mathcal C}$7, and $F_{\mathcal C}$8, with $F_{\mathcal C}$9. MGFiD jointly optimizes answer generation, passage re-ranking, and sentence classification through

$F_{\mathbf D}$0

with $F_{\mathbf D}$1 and $F_{\mathbf D}$2 (Lin et al., 2024, Choi et al., 2024).

A frequent design is to add explicit alignment losses. MGHFT combines classification with stage-wise alignment, where each stage uses an InfoNCE-style contrastive loss and a multi-level cross-entropy loss, summed as

$F_{\mathbf D}$3

MolFusion similarly combines molecular-level and atomic-level objectives,

$F_{\mathbf D}$4

where MolSim matches predicted cross-modal similarities to continuous Tanimoto targets and AtomAlign performs masked-token and masked-indicator prediction (Chen et al., 25 Jul 2025, Cai et al., 2024).

Other modules rely on preprocessing or auxiliary regularization to make fusion meaningful. EMGF uses multi-anchor triplet learning to bring homologous dependency and constituency nodes closer, then applies orthogonal projection to disentangle syntactic features from semantic features before factorized fusion. SFMGTL adds graph reconstruction losses, DiffPool auxiliary losses, prediction losses at every level, and domain-adversarial losses. By contrast, Mg-MRN states that there is no separate fusion-only loss in its Mutual Refinement Module; the fused features are shaped indirectly through semantic cross-entropy and attribute regression in the downstream decoder (Zhao et al., 2024, Chen et al., 2023, Wang et al., 11 Nov 2025).

This suggests that the module boundary is often architectural rather than optimization-isolated: the fusion block is typically coupled to auxiliary objectives that enforce compatibility across granularities.

5. Empirical behavior

Reported ablations consistently treat multi-granularity fusion as an incremental mechanism whose contribution can be isolated against simpler fusion baselines, modality removal, or single-level variants.

Paper/task	Compared settings	Reported effect
MGN-MA (Yan et al., 2021)	MLP 0.01529; MOE 0.01718; MAF+MOE 0.01849	+0.00131 average correlation over MOE
IMG (Lin et al., 6 Aug 2025)	vision-only R1@3 72.37; full local+event+global 75.18, mIoU 55.62	+2.81 R1@3 over vision-only
MGFiD (Choi et al., 2024)	FiD-KD 48.4 EM; MGFiD 50.1 EM; pruning 49.7 EM	+1.7 EM on NQ; avg passages reduced to 4.8
VFFM-UNet (Liu et al., 4 Apr 2025)	baseline mIoU 76.71; full VFFM 79.00	+2.29 mIoU on ISIC-2017
MGP-STR (Da et al., 2023)	char-only 92.73; CFS 93.35; LFS 94.05	+1.32 average accuracy over vision-only

Additional studies reinforce the same pattern. MGFF-TDNN reports EER $F_{\mathbf D}$5 and minDCF $F_{\mathbf D}$6 on VoxCeleb1-O, outperforming ECAPA-TDNN and Res2Net, while ablations that remove DSM, PLP, or TDNN branching all degrade EER. MGML-FENet reports that adding only MGML-FFB yields about $F_{\mathbf D}$7–$F_{\mathbf D}$8 gain, adding only MGML-FEM yields about $F_{\mathbf D}$9–$g_l$0 gain, and the full ensemble yields up to $g_l$1 on AID and $g_l$2 on NWPU-RESISC45. SFMGTL reports that removing hierarchical node clustering raises RMSE from $g_l$3 to $g_l$4 and MAE from $g_l$5 to $g_l$6 in the NY$g_l$7DC taxi setting (Li et al., 6 May 2025, Zhao et al., 2020, Chen et al., 2023).

The reported gains are therefore heterogeneous in scale but consistent in direction. Some modules mainly improve robustness to noisy or missing signals, as in IMG’s audio-importance weighting and cross-modal knowledge distillation. Others improve efficiency simultaneously with accuracy, as in MGFiD’s pruning strategy and VFFM-UNet’s comparison against MISSFormer (Lin et al., 6 Aug 2025, Choi et al., 2024, Liu et al., 4 Apr 2025).

6. Conceptual boundaries and common misunderstandings

One common misunderstanding is to equate multi-granularity fusion with attention alone. The literature does not support that restriction. Some modules use attention over modalities or tokens, but others use learned scalar mixtures, residual addition, factorized bilinear pooling, channel gating, Gaussian diffusion, or even branch selection by confidence or image-text cosine similarity (Yan et al., 2021, Lin et al., 2024, Zhao et al., 2024, Da et al., 2023).

A second misunderstanding is that multi-granularity fusion must be multimodal. Several models are multimodal, but others fuse representations inside one modality or inside one prediction space. VFFM-UNet fuses pixel-, patch-, and window-scale visual features; MGML-FENet fuses multi-level backbone features and aggregates multiple prediction heads; MGP-STR fuses character-, BPE-, and WordPiece-level outputs rather than raw modalities (Liu et al., 4 Apr 2025, Zhao et al., 2020, Da et al., 2023).

A third misunderstanding is that “mutual refinement” implies iterative message passing until convergence. Mg-MRN explicitly states that there is no explicit multi-iteration loop in the published model: each granularity is refined exactly once by the SCAB. Similarly, SFMGTL states that all levels’ predictions are fused only in the loss, not by concatenation, and MGP-STR’s Confidence-Based Fusion has no trainable weights and is used only at inference (Wang et al., 11 Nov 2025, Chen et al., 2023, Da et al., 2023).

Finally, multi-granularity fusion does not imply equal treatment of all branches. MGN-MA adds modal dropout so that the network does not overly rely on any single modality, while IMG predicts an audio-importance score $g_l$8 and anneals $g_l$9 from $u_l$0 to the predicted value during training. These designs indicate that the module often serves not merely to combine branches, but to regulate branch reliability (Yan et al., 2021, Lin et al., 6 Aug 2025).

Taken together, the literature presents the Multi-Granularity Fusion Module not as a single canonical block, but as a reusable design principle: identify complementary resolutions, align them into a compatible representational space, fuse them with an operator matched to the task, and supervise the result with objectives that preserve both cross-granularity consistency and task discriminability.