IMG: Importance-aware Multi-Granularity Fusion

Updated 4 July 2026

The paper introduces IMG, which fuses signals by learning and weighting importance across multiple granularities such as token, block, or event levels.
It details specialized modules like the Audio Importance Predictor in video retrieval and block-level gating in malicious URL detection to enhance task-specific performance.
Empirical results show significant gains over baselines by dynamically emphasizing informative features while suppressing noisy inputs using multi-granularity fusion.

Importance-aware Multi-Granularity Fusion Model (IMG) denotes a class of architectures that do not fuse heterogeneous signals uniformly, but instead estimate which representations are informative at multiple granularities and then weight, gate, or align them accordingly. In the literature provided here, the term appears both as the explicit name of a model for Video Moment Retrieval (VMR) and as a conceptual description for related architectures in malicious URL detection, open-domain question answering, reading comprehension, scene text recognition, aspect-based sentiment analysis, event-driven forecasting, and panoptic-part segmentation. Across these uses, the recurring idea is that fusion is conditioned on learned importance at more than one representational level—such as token, layer, block, event, sentence, sequence, or modality—rather than relying on flat concatenation or symmetric cross-modal mixing (Lin et al., 6 Aug 2025, Tian et al., 14 Oct 2025).

1. Terminological scope and defining characteristics

The phrase “Importance-aware Multi-Granularity Fusion Model” has a narrow and a broad usage. In "Audio Does Matter: Importance-Aware Multi-Granularity Fusion for Video Moment Retrieval" it is the proper name of a VMR model that selectively aggregates audio, vision, and text (Lin et al., 6 Aug 2025). In "IP-Augmented Multi-Modal Malicious URL Detection Via Token-Contrastive Representation Enhancement and Multi-Granularity Fusion," by contrast, “Importance-aware Multi-Granularity Fusion” is explicitly not the name of a single module; it is the design philosophy jointly realized by the Cross-Layer Multi-Scale Aggregator (CLMSA) and the Blockwise Multi-Modal Coupler (BMMC), with TACL-BERT supplying robust token-level embeddings (Tian et al., 14 Oct 2025).

Context	Referent of IMG	Main granularities
VMR	Named model	local, event, global; audio importance
Malicious URL detection	CLMSA + BMMC design philosophy	token, layer/scale, block/modal
ODQA / FiD	Conceptual mapping	passage, sentence
Reading comprehension	Conceptual mapping	character, word, sentence, paragraph
STR	Decision-level fusion over outputs	character, BPE, WordPiece
Financial forecasting	Conceptual mapping	sequence, token/step
Panoptic-part segmentation	Conceptual mapping	semantic area, instance, part

What unifies these usages is not a fixed operator but a recurrent design principle. Importance-awareness is realized through mechanisms such as a pseudo-label-supervised audio importance predictor, block-level attention weights $\alpha_i$ , listwise passage importance $p_i$ , salience-based anchor selection, Granger-supervised feature-wise gates, or parameter-free confidence balancing. Multi-granularity refers to the coexistence of several levels of representation or structure, for example local/event/global temporal structure in VMR, token/layer/block in URL detection, or semantic/instance/part structure in panoptic-part segmentation (Lin et al., 6 Aug 2025, Tian et al., 14 Oct 2025, Choi et al., 2024, Muralidhara et al., 2023).

A common misconception is to treat IMG as a single canonical architecture. The record here suggests otherwise. The named IMG model in VMR is one particular instantiation, while other papers describe comparable importance-aware multi-granularity fusion mechanisms under different names and with different mathematical forms (Lin et al., 6 Aug 2025, Choi et al., 2024).

2. General architectural pattern

Despite domain differences, the architectures follow a similar decomposition. First, each modality or representational stream is encoded independently into a latent space. Second, one or more modules estimate importance, either explicitly through scores or gates, or implicitly through learned filters and aggregation blocks. Third, fusion is carried out at multiple granularities, often with different operators at each level. Fourth, a task-specific prediction head consumes the fused representation.

In the VMR formulation, textual guidance is integrated with vision and audio separately, after which a pseudo-label-supervised Audio Importance Predictor computes a sample-wise score $p \in [0,1]$ and the model fuses audio and visual features at local-, event-, and global-level (Lin et al., 6 Aug 2025). In CURL-IP, TACL-BERT produces layer-wise token representations $H^{(l)} \in \mathbb{R}^{B\times T\times D}$ for $l=1,\dots,L$ with $L=12$ and $D=768$ , CLMSA aggregates them into $f_{\text{url}} \in \mathbb{R}^{B\times128}$ , an MLP maps IP features to $f_{\text{ip}} \in \mathbb{R}^{B\times128}$ , and BMMC learns block-level attention weights from a shared global context $g$ before classification (Tian et al., 14 Oct 2025).

The importance mechanisms differ materially. In VMR, the gating weight is scalar at the sample level:

$p_i$ 0

Fusion at each granularity then takes forms such as

$p_i$ 1

In CURL-IP, BMMC predicts per-block scores through

$p_i$ 2

rescales them to $p_i$ 3, samples block dropout masks $p_i$ 4, and applies

$p_i$ 5

The paper further emphasizes that BMMC does not compute $p_i$ 6 dot-product attention; it uses learned gating from a shared global context instead (Tian et al., 14 Oct 2025).

This suggests that IMG is best understood as a design schema: importance estimation is inserted before or during fusion, and the granularity axis is treated as a first-class modeling choice rather than a byproduct of encoder depth.

3. IMG as a named model for Video Moment Retrieval

In the VMR setting, IMG addresses the problem of retrieving a segment $p_i$ 7 in an untrimmed video $p_i$ 8 matched by a textual query $p_i$ 9, given synchronized visual frames $p \in [0,1]$ 0 and audio $p \in [0,1]$ 1 (Lin et al., 6 Aug 2025). The model uses pre-trained visual backbones such as I3D, C3D, SlowFast, or InternVideo2; audio backbones such as PANNs or VGGish; and 300d GloVe embeddings for text, with modality encoders following Span-based architectures.

Its first distinctive component is the Audio Importance Predictor. Because ground-truth audio-importance labels are unavailable, IMG constructs pseudo-labels from the unimodal retrieval losses $p \in [0,1]$ 2 and $p \in [0,1]$ 3:

$p \in [0,1]$ 4

After thresholding with $p \in [0,1]$ 5 and $p \in [0,1]$ 6, the predictor is trained with binary cross-entropy,

$p \in [0,1]$ 7

Early training uses a neutral fusion weight of approximately $p \in [0,1]$ 8, then gradually increases reliance on $p \in [0,1]$ 9 (Lin et al., 6 Aug 2025).

The second component is the multi-granularity fusion module. Local-level fusion applies multi-kernel 1D convolutions to capture fine temporal neighborhoods. Event-level fusion uses Slot Attention to extract events and a cross-modal transformer to condition sequences on event slots. Global-level fusion pools the entire video context and reinjects it into the sequence through concatenation and MLPs. A set of bidirectional GRUs over pairs $H^{(l)} \in \mathbb{R}^{B\times T\times D}$ 0 re-establishes inter-level relationships and yields the final fused stream $H^{(l)} \in \mathbb{R}^{B\times T\times D}$ 1 (Lin et al., 6 Aug 2025).

The training objective combines retrieval, importance prediction, knowledge distillation, and saliency:

$H^{(l)} \in \mathbb{R}^{B\times T\times D}$ 2

with $H^{(l)} \in \mathbb{R}^{B\times T\times D}$ 3, $H^{(l)} \in \mathbb{R}^{B\times T\times D}$ 4, and $H^{(l)} \in \mathbb{R}^{B\times T\times D}$ 5. Cross-modal knowledge distillation transfers the fused branch into unimodal branches so that the visual student can be used when audio is missing at inference (Lin et al., 6 Aug 2025).

The paper also introduces Charades-AudioMatter, a test subset of Charades-STA containing 1,196 samples where audio provides complementary or dominant cues. Six annotators independently labeled samples, each instance was double-annotated, disagreements were adjudicated by a third annotator, and inter-annotator agreement exceeded 95% (Lin et al., 6 Aug 2025).

Empirically, IMG significantly outperforms prior audio-incorporated methods such as UMT, PMI-LOC, QD-DETR, and ADPN. On Charades-STA with I3D, IMG improves R1@7 by +4.71 points and mIoU by +2.86 over its visual-only counterpart. On Charades-AudioMatter, R1@7 improves by +6.69 and mIoU by +4.41. The reported compute is modest: 0.38 GFlops and 3.31 M parameters in the I3D setting, with the AIP adding approximately $H^{(l)} \in \mathbb{R}^{B\times T\times D}$ 6 GFlops and the multi-granularity fusion module approximately 0.20 GFlops (Lin et al., 6 Aug 2025).

4. Importance-aware multi-granularity fusion in CURL-IP

CURL-IP addresses malicious URL detection under obfuscation, character-level perturbations, adversarial attacks, and the need to incorporate auxiliary network-level signals such as IP addresses (Tian et al., 14 Oct 2025). Its pipeline is explicitly multi-stage: TACL-BERT produces layer-wise token representations, CLMSA performs cross-layer multi-scale aggregation inside the URL encoder, an IP branch projects external IP embeddings to a 128-dimensional vector, and BMMC performs blockwise multi-modal coupling before binary or multi-class classification.

The token-level component is the Token-Contrastive Representation Enhancer, also described as TCRE/TaCL-BERT. With teacher $H^{(l)} \in \mathbb{R}^{B\times T\times D}$ 7 on clean input $H^{(l)} \in \mathbb{R}^{B\times T\times D}$ 8 and student $H^{(l)} \in \mathbb{R}^{B\times T\times D}$ 9 on masked input $l=1,\dots,L$ 0, the token-aware contrastive objective is

$l=1,\dots,L$ 1

combined with masked language modeling:

$l=1,\dots,L$ 2

The paper states that this yields more discriminative and isotropic subword embeddings and improves sensitivity to character-level obfuscations (Tian et al., 14 Oct 2025).

CLMSA then stacks and permutes all hidden layers,

$l=1,\dots,L$ 3

applies four $l=1,\dots,L$ 4 convolution blocks with channels $l=1,\dots,L$ 5, pools to $l=1,\dots,L$ 6, reshapes to $l=1,\dots,L$ 7, projects with $l=1,\dots,L$ 8, applies a gMLP, and temporally averages to produce $l=1,\dots,L$ 9 (Tian et al., 14 Oct 2025). Although no explicit gating weights are computed per scale, the convolutional hierarchy and gMLP are described as learned filters that emphasize informative multi-layer patterns.

BMMC partitions each modality into channel blocks, computes global summaries $L=12$ 0, aggregates them into a shared context $L=12$ 1, and predicts normalized block importance weights $L=12$ 2. The blocks are then scaled and masked through dropout. This is the explicit importance-aware component: salient regions are up-weighted, noisy ones are down-weighted or dropped (Tian et al., 14 Oct 2025).

The reported datasets are URL-Binary with 802,228 samples, URL-Adversarial with 160,000 samples, and URL-MultiClass with 671,957 samples. On binary detection, the model attains Accuracy 0.9799, Precision 0.9737, Recall 0.9858, F1 0.9797, and AUC 0.9946, with TPR@FPR=0.0001 equal to 0.9263 and TPR@FPR=0.001 equal to 0.9263. On the adversarial dataset, Accuracy is 0.9395, F1 is 0.9393, and AUC is 0.9812; the paper notes that baselines degrade sharply, citing URLBERT at Accuracy 0.6853 and AUC 0.8529. For multi-class detection, CURL-IP reports the highest macro-AUC of 0.9543 and class-wise F1 scores of 95.66% for benign, 83.68% for malicious, and 77.96% for phishing (Tian et al., 14 Oct 2025).

Several papers do not use the label IMG as a formal model name, yet they are explicitly described as conceptually matching an importance-aware multi-granularity fusion approach.

In open-domain question answering, MGFiD extends Fusion-in-Decoder by learning passage-level importance through a listwise re-ranking head, sentence-level importance through a focal-loss classifier, and a max-pooled sentence “anchor vector” injected into the decoder’s $L=12$ 3 query. It also reuses passage importance $L=12$ 4 for pruning, reducing the average number of passages to the decoder to 4.8 on Natural Questions and 7.7 on TriviaQA at $L=12$ 5 (Choi et al., 2024).

In reading comprehension, the hierarchical attention fusion network of "Multi-granularity hierarchical attention fusion networks for reading comprehension and question answering" performs attention and fusion horizontally and vertically across layers. Importance-awareness is realized through soft-alignment weights $L=12$ 6, $L=12$ 7, $L=12$ 8, and $L=12$ 9, and through gated fusion functions such as

$D=768$ 0

The granularities include character/subword via ELMo, word via GloVe and BiLSTMs, sentence-level question aggregation, and paragraph-level passage reasoning (Wang et al., 2018).

In scene text recognition, MGP-STR introduces character, BPE, and WordPiece outputs and fuses them with either a confidence-based fusion strategy or a learnable fusion strategy. The learnable strategy scores candidate sequences by image–text similarity in a CLIP-like contrastive space and selects the highest-scoring granularity; the paper states that MGP-STR achieves an average recognition accuracy of $D=768$ 1 on standard benchmarks (Da et al., 2023).

In aspect-based sentiment analysis, EMGF integrates dependency syntax, constituent syntax, attention semantic features, and external knowledge graphs. Importance enters through semantic-attention-based Top-K anchor selection and through learnable transformations in factorized bilinear pooling; the paper emphasizes a cumulative effect as more granularities are added (Zhao et al., 2024).

In event-driven forecasting, GS-FUSE uses a Granger-supervised, causal-aware gated fusion module and a multi-granularity alignment mechanism spanning instance/sequence-level and token/step-level alignment. The gate is trained against an online incremental-utility signal

$D=768$ 2

so that text is incorporated when it adds predictive value beyond prices (Zhang et al., 27 May 2026).

In panoptic-part segmentation, JPPF fuses semantic areas, object instances, and semantic parts with a parameter-free operator

$D=768$ 3

which the paper describes as dynamically balanced fair fusion across granularities (Muralidhara et al., 2023).

6. Empirical themes, limitations, and interpretive issues

Across the cited work, IMG-style designs are consistently motivated by the claim that uniform fusion treats informative and noisy evidence alike. The VMR paper states that not all audios are helpful and that some videos contain complete noise or background sound that is meaningless to moment determination, hence the need for importance prediction (Lin et al., 6 Aug 2025). The CURL-IP paper makes an analogous claim for URL-IP fusion, arguing that early fusion risks dilution of salient cues and that block-level importance can suppress noisy channels while leveraging complementary IP signals (Tian et al., 14 Oct 2025).

The limitations are correspondingly domain-specific. In VMR, performance depends on pseudo-label quality for the Audio Importance Predictor and on the presence and clarity of audio events relevant to the query; if AIP predicts $D=768$ 4 on audio-dominant samples, performance deteriorates, though the model still outperforms visual-only baselines on average (Lin et al., 6 Aug 2025). In CURL-IP, benefits diminish when IP metadata is missing or noisy, fixed channel block sizes may be suboptimal, stacking all Transformer layers increases memory, and pretraining on Wikipedia may underfit URL-specific distributions (Tian et al., 14 Oct 2025).

A further interpretive issue concerns what counts as “importance-aware.” In some models, importance is an explicit scalar or vector gate, as with $D=768$ 5 in VMR or $D=768$ 6 in BMMC. In others, such as CLMSA or hierarchical attention fusion for reading comprehension, importance is partly implicit in learned filters, soft alignments, or gating operations rather than a single standalone score (Tian et al., 14 Oct 2025, Wang et al., 2018). This suggests that the defining property of IMG is not a particular attention formula but the systematic use of learned relevance estimation across more than one granularity.

Another common misconception is that multi-granularity fusion is equivalent to multimodal fusion. The literature here shows a broader scope. The relevant granularities may be temporal levels within one modality, representational depth within a Transformer, linguistic units such as characters and subwords, evidence levels such as passages and sentences, or nested scene labels such as semantic areas, instances, and parts (Lin et al., 6 Aug 2025, Choi et al., 2024, Muralidhara et al., 2023).

Taken together, these works establish IMG not as a single architecture, but as a research pattern: identify meaningful granularities, estimate importance at those levels, and fuse only after that estimation. The named VMR model and the CURL-IP design philosophy are two explicit formulations, while adjacent work in QA, recognition, forecasting, and structured perception indicates that the same principle recurs whenever heterogeneous evidence is useful but unevenly reliable.