Multi-Granularity Adaptive Attention

• Multi-Granularity Adaptive Attention (MGAA) is an architectural design pattern that constructs parallel representations at varying scales to capture both local details and global context.
• It adapts the fusion of these granularities using mechanisms like gating, masking, and learned weights, enabling dynamic selection of the most pertinent features.
• MGAA has demonstrated practical benefits across applications such as vision-language models, audio deepfake detection, and financial modeling, enhancing accuracy and robustness.

Multi-Granularity Adaptive Attention (MGAA) denotes a class of attention mechanisms that model information at more than one granularity and adapt how those granularities are combined, selected, or made to interact. Across the provided literature, the term does not refer to a single canonical module. Instead, it appears as a recurrent architectural principle: a model maintains representations at distinct semantic, spatial, temporal, or time-frequency scales, and then uses conditioning signals, masking, gating, or learned fusion weights to determine which scale should dominate for a given input or task. Representative instances include text-conditioned visual granularity control in multimodal LLMs (Mao et al., 9 Mar 2026), global-local branch fusion over time-frequency maps for audio deepfake detection (Shi et al., 2 Aug 2025), masked interaction across word, line, paragraph, and page queries for unified text detection (Wan et al., 2024), bidirectional interaction across phoneme, word, and utterance levels for pronunciation assessment (Han et al., 5 Jan 2026), and volatility-conditioned temporal-resolution fusion for order-flow modeling (Hu, 20 Mar 2026). In cross-domain recommendation, the abstract of MIMNet introduces a “multi-granularity target-guided attention network” using fine-grained and coarse-grained target signals, although the supplied document does not contain the substantive module description (Zhu et al., 2024).

1. Conceptual definition and scope

MGAA is motivated by a common failure mode of single-scale representations: fixed global abstractions can discard local evidence, whereas purely fine-grained encodings can preserve detail without providing task-relevant compression or higher-level structure. This tension is stated explicitly in Granulon, which contrasts CLIP-based encoders that emphasize global semantic alignment with DINOv3 encoders that preserve strong pixel-level perception but lack coarse-grained semantic abstraction (Mao et al., 9 Mar 2026). A closely related argument appears in audio deepfake detection, where communication degradations alter time-frequency structure in ways that cannot be handled well by a single receptive field, because some degradations are global while others are local and intermittent (Shi et al., 2 Aug 2025).

Within the supplied papers, “granularity” is domain-specific. In vision-language modeling, it ranges from pixel tokens to fine and coarse semantic tokens (Mao et al., 9 Mar 2026). In unified text detection, it refers to word, line, paragraph, and page levels (Wan et al., 2024). In pronunciation assessment, it denotes phoneme, word, and utterance levels (Han et al., 5 Jan 2026). In traffic forecasting and high-frequency finance, it corresponds to temporal resolutions such as original, hourly, daily, weekly, or fine tick-level versus coarse low-frequency dynamics (Zhang et al., 2021, Hu, 20 Mar 2026). In audio deepfake detection, it refers to multiple time-frequency receptive fields together with a global branch (Shi et al., 2 Aug 2025).

This suggests that MGAA is better understood as a design pattern rather than a fixed layer type. The common property is not a shared formula, but a shared objective: preserving useful fine structure while enabling adaptive abstraction or cross-level interaction.

2. Core architectural pattern

Across the corpus, MGAA-style systems repeatedly instantiate three components.

First, they construct parallel or structured representations at different granularities. Granulon begins from dense DINOv3 image tokens and then creates semantic tokens at different abstraction levels (Mao et al., 9 Mar 2026). The audio framework builds one global Time-Frequency Attention branch and several Local Time-Frequency Attention branches indexed by kernel sizes $k_i$ (Shi et al., 2 Aug 2025). DAT organizes learnable object queries into four granularity groups,

$$Q=\{(Q_{k}^{word}, Q_{k}^{line}, Q_{k}^{para}, Q_{k}^{page})\}_{k=1}^{N_q},$$

so that each level can learn its own semantics and geometry priors (Wan et al., 2024). HIA likewise constructs granularity-specific states for phoneme, word, and utterance levels (Han et al., 5 Jan 2026).

Second, they introduce an adaptive or constrained interaction mechanism. In Granulon, a text-conditioned Controller predicts the suitable abstraction level from the question and outputs a distribution over granularity candidates $g_k=(\alpha_k,\beta_k,\gamma_k)$ (Mao et al., 9 Mar 2026). In the audio MGAA, an Adaptive Fusion Module computes a sample-dependent saliency-driven weight distribution over global and local attention branches (Shi et al., 2 Aug 2025). DAT uses masked global self-attention over concatenated query groups, with the binary attention mask $A$ controlling which granularities may interact (Wan et al., 2024). The financial AGA module converts local volatility and transaction intensity into a gate $g_t$ that interpolates between fine and coarse temporal encoders (Hu, 20 Mar 2026).

Third, they either fuse the granularity-specific outputs or preserve them jointly for downstream reasoning. Granulon concatenates semantic tokens with the original pixel tokens before the multimodal projector (Mao et al., 9 Mar 2026). The audio detector computes

$$\textstyle MGAA_{out}(\xi) = \sum_{i=0}^{n} \mathcal{W}^{(i)}(\xi) \cdot \mathcal{A}^{(i)}(\xi),$$

with $\mathcal{A}^{(0)}$ denoting the global branch (Shi et al., 2 Aug 2025). GACAN concatenates the outputs from original, hourly, daily, and weekly temporal-attention streams and fuses them with a fully connected layer after each graph attention layer rather than only at the end (Zhang et al., 2021).

A concise summary is given below.

Domain	Granularity units	Adaptive mechanism
Vision-language	pixel, fine, coarse semantic tokens	text-conditioned Controller + AdaTA (Mao et al., 9 Mar 2026)
Audio deepfake detection	global TF branch + multi-scale local TF branches	Adaptive Fusion Module (Shi et al., 2 Aug 2025)
Text detection	word, line, paragraph, page queries	masked across-granularity interactive attention (Wan et al., 2024)
Pronunciation assessment	phoneme, word, utterance	bidirectional Interactive Attention Module (Han et al., 5 Jan 2026)
Traffic forecasting	original, hourly, daily, weekly series	repeated fusion after each attention layer (Zhang et al., 2021)
Order-flow modeling	fine tick-level, coarse low-frequency dynamics	volatility/intensity-conditioned gate (Hu, 20 Mar 2026)

3. Mechanisms of adaptation

The most explicit adaptive formulation in the provided material is Granulon’s text-conditioned granularity Controller. The Controller aggregates the textual query after the first block of the LLM and maps it into a granularity space $\pi_\theta$: $$\mathbf{h} = \mathbf{W}_{p}\,\sigma\big(\tfrac{1}{L}\sum_{i=1}^{L}\mathbf{E}^{(1)}_{i}\big),$$

$p(g\mid\mathcal{T})=\mathrm{softmax}(\Phi_{\text{MLP}(\mathbf{h})), \qquad g^{*}=\arg\max_{g_k\in \bar g} p(g_k\mid \mathcal{T}).$

Here, $$Q=\{(Q_{k}^{word}, Q_{k}^{line}, Q_{k}^{para}, Q_{k}^{page})\}_{k=1}^{N_q},$$0 controls spatial down-sampling, $$Q=\{(Q_{k}^{word}, Q_{k}^{line}, Q_{k}^{para}, Q_{k}^{page})\}_{k=1}^{N_q},$$1 controls cluster cardinality, and $$Q=\{(Q_{k}^{word}, Q_{k}^{line}, Q_{k}^{para}, Q_{k}^{page})\}_{k=1}^{N_q},$$2 controls projector weights (Mao et al., 9 Mar 2026). The paper interprets this as mapping linguistic scope to perceptual scope: broad questions push the system toward coarse abstraction, while narrow attribute questions trigger fine-grained processing.

The audio MGAA expresses adaptation through branch weighting rather than explicit granularity classification. Its branch weights are computed as

$$Q=\{(Q_{k}^{word}, Q_{k}^{line}, Q_{k}^{para}, Q_{k}^{page})\}_{k=1}^{N_q},$$3

followed by softmax-normalized fusion across the global and local branches (Shi et al., 2 Aug 2025). The paper states that this allows the system to dynamically reallocate its focus according to the characteristics of the degradation.

In DAT, adaptation is structural rather than fully free-form. All query groups are concatenated into

$$Q=\{(Q_{k}^{word}, Q_{k}^{line}, Q_{k}^{para}, Q_{k}^{page})\}_{k=1}^{N_q},$$4

and global self-attention is then applied under a binary attention mask

$$Q=\{(Q_{k}^{word}, Q_{k}^{line}, Q_{k}^{para}, Q_{k}^{page})\}_{k=1}^{N_q},$$5

The interaction factor $$Q=\{(Q_{k}^{word}, Q_{k}^{line}, Q_{k}^{para}, Q_{k}^{page})\}_{k=1}^{N_q},$$6 determines whether only adjacent granularities interact, whether broader skip-level interactions are allowed, or whether the system becomes fully connected (Wan et al., 2024). The mechanism is therefore adaptive in the sense of selective connectivity rather than per-sample soft weighting.

The financial AGA formulation is still more explicit about state dependence. It derives local volatility and order-arrival intensity from a sliding window,

$$Q=\{(Q_{k}^{word}, Q_{k}^{line}, Q_{k}^{para}, Q_{k}^{page})\}_{k=1}^{N_q},$$7

and then computes a feature-wise gate

$$Q=\{(Q_{k}^{word}, Q_{k}^{line}, Q_{k}^{para}, Q_{k}^{page})\}_{k=1}^{N_q},$$8

which interpolates between fine and coarse encoder outputs: $$Q=\{(Q_{k}^{word}, Q_{k}^{line}, Q_{k}^{para}, Q_{k}^{page})\}_{k=1}^{N_q},$$9 The supplied description states that high volatility and high trading intensity shift emphasis toward fine-grained features, whereas stable conditions favor coarse features (Hu, 20 Mar 2026).

4. Representative instantiations

Granulon provides a prototypical MGAA formulation for multimodal reasoning. After the Controller selects a granularity, the Adaptive Token Aggregation module performs granularity-guided pooling, relation-aware clustering, and feature refinement. Pooling jointly downsamples both the feature map and attention field according to $g_k=(\alpha_k,\beta_k,\gamma_k)$0, clustering uses the pooled features and pooled attention to construct representative visual prototypes with cluster count controlled by $g_k=(\alpha_k,\beta_k,\gamma_k)$1, and a refinement stage scores clusters by support size, coherence, and dispersion before keeping the top-$g_k=(\alpha_k,\beta_k,\gamma_k)$2 semantic tokens (Mao et al., 9 Mar 2026). The resulting design enables what the paper calls unified “pixel-to-fine-to-coarse” reasoning within a single forward pass.

The audio deepfake detector instantiates MGAA as a time-frequency attention block composed of Global Time-Frequency Attention, Local Time-Frequency Attention, and an Adaptive Fusion Module (Shi et al., 2 Aug 2025). The global branch computes coarse long-range gating over the full time-frequency tensor, while each local branch uses depth-wise convolutions with kernel sizes $g_k=(\alpha_k,\beta_k,\gamma_k)$3 in both frequency and time directions: $g_k=(\alpha_k,\beta_k,\gamma_k)$4 The paper ties $g_k=(\alpha_k,\beta_k,\gamma_k)$5 to increasingly longer acoustic durations, from approximately 95 ms to approximately 286 ms (Shi et al., 2 Aug 2025). This formulation makes “multi-granularity” synonymous with multiple time-frequency receptive fields plus global context.

DAT replaces unrestricted self-attention with hierarchy-aware interaction among structured detection queries (Wan et al., 2024). The forward detection branch is written as

$g_k=(\alpha_k,\beta_k,\gamma_k)$6

and the central innovation is the across-granularity interactive attention module. Word, line, paragraph, and page queries first undergo group-wise self-attention separately; they are then globally concatenated and passed through masked self-attention. The paper explicitly distinguishes this from ordinary self-attention, emphasizing that unrestricted global interaction introduces noise and harms granularity-specific learning (Wan et al., 2024).

HIA adapts the same broad principle to pronunciation assessment, but the emphasis shifts from masking to bidirectional exchange across linguistic units (Han et al., 5 Jan 2026). Granularity queries are stacked as

$g_k=(\alpha_k,\beta_k,\gamma_k)$7

processed by self-attention,

$g_k=(\alpha_k,\beta_k,\gamma_k)$8

and then grounded in acoustic embeddings through cross-attention,

$g_k=(\alpha_k,\beta_k,\gamma_k)$9

The outputs $A$0, $A$1, and $A$2 are then injected into residual hierarchical scoring branches. Here, MGAA functions as a mechanism for jointly modeling phoneme, word, and utterance correlations rather than selecting among receptive fields.

GACAN presents a more indirect but still MGAA-style design. It does not use the term MGAA, yet it processes original, hourly, daily, and weekly time series with separate attention modules and fuses them after each graph attention layer: $A$3 The paper stresses that, unlike ASTGCN, the multiple granularities are not handled independently until the output stage; instead, their effects are integrated throughout the model (Zhang et al., 2021).

5. Empirical behavior and ablation evidence

The strongest cross-paper regularity is that gains arise from structured or adaptive multi-granularity interaction rather than from simply adding more attention. DAT states this directly through its interaction-factor ablation: the best overall configuration is $A$4, while fully connected interaction $A$5 performs poorly, indicating that unrestricted cross-granularity attention is too noisy (Wan et al., 2024). The interpretation supplied with the paper is that the gain comes from carefully structured, masked, multi-granularity interaction rather than from “more attention.”

Granulon reports large end-task gains from adaptive granularity. The abstract states that it improves accuracy by approximately $A$6 and reduces hallucination by approximately $A$7, and the detailed summary reports that under identical LLaVA settings it outperforms CLIP, SigLIP, DINOv2, and DINOv3 with both Qwen2.5 and Llama3.2 backbones (Mao et al., 9 Mar 2026). The same source also reports that the fixed-token version performs worse than the adaptive version, even though adaptive selection increases token usage by around $A$8, which the paper interprets as evidence that the gain comes from intelligent granularity choice rather than token count (Mao et al., 9 Mar 2026).

The audio MGAA likewise attributes performance gains to adaptive branch weighting rather than to mere multibranch redundancy. The paper reports that replacing the Adaptive Fusion Module with fixed equal-weight fusion leads to a significant performance drop, and that removing MGAA entirely increases EER across all time-frequency representations (Shi et al., 2 Aug 2025). It further states that the best-performing local-scale set is $A$9, while including $g_t$0 hurts performance (Shi et al., 2 Aug 2025). The t-SNE analysis described in the supplied text additionally reports that MGAA-enhanced features become more compact and class-separated under communication degradation (Shi et al., 2 Aug 2025).

In traffic forecasting, the ablation signal is cumulative rather than branch-adaptive. The text states that performance improves consistently as more time granularities are introduced, with the most significant improvement occurring when the daily time series is added (Zhang et al., 2021). This is evidence for the value of multi-granularity integration, although not necessarily for per-sample adaptive weighting in the stricter sense.

In financial modeling, the supplied description reports that AGA-Neural HMM reaches $g_t$1 accuracy on NASDAQ versus $g_t$2 for the best baseline TCN-MultiRes, with a $g_t$3 percentage-point gain, and that removing AGA drops accuracy by $g_t$4 (Hu, 20 Mar 2026). It also reports a $g_t$5 accuracy boost over static approaches in volatile regimes, which is consistent with the paper’s claim that state-dependent granularity selection is especially useful when market conditions change rapidly (Hu, 20 Mar 2026).

HIA reports that the full interactive attention setup improves nearly all metrics on speechocean762, with especially notable gain on word stress, and that removing residual connections or hierarchical score passing reduces performance (Han et al., 5 Jan 2026). This suggests that in hierarchical sequence tasks, MGAA-style interaction can be effective when paired with residual preservation of lower-level evidence.

6. Relation to neighboring concepts, limitations, and terminology

A recurrent misconception is to equate MGAA with standard self-attention or static multi-scale feature fusion. The supplied DAT summary explicitly rejects that equivalence, stating that ordinary self-attention lets every token attend to every other token without explicit granularity structure, whereas DAT organizes tokens by granularity and constrains interaction with masking (Wan et al., 2024). Granulon makes a related distinction by arguing that adaptive multi-granularity semantics are not the same as static multi-scale features, because the abstraction level is conditioned on the question rather than fixed in advance (Mao et al., 9 Mar 2026).

Another limitation is terminological instability. Some papers use “Multi-Granularity Adaptive Attention” directly (Shi et al., 2 Aug 2025), whereas others describe closely related mechanisms with different names: “across-granularity interactive attention” in DAT (Wan et al., 2024), “Interactive Attention Module” in HIA (Han et al., 5 Jan 2026), and “Adaptive Granularity Attention” in order-flow modeling (Hu, 20 Mar 2026). This suggests that the research area is methodologically coherent but not yet standardized lexically.

The supplied material also imposes a boundary on what can be said about recommendation-system variants. The abstract of MIMNet states that the model introduces both fine-grained and coarse-grained target signals and aggregates transformed interest-level representations by incorporating a novel multi-granularity target-guided attention network (Zhu et al., 2024). However, the accompanying details state that the provided paper text is an empty LaTeX skeleton with no title, abstract, body text, equations, architecture, or experiments for that module. Accordingly, the existence of a multi-granularity target-guided attention mechanism in MIMNet can be noted from the abstract, but its mathematical formulation, workflow, or empirical ablations cannot be described reliably from the supplied document (Zhu et al., 2024).

Taken together, the available literature supports a restrained but technically precise conclusion. MGAA is a family of mechanisms for balancing fine and coarse evidence under task- or sample-dependent control. Its concrete implementations differ substantially across modalities, but the recurring themes are parallel granularity-specific representations, learned interaction or weighting, and an explicit attempt to avoid the rigidity of single-scale modeling.