Granularity-Aware Representation Module

• Granularity-aware representation modules are components that encode, align, and fuse multi-scale features, effectively balancing fine and coarse details.
• Canonical designs include alignment–unification frameworks, hierarchical graph models, cross-granularity fusion blocks, and adaptive granularity controllers.
• Empirical results across tasks demonstrate that matching representation scales to evidence levels improves performance in diverse multimodal applications.

Searching arXiv for recent and foundational papers on granularity-aware representation modules across modalities. arXiv search query: "granularity-aware representation module multimodal granularity-aware alignment representation"

A granularity-aware representation module is a model component that explicitly encodes, aligns, or fuses information across multiple semantic, spatial, temporal, or structural scales. Across recent work, the term has come to denote a family of designs rather than a single architecture: in text-to-image person re-identification it addresses the mismatch between fine-grained visual features and coarse textual descriptions; in retrieval-augmented generation it mediates between summaries, sentences, entities, and passages; in medical vision-language pre-training it couples instance-, sentence-, patch-, and word–patch-level cues; and in multimodal LLMs it dynamically moves between pixel-level, fine-grained, and coarse-grained semantics (Shao et al., 2022, Zhan et al., 16 Jun 2026, Wang et al., 10 Sep 2025, Mao et al., 9 Mar 2026).

1. Conceptual foundations

In the cited literature, granularity is defined relative to the structure of the input and the task. In fine-grained visual classification, “small granularity” corresponds to earlier CNN stages with small receptive fields that capture local parts, whereas “larger granularity” corresponds to deeper stages with larger receptive fields that capture bigger regions or the whole object (Song et al., 2021). In salient object detection, “low granularity” denotes coarse object semantics and rough locations, while “high granularity” denotes fine-grained boundaries and local details; the former is associated with high-level features and the latter with low-level features (Wu et al., 11 Aug 2025). These formulations make granularity a property of both representation scale and semantic abstraction.

Cross-modal work generalizes the same idea. SimCroP defines granularity over instance-level CT volume representations, sentence-level report units, patch-level subregions, and word–patch interactions (Wang et al., 10 Sep 2025). FlowRAG organizes retrieval over passages, summaries, sentences, and entities, with summary nodes acting as coarse semantic hubs and sentence/entity nodes providing fine evidence (Zhan et al., 16 Jun 2026). GranAlign makes the precision–coverage trade-off explicit: coarse queries provide high coverage but low precision, whereas fine queries provide high precision but low coverage (Jeon et al., 2 Jan 2026). Taken together, these formulations suggest that a granularity-aware module is not merely a multi-scale feature extractor; it is a mechanism that decides which scale is structurally appropriate, how scales should interact, and how supervision should be matched to those scales.

2. Canonical architectural forms

One canonical form is the explicit alignment–unification decomposition. In “Learning Granularity-Unified Representations for Text-to-Image Person Re-identification” (Shao et al., 2022), the LGUR framework is described as an end-to-end framework based on transformers with two modules: a Dictionary-based Granularity Alignment module and a Prototype-based Granularity Unification module. The first uses a Multi-modality Shared Dictionary to reconstruct both visual and textual features, with cross-modality guidance and foreground-centric reconstruction; the second uses shared and learnable prototypes as queries to extract diverse and semantically aligned features in a granularity-unified feature space. In this formulation, the module exists to reduce the inter-modal gap caused by the fact that visual features are usually fine-grained while textual features are coarse.

A second form is the explicit hierarchical graph. FlowRAG constructs a quad-level heterogeneous graph

$$G = (V, E), \quad V = V_P \cup V_{Sum} \cup V_S \cup V_E,$$

where passages, summaries, sentences, and entities constitute distinct node types (Zhan et al., 16 Jun 2026). Granularity is represented directly in the graph schema rather than only in feature dimensions. Query relevance is then injected through both summary–query alignment and sentence-level matching, so the representation module is simultaneously a hierarchy constructor and a relevance propagation mechanism.

A third form is the cross-granularity fusion block. SimCroP separates a global instance-level visual stream,

$f^I = \mathrm{GAP}(f_v),$

from a fine-grained word–patch stream,

$f^{W} = \mathrm{CrossAttention}(f_t, f_v),$

and reconstructs masked report tokens from their sum,

$\hat{T}_m = D_t(f^I + f^{W}),$

thereby forcing the representation to remain useful at both holistic and local scales (Wang et al., 10 Sep 2025). Similar dual-granularity or multi-granularity decompositions appear in vectorized HD map construction, where MGMapNet uses coarse-grained instance-level queries and fine-grained point-level queries generated from multi-scale BEV features, followed by a Point Instance Interaction module (Yang et al., 2024).

A fourth form is adaptive granularity control. Granulon introduces a text-conditioned granularity Controller that predicts parameters governing spatial down-sampling, cluster cardinality, and multimodal mixing, and an Adaptive Token Aggregation module that performs granularity-guided pooling and relation-aware clustering (Mao et al., 9 Mar 2026). Here granularity is not fixed by the backbone depth or graph schema; it is selected according to the semantic scope of the textual input.

3. Alignment, interaction, and transport mechanisms

The central technical problem is not only to represent multiple scales, but to align them. In LGUR, the DGA module aligns modalities by reconstructing both visual and textual features from a shared dictionary, while the PGU module extracts semantically aligned features using shared prototypes (Shao et al., 2022). Although the supplied technical record does not specify the detailed mathematical definitions of these components, the abstract makes clear that alignment and unification are treated as distinct operations: first reduce the granularity mismatch, then learn a common feature space.

SimCroP makes the alignment mechanism explicit at sentence–patch granularity. For each report sentence and each CT patch, similarity is computed as

$$\mathrm{Sim}_{l,s}(T_{\text{sent}^l}, I_u^s) = (f_{\text{sent}^l})^\top f_v^s,$$

after which the model selects the Top-$K$ most similar patches and aggregates them into a sentence-aligned visual representation (Wang et al., 10 Sep 2025). This converts an implicit correspondence problem into a controlled selection problem. The subsequent symmetric InfoNCE objective then forces the selected patch set and the sentence embedding to cohere.

FlowRAG uses a different but related design: the initial entity activation is the maximum of a micro branch based on sentence matching and a macro branch based on summary alignment,

$$a_j^{(0)} = \max \left( \mathcal{S}_{micro}(q, e_j),\ \mathcal{S}_{macro}(q, e_j) \right).$$

This mechanism is explicitly motivated by paraphrase and abstraction robustness, since summaries can capture thematic relevance even when sentence-level wording is sparse or indirect (Zhan et al., 16 Jun 2026). The module is granularity-aware because it preserves two distinct semantic entry points into the same retrieval graph.

In CONQUER, fine-grained image–text matching is formulated as regularized optimal transport over local token features:

$$\min_{T \in \Pi(\mathbf{a}, \mathbf{b})} \langle T, C \rangle + \lambda \,\mathrm{H}(T),$$

with the resulting transport plan aligned to cosine local similarity through a row-wise KL term,

$$\mathcal{L}_{\mathrm{OT}} = \mathrm{KL}\bigl(\mathrm{softmax}_{\mathrm{row}}(T)\; \big\|\; \mathrm{softmax}_{\mathrm{row}}(S^{\mathrm{loc}})\bigr).$$

This couples coarse semantic context and local attribute-level correspondence without requiring manual attribute annotations (Xie, 26 Jan 2026). MGMapNet provides a spatial counterpart: its Point Instance Interaction module uses point-to-point and point-to-instance attention so that geometry-level point queries and semantics-level instance queries repeatedly refine one another (Yang et al., 2024).

4. Supervision and optimization

Granularity-aware modules are typically trained with objectives that attach losses to different scales rather than only to the final fused embedding. SimCroP combines masked image modeling, sentence–patch alignment, and masked language modeling:

$$\mathcal{L} = \mathcal{L}_{\text{MIM}} + \lambda_1 \mathcal{L}_{\text{align}} + \lambda_2 \mathcal{L}_{\text{MLM}},$$

with $f^I = \mathrm{GAP}(f_v),$0 (Wang et al., 10 Sep 2025). This objective assigns complementary roles to local structural reconstruction, cross-granularity alignment, and language-grounded fusion.

GAPNet is explicit about matching supervision granularity to feature granularity. Its three decoder outputs are supervised by different saliency targets, and the overall loss is

$f^I = \mathrm{GAP}(f_v),$1

where $f^I = \mathrm{GAP}(f_v),$2 is a boundary-others map, $f^I = \mathrm{GAP}(f_v),$3 is a center map, and $f^I = \mathrm{GAP}(f_v),$4 is the full saliency map (Wu et al., 11 Aug 2025). The module is therefore not only multi-scale in architecture but also granularity-specific in supervision.

MGS$f^I = \mathrm{GAP}(f_v),$5 adopts a multi-granularity contrastive objective for code search,

$f^I = \mathrm{GAP}(f_v),$6

so that function-, block-, and statement-level embeddings are jointly optimized in the same representation space (Li et al., 30 May 2025). This is structurally analogous to multimodal systems that combine global and local losses: granularity becomes part of the training target, not just a by-product of the encoder.

RaCMC shows the same principle in a multimodal classification setting through two kinds of constraints: a news-overall constraint based on

$f^I = \mathrm{GAP}(f_v),$7

and a news-internal constraint that shapes text–image similarity at the pair level (Yu et al., 2024). This suggests a broader pattern: granularity-aware modules are often most effective when global distributional structure and local correspondence structure are regularized simultaneously.

5. Empirical behavior across applications

Representative empirical results show that granularity-aware modules are not confined to one modality or one task family. They appear in person retrieval, graph-based retrieval, medical vision-language pre-training, fine-grained classification, video moment retrieval, zero-shot temporal grounding, and multimodal LLMs (Shao et al., 2022, Zhan et al., 16 Jun 2026, Wang et al., 10 Sep 2025, Song et al., 2021, Lin et al., 6 Aug 2025, Jeon et al., 2 Jan 2026, Mao et al., 9 Mar 2026).

Domain	Representative module pattern	Representative evidence
Text-to-image person ReID	Alignment + unification	LGUR consistently outperforms state-of-the-arts by large margins on CUHK-PEDES and ICFG-PEDES
GraphRAG	Summary–sentence dual activation	FlowRAG attains highest Recall of 92.90%
CT vision-language pre-training	Sentence–patch alignment + instance/word–patch fusion	SimCroP improves AUC by about +1.6 on CT-RATE with only 1% labeled data
Fine-grained visual classification	Multi-stage receptive-field supervision	ResNet-50 baseline 83.7%, +GSC 88.7%, +GSC+OAM 90.4%
Video moment retrieval	Local/event/global fusion with importance gating	IMG reaches R1@7 = 44.23 and mIoU = 55.62 on Charades-STA
Zero-shot video moment retrieval	Coarse–coarse and fine–fine alignment paths	GranAlign reports a 3.23% mAP@avg improvement on QVHighlights
MLLMs	Text-conditioned adaptive granularity	Granulon improves accuracy by ~30% and reduces hallucination by ~20%

The significance of these results is not merely that “more features” help. The recurring empirical pattern is that performance improves when the module explicitly matches the scale of representation to the scale of evidence and then imposes interactions across those scales. FlowRAG’s ablations report a $f^I = \mathrm{GAP}(f_v),$8 GPT-Acc drop on MuSiQue when Dual-Granularity Activation is removed, and SimCroP’s full model improves over the best non-SA variant by more than $f^I = \mathrm{GAP}(f_v),$9 AUC and more than $f^{W} = \mathrm{CrossAttention}(f_t, f_v),$0 Dice in the reported ablation (Zhan et al., 16 Jun 2026, Wang et al., 10 Sep 2025). In fine-grained visual classification, the main gain in GA-CNN comes from the granularity-specific classifiers themselves, with the object-attentive module providing an additional gain (Song et al., 2021). This suggests that granularity-aware design is often a primary representational factor rather than a secondary refinement.

6. Limitations, misconceptions, and open directions

A common misconception is that finer granularity is always preferable. The literature does not support that claim. GranAlign reports that coarse queries tend toward high coverage and low precision, while fine queries tend toward high precision and low coverage (Jeon et al., 2 Jan 2026). Granulon reports that coarse configurations perform best on A-OKVQA, whereas increasing cluster count improves GPTscore on FLUX-Reason (Mao et al., 9 Mar 2026). The relevant issue is therefore not maximal detail, but appropriate detail conditioned on task structure.

A second misconception is that granularity awareness is equivalent to generic multi-scale processing. GAPNet explicitly argues otherwise by assigning coarse center supervision to high-level outputs and fine boundary-related supervision to low-level outputs, instead of supervising every stage with the same full saliency map (Wu et al., 11 Aug 2025). FlowRAG similarly shows that multi-granularity retrieval still requires noise control: removing Frequency-Aware Weighted Flow causes consistent performance drops, including $f^{W} = \mathrm{CrossAttention}(f_t, f_v),$1 on HotpotQA and $f^{W} = \mathrm{CrossAttention}(f_t, f_v),$2 on Medical (Zhan et al., 16 Jun 2026). Granularity-aware modules therefore depend on how scales are connected and regularized, not only on how many scales are present.

Open problems remain. SimCroP explicitly notes that it does not perform instance-level cross-modal alignment, which limits zero-shot performance, and points to report-level or impression-level alignment as a future direction (Wang et al., 10 Sep 2025). GranAlign identifies hallucination and over-mimicry risks in query-aware captions and suggests semantic verification and adaptive weighting between coarse and fine paths (Jeon et al., 2 Jan 2026). More broadly, the surveyed work points toward adaptive granularity selection, stronger uncertainty modeling, and tighter coupling between global and local supervision as the next phase of development. This suggests that the mature form of a granularity-aware representation module is likely to be dynamic rather than static: not only multi-granular, but also context-sensitive in how it activates, fuses, and verifies those granularities.