---
title: Granularity-Aware Representation Module
url: https://www.emergentmind.com/topics/granularity-aware-representation-module
type: topic
---

# Granularity-Aware Representation Module

Searching arXiv for recent and foundational papers on granularity-aware representation modules across modalities.
arXiv search query: "granularity-aware representation module multimodal granularity-aware alignment representation"

A granularity-aware representation module is a model component that explicitly encodes, aligns, or fuses information across multiple semantic, spatial, temporal, or structural scales. Across recent work, the term has come to denote a family of designs rather than a single architecture: in text-to-image person re-identification it addresses the mismatch between fine-grained visual features and coarse textual descriptions; in retrieval-augmented generation it mediates between summaries, sentences, entities, and passages; in medical vision-language pre-training it couples instance-, sentence-, patch-, and word–patch-level cues; and in multimodal large language models it dynamically moves between pixel-level, fine-grained, and coarse-grained semantics [2207.07802] [2606.17856] [2509.08311] [2603.08800].

## 1. Conceptual foundations

In the cited literature, *granularity* is defined relative to the structure of the input and the task. In fine-grained visual classification, “small granularity” corresponds to earlier CNN stages with small receptive fields that capture local parts, whereas “larger granularity” corresponds to deeper stages with larger receptive fields that capture bigger regions or the whole object [2103.02788]. In salient object detection, “low granularity” denotes coarse object semantics and rough locations, while “high granularity” denotes fine-grained boundaries and local details; the former is associated with high-level features and the latter with low-level features [2508.07585]. These formulations make granularity a property of both representation scale and semantic abstraction.

Cross-modal work generalizes the same idea. SimCroP defines granularity over instance-level CT volume representations, sentence-level report units, patch-level subregions, and word–patch interactions [2509.08311]. FlowRAG organizes retrieval over passages, summaries, sentences, and entities, with summary nodes acting as coarse semantic hubs and sentence/entity nodes providing fine evidence [2606.17856]. GranAlign makes the precision–coverage trade-off explicit: coarse queries provide high coverage but low precision, whereas fine queries provide high precision but low coverage [2601.00584]. Taken together, these formulations suggest that a granularity-aware module is not merely a multi-scale feature extractor; it is a mechanism that decides which scale is structurally appropriate, how scales should interact, and how supervision should be matched to those scales.

## 2. Canonical architectural forms

One canonical form is the explicit *alignment–unification* decomposition. In “Learning Granularity-Unified Representations for Text-to-Image Person Re-identification” [2207.07802], the LGUR framework is described as an end-to-end framework based on transformers with two modules: a Dictionary-based Granularity Alignment module and a Prototype-based Granularity Unification module. The first uses a Multi-modality Shared Dictionary to reconstruct both visual and textual features, with cross-modality guidance and foreground-centric reconstruction; the second uses shared and learnable prototypes as queries to extract diverse and semantically aligned features in a granularity-unified feature space. In this formulation, the module exists to reduce the inter-modal gap caused by the fact that visual features are usually fine-grained while textual features are coarse.

A second form is the explicit *hierarchical graph*. FlowRAG constructs a quad-level heterogeneous graph
$$
G = (V, E), \quad V = V_P \cup V_{Sum} \cup V_S \cup V_E,
$$
where passages, summaries, sentences, and entities constitute distinct node types [2606.17856]. Granularity is represented directly in the graph schema rather than only in feature dimensions. Query relevance is then injected through both summary–query alignment and sentence-level matching, so the representation module is simultaneously a hierarchy constructor and a relevance propagation mechanism.

A third form is the *cross-granularity fusion block*. SimCroP separates a global instance-level visual stream,
$$
f^I = \mathrm{GAP}(f_v),
$$
from a fine-grained word–patch stream,
$$
f^{W} = \mathrm{CrossAttention}(f_t, f_v),
$$
and reconstructs masked report tokens from their sum,
$$
\hat{T}_m = D_t(f^I + f^{W}),
$$
thereby forcing the representation to remain useful at both holistic and local scales [2509.08311]. Similar dual-granularity or multi-granularity decompositions appear in vectorized HD map construction, where MGMapNet uses coarse-grained instance-level queries and fine-grained point-level queries generated from multi-scale BEV features, followed by a Point Instance Interaction module [2410.07733].

A fourth form is *adaptive granularity control*. Granulon introduces a text-conditioned granularity Controller that predicts parameters governing spatial down-sampling, cluster cardinality, and multimodal mixing, and an Adaptive Token Aggregation module that performs granularity-guided pooling and relation-aware clustering [2603.08800]. Here granularity is not fixed by the backbone depth or graph schema; it is selected according to the semantic scope of the textual input.

## 3. Alignment, interaction, and transport mechanisms

The central technical problem is not only to represent multiple scales, but to align them. In LGUR, the DGA module aligns modalities by reconstructing both visual and textual features from a shared dictionary, while the PGU module extracts semantically aligned features using shared prototypes [2207.07802]. Although the supplied technical record does not specify the detailed mathematical definitions of these components, the abstract makes clear that alignment and unification are treated as distinct operations: first reduce the granularity mismatch, then learn a common feature space.

SimCroP makes the alignment mechanism explicit at sentence–patch granularity. For each report sentence and each CT patch, similarity is computed as
$$
\mathrm{Sim}_{l,s}(T_{\text{sent}^l}, I_u^s) = (f_{\text{sent}^l})^\top f_v^s,
$$
after which the model selects the Top-$K$ most similar patches and aggregates them into a sentence-aligned visual representation [2509.08311]. This converts an implicit correspondence problem into a controlled selection problem. The subsequent symmetric InfoNCE objective then forces the selected patch set and the sentence embedding to cohere.

FlowRAG uses a different but related design: the initial entity activation is the maximum of a micro branch based on sentence matching and a macro branch based on summary alignment,
$$
a_j^{(0)} = \max \left( \mathcal{S}_{micro}(q, e_j),\ \mathcal{S}_{macro}(q, e_j) \right).
$$
This mechanism is explicitly motivated by paraphrase and abstraction robustness, since summaries can capture thematic relevance even when sentence-level wording is sparse or indirect [2606.17856]. The module is granularity-aware because it preserves two distinct semantic entry points into the same retrieval graph.

In CONQUER, fine-grained image–text matching is formulated as regularized optimal transport over local token features:
$$
\min_{T \in \Pi(\mathbf{a}, \mathbf{b})} \langle T, C \rangle + \lambda \,\mathrm{H}(T),
$$
with the resulting transport plan aligned to cosine local similarity through a row-wise KL term,
$$
\mathcal{L}_{\mathrm{OT}} = \mathrm{KL}\bigl(\mathrm{softmax}_{\mathrm{row}}(T)\; \big\|\; \mathrm{softmax}_{\mathrm{row}}(S^{\mathrm{loc}})\bigr).
$$
This couples coarse semantic context and local attribute-level correspondence without requiring manual attribute annotations [2601.18625]. MGMapNet provides a spatial counterpart: its Point Instance Interaction module uses point-to-point and point-to-instance attention so that geometry-level point queries and semantics-level instance queries repeatedly refine one another [2410.07733].

## 4. Supervision and optimization

Granularity-aware modules are typically trained with objectives that attach losses to different scales rather than only to the final fused embedding. SimCroP combines masked image modeling, sentence–patch alignment, and masked language modeling:
$$
\mathcal{L} = \mathcal{L}_{\text{MIM}} + \lambda_1 \mathcal{L}_{\text{align}} + \lambda_2 \mathcal{L}_{\text{MLM}},
$$
with $\lambda_1 = \lambda_2 = 1$ [2509.08311]. This objective assigns complementary roles to local structural reconstruction, cross-granularity alignment, and language-grounded fusion.

GAPNet is explicit about matching supervision granularity to feature granularity. Its three decoder outputs are supervised by different saliency targets, and the overall loss is
$$
\mathcal{L}_{overall} = \sum_{i=1}^3 \mathcal{L}(P_i, G_i),
$$
where \(G_1\) is a boundary-others map, \(G_2\) is a center map, and \(G_3\) is the full saliency map [2508.07585]. The module is therefore not only multi-scale in architecture but also granularity-specific in supervision.

MGS\(^3\) adopts a multi-granularity contrastive objective for code search,
$$
\mathcal{L}_{MG} = \mathcal{L}_{f} + \alpha \mathcal{L}_{b} + \beta \mathcal{L}_{s},
$$
so that function-, block-, and statement-level embeddings are jointly optimized in the same representation space [2505.24274]. This is structurally analogous to multimodal systems that combine global and local losses: granularity becomes part of the training target, not just a by-product of the encoder.

RaCMC shows the same principle in a multimodal classification setting through two kinds of constraints: a news-overall constraint based on
$$
l_{no} = -\mathrm{MMD}^2(\mathcal{F},\mathbf{M}'^{+},\mathbf{M}'^{-}),
$$
and a news-internal constraint that shapes text–image similarity at the pair level [2412.18254]. This suggests a broader pattern: granularity-aware modules are often most effective when global distributional structure and local correspondence structure are regularized simultaneously.

## 5. Empirical behavior across applications

Representative empirical results show that granularity-aware modules are not confined to one modality or one task family. They appear in person retrieval, graph-based retrieval, medical vision-language pre-training, fine-grained classification, video moment retrieval, zero-shot temporal grounding, and multimodal large language models [2207.07802] [2606.17856] [2509.08311] [2103.02788] [2508.04273] [2601.00584] [2603.08800].

| Domain | Representative module pattern | Representative evidence |
|---|---|---|
| Text-to-image person ReID | Alignment + unification | LGUR consistently outperforms state-of-the-arts by large margins on CUHK-PEDES and ICFG-PEDES |
| GraphRAG | Summary–sentence dual activation | FlowRAG attains highest Recall of 92.90% |
| CT vision-language pre-training | Sentence–patch alignment + instance/word–patch fusion | SimCroP improves AUC by about +1.6 on CT-RATE with only 1% labeled data |
| Fine-grained visual classification | Multi-stage receptive-field supervision | ResNet-50 baseline 83.7%, +GSC 88.7%, +GSC+OAM 90.4% |
| Video moment retrieval | Local/event/global fusion with importance gating | IMG reaches R1@7 = 44.23 and mIoU = 55.62 on Charades-STA |
| Zero-shot video moment retrieval | Coarse–coarse and fine–fine alignment paths | GranAlign reports a 3.23% mAP@avg improvement on QVHighlights |
| MLLMs | Text-conditioned adaptive granularity | Granulon improves accuracy by ~30% and reduces hallucination by ~20% |

The significance of these results is not merely that “more features” help. The recurring empirical pattern is that performance improves when the module explicitly matches the scale of representation to the scale of evidence and then imposes interactions across those scales. FlowRAG’s ablations report a \(-2.7\%\) GPT-Acc drop on MuSiQue when Dual-Granularity Activation is removed, and SimCroP’s full model improves over the best non-SA variant by more than \(1\%\) AUC and more than \(1.1\%\) Dice in the reported ablation [2606.17856] [2509.08311]. In fine-grained visual classification, the main gain in GA-CNN comes from the granularity-specific classifiers themselves, with the object-attentive module providing an additional gain [2103.02788]. This suggests that granularity-aware design is often a primary representational factor rather than a secondary refinement.

## 6. Limitations, misconceptions, and open directions

A common misconception is that finer granularity is always preferable. The literature does not support that claim. GranAlign reports that coarse queries tend toward high coverage and low precision, while fine queries tend toward high precision and low coverage [2601.00584]. Granulon reports that coarse configurations perform best on A-OKVQA, whereas increasing cluster count improves GPTscore on FLUX-Reason [2603.08800]. The relevant issue is therefore not maximal detail, but appropriate detail conditioned on task structure.

A second misconception is that granularity awareness is equivalent to generic multi-scale processing. GAPNet explicitly argues otherwise by assigning coarse center supervision to high-level outputs and fine boundary-related supervision to low-level outputs, instead of supervising every stage with the same full saliency map [2508.07585]. FlowRAG similarly shows that multi-granularity retrieval still requires noise control: removing Frequency-Aware Weighted Flow causes consistent performance drops, including \(-1.0\%\) on HotpotQA and \(-1.17\%\) on Medical [2606.17856]. Granularity-aware modules therefore depend on how scales are connected and regularized, not only on how many scales are present.

Open problems remain. SimCroP explicitly notes that it does not perform instance-level cross-modal alignment, which limits zero-shot performance, and points to report-level or impression-level alignment as a future direction [2509.08311]. GranAlign identifies hallucination and over-mimicry risks in query-aware captions and suggests semantic verification and adaptive weighting between coarse and fine paths [2601.00584]. More broadly, the surveyed work points toward adaptive granularity selection, stronger uncertainty modeling, and tighter coupling between global and local supervision as the next phase of development. This suggests that the mature form of a granularity-aware representation module is likely to be dynamic rather than static: not only multi-granular, but also context-sensitive in how it activates, fuses, and verifies those granularities.

Source: https://www.emergentmind.com/topics/granularity-aware-representation-module