Multi-Granular Language Learning

• MGLL is a framework that organizes language supervision across multiple granularities—from words to nested expressions—to capture both fine details and broader context.
• It leverages progressive decomposition and explicit cross-granular interactions to reduce ambiguity and promote faithful cross-modal alignment.
• Empirical results show MGLL improves performance in vision-language reasoning, multi-hop QA, document understanding, speech-text pre-training, and medical imaging tasks.

Multi-Granular Language Learning (MGLL) denotes a class of learning principles in which language supervision is organized across multiple levels of abstraction and coupled to aligned structure in another modality or task. Across papers, these granularities include nested referring expressions in grounded vision-language reasoning, local reasoning steps and global answers in multi-hop question answering, page-region-word hierarchies in document understanding, label-phrase-sentence hierarchies in vision-language pre-training, global and fine-grained style captions in speech-text pre-training, and sentence-word-phoneme assessments in speech evaluation (Le et al., 2024, Coleman et al., 9 Feb 2025, Wang et al., 2022, Li et al., 2022, Yang et al., 6 Jan 2026, Parikh et al., 8 Jun 2026). The common premise is that single-granularity supervision is often too shallow: lower-level units provide detail, higher-level units provide context, and explicit relations across levels constrain ambiguity, improve compositionality, and support more faithful alignment.

1. Conceptual scope

In the grounded LVLM setting, MGLL is formulated as a hierarchy of aligned linguistic and visual units. "Progressive Multi-granular Alignments for Grounded Reasoning in Large Vision-LLMs" defines level-one expressions as non-relational object-centric concepts such as “the woman” or “a horse,” level-two expressions as relational combinations such as “the woman riding a horse,” and higher levels as deeper nested structures such as “the man is behind the woman riding a horse” (Le et al., 2024). Lower-level expressions are treated as building blocks for higher-level expressions rather than as separate samples.

In multi-hop QA, MGLL is defined differently: the relevant granularities are local reasoning steps and global answer correctness. "Multi-granular Training Strategies for Robust Multi-hop Reasoning Over Noisy and Heterogeneous Knowledge Sources" trains AMKOR both on intermediate reasoning steps $t_i$ in a trajectory $\mathcal{T}=\{t_1,\dots,t_m\}$ and on the final answer $a$, with the stated aim of reducing cascading errors (Coleman et al., 9 Feb 2025).

In document understanding, MGLL is explicitly hierarchical in spatial structure. MGDoc models a page $P$, regions $\{R_1,\dots,R_m\}$, and words $\{w_1,\dots,w_n\}$ jointly, arguing that document images are organized from fine granularity to coarse granularity and that many downstream tasks require reasoning across all three levels (Wang et al., 2022).

In language-speech and vision-language pre-training, the same term spans different linguistic decompositions. Uni-EDEN uses label $\rightarrow$ phrase $\rightarrow$ sentence supervision (Li et al., 2022); CLSP uses global utterance-level captions together with fine-grained narrative captions that track within-clip dynamics (Yang et al., 6 Jan 2026); and $\beta$-CLIP aligns full captions, sentences, and phrases with contextualized visual embeddings (Zohra et al., 14 Dec 2025).

Setting	Granularities	Representative paper
Grounded LVLMs	object phrase, relation phrase, nested expression	(Le et al., 2024)
Multi-hop QA	local reasoning step, global answer	(Coleman et al., 9 Feb 2025)
Document understanding	page, region, word	(Wang et al., 2022)
Vision-language pre-training	label, phrase, sentence	(Li et al., 2022)
Speech-text pre-training	global caption, fine-grained caption	(Yang et al., 6 Jan 2026)
L2 speech assessment	sentence, word, phoneme, rationale	(Parikh et al., 8 Jun 2026)

This suggests that MGLL is best understood not as a single architecture, but as a recurring organizational principle: supervision is distributed across multiple semantic or structural resolutions, and those resolutions are linked rather than optimized independently.

2. Core mechanisms

A central MGLL mechanism is progressive decomposition. PromViL decomposes a complex input sentence into a chain of nested subsequences, from the smallest referential pieces to the full compositional expression, and trains the model to predict spatial tokens at each level while conditioning on feedback from lower levels (Le et al., 2024). The paper gives the objective

$$\mathcal{L}=\frac{1}{|E|}\sum_{i=1}^{c}\log P(y_i \mid V, E_i, \langle E_{i-1}, y_{i-1}\rangle),$$

where the conditioning on $\mathcal{T}=\{t_1,\dots,t_m\}$0 is the progressive element. At inference, a constituency parser and a dependency parser extract nested subsequences, and grounding proceeds level by level.

A second mechanism is explicit cross-granular interaction. MGDoc introduces multi-granular attention with hierarchy and relative-position biases to model containment relations such as region-word and page-region, and it applies this attention separately to text embeddings and visual embeddings before cross-modal fusion (Wang et al., 2022). $\mathcal{T}=\{t_1,\dots,t_m\}$1-CLIP similarly avoids a single static image embedding: for each caption-, sentence-, or phrase-level text query, it performs text-conditioned cross-attention pooling over image patches, creating contextualized visual embeddings specific to each linguistic scale (Zohra et al., 14 Dec 2025).

A third mechanism is contrastive or probabilistic coupling across levels. GeoAlignCLIP keeps global image-caption alignment as Stage I, then adds Stage II with region-phrase alignment, hard-negative alignment, visual intra-consistency, and hierarchical textual consistency (Yang et al., 10 Mar 2026). CLSP first aligns speech with fine-grained captions through symmetric InfoNCE, then introduces a Stage-2 multi-positive objective in which one speech clip is paired with both a global caption and a fine-grained caption, with $\mathcal{T}=\{t_1,\dots,t_m\}$2 in the final model (Yang et al., 6 Jan 2026).

A fourth mechanism is unified generation across granularities. Uni-EDEN uses a two-stream Transformer encoder-decoder and combines Masked Object Classification, Masked Region Phrase Generation, Image-Sentence Matching, and Masked Sentence Generation so that object-level grounding, phrase-level description, sentence-level alignment, and sentence generation are learned in one model (Li et al., 2022). A related unification appears in MVP-LM, which treats word-based and sentence-based instructions together with box and mask predictions under one VLLM framework, using a CoT-inspired dataset unification strategy and a multi-granularity decoder (Xiang et al., 22 Jul 2025).

3. Representative instantiations across domains

In grounded and perception-oriented vision-language work, MGLL is tightly connected to compositional grounding. PromViL fine-tunes existing grounded LVLMs, including Kosmos-2 and one MiniGPTv2 variant, around nested expressions and recursive grounding (Le et al., 2024). MMGiC extends a general autoregressive MLLM framework beyond coarse image captions by combining captions, object labels, attribute labels, relationship labels, label descriptions, and object-region crops in one structured template, with the explicit claim that captions provide breadth while fine-grained annotations provide depth (Xu et al., 2024). $\mathcal{T}=\{t_1,\dots,t_m\}$3-CLIP and GeoAlignCLIP move CLIP-style alignment beyond global image-text matching into hierarchical dense correspondence, while MVP-LM unifies word-based versus sentence-based instructions and box versus mask predictions in a single architecture (Zohra et al., 14 Dec 2025, Yang et al., 10 Mar 2026, Xiang et al., 22 Jul 2025).

In document understanding, MGDoc treats the page as a hierarchy in which words belong to regions and regions belong to the page. Its unified text-visual encoder projects multi-granular features into the same hyperspace, and its Multi-Granularity Modeling task explicitly predicts which region contains a given word (Wang et al., 2022). This is a particularly literal form of MGLL because the language units and spatial units are both nested.

In speech and speech-language work, MGLL appears in two distinct forms. CLSP models speaking style as inherently multi-scale, pairing a speech encoder with a text encoder and training on global captions, fine-grained narrative captions, and multiple fine-grained captions for the same audio (Yang et al., 6 Jan 2026). "A Finetuned SpeechLLM for Joint Multi-Granular L2 Assessment and Natural-Language Rationales" applies MGLL to assessment rather than pre-training: a rubric-guided SpeechLLM jointly predicts sentence-level Accuracy, Fluency, and Prosody labels, word-level Accuracy labels, phoneme-level Accuracy labels, and a free-text rationale in a single response (Parikh et al., 8 Jun 2026).

In knowledge-intensive reasoning, MGLL is instantiated as supervision over reasoning trajectories. AMKOR combines parametric knowledge from the LLM with retrieved external knowledge, explores multiple candidate trajectories through probabilistic beam reasoning, and optimizes both local reasoning-step loss and global answer loss (Coleman et al., 9 Feb 2025).

In medical imaging, the term MGLL is also used as the name of a specific contrastive framework. "Boosting Medical Visual Understanding From Multi-Granular Language Learning" preserves a CLIP-style dual encoder but changes the supervision: soft multi-label image-text alignment, point-wise supervision, and smooth KL divergence enforce cross-granularity consistency across disease categories, study descriptions, series descriptions, and clinical explanations (Li et al., 20 Nov 2025).

4. Objectives, data construction, and evaluation

MGLL methods are typically data-centric. PromViL constructs CompoVL from Visual Genome, including about 29K lists of nested expressions and up to 115K individual V-L pairs, then extends it with 1.2K instances from VSR, 8K VG-VQA, and 22K LLaVA-Instruct150K samples to yield 60.3K instances; it also defines CompoVL-hard using only level-two and above pairs (Le et al., 2024). MMGiC aggregates 3.48M images, 25.55M regions, and 35,048 unique concept label–description pairs, with 4.06M samples after tokenization and template construction (Xu et al., 2024). GeoAlignCLIP introduces RSFG-100k with 100k images and 400k+ hierarchical textual annotations, plus hard negatives and reported quality controls including a 94.1% overall pass rate and Cohen’s $\mathcal{T}=\{t_1,\dots,t_m\}$4 (Yang et al., 10 Mar 2026). FCaps contains 18,131,371 fine-grained captions and 46,787 hours of speech in FCaps-Emilia, plus 140,602 global captions, 930,917 fine-grained captions, and 267 hours in FCaps-PSCBase (Yang et al., 6 Jan 2026). The medical MGLL paper constructs MGLL-Fundus with 246,389 fundus image-text pairs from 49 public fundus datasets and MGLL-Xray with 190,882 X-ray images from MIDRC (Li et al., 20 Nov 2025).

The loss designs reflect the same principle. AMKOR combines local and global supervision as

$\mathcal{T}=\{t_1,\dots,t_m\}$5

where local loss scores intermediate reasoning steps and global loss scores the final answer (Coleman et al., 9 Feb 2025). The medical MGLL framework combines soft multi-label alignment, point-wise supervision, and smooth KL consistency:

$\mathcal{T}=\{t_1,\dots,t_m\}$6

with the reported best default weights $\mathcal{T}=\{t_1,\dots,t_m\}$7 (Li et al., 20 Nov 2025).

Evaluation is correspondingly multi-regime. Grounded vision-language work uses RefCOCO, RefCOCO+, RefCOCOg, GQA, GQA-OOD, Visual7W, open-set segmentation, referring expression comprehension, and compositional subsets such as CompoVL-hard (Le et al., 2024, Xiang et al., 22 Jul 2025). Document work uses FUNSD, CORD, and RVL-CDIP (Wang et al., 2022). Multi-hop reasoning uses HotpotQA, 2WikiMQA, MuSiQue, and Bamboogle (Coleman et al., 9 Feb 2025). Speech work evaluates global and fine-grained retrieval, zero-shot paralinguistic classification, and speech style similarity scoring (Yang et al., 6 Jan 2026). Medical work evaluates fundus and X-ray classification, zero-shot classification, segmentation, and MLLM integration (Li et al., 20 Nov 2025).

5. Empirical findings

Across the cited studies, the strongest recurring empirical claim is that hierarchical organization matters more than simply adding heterogeneous supervision. PromViL reports 64.07% on CompoVL-hard for the 1.6B Kosmos-2-based model versus 55.37% for vanilla Kosmos-2 and 59.12% for the finetuned baseline without the progressive structure; with MiniGPTv2 as backbone, PromViL$\mathcal{T}=\{t_1,\dots,t_m\}$8 reaches 68.92% (Le et al., 2024). The paper explicitly argues that simply adding more VQA data or training on CompoVL without nesting gives only small gains.

AMKOR reports state-of-the-art F1 on all four reported datasets: 63.2 on HotpotQA, 73.4 on 2WikiMQA, 37.4 on MuSiQue, and 75.2 on Bamboogle. Under injected irrelevant retrieval on HotpotQA, AMKOR still achieves 54.3 F1 at 40% noise, compared with 43.8 for ProbTree and 32.1 for CoT (Coleman et al., 9 Feb 2025).

MGDoc reaches 89.44 F1 on FUNSD and 97.11 F1 on CORD, with ablations showing improvement from 83.01 without pre-training to 87.20 with MTM+MVM and 89.44 with MTM+MVM+MGM on FUNSD (Wang et al., 2022). CLSP reports 68.1 R@1 for speech-to-text fine-grained retrieval and correlations up to 0.903 Pearson on situational traits in speech style similarity scoring (Yang et al., 6 Jan 2026). MMGiC reports that an appropriate combination with image-caption data achieves 3.95% and 2.34% absolute improvements over image-caption data alone on POPE and SEED-Bench (Xu et al., 2024). The medical MGLL paper reports 79.62 AUC on RFMiD in linear probing versus CLIP’s 44.66 AUC, and average ophthalmic multiple-choice accuracy gains such as +14.7% for InstructBLIP and +34.1% for LLaVA-Med when MGLL is used as the pretrained vision encoder (Li et al., 20 Nov 2025).

A plausible implication is that MGLL is especially useful when errors propagate across levels: compositional grounding, deep multi-hop reasoning, dense alignment, and multi-label medical semantics all exhibit this property in the reported experiments.

6. Misconceptions, limitations, and open problems

A common misconception is that MGLL is equivalent to adding finer annotations to a model trained on coarse supervision. Several papers explicitly contradict this. In MMGiC, the C+L recipe can underperform C alone, while C+L+D and especially C+L+D+R recover and then surpass caption-only training; the paper argues that labels without descriptions or regions can be treated as noise (Xu et al., 2024). PromViL similarly argues that the main gain comes from the hierarchical organization of supervision and reasoning, not merely more data (Le et al., 2024).

Another misconception is that MGLL is just ordinary multi-task learning. The L2 assessment paper states that its formulation is not just multi-task learning but joint multi-granular assessment with explanation, because sentence-level, word-level, and phoneme-level outputs are produced in one pass and tied to a shared rubric prompt and a rationale (Parikh et al., 8 Jun 2026). In a related way, CLSP’s Stage-2 objective does not merely mix captions; it teaches that one speech clip can correspond to multiple valid textual views at different granularities (Yang et al., 6 Jan 2026).

The gains are also uneven across levels. In the SpeechLLM study, rationale sentiment is highly self-consistent at sentence level—88.1% for negative, 89.2% for neutral, and 91.1% for positive under the Internal (Pred) reference—but phoneme-level mention-based agreement is only 0.20 internal and 0.07 external (Parikh et al., 8 Jun 2026). The paper therefore concludes that token-level faithful explanation remains weak. MVP-LM reports that replacing query selection with separate learnable query embeddings improves RefCOCO val to 78.3 but hurts COCO performance to 54.4 PQ / 65.1 mIoU, indicating that some granularity choices favor simpler single-target referring tasks while others favor generic perception (Xiang et al., 22 Jul 2025).

Open problems recur across domains. Data quality and annotation completeness remain central constraints: the medical MGLL paper notes dependence on constructing good multi-granular text descriptions and reports validation mainly on retinal and X-ray data (Li et al., 20 Nov 2025); MMGiC notes that its annotations are still mainly concrete concepts and that some attributes and relationships are only partially available (Xu et al., 2024). The MG-LLM roadmap elevates the problem to a broader systems question, calling for a unified multi-modal graph vocabulary and tokenizer capable of spanning pixels, words, nodes, edges, subgraphs, and graphs in a shared space (Wang et al., 11 Jun 2025). This suggests that the long-term challenge for MGLL is not only designing better losses, but also defining interoperable representations across semantic, structural, and modal scales.