Multi-Granularity Prompt Learning

• Multi-Granularity Prompt Learning is a method that decomposes prompts into multiple, distinct levels—such as semantic, structural, and modality-specific—that provide enhanced control over model adaptation.
• It leverages task metadata and hierarchical token structures to improve few-shot, domain-adaptive performance across text, vision, graph, and federated learning tasks.
• Empirical evidence shows significant gains in classification and interpretability while highlighting challenges in managing increased prompt complexity and control.

Searching arXiv for the provided topic and representative papers to ground the article in current literature. Multi-granularity prompt learning denotes a family of prompt-based adaptation methods in which the prompt is not treated as a single homogeneous context string or vector, but is decomposed into multiple semantic, structural, or modality-specific levels that serve different roles in inference. Across recent work, these levels include task metadata such as object, summary, and task descriptions in few-shot classification; task-, domain-, and context-specific prompts in machine reading comprehension; low-, high-, and global-level textual prompts in vision-LLMs; probability and location prompts in face forgery detection; low-frequency and high-frequency visual bands in spectral prompt learning; coarse-to-fine structural prompts in graph models; and coarse global versus fine local prompts in personalized federated continual learning (Weng et al., 2023, Chen et al., 2023, Wang et al., 2024, Chen et al., 16 Jul 2025, Zhou et al., 6 May 2026, Zheng et al., 10 Oct 2025, Yu et al., 2024, Yu et al., 2023, Liu et al., 2023). The shared premise is that downstream performance depends not only on whether prompting is used, but on how prompt information is partitioned, aligned with pretrained knowledge, and injected into the model.

1. Conceptual basis and motivation

A common motivation across the literature is that single-prompt adaptation is often too coarse for the heterogeneity of modern downstream tasks. In few-shot language classification, prompt construction is described as fragile: minor changes in prompt form can lead to widely different results, manual prompt design is heuristic and expensive, and the search space grows quickly with template tokens and label words (Weng et al., 2023). In machine reading comprehension, existing soft-prompt methods are characterized as input-independent and therefore inattentive to domain variation and passage-specific evidence (Chen et al., 2023). In graph prompting, both MultiGPrompt and MSGCOT identify a similar narrowness: single-pretext or single-granularity prompts capture only one facet of graph structure, leaving downstream few-shot adaptation with limited reusable knowledge (Yu et al., 2023, Zheng et al., 10 Oct 2025).

In vision-language settings, the same critique appears in two forms. HPT++ argues that category names alone are often too weak for CLIP-like models when classes are ambiguous, fine-grained, or domain-shifted, and that conventional descriptions lack explicit structured information about entities, attributes, and their relations (Wang et al., 2024). SpecPL frames the issue as modality asymmetry: prompt learning methods mostly optimize text tokens while relying on a frozen visual encoder as a holistic extractor, thereby neglecting the spectral granularity needed for fine-grained discrimination (Zhou et al., 6 May 2026). MGFFD-VLM reaches an analogous conclusion in face forgery detection, where a VLM should not only answer whether a face is fake, but should also condition on authenticity probability cues and spatial forgery cues to support explanation (Chen et al., 16 Jul 2025).

A related but distinct motivation appears in personalized federated continual learning. FedMGP argues that full-parameter sharing is too fine-grained and destructive under simultaneous temporal updates and cross-client aggregation, and therefore proposes prompt-level decomposition into coarse-grained global knowledge and fine-grained local knowledge to mitigate spatial-temporal catastrophic forgetting (Yu et al., 2024). Taken together, these formulations suggest that multi-granularity prompting is less a single algorithm than a design principle: separate prompt channels are introduced when one prompt cannot simultaneously encode stable invariants, discriminative detail, and task-specific control.

2. Forms of granularity across modalities

The granularity axis differs by domain, but the underlying pattern is systematic. In "Helping LLMs Learn More: Multi-dimensional Task Prompt for Few-shot Tuning" (Weng et al., 2023), MTPrompt defines three prompt dimensions: Object Description (OD), Summary Description (SD), and Task Description (TD). The prompt transforms an input as

$T_{}(x^{i})= x^{i}.[t_{Od}][t_{Sd}][t_{Td}][MASK],$

so the model is guided by domain anchoring, topical context, and an explicit statement of the prediction objective. The SST-2 example—“A movie review,” “Talking about its director, actor, performance, character skill, and story.”, and “The emotion of this review was [MASK]”—makes the three-way decomposition concrete (Weng et al., 2023).

In "Exploring Multi-level Prompt Tuning for Machine Reading Comprehension" (Chen et al., 2023), the levels are task-specific, domain-specific, and context-specific. The task-specific prompt captures dataset-wide QA behavior; domain-specific prompts are assigned after KMeans clustering of contexts using SentenceTransformers embeddings; and context-specific prompts are modulated by a prompt generator conditioned on the actual passage. Here granularity is defined by the scope of shared semantics: global across the dataset, shared within latent domains, or specific to an individual instance (Chen et al., 2023).

Other systems define granularity in ways native to their modality. HPT++ uses low-level prompts grounded in generated descriptions and structured relations, high-level prompts derived from description representations, and global-level prompts that are category-agnostic and shared across classes (Wang et al., 2024). MGFFD-VLM uses a probability prompt for coarse image-level authenticity evidence and a location prompt for dense spatial forgery evidence, with the full LLM input given by

$$P_{i}^{all} = \left\{ P_{i}^{probability},P_{i}^{segment},P_{i}^{question},P_{i}^{vision} \right\}$$

(Chen et al., 16 Jul 2025). SpecPL makes granularity explicitly spectral by decomposing a frozen VAE latent into a low-frequency base component and a high-frequency detail component, assigning the former to semantic anchoring and the latter to fine-grained discrimination (Zhou et al., 6 May 2026).

Graph methods expand the same idea into structural scale. MSGCOT is explicitly multi-scale, treating node-, subgraph-, and coarse graph-level abstractions as a hierarchy from which prompts are generated in a coarse-to-fine chain (Zheng et al., 10 Oct 2025). MultiGPrompt organizes prompt knowledge differently: it combines multiple pretext tasks during pre-training and later separates downstream prompts into composed prompts, which recover task-specific knowledge from pretext tokens, and open prompts, which capture global inter-task knowledge (Yu et al., 2023). In low-resource scientific NLP, MPT mixes manual hard prompts and automatically learned soft prompts, so granularity is represented as complementary knowledge sources rather than spatial or semantic scales (Liu et al., 2023). In FedMGP, the decomposition is coarse-grained global prompt versus fine-grained local prompt, where the former supports shareable common cognition and the latter supports client-specific personalization and temporal retention (Yu et al., 2024).

3. Prompt construction and architectural mechanisms

The simplest multi-granularity mechanism appends multiple prompt fragments to the original input. MTPrompt follows this pattern by concatenating task-related tokens and using the prediction at [MASK] for classification. After prompt selection, the input becomes

$$\overline{x}^{i} = T_{select}(x^{i}) = x^{i}.[t_{1}][t_{2}][t_{..}][t_{N}][MASK],$$

with label verbalization handled by

$$F(y): y_{t} \rightarrow v_{t}, y_{t} \in {Y}, v_{t} \in {V}$$

(Weng et al., 2023). The method’s theoretical claim is geometric: prompt tokens change the position of task-related tokens in semantic space, and therefore affect whether the representation is moved toward a task-appropriate surface (Weng et al., 2023).

Prefix-style mechanisms dominate when prompt information must be inserted into transformer internals rather than appended as plain text. MPrompt learns task-specific prefixes and combines them with prompt-generator outputs so that encoder self-attention receives

$K'_l = [\mathcal{T}_{E_{l,K} ; \mathcal{P}_{l,K} ; K_l], \qquad V'_l = [\mathcal{T}_{E_{l,V} ; \mathcal{P}_{l,V} ; V_l],$

while decoder layers use task-specific prompt states in self-attention and cross-attention (Chen et al., 2023). FedMGP also adopts prefix-style local prompts, with the paper writing

$$\operatorname{MSA}' = \operatorname{MSA}(h_Q,[p_V;h_K],[p_V;h_V]),$$

to inject fine-grained client-specific knowledge directly into attention while keeping the pretrained ViT frozen (Yu et al., 2024).

Graph methods employ layer-wise modulation. In MultiGPrompt, each pretext task has a token set

$$_{\langle k \rangle}=\{\mathbf{t}_{\langle k\rangle,0}, \mathbf{t}_{\langle k\rangle,1}, \ldots, \mathbf{t}_{\langle k\rangle,L}\},$$

and each token reweights the hidden state through

${H}^{l+1} = MP({t}_{\langle k\rangle,l}\odot {H}^{l},\Vec{A};\theta^l).$

Downstream, composed prompts are formed as learnable compositions of frozen pretext tokens, while open prompts are freely learned, and the two outputs are aggregated before prototype-based classification (Yu et al., 2023). MSGCOT instead generates prompts recursively from a hierarchy of coarsened graphs; at reasoning step $l+1$,

$\mathbf{p}_i^{l+1} = \sum_{j}^{C^l} \alpha_{ij}^{l+1}\mathbf{t}_j^{l}, \qquad \hat{\mathbf{h}_i^{l+1} = \hat{\mathbf{h}_i^{l} + \mathbf{p}_i^{l+1},$

so prompts are not static parameters but intermediate “thoughts” derived from multi-scale structure (Zheng et al., 10 Oct 2025).

Vision-LLMs add structured language and visual-side guidance. HPT++ propagates low-level, high-level, and global-level prompts across layers of a hierarchical text encoder and injects structured relation information by reweighting attention. The improved formulation is

$$P_{i}^{all} = \left\{ P_{i}^{probability},P_{i}^{segment},P_{i}^{question},P_{i}^{vision} \right\}$$0

where related token pairs are amplified and unrelated pairs are suppressed (Wang et al., 2024). MGFFD-VLM constructs prompts directly from vision outputs. Its probability prompt is

$$P_{i}^{all} = \left\{ P_{i}^{probability},P_{i}^{segment},P_{i}^{question},P_{i}^{vision} \right\}$$1

with learnable CoOp-style vectors and textual suffixes “it is fake” or “it is real,” and its location prompt is

$$P_{i}^{all} = \left\{ P_{i}^{probability},P_{i}^{segment},P_{i}^{question},P_{i}^{vision} \right\}$$2

thereby turning classification confidence and segmentation features into language-side control signals (Chen et al., 16 Jul 2025). SpecPL similarly enriches prompt learning from the visual side, although through spectral decomposition rather than explicit textual templates. After computing

$$P_{i}^{all} = \left\{ P_{i}^{probability},P_{i}^{segment},P_{i}^{question},P_{i}^{vision} \right\}$$3

it maps the two bands into CLIP space and uses a frozen Visual Semantic Bank to refine class text features (Zhou et al., 6 May 2026).

4. Learning objectives, search procedures, and supervision

Multi-granularity prompt learning is usually coupled with nontrivial optimization schemes because the prompt decomposition must be made operational. MTPrompt explicitly supports manual and automatic prompt selection. Candidate prompts are generated from OD, SD, and TD combinations, evaluated on training data, and ranked by

$$P_{i}^{all} = \left\{ P_{i}^{probability},P_{i}^{segment},P_{i}^{question},P_{i}^{vision} \right\}$$4

so the chosen discrete prompt is the one assigning the highest probability to correct label verbalizers (Weng et al., 2023). This differs from purely continuous prompt tuning by treating prompt construction as a search problem over interpretable task metadata.

MPT adds semi-supervised pseudo-labeling to the prompt ensemble. Its prompt-tuned models, built from both manual hard templates and soft continuous prompts, are aggregated through

$$P_{i}^{all} = \left\{ P_{i}^{probability},P_{i}^{segment},P_{i}^{question},P_{i}^{vision} \right\}$$5

and only high-confidence pseudo-labels are retained to build enlarged training sets generation by generation (Liu et al., 2023). This makes multi-granularity prompting a mechanism for harvesting unlabeled scientific text rather than only adapting to a fixed labeled set.

MPrompt regularizes prompt specialization directly. Domain-specific prompts are encouraged to be distinct through a Hilbert-Schmidt Independence Criterion and its normalized CKA form,

$$P_{i}^{all} = \left\{ P_{i}^{probability},P_{i}^{segment},P_{i}^{question},P_{i}^{vision} \right\}$$6

with the total loss

$$P_{i}^{all} = \left\{ P_{i}^{probability},P_{i}^{segment},P_{i}^{question},P_{i}^{vision} \right\}$$7

The stated purpose is to prevent different domain prompts from collapsing into redundant representations (Chen et al., 2023).

Several later systems rely on auxiliary supervision aligned to prompt semantics. MGFFD-VLM is trained in three stages: first, vision-side detection and segmentation are learned while the LLM is frozen; second, projector and prompt-related parameters are trained with all DD-VQA+ QA types and a fine-grained contrastive loss

$$P_{i}^{all} = \left\{ P_{i}^{probability},P_{i}^{segment},P_{i}^{question},P_{i}^{vision} \right\}$$8

third, full instruction tuning is performed with

$$P_{i}^{all} = \left\{ P_{i}^{probability},P_{i}^{segment},P_{i}^{question},P_{i}^{vision} \right\}$$9

The text calibration term sharpens the real/fake decision at the generated authenticity phrase (Chen et al., 16 Jul 2025). SpecPL likewise combines a main CLIP-style classification loss with semantic anchoring and both factual and counterfactual granule supervision:

$$\overline{x}^{i} = T_{select}(x^{i}) = x^{i}.[t_{1}][t_{2}][t_{..}][t_{N}][MASK],$$0

By permuting high-frequency granules across samples, it forces the model to discriminate detail from semantic invariance (Zhou et al., 6 May 2026).

Graph and federated methods introduce specialized objectives tied to their own notion of granularity. MSGCOT uses a similarity-based downstream loss together with a cosine reconstruction loss to ensure that coarse-grained prompts do not overwrite node-specific information (Zheng et al., 10 Oct 2025). MultiGPrompt optimizes a weighted multi-task pre-training loss across DGI, GraphCL, and link prediction, then freezes the encoder and learns only prompt-related parameters for downstream adaptation (Yu et al., 2023). FedMGP trains global and local prompts separately and aggregates only global prompts through Selective Prompt Fusion, using a server proxy dataset and an MSE-based distillation term to align prompt pools from different clients (Yu et al., 2024). This suggests that prompt granularity is often inseparable from supervision granularity: once prompts are factorized, the loss design typically follows.

5. Representative empirical behavior

Reported results indicate that multi-granularity prompting is most consistently beneficial under data scarcity, distribution shift, or explanation-heavy settings. MTPrompt reports strong few-shot classification performance with RoBERTa-large under Gao et al.’s $$\overline{x}^{i} = T_{select}(x^{i}) = x^{i}.[t_{1}][t_{2}][t_{..}][t_{N}][MASK],$$1 protocol, reaching 85.0 on TREC, 77.2 on SNLI, 64.9 on QNLI, 50.4 on SST-5, and 91.9 on SST-2, while also showing effectiveness and stability across multiple few-shot subsets, batch sizes, and random seeds (Weng et al., 2023). Its ablation on SST-5 is notable because OD+SD obtains 50.1 accuracy, whereas OD+SD+TD yields 47.3 mean accuracy but 50.0 median, which the paper uses to argue that more metadata is not always better (Weng et al., 2023).

In low-resource academic function recognition, MPT reports an average increase of 5% in Macro-F1 score compared with fine-tuning and 6% in Macro-F1 score compared with other semi-supervised methods. The paper further gives concrete few-shot examples, such as 78.69 Macro-F1 on SciCite with $$\overline{x}^{i} = T_{select}(x^{i}) = x^{i}.[t_{1}][t_{2}][t_{..}][t_{N}][MASK],$$2 and 78.35 Macro-F1 on PMO-kw with $$\overline{x}^{i} = T_{select}(x^{i}) = x^{i}.[t_{1}][t_{2}][t_{..}][t_{N}][MASK],$$3, and states that improvements are statistically significant against strong baselines (Liu et al., 2023). MPrompt, targeting twelve QA benchmarks, reports an average improvement of 1.94% over prior state-of-the-art soft-prompt methods and average gains over Prefix-tuning of about +2.17% on UnifiedQA-Base, +1.85% on UnifiedQA-Large, and +1.82% on UnifiedQA-XL (Chen et al., 2023).

Vision-language results show a similar pattern. HPT++ reports that it improves harmonic mean by about 0.5% over HPT/CoPrompt on average in base-to-new generalization and achieves the best average target accuracy of 68.02% in cross-dataset evaluation (Wang et al., 2024). SpecPL reports harmonic-mean gains of +4.86 for CoOp, +2.00 for CoCoOp, +0.80 for MaPLe, and +0.31 for MMRL, culminating in a new performance ceiling of 81.51% harmonic-mean accuracy on MMRL + SpecPL across 11 benchmarks (Zhou et al., 6 May 2026). MGFFD-VLM shows a staged improvement on DD-VQA+: the baseline MobileVLM achieves 0.8802 Acc and 0.9249 F1; adding the Location Prompt yields 0.8954 Acc; adding both Location Prompt and Probability Prompt yields 0.9030 Acc and 0.9396 F1; and the final MGFFD-VLM with Attribute-Driven Hybrid LoRA reaches 0.9072 Acc and 0.9420 F1 (Chen et al., 16 Jul 2025). The same paper states that the location prompt improves accuracy by 1.52% over the version with only region/type augmentation and the probability prompt adds another 0.76% (Chen et al., 16 Jul 2025).

Graph and federated systems also report gains concentrated in few-shot or heterogeneous regimes. MSGCOT reports improvements of about 5–8% in node classification under extremely limited labels and consistently outperforms GCOT and other baselines on eight datasets (Zheng et al., 10 Oct 2025). MultiGPrompt reports best one-shot node-classification accuracies of 57.72 on Cora, 54.74 on Citeseer, 48.09 on PROTEINS, and 54.47 on ENZYMES, as well as best five-shot graph-classification accuracies of 60.07 on BZR, 56.17 on COX2, 56.02 on PROTEINS, and 26.63 on ENZYMES (Yu et al., 2023). It remains parameter-efficient, with 522 downstream tunable parameters versus 256 for GraphPrompt and hundreds of thousands for GCN (Yu et al., 2023). FedMGP reports best accuracy in both synchronous and asynchronous PFCL settings on CIFAR-100, with 83.46% and 90.56% respectively, and associates global prompts with spatial retention and local prompts with temporal retention (Yu et al., 2024).

6. Interpretability, limitations, and broader significance

One recurrent claim is that multi-granularity prompting improves interpretability because prompt components have explicit functional roles. MTPrompt emphasizes that each prompt part is semantically inspectable and ties its interpretability to the geometric view that prompt words reposition task-related content in representation space (Weng et al., 2023). MGFFD-VLM makes the same point in a multimodal form: the probability prompt supplies a soft class prior, the location prompt anchors explanations in spatial evidence, and the generated rationale is trained against region/type and quality descriptions rather than left as a generic free-form explanation (Chen et al., 16 Jul 2025). HPT++ argues that linguistic knowledge becomes more discriminative when coarse descriptions, fine-grained distinctions, and structured relation graphs are all modeled together, and that semantic diversity rather than semantic quantity is the important factor (Wang et al., 2024).

At the same time, the literature repeatedly notes that increasing prompt granularity introduces its own control problem. MTPrompt explicitly warns that too many task descriptors may introduce noise and that automatic and faster prompt searching remains future work (Weng et al., 2023). MPrompt identifies prompt length as critical, notes sensitivity on some reasoning-heavy datasets, and states that the framework is limited to encoder-decoder models and is not compatible with black-box APIs (Chen et al., 2023). FedMGP isolates only the global prompt for aggregation because indiscriminate fusion of fine-grained information can destroy personalization (Yu et al., 2024). SpecPL addresses a closely related tension in another vocabulary, describing prompt learning as a stability-generalization trade-off and using spectral disentanglement to separate stable semantics from discriminative detail (Zhou et al., 6 May 2026).

The broader significance of the field lies in the fact that “granularity” is becoming a general organizing principle for prompt design rather than a narrow task-specific trick. In text, it can mean task metadata, latent domains, or manual-versus-soft prompt knowledge (Weng et al., 2023, Chen et al., 2023, Liu et al., 2023). In vision-LLMs, it can refer to semantic levels, relation structure, confidence priors, spatial evidence, or spectral bands (Wang et al., 2024, Chen et al., 16 Jul 2025, Zhou et al., 6 May 2026). In graphs, it can refer to pretext-task provenance or explicit structural scale (Yu et al., 2023, Zheng et al., 10 Oct 2025). In federated continual learning, it becomes a way to partition shareable and personalizable knowledge (Yu et al., 2024). This suggests that multi-granularity prompt learning is best understood as a structured interface between pretrained representations and downstream objectives: prompt channels are separated when different kinds of evidence must be preserved, selected, or fused without collapsing into a single undifferentiated context.