Semantic-Aware Prompt Grouping (SAPG)

• Semantic-Aware Prompt Grouping (SAPG) is a methodological pattern that clusters semantically equivalent text prompts to reduce sensitivity and improve training consistency.
• It employs techniques such as soft weighting, compatibility gating, and prompt aggregation to enhance performance in diverse tasks including medical segmentation, 3D detection, and symmetry analysis.
• By leveraging structured semantic relationships, SAPG boosts model robustness and facilitates effective task and label adaptation across various vision-language applications.

Searching arXiv for the cited SAPG-related papers to ground the article in the latest records. Semantic-Aware Prompt Grouping (SAPG) denotes a family of methods that treat prompt variability as structured semantic organization rather than as incidental wording noise. Across recent work, SAPG is instantiated in several distinct but related ways: grouping semantically equivalent text prompts that share a segmentation target in medical vision-language segmentation, partitioning scene-adaptive category vocabularies into semantic compatibility groups for open-vocabulary 3D detection, aggregating multiple object-based prompts for symmetry detection, and organizing prompts, tasks, labels, or task-conditioned prompt vectors according to semantic relatedness or compatibility (Wu et al., 6 Mar 2026, Kim et al., 23 Mar 2026, Yang et al., 19 Aug 2025, Zhang et al., 23 Mar 2025). The common principle is that prompts are not handled independently: they are grouped, weighted, or constrained according to semantic structure, and that structure is then injected into training, inference, or prompt analysis.

1. Conceptual scope and core definitions

In the most explicit formulation, SAPG organizes “multiple, semantically equivalent text prompts that describe the same anatomical target into a single prompt group” and trains a text-guided segmentation model to be both quality-aware and group-consistent (Wu et al., 6 Mar 2026). In that setting, a prompt group is a set of natural-language prompts intended to identify the same segmentation target, and all prompts in the group share an identical ground-truth mask. This reframes prompt sensitivity as a many-to-one mapping from diverse linguistic expressions to a single target.

A broader formulation appears in Group3D, where SAPG refers to a language-driven mechanism that extracts a scene-adaptive vocabulary via multimodal LLM prompting and partitions that vocabulary into semantic compatibility groups. These groups encode which category names can plausibly refer to the same physical object across views, and they act as hard merge-time constraints during multi-view 3D instance construction (Kim et al., 23 Mar 2026).

In CLIPSym, SAPG is a prompting and aggregation strategy rather than an equivalence-class training loss. Multiple object-based prompts are encoded by CLIP’s text encoder, used to FiLM-condition image features, and aggregated with learned non-negative weights. Here the grouping unit is the prompt set itself, and the objective is to integrate complementary semantic cues relevant to symmetry detection (Yang et al., 19 Aug 2025).

Several other works implement the same organizing idea without always using the term SAPG explicitly. Dynamic Task Vector Grouping selects a semantically coherent subset of source tasks by balancing Target Similarity and Knowledge Consistency in task-prompt-vector space (Zhang et al., 23 Mar 2025). AdaPromptCL groups tasks by semantic similarity in continual learning through an assign-and-refine mechanism over prompt-derived task representations (Kim et al., 2023). Multi-Label Visual Prompt Tuning groups labels into correlative and discriminative subsets and allocates group-aware prompt tokens to each subset (Ma et al., 14 Apr 2025). PromptPrism does not propose a grouping algorithm, but it provides a taxonomy and feature extraction pipeline that expose structural, semantic, and syntactic features that can support semantic-aware prompt grouping (Jeoung et al., 19 May 2025).

This diversity suggests that SAPG is best understood as a methodological pattern rather than a single algorithm. A plausible implication is that the term now spans at least four operational meanings: semantic equivalence grouping, semantic compatibility gating, multi-prompt semantic aggregation, and semantic partitioning of tasks or labels.

2. Mechanisms for forming groups

The literature differs primarily in how groups are constructed and what kind of semantic relation a group is intended to encode.

Setting	Grouping unit	Grouping criterion
Nuclei segmentation	Natural-language prompts	Same target, same ground-truth mask
Group3D	Category names in a scene vocabulary	Plausible cross-view category equivalence
CLIPSym	Object-based prompts	Diverse frequent object semantics
DTVG / AdaPromptCL / ML-VPT	Tasks, task vectors, or labels	Similarity, consistency, co-occurrence, exclusivity

In prompt group-aware nuclei segmentation, groups are template- and curation-based rather than learned via text embeddings or ontology lookup. Group construction varies linguistic completeness and specificity while keeping supervision unchanged. Two axes define the grouping protocol: prompt type, namely T1 for all-nuclei segmentation and T2 for category-specific nuclei segmentation, and prompt quality, with low, medium, and high levels. Lower-quality prompts are short and underspecified, whereas higher-quality prompts provide richer semantic context, spatial constraints, or exclusion clauses. Ablations report best performance around group size $K=3$–$4$, with diminishing returns for $K \ge 5$ (Wu et al., 6 Mar 2026).

Group3D constructs groups in two stages. First, an MLLM is queried per view to generate up to five category hypotheses, which are canonicalized and unioned into a Scene Vocabulary Memory. Second, the MLLM partitions this vocabulary into semantic compatibility groups under explicit constraints: categories may be grouped if they could plausibly refer to the same physical object across views, but not merely because they co-occur, belong to the same structure, or stand in part–whole relations. The resulting group map $g(\ell)$ defines a hard compatibility function $C(\ell_i,\ell_j)=1$ iff $g(\ell_i)=g(\ell_j)$ (Kim et al., 23 Mar 2026).

In CLIPSym, prompt construction is object-based and diversity-driven. Frequent object classes extracted from symmetry datasets are concatenated into prompts of length $K$, and a fixed global set of $M$ prompts is used for all images. The best reported configuration on DENDI reflection uses $M=25$ prompts and $K=4$ tokens per prompt (Yang et al., 19 Aug 2025).

Task- and label-level variants use learned or empirical semantics instead of prompt paraphrases. DTVG defines a Task Prompt Vector $4$0 and groups source tasks dynamically according to Target Similarity and Knowledge Consistency (Zhang et al., 23 Mar 2025). AdaPromptCL derives a normalized task semantic representation from a warmed-up prompt and forms super-groups and fine-grained groups using radius-based assignment and refinement (Kim et al., 2023). ML-VPT estimates empirical label co-occurrence, builds correlative and dis-occurrence affinity matrices, and applies spectral clustering to obtain two label partitions, one emphasizing co-occurrence and the other emphasizing mutual exclusivity (Ma et al., 14 Apr 2025).

3. Optimization patterns and computational forms

SAPG methods typically intervene at one of three points: training objectives, merge-time constraints, or feature aggregation.

In nuclei segmentation, the formal setting is defined by an image $4$1 and a prompt group $4$2 with shared mask $4$3. A text-conditioned segmentation model produces per-prompt predictions $4$4 and logits $4$5. The per-prompt segmentation loss is

$4$6

Prompt quality is induced implicitly by the negative segmentation loss,

$4$7

and soft weights are defined by

$4$8

The group regularizer is

$4$9

and the logit-level consistency term uses a shuffled single-anchor stop-gradient design,

$K \ge 5$0

The total loss is

$K \ge 5$1

with default $K \ge 5$2, $K \ge 5$3, and $K \ge 5$4 (Wu et al., 6 Mar 2026). The stop-gradient is used to prevent a mutual-reinforcement failure mode in which all prompts drift toward a poor consensus.

Group3D does not use a training loss for grouping, because the method is training-free with respect to target 3D supervision. Instead, SAPG enters as a hard gating rule during fragment merging. Given two fragments with labels $K \ge 5$5, merging is allowed only if semantic compatibility and geometric consistency both hold:

$K \ge 5$6

with $K \ge 5$7 and $K \ge 5$8 in the supplementary setting (Kim et al., 23 Mar 2026).

CLIPSym uses semantic-aware prompt grouping as a weighted multi-branch conditioning mechanism. Each prompt $K \ge 5$9 is encoded into $g(\ell)$0, used to FiLM-modulate image tokens $g(\ell)$1,

$g(\ell)$2

refined by a Transformer, and then aggregated across prompts:

$g(\ell)$3

No SAPG-specific auxiliary supervision is introduced; the whole pipeline is trained end-to-end via focal loss on the symmetry heatmap (Yang et al., 19 Aug 2025).

Task- and label-level SAPG methods use yet other optimization patterns. DTVG optimizes source-group selection by balancing

$g(\ell)$4

and within-group

$g(\ell)$5

followed by prompt-vector merging

$g(\ell)$6

at each iteration (Zhang et al., 23 Mar 2025). ML-VPT uses grouped prompt tokens in ViT blocks and a mixture-of-experts layer to convert group-aware representations into label-aware ones after semantic grouping by spectral clustering (Ma et al., 14 Apr 2025).

4. Biomedical and geometric vision instantiations

The clearest empirical demonstration of SAPG as prompt-robust training appears in text-guided nuclei segmentation. Training on PanNuke and CoNSeP with only 10% of training images, the prompt group-aware method improves over SAM3* on both T1 and T2 settings: on PanNuke (Text), 79.42 / 62.01 versus 78.45 / 48.98, corresponding to gains of +0.97 and +6.20 Dice; on CoNSeP (Text), 76.81 / 46.86 versus 75.03 / 40.89, yielding gains of +1.78 and +3.24 Dice. On six zero-shot cross-dataset all-nuclei tasks, the method reports an average Dice improvement of 2.16 points. Under prompt-quality perturbations, the largest improvement occurs for low-quality T2 prompts, where performance rises from 47.20 to 62.54, a gain of +15.34 Dice (Wu et al., 6 Mar 2026). These results support the paper’s claim that SAPG reduces sensitivity to prompt formulation and improves robustness under textual prompting.

Group3D uses SAPG in a different manner: not to stabilize paraphrases of a fixed prompt, but to constrain open-vocabulary 3D instance construction. On ScanNet20 in the pose-free setting, the semantic compatibility grouping variant achieves mAP25/mAP50 of 41.2/18.5, compared with 35.9/14.8 for “Same Category only” and 28.2/9.9 for geometry-only merging. Similar gains hold across ScanNet60, ScanNet200, and ARKitScenes, in both pose-known and pose-free regimes, and the paper attributes especially strong benefits to the semantically gated merging under reconstruction noise and incomplete geometry (Kim et al., 23 Mar 2026).

CLIPSym demonstrates SAPG as semantic cue aggregation for symmetry detection. Single literal prompts such as “reflection axis” and “symmetry axes in the image” yield approximately 64.4% and 64.8% F1 on DENDI-reflection, while a single long prompt built from 25 frequent objects yields 65.8%. The multi-prompt SAPG configuration with $g(\ell)$7 and $g(\ell)$8 reaches approximately 66.5% and corresponds to the best reported DENDI-reflection result for the CLIPSym equivariant model, which also surpasses EquiSym* on DENDI-rotation, SDRW, and LDRS (Yang et al., 19 Aug 2025). This indicates that grouped prompt diversity, rather than a single semantic phrase, is the effective signal carrier in that setting.

A related but architecturally distinct nuclei application appears in the Prompt-based Grouping Transformer. There, learnable group embeddings are simultaneously used as grouping tokens in a hierarchical classifier and as prompt tokens injected into every Swin window. The model reports state-of-the-art results on CoNSeP, BRCA-M2C, and Lizard; in prompt-tuning mode with the backbone frozen, it trains 15.379M parameters instead of 102.2M for full fine-tuning while maintaining nearly identical performance, and sharing embeddings between prompts and groups outperforms a detached variant by +2.4% $g(\ell)$9 and +3.1% $C(\ell_i,\ell_j)=1$0 (Huang et al., 2023).

5. Task-level, label-level, and continual-learning variants

At the task level, DTVG treats semantic-aware grouping as dynamic source-task subset selection for prompt transfer. The method replaces soft-prompt similarity with Task Prompt Vectors and recomputes source-target similarity at every training step, reflecting the observation that the “most similar” source may change during fine-tuning. On GLUE and SuperGLUE, DTVG reports the highest average performance with minimal parameters, including GLUE average 87.4 with 0.035% of full fine-tuning parameters and SuperGLUE average 78.9. The TPV-based transfer result on SuperGLUE averages 76.4, compared with 72.6 for SPoT and 72.1 for prompt tuning, and the method generalizes to Llama-3.2-1B and 3.2-3B as well (Zhang et al., 23 Mar 2025). Here SAPG is not about grouping natural-language prompt strings, but about grouping prompt-derived task representations according to semantic alignment and internal consistency.

In continual learning, AdaPromptCL uses an assign-and-refine semantic grouping mechanism over tasks with mixed mild, moderate, and abrupt semantic shifts. The method maintains semantic super-groups and fine-grained groups, guided by a radius threshold $C(\ell_i,\ell_j)=1$1 and a refinement radius $C(\ell_i,\ell_j)=1$2 with $C(\ell_i,\ell_j)=1$3. It reports up to 21.3% relative improvement in last accuracy over LAE on VTAB-19T, and the average number of prompts adapts to shift severity, using approximately one prompt on ImageNet-R and approximately eighteen prompts on VTAB-19T (Kim et al., 2023). This makes SAPG a mechanism for balancing shared prompt reuse against specialization under nonstationary task streams.

In multi-label classification, ML-VPT instantiates semantic-aware grouping through dual label partitions: one correlative and one discriminative. Labels are grouped by spectral clustering over affinity matrices derived from empirical co-occurrence and dis-occurrence statistics, and each group receives multiple prompt tokens within ViT. Group-aware representations are later routed to label-aware ones through a mixture-of-experts layer. On MS-COCO, PASCAL-VOC 2007, NUS-WIDE, and Visual Genome 256, the full method achieves competitive results and outperforms recent VPT variants; for example, on DINOv2/B at 448 resolution it reports mAP 90.6 on MS-COCO and 97.0 on PASCAL-VOC 2007 (Ma et al., 14 Apr 2025).

These task- and label-centric formulations broaden SAPG beyond literal prompt paraphrases. A plausible implication is that the essential object being grouped is often not the surface prompt itself, but a semantically structured proxy: a task vector, a task embedding, or a label-relation partition.

6. Taxonomy, interpretability, and recurring limitations

PromptPrism provides a linguistically grounded decomposition that can support SAPG by exposing semantic components, role structure, and syntactic patterns. It formalizes a prompt as an ordered sequence of role-content pairs $C(\ell_i,\ell_j)=1$4, defines semantic component categories such as Instruction, Contextual/Reference information, Output Constraints, Tools, and User request / Question, and extracts component spans, delimiter types, directive markers, and other structural metadata (Jeoung et al., 19 May 2025). The paper does not define a clustering objective or similarity function, but it explicitly states that these extracted features can be used to group prompts according to semantically meaningful components and their organization.

Its controlled sensitivity analysis is especially relevant for SAPG design. On Task067, semantic reordering of components has statistically significant effects on Rouge-L, whereas delimiter modifications are not statistically significant. For Claude-Sonnet-3.5, moving Instruction to the last position improves Rouge-L from 56.49 to 63.37, while moving Request Query to the middle position reduces it by 76%; for LLaMA3.2-3b-instruct, placing Request Query last raises Rouge-L from 47.21 to 58.58 (Jeoung et al., 19 May 2025). This suggests that semantic organization, not merely lexical content, should be treated as a primary grouping signal.

Across the surveyed work, several limitations recur. Prompt group-aware segmentation assumes that all prompts within a group refer to the same target; if semantically dissimilar prompts are grouped together, gradients may conflict and harm convergence, and the method depends on the availability of multiple semantically equivalent prompts during training (Wu et al., 6 Mar 2026). Group3D inherits failure modes from both semantics and geometry: ambiguous or overly broad groups can permit incorrect merges, while reconstruction noise can still leave sparse fragments unmerged (Kim et al., 23 Mar 2026). CLIPSym notes an object-centric bias: fixed prompt sets may favor typical symmetric objects and may miss unusual or fine-grained symmetries in cluttered scenes (Yang et al., 19 Aug 2025). DTVG reports extra per-step overhead for dynamic grouping, though the added computation is gradient-free and the paper measures only 8.7% slower training than prompt tuning in one appendix example on RTE (Zhang et al., 23 Mar 2025). PromptPrism explicitly notes that it does not itself define grouping algorithms, similarity metrics, or clustering evaluations (Jeoung et al., 19 May 2025).

Taken together, these works show that SAPG is not a single canonical recipe. It is a design principle for making prompt-conditioned systems respect semantic structure at the level most relevant to the task: target-equivalent prompts, scene labels, object descriptors, source tasks, task streams, or correlated label subsets. The present literature consistently links that principle to improved robustness, reduced negative transfer, or better semantic control, but it also shows that effectiveness depends critically on how groups are defined and on whether the assumed semantic relation truly matches the downstream objective (Wu et al., 6 Mar 2026, Kim et al., 23 Mar 2026).