Momentum-Guided Semantic Clustering (MGSC)

• The paper introduces MGSC as a mechanism that stabilizes prompt optimization by consolidating conflicting textual gradients through temporal decay and semantic grouping.
• MGSC maintains a decayed historical gradient pool and clusters latent textual embeddings, ensuring persistent semantic consensus for iterative prompt evolution.
• By addressing gradient conflict with focused clustering, MGSC enables robust, API-compatible prompt optimization that significantly improves task performance.

Searching arXiv for the primary paper and closely related works to ground the article. {"query":"(Yan et al., 11 Feb 2026) C-MOP Integrating Momentum and Boundary-Aware Clustering for Enhanced Prompt Evolution", "max_results": 5} {"query":"Momentum-Guided Semantic Clustering C-MOP prompt optimization MGSC arXiv", "max_results": 10} {"query":"(Yang, 20 Aug 2025) MGSC Multi-Granularity Consistency Framework Robust End-to-end ASR arXiv", "max_results": 5} Momentum-Guided Semantic Clustering (MGSC) is the gradient-refinement module inside C-MOP, a prompt optimization framework for iteratively improving a prompt $p$ for a task model $F$. In C-MOP, prompt search is formalized as

$$p^* = \arg \max_p \mathbb{E}_{(q,a)\sim D}[s(F(p, q), a)],$$

where $D=\{(q_i,a_i)\}$ is a query-answer dataset and $s(\cdot,\cdot)$ is the task metric, such as accuracy, F1, or exact match. MGSC is introduced to stabilize optimization when the update directions extracted from different batches or failure cases conflict semantically. Its central mechanism is a temporally decayed memory of textual gradients, followed by semantic clustering and weight-based selection, so that prompt evolution is guided by persistent semantic consensus rather than transient batch-specific contradictions (Yan et al., 11 Feb 2026).

1. Conceptual role in automatic prompt optimization

MGSC addresses a specific diagnosis of automatic prompt optimization: prompt updates are not ordinary differentiable gradients but “textual gradients,” namely LLM-generated natural-language critiques or improvement suggestions derived from model errors. These signals are discrete, noisy, often redundant, and can directly contradict one another. The core limitation identified in C-MOP is “textual gradient conflict and instability”: in large-scale batch optimization, “different error cases suggest opposing optimization directions,” and because textual gradients are discrete rather than numerically averaged vectors, they are “more susceptible to mutual interference” (Yan et al., 11 Feb 2026).

Within C-MOP, MGSC occupies the stage after Boundary-Aware Contrastive Sampling (BACS) and before prompt rewriting. BACS acts in example space: it selects informative examples near decision boundaries through tripartite features—Hard Negatives, Anchors, and Boundary Pairs. MGSC acts in gradient space: it refines the textual gradients produced from those examples across iterations. This division of labor is explicit in the paper’s interpretation of the ablations: BACS provides “high-quality raw signals,” whereas MGSC ensures their “stable refinement into consistent optimization directions” (Yan et al., 11 Feb 2026).

The semantic conflict targeted by MGSC is not merely stochastic noise. One batch may suggest a more cautious and extraction-focused prompt, another may push toward more explicit reasoning structure, and another may emphasize subtype-specific rules that are locally plausible but globally incompatible. Because such update directions are expressed in natural language rather than in a shared numeric parameter space, the problem is not solved by simple averaging. MGSC is therefore a mechanism for historical aggregation and semantic consolidation, rather than a direct analogue of numerical gradient momentum.

2. Formal definition and update equations

MGSC is defined through a historical gradient pool with temporally decayed weights. The paper states: “Dynamic Gradient Pool Maintenance: We maintain a historical gradient pool $G_t = \{(g_i, w_i, t_i)\}$, where each entry consists of a textual gradient $g_i$, its importance weight $w_i$, and the iteration index $t_i$ when the gradient is generated.” At iteration $t$, historical gradients are decayed according to

$F$0

This is the module’s temporal momentum rule: each textual gradient is retained after creation, but its influence fades exponentially over time (Yan et al., 11 Feb 2026).

The current iteration’s gradients are then merged with the historical pool: $F$1 MGSC subsequently partitions this unified set into $F$2 semantic clusters $F$3 based on latent embeddings. Within each cluster, it performs weight-based selection: $F$4 The paper explicitly characterizes this as “a semantic moving average”: semantically similar textual directions reinforce one another through accumulated weight rather than through coordinate-wise vector addition (Yan et al., 11 Feb 2026).

The refined gradient set used for prompt evolution is then

$F$5

That is, the top-2 gradients from each semantic cluster define the primary optimization directions for candidate prompt generation. The paper does not provide centroid-update equations, prototype formulas, or a separate MGSC clustering objective. Momentum influences which textual gradients survive and dominate inside each semantic cluster; it does not directly modify the embedding space itself (Yan et al., 11 Feb 2026).

3. Textual gradients, embeddings, and semantic consensus

The objects clustered by MGSC are textual gradients, not prompts, raw queries, or task instances. These textual gradients are the optimizer-model-generated natural-language critiques produced after BACS has selected informative examples. The optimizer is prompted to “propose three precise, atomic Textual Gradients (prompt modifications or additional constraints) that fix these errors while preserving the Anchors,” and each output is wrapped between <START> and <END>. These are the units stored in the historical pool and reclustered over time (Yan et al., 11 Feb 2026).

Clustering is performed “based on their latent embeddings.” The implementation details specify that “For text clustering, all-MiniLM-L6-v2 is used as the embedding model.” Because no separate embedding model is introduced for MGSC, the paper supports the conclusion that MGSC uses this text embedding model to encode textual gradients before clustering. However, the paper does not specify the exact textual formatting fed into the embedder, does not state a similarity function explicitly, and does not restate a distinct clustering algorithm for MGSC. It mentions “K-means clustering” explicitly for BACS, while MGSC is described as partitioning $F$6 into $F$7 semantic clusters. This suggests embedding-based hard clustering, likely analogous to the K-means used in BACS, but that implication is not explicitly formalized (Yan et al., 11 Feb 2026).

The conceptual core of MGSC is the extraction of “persistent semantic consensus.” A recurring critique that reappears across multiple iterations, even when phrased differently, accumulates weight inside a shared semantic cluster. A one-off critique fades as its weight decays. In this sense, MGSC is temporally cumulative and online rather than offline: historical gradients are accumulated, decayed, reclustered, and reselected at every iteration. It does not require access to model backpropagation gradients; it operates entirely with black-box model outputs, task scoring, and optimizer-model textual feedback. This makes it compatible with API-based or black-box prompt optimization settings (Yan et al., 11 Feb 2026).

4. Integration in the C-MOP optimization loop

Algorithmically, MGSC is integrated into C-MOP as follows. An empty gradient pool is initialized. At each iteration, the current prompt is evaluated on a batch; task instances are embedded and clustered; BACS selects anchors, hard negatives, and boundary pairs; textual gradients are generated from those examples; the existing pool is decayed; new gradients are added with initial weights; the full pool is clustered semantically; cluster weights are accumulated; the top-$F$8 gradients with highest weights are selected; those gradients are used to generate candidate prompts; and the best next prompt is chosen with UCB-based prompt selection. The pseudocode is slightly inconsistent in variable naming—$F$9 versus $$p^* = \arg \max_p \mathbb{E}_{(q,a)\sim D}[s(F(p, q), a)],$$0—but the intended procedure is clear (Yan et al., 11 Feb 2026).

The complete pipeline therefore has a strict ordering. BACS improves feedback quality by selecting informative examples in instance space. MGSC consolidates and stabilizes the resulting textual feedback in gradient space. Prompt rewriting then uses the refined gradients, and UCB performs candidate ranking or exploration-exploitation balancing. This placement is central: MGSC is neither a preprocessing step nor a final reranker, but an online inter-iteration refinement module.

The reported implementation defaults are explicit. The task model is usually Qwen3-30B-A3B-Instruct-2507, the optimizer model is Qwen3-30B-A3B-Thinking-2507, and the embedding model is all-MiniLM-L6-v2. Default optimization uses batch size $$p^* = \arg \max_p \mathbb{E}_{(q,a)\sim D}[s(F(p, q), a)],$$1, total iterations $$p^* = \arg \max_p \mathbb{E}_{(q,a)\sim D}[s(F(p, q), a)],$$2, beam size $$p^* = \arg \max_p \mathbb{E}_{(q,a)\sim D}[s(F(p, q), a)],$$3, total candidate prompts per iteration $$p^* = \arg \max_p \mathbb{E}_{(q,a)\sim D}[s(F(p, q), a)],$$4, total BACS sample quota $$p^* = \arg \max_p \mathbb{E}_{(q,a)\sim D}[s(F(p, q), a)],$$5, BACS cluster count $$p^* = \arg \max_p \mathbb{E}_{(q,a)\sim D}[s(F(p, q), a)],$$6, MGSC gradient cluster count $$p^* = \arg \max_p \mathbb{E}_{(q,a)\sim D}[s(F(p, q), a)],$$7, decay factor $$p^* = \arg \max_p \mathbb{E}_{(q,a)\sim D}[s(F(p, q), a)],$$8, and UCB exploration parameter $$p^* = \arg \max_p \mathbb{E}_{(q,a)\sim D}[s(F(p, q), a)],$$9. The paper does not quantify MGSC-specific computational overhead or memory footprint, and it does not mention pruning or pool-cap mechanisms, although it implies that historical gradients remain in the pool with decayed weights (Yan et al., 11 Feb 2026).

5. Empirical behavior and ablation evidence

The strongest evidence for MGSC comes from gradient-side ablations. Starting from zero-shot, average performance on Liar and BBH is 61.29. Using BACS alone raises this to 72.18. Adding plain momentum without semantic gradient clustering yields 71.82, which is slightly worse than BACS alone. Full BACS + MGSC reaches 75.35. The same pattern holds on individual datasets: on Liar (F1), BACS alone scores 60.38, BACS + Momentum 61.11, and BACS + MGSC 64.46; on BBH (EM), BACS alone scores 83.97, BACS + Momentum 82.53, and BACS + MGSC 86.24 (Yan et al., 11 Feb 2026).

Method	Liar (F1)	BBH (EM) / Avg
Zero-shot	—	61.29 Avg
BACS alone	60.38	83.97 / 72.18 Avg
BACS + Momentum	61.11	82.53 / 71.82 Avg
BACS + MGSC	64.46	86.24 / 75.35 Avg

These numbers are central to interpreting MGSC. They show that naive momentum is not merely insufficient but can be detrimental: the paper states that “applying momentum without secondary semantic clustering leads to a performance dip on BBH to 82.53%, an outcome even lower than the 83.97% attained by BACS alone.” The stabilizing effect therefore comes from the combination of temporal decay and semantic clustering, not from temporal accumulation alone (Yan et al., 11 Feb 2026).

A second empirical result is the interaction between instance clustering and gradient clustering. With MGSC fixed, random instance sampling scores 71.54 average, naive clustering of failure instances scores 72.01, and full BACS + MGSC scores 75.35. The paper interprets this as evidence that “BACS and MGSC are highly complementary.” Sensitivity analysis on the Liar dataset further reports that the best result, 64.46, is obtained at $D=\{(q_i,a_i)\}$0 and $D=\{(q_i,a_i)\}$1. When $D=\{(q_i,a_i)\}$2, increasing $D=\{(q_i,a_i)\}$3 gives only limited gains and can dilute the optimization signal; when $D=\{(q_i,a_i)\}$4, increasing $D=\{(q_i,a_i)\}$5 gives only marginal benefits because poor sample diversity constrains the value of gradient refinement. The paper summarizes the successful $D=\{(q_i,a_i)\}$6 regime as evidence that “high-diversity sample exploration must be coupled with high-precision gradient grouping” (Yan et al., 11 Feb 2026).

At the C-MOP level, the abstract reports that the full framework consistently outperforms baselines such as PromptWizard and ProTeGi, yielding average gains of 1.58% and 3.35%, and that it enables a general LLM with 3B activated parameters to surpass a 70B domain-specific dense LLM. These are framework-level results rather than MGSC-only measurements, but they provide the broader performance context in which MGSC functions as the stabilization component (Yan et al., 11 Feb 2026).

6. Relation to adjacent methods and acronym ambiguity

MGSC differs from ordinary numerical momentum because its objects are natural-language suggestions rather than continuous vectors. In SGD-like methods, momentum accumulates vectors in parameter space by coordinate-wise addition. In MGSC, the paper argues that such accumulation is inadequate because textual gradients are discrete and semantically interfering. It also differs from prompt ensembling and self-consistency: it does not vote over multiple prompts or multiple final answers, but consolidates recurring optimization advice across iterations.

In relation to nearby representation-learning methods, its closest analogies are partial rather than exact. “Deep Clustering by Semantic Contrastive Learning” organizes memory by pseudo-cluster and uses a momentum encoder plus semantic memory banks to perform cross-cluster instance discrimination (Huang et al., 2021). “Unsupervised Visual Representation Learning by Synchronous Momentum Grouping” updates group representatives by an EMA-like rule and uses group-level contrastive learning, but its grouping is feature-space prototype learning rather than textual-gradient consolidation (Pang et al., 2022). “Federated Momentum Contrastive Clustering” uses online and target networks with cluster-level contrastive learning, again operating on learned embeddings rather than natural-language critiques (Miao et al., 2022). “Semantic-Enhanced Image Clustering” uses CLIP-derived semantic pseudo-labels and iterative semantic center refinement, but explicitly lacks momentum guidance (Cai et al., 2022). These comparisons indicate that MGSC is unusual in making semantic clustering operate over textual optimization directions rather than visual or latent-instance features.

The acronym is also ambiguous across the literature. In “MGSC: A Multi-Granularity Consistency Framework for Robust End-to-end ASR,” MGSC stands for Multi-Granularity Soft Consistency, not Momentum-Guided Semantic Clustering. That ASR work introduces sentence-level semantic consistency and token-level alignment consistency, and explicitly does not contain a momentum-based mechanism or a semantic clustering module. The acronym overlap is coincidental and should not be conflated with the prompt-optimization module defined in C-MOP (Yang, 20 Aug 2025).

Taken in its own setting, Momentum-Guided Semantic Clustering is best understood as C-MOP’s mechanism for turning unstable natural-language feedback into stable prompt-optimization directions. It preserves historical evidence through temporal decay, groups semantically compatible critiques, and selects high-weight representatives that embody recurring cross-iteration advice. Its defining contribution is therefore not momentum alone, nor clustering alone, but the combination of the two in a discrete textual-gradient regime where ordinary numerical optimization analogies break down (Yan et al., 11 Feb 2026).