Self-Supervised Prompt Enhancement Module (SPEM)

• SPEM is a self-supervised prompt enhancement module that optimizes prompts using unlabeled data and internal model feedback.
• It employs an iterative Optimize–Execute–Evaluate loop in language models and a PCA+K-means+MLP pipeline in vision transformers for cost-effective performance.
• Empirical results demonstrate state-of-the-art accuracy and dramatic cost savings, highlighting robustness across both textual and visual tasks.

The Self-Supervised Prompt Enhancement Module (SPEM) is a module-class methodology that occupies a central role in modern prompt-based learning frameworks for both vision and LLMs. SPEM enables the automatic discovery, generation, and optimization of prompts using only unlabeled data and self-supervised objectives. Unlike traditional prompt engineering approaches requiring ground-truth feedback or human annotation, SPEM algorithms are designed to leverage internal model assessments and data-driven consistency signals. Leading SPEM variants are instantiated for both LLMs and vision transformers (ViT), with distinct architectural, algorithmic, and mathematical underpinnings (Xiang et al., 7 Feb 2025, Xiao et al., 16 Nov 2025).

1. Objective and Problem Setting

SPEM frameworks are motivated by the need for scalable, reference-free prompt optimization that is robust across domains and data modalities. In LLM settings, well-designed prompts are essential for enhancing reasoning and aligning outputs to user requirements but existing approaches require costly iterative human-in-the-loop refinement or rely on gold-standard outputs. SPEM overcomes this barrier by employing self-supervised objectives to assess and evolve prompts without recourse to external labels (Xiang et al., 7 Feb 2025). In computer vision, specifically in cross-domain road damage detection, SPEM is utilized to mine defect-aware prompts from unlabeled target-domain images to steer a frozen vision backbone (ViT) toward improved domain-adaptive feature extraction (Xiao et al., 16 Nov 2025).

2. SPEM Algorithms in Language and Vision Models

LLM SPEM

The LLM instantiation of SPEM, also presented as Self-Supervised Prompt Optimization (SPO), is operationalized through an Optimize–Execute–Evaluate loop:

• Prompt Proposal ($\phi_\mathrm{opt}$): Proposes a new prompt $P'$ via an optimizer LLM, given the current best prompt $P^*$ and associated outputs $A^*$.
• Execution ($\phi_\mathrm{exe}$): Applies the candidate prompt to an LLM to obtain model outputs.
• Evaluation ($\phi_\mathrm{eval}$): Performs pairwise comparisons of output sets (Output-vs-Output/OvO), using an evaluator LLM to decide which prompt leads to superior outputs with respect to requirements $R$.

At each iteration, the candidate prompt $P'$ and its outputs $A'$ are compared to $P^*$ and $A^*$. The preferred prompt is selected via a majority vote over $m$ randomized pairwise judgments. The update rule is:

$$P_{t+1} = \begin{cases} P' & \text{if } \frac{1}{m}\sum_{i=1}^{m} \mathbf{1}[\phi_{\text{eval}}^{(i)}(A',A^*)=1] \ge 0.5, \ P^* & \text{otherwise.} \end{cases}$$

This procedure is entirely self-supervised, as all optimization and evaluation signals are generated by the LLMs themselves without any need for external references (Xiang et al., 7 Feb 2025).

Visual Transformer SPEM

In visual domains, as exemplified by the PROBE framework, SPEM constructs and injects defect-aware visual prompts to a frozen ViT backbone via a multi-stage process:

• Extract patch embeddings $z_i^{(0)}$ for each image from the frozen ViT.
• Apply PCA for dimensionality reduction ($D=768$ to $d'=50$).
• Perform K-means clustering (typically $K=10$) in the reduced space to discover prompt prototypes $\mathcal{C} = \{c_1,...,c_K\}$.
• Map prototypes back to the ViT embedding dimension using a shallow 2-layer MLP: $$P^t = \mathrm{MLP}_{\theta_p}(\mathcal{C}) \in \mathbb{R}^{K \times D}$$.
• Inject prompts at shallow and mid-level transformer layers (e.g., layers 0 and 6) by prepending them to the sequence of patch tokens.

The design is parameter efficient, as only the prompt MLP and small detection heads are updated during training (Xiao et al., 16 Nov 2025).

3. Mathematical Formulation and Loss Functions

LLM SPEM

Evaluation and optimization are formalized as follows:

• Output vs. Output function: $$f_\mathrm{OvO}(O_1,...,O_k) = \phi_\mathrm{eval}(\{\phi_\mathrm{exe}(Q_i,P_j)\}_{j=1..k})$$, enabling reference-free scoring.
• Binary scoring: Each pairwise comparison yields $s \in \{0,1\}$, and majority voting over $m$ shuffles mitigates possible order bias.

Vision Model SPEM

For visual prompt enhancement, three core objectives are used:

• Prompt consistency loss ($\mathcal{L}_\mathrm{prompt}$): InfoNCE-style contrastive loss measuring alignment between the final frozen backbone features $h_i^t$ and the mean of its $K$ prompts $\overline{p}_i^t$:

$$\mathcal{L}_\mathrm{prompt} = -\sum_{i=1}^B \log\frac{ \exp(\mathrm{sim}(h_i^t, \overline{p}_i^t)/\tau) }{ \sum_{j=1}^B \exp(\mathrm{sim}(h_i^t, \overline{p}_j^t)/\tau) }$$

• Domain-Aware Prompt Alignment (DAPA) loss ($\mathcal{L}_\mathrm{DAPA}$): Linear-kernel MMD loss between prompt-conditioned representations of source and target images: $$\mathcal{L}_\mathrm{DAPA} = \left\| \mathbb{E}_{x^s \sim X^s}[f_p(h^s)] - \mathbb{E}_{x^t \sim X^t}[f_p(h^t)] \right\|_2^2$$
• Total loss: $$\mathcal{L}_\mathrm{total} = \mathcal{L}_\mathrm{ssl} + \lambda_1 \mathcal{L}_\mathrm{prompt} + \lambda_2 \mathcal{L}_\mathrm{DAPA}$$, with $\lambda_1 = 1.0$, $\lambda_2 = 0.5$ used in practice (Xiao et al., 16 Nov 2025).

4. Architectural and Training Details

SPEM modules are designed for parameter and compute efficiency in both domains.

LLM Setting:

• Prompts are sequences of natural language.
• Optimizer LLM: Claude-3.5-Sonnet (GPT-4o for ablation).
• Evaluator and Executor LLM: GPT-4o-mini, temperature-controlled.
• Greedy hill-climbing over $N_\mathrm{max}=10$ iterations, with $n=3$ samples per step and $m=4$ pairwise comparisons.
• Cost per dataset is approximately \$0.15 (1.1%–5.6% of baselines).

Vision Model Setting:

• Backbone: Frozen ViT-B/16 (86M parameters).
• SPEM prompt MLP (0.5M parameters) and DAPA head (0.06M) trained alongside a detection head (2.7M).
• Prompts (K=10) injected at layers 0 and 6, derived via a PCA+K-means+MLP pipeline.
• Training proceeds for 200 epochs, AdamW optimizer, batch size 64, with SimSiam as the self-supervised backbone criterion.

Ablation studies demonstrate:

• Peak sample size $n=3$ for language tasks; smaller values cause overfitting, larger lead to evaluator context overload.
• Mid-layer prompt injection and $K=10$ prompts are optimal for vision tasks; fewer prompts lead to performance drops, more grants negligible improvement (Xiang et al., 7 Feb 2025, Xiao et al., 16 Nov 2025).

5. Empirical Results and Benchmarking

LLM Prompt Enhancement:

Experiments on closed (GPQA-Diamond, AGIEval-MATH, LIAR, WSC, BBH-Navigate) and open-ended (MT-Bench) tasks show that SPEM achieves competitive or state-of-the-art performance with dramatically reduced compute cost and samples. Closed task performance (average F1 or accuracy) and cost (\$):

Method	Avg. Perf.	Cost (\$)
APE	64.8	9.07
OPRO	66.6	4.51
PromptBreeder	64.5	4.82
TextGrad	63.9	13.14
SPO (SPEM)	66.9	0.15

Best results for model-role transferability in the BBH-Navigate setting reached accuracy 97.8 using GPT-4o-mini for all roles (Xiang et al., 7 Feb 2025).

Vision Model Prompt Enhancement:

Zero-shot and few-shot performance on road damage transfer benchmarks (mAP@50):

Dataset	CDTrans	PROBE (SSL+SPEM+DAPA)
TD-RD	87.8	90.2
CNRDD	32.5	38.1
CRDDC’22	48.2	50.3

Ablation analysis confirms that the joint use of SPEM and DAPA is essential for maximal improvement. In few-shot settings, PROBE (with SPEM) achieves comparable mAP with $\sim$5× label efficiency relative to supervised competitors (Xiao et al., 16 Nov 2025).

6. Extensions, Limitations, and Interpretations

SPEM frameworks are modular and extensible. In LLMs, $\phi_\mathrm{opt}$ and $\phi_\mathrm{eval}$ can be replaced by any LLM of sufficient capability, multi-candidate proposals can be attempted, and $m$ (number of comparisons) can be tuned for cost/performance tradeoff. The vision model design allows prompt consistency and DAPA alignment objectives to be ported to other visual domains and backbone architectures.

All SPEM approaches are strictly self-supervised in the sense of requiring no extrinsic reference signals: only unlabeled queries and the model itself are needed at training time. This enables strong domain transfer, robust performance under data shift, and superior cost-efficiency.

A plausible implication is that the SPEM methodology can be generalized across modalities, as both textual and visual variants rely on self-generated output consistency, pairwise evaluation, and modular prompt transformation networks. However, fundamental limitations arise where output signals are insufficiently informative to guide prompt improvement—such as in domains lacking coherence or where model self-critique is unreliable.

7. Core Contributions and Future Directions

SPEM delivers a unifying framework for the self-supervised discovery and optimization of prompts for both language and vision tasks. Core contributions include:

• A fully reference-free prompt optimization loop for LLMs with OvO comparison and greedy selection.
• A defect-aware visual prompt module for domain adaptation in frozen transformer backbones, paired with domain-alignment regularizers.
• Demonstrated cost savings, label efficiency, and competitive or superior accuracy in both textual and visual domains.

Future research may explore enhanced prompt diversity, prompt evolution through reinforcement learning or population-based search, or hybrid integration with semi-supervised or weakly-supervised signals. The generalization capability of SPEM across architectures, tasks, and evaluation regimes remains a significant, open research direction (Xiang et al., 7 Feb 2025, Xiao et al., 16 Nov 2025).