Prompt-Based Classifiers: Methods & Insights

• Prompt-Based Classifiers are models that reframe traditional supervised tasks, such as sentiment analysis and image categorization, into prompt-driven inference using pre-trained foundation models.
• They employ handcrafted or learned prompt templates and verbalizers to guide next-token prediction or masked token infilling, addressing label-word bias with unsupervised reweighting.
• Empirical performance demonstrates that optimized prompt-based classifiers can match or exceed fine-tuned baselines in zero-shot, few-shot, and multimodal settings while enabling rapid task adaptation.

Prompt-based classifiers are a class of models that leverage either static or learned natural-language prompts as an interface to pre-trained or frozen foundation models such as LLMs or vision-LLMs (VLMs). These classifiers reframe traditional supervised learning tasks—such as sentiment analysis, topic identification, or image categorization—into prompt-driven inference, operating in zero-shot, few-shot, or parameter-efficient settings. The core mechanism involves transforming inputs and label spaces into prompt templates that elicit model-internal knowledge through next-token prediction, masked token infilling, or conditional generation. This paradigm enables rapid deployment across new tasks with minimal annotation, supports efficient black-box adaptation, and exposes novel axes for optimization and interpretation in both NLP and vision contexts.

1. Foundational Formulation and Mathematical Structure

The typical prompt-based classifier is defined by a tuple $Q = \{p, \{w_1, ..., w_K\}\}$, where $p(x)$ is a prompt template mapping the original input $x$ to a model-ingestible string (e.g., “What is the sentiment of: <x>?”); $w_k$ is the label-word or verbalizer token assigned to class $y_k$. Given a pre-trained LLM with next-token probability $P_e$, classification reduces to:

$$p(y_k | x, Q) = \frac{P_e(w_k \mid p(x))}{\sum_{i=1}^K P_e(w_i \mid p(x))}$$

For zero-shot or few-shot classification, prompt templates and label tokens are typically designed manually or via lightweight search. Variants for models based on masked language modeling use $[\mathrm{MASK}]$ as a place-holder and map labels to candidate tokens via a verbalizer function.

Extending to multi-class or multi-label problems, some frameworks aggregate over multiple label words per class, such that for $k$-class classification:

$$p(y \mid x) = \sum_{v \in \mathcal{S}(y)} p([\mathrm{MASK}] = v \mid \mathcal{T}(x))$$

where $\mathcal{S}(y)$ denotes the set of verbalizer tokens assigned to class $y$ (Wang et al., 2022).

In vision-LLMs (e.g., CLIP), prompts become templated class descriptors fed to the text encoder, and classification is performed via cosine similarity between image and textual feature embeddings (Qu et al., 27 Feb 2025).

2. Prompt Sensitivity, Label-Word Bias, and Unsupervised Correction

Prompt-based classifiers are highly sensitive to both prompt template and label-word selection. Liusie et al. (Liusie et al., 2023) demonstrated that even semantically equivalent prompt styles yield dramatically different output distributions—a phenomenon attributed to internal word priors of LMs, which introduce hidden class biases.

The marginal class prior under $Q$ is:

$$p(y_k \mid Q) = E_{x\sim D_{\mathrm{unlab}}}[p(y_k | x, Q)] \approx \frac{1}{N} \sum_{j=1}^N p(y_k | x^{(j)}, Q)$$

Word bias manifests when $p(y_k | Q) \neq p^*(y_k)$, with $p^*(y_k)$ typically assumed uniform unless prior knowledge suggests otherwise.

To mitigate this, an unsupervised reweighting procedure assigns class weights $\alpha_k$:

$$\widehat p(y_k | x, Q, \alpha) = \frac{\alpha_k \, p(y_k | x, Q)}{\sum_{i=1}^K \alpha_i \, p(y_i | x, Q)}$$

$\alpha$ is selected such that $E_x[\widehat p(y_k | x)] = 1/K$ for all $k$, either empirically from unlabeled data or in a zero-resource setting by normalizing with respect to the LM’s null-input word priors, i.e., $\alpha_k \propto 1/P_e(w_k | p(\varnothing))$. This yields robust, unsupervised improvement in classification performance—often matching the “oracle” performance attained by tuning thresholds on labeled data (Liusie et al., 2023).

3. Prompt Design, Verbalizer Learning, and Optimization

Prompt-based classifiers differ in mechanism according to the degree of search or learning over prompt and verbalizer space:

• Manual/discrete prompt design: Prompts and label tokens are hand-written or selected by small-scale search for interpretable mappings. Suitable for small-$K$ settings or where interpretability is paramount.
• Automatic verbalizer learning: For multi-class problems, manual label-word mapping is sub-optimal. Methods such as Mapping-Free Automatic Verbalizer (MAV) (Kho et al., 2023) replace manual assignment with a trainable, two-layer neural network that learns to map the model’s entire output vocabulary to class logits. MAV receives the full masked position output $\mathbf{v} \in \mathbb{R}^{|\mathcal{V}|}$ and outputs class probabilities via:

  $\mathbf{u} = \mathrm{Tanh}(W_1^T \mathbf{v} + b_1)$

  $\mathbf{z} = W_2^T \mathbf{u} + b_2$

  $$P(y = i | x') = \frac{\exp(z_i)}{\sum_{j=1}^K \exp(z_j)}$$

This approach enables scaling to large-$K$ tasks and systematically aggregates over vocabulary, outperforming both single- and multi-token manual verbalizers in semi-supervised few-shot learning.

• Black-box and zero-shot prompt optimization: In absence of gradient-based access, prompt optimization is achieved via techniques such as boosting ensembles (PromptBoosting (Hou et al., 2022)), pseudo-label–aligned prompt reinforcement (PAPO/PPD (Zhang et al., 4 Oct 2024)), or evolutionary strategies (ProAPO in vision (Qu et al., 27 Feb 2025)). These methods construct pools of diverse prompts or weak learners and combine them through adaptive weighting or policy-gradient updates, trading direct prompt search for ensemble robustness or joint prompt–pseudo-label refinement.

4. Empirical Performance across Domains and Task Regimes

Prompt-based classifiers have demonstrated strong empirical performance across NLP and vision benchmarks, frequently matching or surpassing fine-tuned baselines in resource-constrained, zero-, and few-shot regimes.

• In text classification, zero-shot or few-shot prompt-based LLMs (especially with few-shot exemplars) closed the gap or exceeded classical transformers across binary and multi-class software requirements datasets, with macro-F1 gains up to +0.35 over zero-shot (Binkhonain et al., 17 Sep 2025). Persona augmentation delivered minor gains, but verbose chain-of-thought instructions often degraded label assignment accuracy.
• Large-scale studies on German tweets confirmed that zero-shot prompt-based classifiers, using only human-in-the-loop guidelines, outperformed classical statistical baselines and approached the performance of domain-adapted BERT models without seeing a single labeled training example. Macro-F1, for a five-class challenging dataset, reached 0.769 for prompt-based Flan-T5, compared to 0.643 for Naive Bayes and up to 0.898 for domain-adapted BERT (Münker et al., 26 Jun 2024).
• In visual classification (CLIP/VLM), progressive automatic prompt optimization (ProAPO) (Qu et al., 27 Feb 2025) and class-aware prompt-tuning with textual knowledge embedding (TCP) (Yao et al., 2023) consistently improved base-to-new class accuracy, with TCP achieving up to 79.5% harmonic mean in 16-shot ImageNet splits—surpassing previous textual-only prompt-tuning baselines.
• For group fairness, adaptation of classical in-processing (MMD-based fine-tuning) and post-processing (FRAPP) strategies to prompt-based classifier outputs yielded the strongest reduction in group false-positive disparity, with prompt-only (suffix) interventions ineffective on their own (Atwood et al., 24 Jun 2024).

Empirical observations repeatedly underscore the volatility of naive prompt selection and verbalizer assignment, but also the capability of minimal, carefully optimized or reweighted prompt–verbalizer pipelines to rival full model retraining with a fraction of the data or computational cost.

5. Advanced Topics: Interpretability, Personalization, and Value Alignment

Beyond standard accuracy metrics, several recent advances address interpretability, inclusivity, and personalization—each exploiting the flexible, human-interpretable nature of prompts:

• Interpretable prompt optimization: The Promptimizer framework (Wang et al., 10 Oct 2025) introduces human-in-the-loop, rubric-based optimization strategies, clustering model errors into “failure patterns” and restricting edits to structured prompt components (positive/negative rubrics, in-prompt examples). This process delivers interpretable, user-controllable filters for social content, validated by higher user preference and comparable accuracy to fully automatic prompt optimization baselines.
• Dynamic value alignment: In socially sensitive NLP (e.g., hate speech, fairness), prompt-based classifiers can ingest explicit value statements as part of the input, enabling dynamic realignment at inference without retraining. Value-aligned models (VAM) (Bang et al., 2022) trained on synthetic LLM-generated data outperformed both direct few-shot LLMs and semantic augmentation, enabling per-query specification of human or group values.
• Any-modality prompt diffusion: Prompt Diffusion (Du et al., 26 Oct 2024) applies a diffusion model in prompt space, generating sample-specific prompts via a learned denoising process. This approach improves generalization under distributional shift (domain and cross-dataset robustness) and is compatible with textual, vision, and multi-modal prompt learning, achieving robust average gains of 1–3 percentage points in accuracy or harmonic mean over strong baselines.

6. Limitations, Open Problems, and Prospects

Although prompt-based classifiers enjoy broad success and practical efficiency, several limitations and open questions persist:

• Prompt/label-word brittleness: Word bias and template sensitivity remain fundamental challenges, partly mitigated by reweighting, multi-label verbalizers, or ensembling, but still problematic outside the tested class of models and tasks (Liusie et al., 2023, Wang et al., 2022).
• Scalability of discrete search: Techniques such as PromptBoosting and ProAPO become computationally intensive as the number of candidate prompts or classes grows, though evolutionary/pruning strategies partially contain this (Hou et al., 2022, Qu et al., 27 Feb 2025).
• Fairness and ethical constraints: Prompt-level interventions have minimal impact on group fairness unless combined with explicit post-processing or in-processing methods. Architectural support for dynamic control over fairness–accuracy tradeoffs remains limited (Atwood et al., 24 Jun 2024).
• Multimodal and generative extension: Most research has focused on classification; extending prompt-based learning to open-ended or dense prediction tasks is an active area. Likewise, insights about prompt sensitivity and kernelization of prompt search in text may not straightforwardly generalize to sequence generation or cross-modal retrieval.
• Cost and efficiency in practice: While parameter-efficient, large LLM API access and prompt iteration can be a limiting factor for widespread deployment, especially in settings with privacy, latency, or cost constraints (Wang et al., 10 Oct 2025).

Prospective work is likely to focus on integrating more sophisticated pseudo-labeling and self-training objectives, prompt search algorithms tailored to domain generalization, and collective or groupwise optimization protocols that exploit interpretability for both model developers and end users.

7. Representative Methods and Experimental Benchmarks (Summary Table)

Method/Class	Domain(s)	Core Mechanism	Key Results/Findings
Prior-matching (Liusie et al.) (Liusie et al., 2023)	NLP	Unsupervised class-prior reweighting	Matches oracle threshold search; reduces prompt sensitivity
AMuLaP (Wang et al., 2022)	NLP (GLUE)	Multi-label, statistics-based verbalizer selection	Zero-param approach; outperforms in-context and prompt-tuned baselines
MAV (Kho et al., 2023)	NLP multi-class	Trainable, mapping-free verbalizer	8–12% accuracy gain (semi-supervised) over manual verbalizers
PromptBoosting (Hou et al., 2022)	NLP	Black-box, boosting ensemble of weak prompt+verbalizer classifiers	Outperforms white-box fine-tuning in few-shot, 10× faster than prior black-box
ProAPO (Qu et al., 27 Feb 2025)	Vision (VLM)	Evolutionary search over template and description library	SOTA one-shot generalization, transferable to new backbones
TCP (Yao et al., 2023)	Vision (CLIP)	Textual Knowledge Embedding (class-aware prompt)	Best prior-free generalization to unseen classes, plug-and-play
Prompt Diffusion (Du et al., 26 Oct 2024)	Vision/NLP/Multi	Diffusion-driven sample-specific prompt generation	+1–3% acc/H on 11–15 domain shift and cross-dataset settings
Promptimizer (Wang et al., 10 Oct 2025)	Personal content moderation	Human-in-the-loop, rubric-structured, iterative	Interpretable, user-controllable, achieves comparable accuracy to SOTA APO

This encapsulates the essential mechanisms, empirical outcomes, and theoretical insights of prompt-based classifiers as a foundational component in contemporary machine learning across modalities.