Prompt-based Classifiers

• Prompt-based classifiers are algorithms that reframe tasks into natural language prompts, leveraging large pre-trained models in zero- and few-shot settings.
• They utilize techniques like template engineering and automatic verbalizer selection (e.g., AMuLaP, MAV) to efficiently map model outputs to class labels.
• Their applications span NLP and vision-language tasks, improving robustness, reducing data dependency, and addressing fairness concerns in predictions.

Prompt-based classifiers are a class of learning algorithms that rephrase downstream classification tasks as prompts to leverage the LLMing or multi-modal alignment capabilities of large pre-trained models. Rather than learning a new classification head per downstream task, prompt-based methods exploit a model’s ability to fill in masked tokens or generate outputs in a format induced by carefully designed prompts, which act as functional templates. This paradigm has become central in both zero-shot and few-shot settings for language and vision-LLMs, where task-specific supervision is limited but large-scale pre-training enables strong task generalization. Prompt-based classifiers subsume a range of methodologies: defining templates and mapping label words, prompt engineering and optimization, composing modular instructions, automatic verbalizer construction, ensembling and boosting methods, and fairness or robustness adaptations. Their practical utility and empirical properties have been explored extensively across natural language understanding, attribute and relation classification, image recognition, adversarial robustness, and software requirements analysis.

1. Prompt Construction and Label Mapping

The core mechanism of prompt-based classification is the transformation of raw inputs into prompt-augmented queries, followed by mapping model outputs back to task labels. A prompt typically consists of a template (e.g., “The sentiment of the following review is [MASK]: ...”) fed into a large pre-trained language or vision-LLM, which produces a distribution over possible outputs (at the [MASK] position or via sequence generation).

Early prompt-based classifiers associated each output class with a single label word from the model’s vocabulary (“verbalizer”), but this mapping is rarely optimal. AMuLaP (Wang et al., 2022) generalizes this approach through a statistics-based algorithm that automatically maps each label $y$ to a set $S(y)$ of $k$ label words:

$$p(y|x) = \sum_{v \in S(y)} p(\text{[MASK]} = v ~|~ x').$$

AMuLaP builds these sets by averaging the probability distributions of the [MASK] slot for examples of each class within a few-shot training set. Each token $v$ is assigned to the class where the expected $\mathbb{E}[p(\text{[MASK]}=v|x')]$ is maximized, then the top $k$ tokens are selected. This approach is parameter-free and circumvents the need for gradient-based optimization or human label word engineering, providing strong interpretability and robustness in few-shot text classification.

Compositional prompt design has been extended in works such as Tailor (Yang et al., 2022), which represents each attribute as a pretrained continuous “prompt” vector; these vectors can be composed (e.g., concatenated) and masked in multi-attribute settings, promoting modularity and efficient adaptation.

2. Optimization Strategies and Verbalizer Learning

Prompt-based classification performance is sensitive to the choice of prompt templates and label word mappings. Instead of handcrafting verbalizers, recent methods automate verbalizer selection or learning.

Mapping-Free Automatic Verbalizer (MAV) (Kho et al., 2023) replaces manually chosen output words with a trainable module:

$$\hat{y} = \text{softmax}( W_c^\top \cdot \tanh(W_{ve}^\top \cdot \tanh(v)) ),$$

where $v$ is the MLM output at [MASK], $W_{ve}$ compresses the vocabulary dimension, and $W_c$ maps to the class space. This end-to-end optimization leverages all vocabulary logits rather than a single token, significantly improving self-training efficacy in multi-class few-shot tasks.

In black-box settings, PromptBoosting (Hou et al., 2022) avoids direct prompt space optimization by assembling a pool of discrete prompt templates and pairing each with a trained verbalizer, then ensembles their predictions using AdaBoost. Token-class assignments in the verbalizer are optimized with a closed-form solution under an $\ell_1$ loss, and boosting data weights focus future weak learners on hard examples. This strategy achieves competitive accuracy with as few as 10 forward passes per batch, improving efficiency in scenarios lacking gradient access or model internal states.

LabelPrompt (Zhang et al., 2023) addresses the challenge of mapping outputs to complex relation labels in relation classification by injecting explicit, semantically-initialized label tokens as additional vocabulary entries—these are directly linked to relation classes via projector-based verbalizers and embedded into custom prompt templates.

3. Robustness, Bias Correction, and Fairness Remediation

Prompt-based classifiers display susceptibility to word prior bias—baseline probabilities dependent on label word frequency in pre-training data—which can distort zero-shot predictions even for semantically equivalent prompt and verbalizer choices. The inherent word bias is typically manifested when a common label word (e.g. “good”) overshadows a valid alternative (“great”), biasing the classifier output regardless of the input signal (Liusie et al., 2023).

This can be mitigated with unsupervised probability reweighting:

$$P_0(y_k | x, Q, \alpha) = \frac{\alpha_k P_e(w_k | p(x))}{\sum_i \alpha_i P_e(w_i | p(x))}, \quad \alpha_k \approx 1 / P_e(w_k | \varnothing),$$

where $P_e(w_k|\varnothing)$ is the model’s word prior. This adjustment harmonizes the output prior to a target distribution (typically uniform), yielding empirical improvements of 6–25% over baseline accuracies across NLI, sentiment, and paraphrase tasks, with near-oracle performance compared to supervised threshold calibration.

For fairness, methods adapted from classic classifiers have been applied in prompt-based LLM regimes (Atwood et al., 24 Jun 2024). In-processing remediation attaches regularization penalties (e.g., Maximum Mean Discrepancy penalties on score distributions) to the fine-tuning loss, aligning false positive rates across groups. Post-processing methods combine base model logits with an “emfairening” model’s corrections, enforcing statistical parity in the output distribution even when model internals are inaccessible. Rudimentary prompt-level interventions (wrapping the input with fairness instructions) provide only modest improvement to group fairness (as measured by the FPR gap), underscoring the limits of prompt-only remediation for group-level disparities.

4. Application Domains and Evaluation Paradigms

Prompt-based classifiers have demonstrated competitive or superior results with dramatically reduced dependence on labeled data across a spectrum of applications:

• Text and Relation Classification: On GLUE, multi-label prompt learners like AMuLaP and prompt ensemble methods like PromptBoosting surpass or match full fine-tuning in few-shot regimes. In software requirements analysis, few-shot or persona-augmented prompting with LLMs matches or exceeds fine-tuned BERT on macro-F1 metrics in tasks such as functional/non-functional and security requirement discrimination (Binkhonain et al., 17 Sep 2025).
• Vision-Language and Image Recognition: Evolutionary prompt optimization (ProAPO (Qu et al., 27 Feb 2025)) augments task and class-specific prompts without human intervention, combining edit-based (add, delete, replace) and evolution-based (crossover, mutation) operations over an LLM-generated prompt pool to maximize an entropy-regularized accuracy objective. This method outperforms gradient-based tuning methods and LLM-generated description baselines on one-shot recognition across diverse datasets, and shows transferability across model backbones.
• Controlled Text Generation and Attribute Modeling: Tailor (Yang et al., 2022) demonstrates plug-and-play attribute control with continuous prompts, applicable to classification via modular label representation, compositionality, and prompt masking.
• Data Augmentation and Model Distillation: PromptMix (Sahu et al., 2023) uses LLM-driven prompt generation and relabeling to synthesize high-fidelity, challenging (class-boundary) augmentations. After LLM relabeling, these data points effectively distill the nuanced classification boundaries of massive models like GPT-3.5-turbo into smaller transformers (e.g., DistilBERT), helping close the performance gap in low-shot or zero-shot regimes.
• Security and Robustness: Embedding-based prompt classifiers (using embeddings from models like OpenAI’s text-embedding-3-small, MiniLM, or GTE) combined with tree-based ML classifiers (e.g., Random Forest, XGBoost) reliably detect prompt injection attacks, matching or exceeding state-of-the-art encoder-only neural models in AUC and F1, and revealing the value of non-linear tabular approaches for adversarial LLM input filtering (Ayub et al., 29 Oct 2024).

5. Advances in Prompt Optimization and Representation

Prompt generalization is hindered by prompt overfitting and sample-dependent variation. Recent work leverages per-sample prompt overfitting and diffusion-based refinement (Du et al., 26 Oct 2024): overfitted instance-specific prompts are used to train a diffusion model that transforms random noise into refined, customized prompts per test sample. This model is agnostic to modality (text, vision, or multi-modal), integrates with standard prompt tuning pipelines, and employs ODE-based sampling for computational efficiency. The outcome is robust generalization under distribution shift and cross-dataset adaptation.

Unsupervised prompt learning is advanced through pseudo-supervised demonstration alignment (Zhang et al., 4 Oct 2024), where high-confidence pseudo labels and associated demonstrations (selected via K-NN clustering) are used for ICL-aligned prompt optimization. A variance-reduced policy gradient estimator refines the discrete prompt sequence distributions, incorporating an entropy penalty and aligning pseudo-supervision clusters to mitigate overfitting.

Explicit label token injection (LabelPrompt (Zhang et al., 2023)) and projected embedding strategies (TCP (Yao et al., 2023)) transfer class-level knowledge directly into prompt tokens; these approaches enhance discriminability, especially in vision-LLMs, and allow rapid adaptation to unseen classes with minimal training overhead.

6. Evaluation Metrics, Benchmarking, and Theoretical Considerations

Standard evaluation often relies on macro-averaged F1, accuracy, and auxiliary metrics (precision, recall), as well as specialized measures. For prompt-based class-agnostic counting, PrACo (Ciampi et al., 24 Sep 2024) introduces prompt-aware evaluation—quantifying how well models heed prompt semantics in multi-class and distractor scenarios—using metrics such as normalized mean of negative predictions (NMN), positive class count nearness (PCCN), counting precision (CntP), and F1-aggregated count accuracy. These metrics expose deficiencies hidden by traditional counting error rates.

Theoretical connections have been established between model word priors and zero-shot output bias, formalizing how unsupervised reweighting can neutralize the impact of pre-training frequency asymmetries (Liusie et al., 2023). In prompt-optimized diffusion, the denoising objective and ODE-based refinement are formalized in the loss landscape of prompt spaces (Du et al., 26 Oct 2024).

A unifying theme, highlighted by the widespread adoption of prompt-based methods, is a shift in NLP and multi-modal learning: from task- and architecture-specific heads requiring extensive labeled data to instruction-following models adaptable via textual meta-programming. This shift dramatically reduces data annotation costs and unifies downstream task adaptation under a single text-to-text or prompt-to-prediction framework (Münker et al., 26 Jun 2024).

7. Future Directions and Open Challenges

Current prompt-based classifier advances suggest several future research avenues:

• Enhanced fairness requires moving beyond prompt-level instruction and developing instance- or population-aware regularization that operates at the embedding or output level, especially under constraints of model accessibility (Atwood et al., 24 Jun 2024).
• Prompt robustness against adversarial prompts and distribution shifts will benefit from further work in sample-specific prompt adaptation, e.g., diffusion-based or gradient-informed strategies. Embedding-based detection and policy gradient optimization present promising directions.
• Efficient and transferable prompt optimization (as in ProAPO) remains a key challenge as tasks become more fine-grained and class sets scale, with sampling and group optimization heuristics helping reduce the combinatorial explosion.
• Integrating compositional and modular prompt representations, with masking and connector patterns inspired by controlled text generation, may allow more generalizable and extensible multi-label or multi-attribute classification frameworks (Yang et al., 2022).
• A plausible implication is that continued advances in prompt automation, bias mitigation, and compositionality will further democratize AI deployment by aligning model outputs with user-specified semantics and real-world constraints, minimizing reliance on task-specific training or annotation.

Prompt-based classifiers thus represent an intersection of template-driven meta-learning, generative modeling, and task adaptation, driving the current paradigm shift in supervised and semi-supervised learning across NLP and vision-language domains.