RLPrompt: RL for Discrete Prompt Optimization

Updated 27 October 2025

RLPrompt is a reinforcement learning framework that recasts discrete prompt design as a sequential decision process, selecting tokens based on task-specific rewards.
It employs a compact trainable MLP on a frozen language model and leverages techniques like input-specific z-score normalization and piecewise rewards to stabilize learning.
RLPrompt-optimized prompts, though often unintelligible, are highly transferable across various language models, achieving superior performance in few-shot classification and text style transfer tasks.

RLPrompt is a reinforcement-learning–based framework for optimizing discrete text prompts for large pre-trained LMs in both few-shot learning and unsupervised tasks. Distinct from soft prompt tuning methods, RLPrompt directly searches the space of human-readable token sequences without requiring access to model gradients or internal representations. By formulating prompt selection as a sequential decision process and leveraging tailored reward-engineering strategies, RLPrompt systematically explores the combinatorial space of possible prompts. Although the resulting optimized prompts often appear as unintelligible, they are surprisingly transferrable, achieving strong performance even across very different LMs. RLPrompt thus establishes a new paradigm for automatic discrete prompt optimization in black-box and constrained settings.

1. Discrete Prompt Optimization as a Sequential RL Problem

RLPrompt recasts discrete prompt design as a Markov Decision Process (MDP) where each token in the prompt is selected sequentially by an RL policy. Given a fixed prompt length $T$ and vocabulary $\mathcal{V}$ , the space of possible prompts is $|\mathcal{V}|^T$ . The formal objective is:

$\max_\theta~ \mathbb{E}_{\hat{\mathbf{z}} \sim \prod_{t=1}^T \pi_\theta(z_t|\mathbf{z}_{<t})} [ R(y_{\text{LM}}(\hat{\mathbf{z}}, x)) ]$

where $\pi_\theta$ is the policy that generates prompt tokens, $x$ is the input, $y_{\text{LM}}$ is the LM's output given the prompt and input, and $R(\cdot)$ is a task-specific reward (e.g., accuracy, style alignment).

To achieve parameter efficiency, RLPrompt introduces a compact, trainable MLP on top of a frozen small LM like distilGPT-2. The MLP maps the partial prompt context to a probability distribution over the next token. Importantly, the base LM remains frozen, and only the MLP is optimized.

2. Reward Engineering and Stabilization Methods

Learning in RLPrompt is fundamentally challenging due to two forms of stochasticity: the black-box nature of the LM and the non-stationarity of prompt-induced behaviors. Two mechanisms are introduced to stabilize learning and improve sample efficiency:

Input-Specific z-Score Normalization: Rewards are normalized by input for each prompt using a z-score, reducing variations due to intrinsic instance difficulty. The reward for prompt $z$ and input $x$ :

$\text{z-score}(z, x) = \frac{R_x(z) - \mathbb{E}_{z'}[R_x(z')]}{\text{Std}_{z'}[R_x(z')]}$

where $R_x(z)$ is the reward for prompt $z$ on input $x$ , and the expectation is over sampled prompts for $x$ .

Piecewise Reward Construction: For tasks like classification, the reward $R$ is defined as a sum of a dense signal (e.g., predicted label probability) and a sparse higher-magnitude bonus if a threshold (e.g., correct prediction) is met, preventing the policy from exploiting misspecified continuous rewards.

3. Empirical Performance and Task Coverage

RLPrompt has been empirically validated on two fronts:

Few-Shot Text Classification: Tasks include sentiment (SST-2, Yelp, MR, CR, SST-5) and topic classification (AG’s News). In a 16-shot regime, RLPrompt with a 5-token prompt achieves higher accuracy (average $\sim$ 75.8%) and lower standard deviation than manual, template, soft-prompt, in-context demonstration baselines, and enumeration-based approaches like AutoPrompt or GrIPS.
Unsupervised Text Style Transfer: RLPrompt is also applied to tasks such as sentiment flipping and Shakespearean authorship transfer using GPT-2 variants. Reward is the sum of style classifier confidence and content preservation. RLPrompt exceeds null, random, and manual prompt baselines, and competes with full fine-tuning models such as DiRR in metrics combining style, content, and fluency (e.g., BERTScore, BLEU, perplexity, human rating).

In terms of training efficiency, RLPrompt converges in a similar number of steps as soft prompt tuning, despite lacking access to gradients from the task model.

4. Model-Agnostic Prompt Transferability

A central finding is that RLPrompt-learned discrete prompts are robustly transferrable between different LMs. Prompts, although often "gibberish" and nongrammatical to humans, perform well when ported from, for example, a compact model (distilGPT-2) to a larger or structurally different LM (e.g., GPT-2-xl or RoBERTa). This indicates that the efficiency of prompt-based control may reside in features orthogonal to human syntax, exposing underlying shared structures among large LMs.

This property enables cost-saving strategies: prompts can be optimized on smaller, cheap-to-query LMs, then deployed in larger and more capable systems with retained performance.

The RLPrompt approach differs fundamentally from:

Soft Prompt Tuning: Operates in a continuous embedding space, supports gradient optimization, but results in non-interpretable and non-transferable prompts, and requires access to model weights/gradients.
Enumeration/Selection (AutoPrompt/GrIPS): Heuristic approaches using paraphrase, fill-in-the-blank, or nearest neighbor selection, do not scale in prompt space and cannot systematically explore diverse tokens/structures.
Evolutionary/Biological Algorithms (SPELL, EvoPrompt): Population dynamics and mutation-based search offer global semantic explorations, but may face convergence and stability limitations compared to RLPrompt’s reward-driven local search mechanisms (Li et al., 2023).

6. Implications, Limitations, and Future Directions

RLPrompt establishes a versatile, model-agnostic, and black-box–friendly prompting optimization framework. It achieves high performance for discrete prompt selection in both classification and conditional generation with few-shot data.

However, prompt interpretability is an outstanding issue: RLPrompt-optimized prompts are often ungrammatical, challenging their use as instructive guidance or for debugging. Ongoing research investigates regularization and search strategies (such as entropy constraints) to recover interpretable and well-formed prompts without sacrificing effectiveness (Choi et al., 20 Jul 2024, Patel et al., 2 Apr 2025). Another active area is reward design; richer or learned reward functions (including inverse RL) can better align prompt optimization with nuanced downstream objectives.

Extending RLPrompt to LMs with massive scale (e.g., GPT-3 and closed black-box APIs), and investigating the intrinsic structure of transferable, nonintuitive prompts are key future directions. There is also interest in hybridizing RLPrompt with other optimization methods (such as evolutionary search or optimal control formulations) to further scale and generalize prompt engineering (Luo et al., 2023, Kong et al., 16 Jan 2024, Chang et al., 1 Apr 2024).

In summary, RLPrompt articulates a principled path for discrete, reinforcement-learning–based prompt optimization that is robust, broadly applicable, and demonstrates unanticipated generalization properties across LLM families.