Optimal Prompt Learning (OPL)

• Optimal Prompt Learning (OPL) is a framework that adapts frozen models by optimizing prompt interfaces, including discrete, continuous, and hybrid forms.
• It leverages various algorithmic paradigms such as adversarial, reinforcement, and Bayesian methods to optimize prompts at both task-level and query-specific granularity.
• Empirical findings show that OPL methods achieve notable performance gains and efficiency improvements in language, vision, and multimodal applications under limited data conditions.

Optimal Prompt Learning (OPL) denotes a family of prompt-optimization formulations in which a frozen foundation model is adapted by optimizing prompts rather than model parameters. In one formulation, OPL asks: given a frozen LLM, how can we find the best discrete prompt, consisting of an instruction and (k) exemplars, under limited labeled data and compute; in Visual In-Context Learning (VICL), OPL is formulated as selecting a demonstration subset (P\subseteq S) that minimizes validation loss; in vision-language models, OPL also appears as learning multiple continuous prompts aligned to local visual features by optimal transport [2312.02614, 2501.08841, 2210.01253].

1. Conceptual scope

Across recent work, OPL is not a single algorithm but a problem class. The optimized object may be a discrete natural-language instruction, a set of in-context demonstrations, an ordering over examples, a role-playing system prompt, a query-dependent prompt generated by a smaller policy model, a prompt pool selected by a lightweight controller, or a set of continuous prompt embeddings attached to a frozen encoder. The common constraint is that the backbone model remains frozen or nearly frozen, while the prompt interface becomes the locus of adaptation [2312.02614, 2408.10504, 2506.02480].

The term also spans several granularity levels. Some methods seek a single task-level prompt for an entire dataset, as in VICL task-level prompting. Others explicitly pursue query-specific prompts, arguing that task-level optimization overlooks query-preferred prompts. Still others treat prompt selection as instance-dependent routing over a fixed prompt pool, or as a sequential refinement process under a limited evaluation budget [2501.08841, 2408.10504, 2308.07272, 2501.03508].

A recurrent motivation is the trade-off between prompt quality and optimization cost. Prior work cited in these papers includes heuristic re-scoring, soft prompts requiring back-propagation, reinforcement-learning-based methods with additional policy networks or reward models, and manual prompt engineering by trial and error. OPL methods therefore differ not only in what they optimize, but in how they exploit labeled data, feedback, and search structure [2312.02614, 2411.14479, 2501.03508].

2. Mathematical formulations

A generic discrete OPL formulation for language models defines prompt parameters (P) as the exact wording of the instruction and demonstration pairs, with the objective of maximizing end-task performance when the prompt is fed to the LLM in an in-context fashion. In adversarial in-context learning, this becomes a minimax problem over generator and discriminator prompts:
[

\mathcal{J}(D_V, G_U)

\mathbb{E}{(x,y)\sim \mathcal{D}}\big[\log D_V(x,y)\big]
+
\mathbb{E}{x\sim \mathcal{D}}\big[\log(1 - D_V(x,\,G_U(x)))\big],
]
with
[
\min_U \max_V \mathcal{J}(D_V, G_U).
]
Here the prompts (U) and (V) replace parameter updates: the generator prompt is optimized to lower the adversarial objective, while the discriminator prompt is optimized to raise it [2312.02614].

In VICL, the prompt is a (K)-shot demonstration set (P\subseteq S), where (S={(x_1,y_1),\dots,(x_N,y_N)}) is a labeled validation pool and (f(P,x_q)\to \hat y_q) is a pre-trained Vision Foundation Model. The sample-level objective seeks
[
P_q^*=\arg\min_{P\subseteq S}\mathcal{L}(f(P,x_q),y_q),
]
but because (y_q) is unknown at test time, prior work relies on a scoring function (g(x,x_q)) to select demonstrations. The task-level reformulation instead defines a single prompt
[
P^*=\arg\min_{P\subseteq S}\sum_{(x_q,y_q)\in S-P}\mathcal{L}(f(P,x_q),y_q),
]
thereby replacing per-query prompt search with average-loss minimization on a held-out validation split [2501.08841].

Query-dependent Prompt Optimization casts prompt generation as a single-step MDP (\mathcal{M}=(\mathcal{S},\mathcal{A},f,R)), where states are natural-language queries and actions are prompts. The policy factorizes autoregressively over prompt tokens, and the reward combines query-level and task-level components. Training uses an offline objective
[
\mathcal{L}=\mathcal{L}{\rm ll}+\lambda \mathcal{L}_r,\qquad \lambda=0.1,
]
where (\mathcal{L}{\rm ll}) is the prompt log-likelihood under teacher forcing and (\mathcal{L}_r) is reward-prediction MSE [2408.10504].

A different formalization appears in sequential optimal learning for automated prompt engineering. There, prompts are encoded as a (D)-dimensional feature vector (x\in\mathbb{R}^D), the scalar utility is the logit-transform of the observed score, and a Bayesian regression posterior defines a knowledge state (S_n=(\theta_n,\Sigma_n,a_n,b_n)). The acquisition function is the forward-looking Knowledge-Gradient
[
\nu_x^n = \mathbb{E}[V_N(S_{n+1})\mid S_n,x]-V_N(S_n),
]
and the next prompt is selected by (\pi^{KG}(S_n)=\arg\max_{x\in\mathcal{X}} \nu_x^n). This turns prompt search into sequential optimal learning under a finite evaluation budget [2501.03508].

3. Algorithmic paradigms

Representative OPL methods span adversarial optimization, reinforcement learning, Bayesian sequential decision-making, and structured prompt selection [2312.02614, 2411.14479, 2308.07272, 2408.10504, 2501.03508, 2506.02480, 2603.21877, 2605.19102].

Method	Optimized prompt object	Optimization mechanism
adv-ICL	Discrete generator and discriminator prompts	Two-player minimax game with a prompt modifier
GRL-Prompt	In-context examples and their order	RL over a knowledge-graph state with HGT, ICMN, and PEC
(DP_2O)	Readable discrete prompt set and input-specific prompt choice	GPT-4 dialogue generation, SUE screening, and policy-gradient matching
QPO	Query-specific natural-language prompt	Multi-loop offline RL with dataset augmentation
ORPP	Role-playing system prompt	Iterative optimization on a small subset plus few-shot transfer
SOPL-KG	Feature-based prompt vector	Bayesian regression with Knowledge-Gradient and MISOCP
P(^2)O	Prompt templates for hard samples	GEPA prompt evolution plus policy optimization
PPO code prompting	Refined code-generation prompt	PPO with direct, lexical-mutation, and semantic-rewrite actions

Adversarial and prompt-editing methods treat prompt search as black-box optimization over discrete text. adv-ICL employs one LLM as generator, another as discriminator, and a third as prompt modifier. Each iteration uses (m=5) samples, (r=5) candidates per edit, and (T=3) rounds, and the method updates only prompts rather than model parameters [2312.02614]. ORPP restricts the search space to role-playing descriptions, iteratively optimizes prompts on a small subset (Q_{\mathrm{sub}}), and then transfers the optimization experience by using the top (m) optimized prompt-question pairs as few-shot exemplars for the remaining questions; its reported hyperparameters are (N=10) optimization rounds, (k=3) candidate prompts per round, and (m=3) few-shot exemplars [2506.02480].

RL-based OPL methods differ in the state and feedback they expose to the optimizer. GRL-Prompt constructs a knowledge graph (G(q)=(V,E,R)) with a query node and candidate nodes, uses a two-layer Heterogeneous Graph Transformer to obtain contextualized node embeddings, and factorizes the joint policy into an In-Context Matching Network for selection and a Pairwise Edge Classifier for ordering. Its reward is (R(a,\hat a)=\lambda R_m(a,\hat a)+(1-\lambda)R_e(a,\hat a)), combining fuzzy textual similarity with cosine similarity between sentence embeddings [2411.14479]. (DP_2O) generates a readable candidate prompt pool by multi-round dialogue with GPT-4, scores candidates with the SUE metric, and trains a two-layer MLP policy network with only (\sim 0.62) M new parameters, approximately (0.6\%-0.7\%) of RoBERTa-Large’s 354 M [2308.07272]. QPO instead learns a small GPT-2 policy entirely from offline prompting demonstration data and bootstraps both the dataset and the policy over (T=4) loops [2408.10504].

Joint policy-prompt methods optimize prompts to improve exploration or dense supervision. P(^2)O introduces a finite set of discrete prompt templates (Z={z_1,\dots,z_M}\cup{\epsilon}), identifies hard samples when empirical success falls below a near-zero threshold, evolves templates via Genetic-Pareto prompt optimization, and updates model parameters with context distillation so that prompt-induced reasoning gains are internalized by the policy [2603.21877]. In code generation, prompt refinement is treated as a finite-horizon MDP with a hybrid action space ({\mathtt{Direct},\mathtt{LexMut},\mathtt{SemRew}}), MiniLM prompt embeddings as states, and PPO as the optimizer [2605.19102].

4. Vision and multimodal instantiations

Visual OPL introduces a distinct question: whether prompts should be optimized per query or once per task. In task-level VICL, more than (27\%) of test samples achieve their best performance under the same prompt (P^*), whereas sample-level search methods such as UnsupPR and SupPR find the “optimal” prompt for approximately (15\%) of samples. This motivates replacing per-sample search with task-level search. The Top-(K) strategy computes one-shot scores for each candidate and selects the top (K) examples, with (O(N)) forward-passes or (O(N\log N)) including sorting; the Greedy strategy iteratively adds the example whose inclusion most lowers validation loss and has worst-case (O(N^2)) forward-passes [2501.08841].

The empirical effect is substantial. On single-object detection, Greedy achieves (28.25) mIoU versus the best sample-level method at approximately (24.0). On foreground segmentation, Greedy reaches (36.76) average mIoU versus the best sample-level method at approximately (35.0). On colorization, Greedy obtains (61.56) MSE ((\times 100)), matching the Oracle upper bound. Across tasks, Greedy comes within (3\%-6\%) of the Oracle while using less than (2\%) of the search time of sample-level methods [2501.08841].

In vision-language models, PLOT uses OPL in a different sense: learning multiple continuous prompts per class and aligning them to local visual features by optimal transport. Visual and prompt features are modeled as discrete distributions, the cost matrix is defined by (C_{m,n}=1-\mathrm{sim}(f_m,g_n)), and the entropically regularized OT distance is optimized by Sinkhorn iterations in an inner loop, while supervised classification loss updates the prompt vectors in an outer loop. With (N=4) prompts, local features from RN50’s last attention pooling, and (\lambda=0.1), PLOT improves the average over 11 few-shot datasets from CoOp’s (59.6\%, 61.8\%, 66.5\%, 69.9\%, 73.3\%) to (62.6\%, 65.2\%, 68.6\%, 71.2\%, 73.9\%) at shots (1,2,4,8,16), corresponding to gains of (+3.0\%, +3.4\%, +2.1\%, +1.3\%, +0.6\%) [2210.01253].

RPLKG emphasizes interpretability and efficiency rather than continuous prompt optimization. It converts ConceptNet triplets into human-readable prompts, caches CLIP text and image embeddings after a one-time forward pass, and learns only a small Gumbel-Softmax selection head with approximately (0.79) M parameters, less than (1\%) of CLIP. Reported efficiency numbers are training time per step of approximately (0.006) s versus CoOp’s (0.06) s and CoCoOp’s (1.1) s, and peak GPU memory of approximately (0.65) GB versus approximately (1) GB to (45) GB [2304.10805].

5. Empirical findings and efficiency patterns

Several language-side OPL methods report gains under strict data or compute constraints. adv-ICL is evaluated on 13 benchmarks across generation, classification, reasoning, MMLU, and BIG-Bench Hard. Relative to few-shot prompting, average absolute gains are (+2.3) ROUGE-L, (+4.0) percentage points, and (+2.7) percentage points for Text-Davinci-002; (+2.9) ROUGE-L, (+2.9) percentage points, and (+3.2) percentage points for Vicuna-13B; and (+1.2) ROUGE-L, (+0.8) percentage points, and (+2.4) percentage points for ChatGPT. On MMLU with 5-shot ChatGPT, the average rises from (71.0) to (74.0); on BBH with 3-shot CoT ChatGPT, performance rises from (68.2) to (70.6) [2312.02614].

(DP_2O) reports (91.80\%) average accuracy on four sentiment datasets versus RLPrompt’s (90.28\%), a (+1.52\%) gain, while training only (0.62) M new parameters and requiring (23.8) minutes on SST-2, which is (10.9\%) of RLPrompt’s (218.6) minutes on a single RTX 3090 [2308.07272]. QPO reports average accuracy across six NLU tasks of (70.9\%) in zero-shot versus Prompt-OIRL’s (68.5\%), (74.2\%) in 6-shot versus (72.1\%), and (69.8\%) in 3-shot versus the best baseline at (67.1\%); on math reasoning it reports (88.6\%) versus Prompt-OIRL’s (85.6\%), while using (10\times)-(20\times) lower LLM-inference cost [2408.10504].

Structured and role-based methods also show consistent gains. GRL-Prompt outperforms recent state-of-the-art methods with average improvements of (0.10) ROUGE-1, (0.07) ROUGE-2, (0.07) ROUGE-L, and (0.05) BLEU, and its ablations report drops of (0.06) ROUGE-1, (0.05) ROUGE-2, (0.04) ROUGE-L, and (0.03) BLEU without the knowledge graph, plus further drops without RL [2411.14479]. ORPP reports, for example, on Qwen-32B, (49.49) on GPQA versus the base model’s (42.93), (67.45) on MATH versus (65.33), and (76.86) on MMLU-Pro versus (75.62); it also reports that ORPP + CoT often yields additional gains, including (+2.17\%) on MMLU-Pro for 14B and (+2.35\%) for 32B [2506.02480].

Prompt optimization can also be coupled directly to policy learning. P(^2)O reports DeepMath-5K performance rising from a GRPO baseline of (54.8\%) to (61.7\%) for P(^2)O(Self-Ref), and DeepScaler-5K performance rising from (60.5\%) to (65.2\%) for P(^2)O(Teacher-Ref), corresponding to (+4.7\%) on the out-of-distribution benchmark; on AIME24 and AIME25 it reports gains of (+12.9\%) and (+11.7\%) over GRPO [2603.21877]. In code generation, PPO-based prompt optimization achieves strict Pass@1 on the 500-task MBPP+ test set of (57.58\%), (64.80\%), and (85.50\%) for CodeT5+, CodeLLaMA, and DeepSeek-Coder, respectively, with corresponding soft Pass@1 of (67.90\%), (73.10\%), and (88.20\%) [2605.19102].

6. Limitations, debates, and open directions

A central debate concerns whether optimal prompts are task-level or query-level. The VICL results suggest that searching sample-level prompts can be redundant because many queries share the same best prompt, and Greedy task-level search reaches near-oracle performance at a fraction of the search cost [2501.08841]. QPO argues the opposite for language prompting: most existing prompt optimization methods only focus on task-level performance, overlooking the importance of query-preferred prompts, which leads to suboptimal performances [2408.10504]. These results do not strictly contradict one another; rather, they suggest that the appropriate granularity of OPL depends on modality, task structure, and the geometry of the prompt-response landscape.

Another misconception is that prompt optimization concerns prompt wording alone. The surveyed methods optimize example selection, example ordering, role framing, prompt pools, continuous prompt embeddings, prompt features, and prompt templates for hard-sample exploration. This suggests that “prompt” in OPL is best understood as an interface variable between data and a frozen model, not merely as a sentence prefix [2411.14479, 2210.01253, 2501.03508, 2603.21877].

The limitations reported across papers are also heterogeneous. VICL task-level OPL requires a labeled validation pool (S), depends on the representativeness of (S), and offers no formal convergence guarantee for the Greedy procedure [2501.08841]. adv-ICL relies on the prompt modifier’s paraphrasing quality and on balancing generator and discriminator capacity; as a black-box method, it has no formal guarantee of global optimum in finite time [2312.02614]. GRL-Prompt uses purely automatic reward based on ROUGE, BLEU, and embedding similarity, which may not align with human preference [2411.14479]. ORPP depends on a separate reward model, and some prompt combinations can hurt performance [2506.02480]. Sequential optimal learning requires manual feature construction and may face expensive MISOCP solves as the feature dimension grows [2501.03508]. PPO-based code prompting incurs high computational cost because each step involves multiple LLM generations and sandboxed execution [2605.19102].

The research directions proposed in these works are convergent. They include scaling to multimodal or vision-language settings, integrating human feedback or RLHF-style supervision, jointly optimizing prompt length and content, learning adaptive numbers of prompts per class, extending from discrete prompts to continuous or hybrid prompt spaces, and developing stronger theory for convergence and sample efficiency under discrete search [2312.02614, 2210.01253, 2411.14479, 2506.02480, 2501.03508]. A plausible implication is that future OPL systems will combine structured prompt representations, cheap proxy models or reward models, and explicit budget-aware search, rather than relying on any single prompt-engineering heuristic.