Prompt Duel Optimizer Framework

• Prompt Duel Optimizer is a framework for automatic prompt optimization that employs pairwise comparisons to identify top-performing prompts.
• It utilizes advanced methods like dueling-bandit algorithms, Bayesian optimization, and mutation strategies to reduce reliance on costly ground-truth labels.
• Empirical results show significant accuracy gains and cost efficiency, making PDO effective across diverse benchmarks in LLM prompt engineering.

A Prompt Duel Optimizer (PDO) is a framework for automatic prompt optimization in scenarios where only pairwise comparisons between candidate prompts are available, typically for optimizing natural language prompts for LLMs. PDOs reduce the reliance on costly ground-truth labels by leveraging ordinal (preference-based) feedback, using dueling-bandit, Bayesian optimization, or related online learning methodologies. These optimizers are particularly crucial for efficient, scalable prompt engineering, label-free evaluation, or scenarios in which direct function measurement is impractical or infeasible (Wu et al., 14 Oct 2025, Xiang et al., 7 Feb 2025, Gonzalez et al., 2017, Xu et al., 2019).

1. Mathematical Formulation and Problem Statement

The canonical PDO problem is cast as black-box optimization over a prompt space $P$ (finite or infinite), for an objective $F$ that cannot be measured directly but can be compared via pairwise duels. The central objects are:

• Candidate prompts $P = \{p_1,\ldots,p_K\}$ in the pool (with $K$ potentially growing via mutation).
• Input space $X = \{x_1,\ldots,x_n\}$ (task queries), and output function $f_p(x)$ (LLM output for prompt $p$ and input $x$).
• Unknown “win rate” or preference function $\mu(i,j)=\Pr[p_i \succ p_j]$, given by the probability $p_i$ is judged superior to $p_j$ over $X$ as evaluated by an LLM judge or human.
• The Copeland score $C(i) = \sum_{j\neq i} 1\{\mu(i,j) > 0.5\}$ (normalized as $\zeta_i = C(i)/(K-1)$) as a practical optimality criterion in the absence of a Condorcet winner (Wu et al., 14 Oct 2025).
• The optimization goal is to identify a prompt $p^* = \arg\max_{p\in P} E_{x\sim D}[F(M(p,x),x)]$, where the signal $F$ is accessible only indirectly via pairwise duels (Xiang et al., 7 Feb 2025).

The most common feedback is ordinal: for each comparison (duel) between prompts on a sampled $x$, an evaluator returns which prompt is better (binary or scalar, but in practice usually binary). Regret is defined in terms of the gap in Copeland score between the best pair in the round and global optimum: $r_t = \zeta^* - \max\{\zeta_{a_t}, \zeta_{b_t}\}$, with cumulative regret $R_T = \sum_{t=1}^T r_t$ to be minimized (Wu et al., 14 Oct 2025).

2. Core Algorithms and Sampling Strategies

2.1 Double Thompson Sampling (D-TS) and Dueling Bandits

Prompt Duel Optimization frequently adopts dueling-bandit algorithms. In label-free prompt optimization, D-TS is used to efficiently allocate duel queries to informative pairs. For each pair $(i, j)$, the algorithm:

• Tracks $W_{ij}$ (wins of $i$ over $j$), $N_{ij}$ (duel counts), and models $\theta_{ij} \sim \mathrm{Beta}(W_{ij}+1, W_{ji}+1)$.
• Computes upper/lower confidence bounds to define the "optimistic Copeland set".
• Uses two-stage sampling to select a challenger $i^*$ from the Copeland-maximizers and an opponent $j^*$ among rivals with uncertain comparisons.
• Updates statistics after batch duels on sampled $x_k$ (Wu et al., 14 Oct 2025).

This method yields $O(K^2 \log T)$ cumulative regret, where per-round regret vanishes asymptotically and Copeland winners are identified.

2.2 Prompt Mutation and Population Management

Given the combinatorial nature of prompt space, D-TS is augmented with "top-performer guided mutation". At regular intervals:

• The bottom-performers (by Copeland score) are pruned.
• Champions are mutated using LLM meta-prompts (e.g., by expansion, minimal change, few-shot addition, or emphasis).
• Mutants are added to the pool and explored, based on the hypothesis of local Lipschitzness in prompt performance (Wu et al., 14 Oct 2025).

2.3 Preferential Bayesian Optimization (PBO) and Gaussian Processes

An alternative, fully Bayesian PDO approach is Preferential Bayesian Optimization:

• Uses a Gaussian process prior $f \sim \mathcal{GP}(0, k(\cdot,\cdot))$ over prompt embeddings.
• Observes only pairwise preference labels $y\sim \mathrm{Bernoulli}(\sigma(f(x_1)-f(x_2)))$ (with $\sigma$ the logistic link).
• Updates posterior over $f$ using Laplace approximation or EP.
• Employs dueling acquisition functions: Dueling Expected Improvement (D-EI), Dueling Probability of Improvement (D-PI), and Dueling Upper Confidence Bound (D-UCB) to select next prompt pairs.
• Returns the Condorcet or Copeland winner as the optimal prompt (Gonzalez et al., 2017).

This approach requires 10–200 comparisons in typical scenarios to surface top-performing prompts, an order of magnitude reduction compared to direct “pointwise” evaluation.

2.4 Mixed-Feedback and Dueling-Choice Bandits

Some frameworks blend direct and comparative feedback via specialized bandit algorithms (e.g., COMP-GP-UCB):

• Leverages both expensive labels and cheap duels, with duels localizing the feasible region and labels providing fine optimization.
• Maintains GP posteriors for Borda probability from comparisons and for function values from direct queries.
• Two-phase strategy: use comparison UCB to localize, then label-UCB over the filtered set for efficient exploitation (Xu et al., 2019).

3. Practical Implementation Pipeline

Exemplar implementations follow these stages:

1. Prompt Pool Initialization: Generate a diverse seed set of 20–50 prompts via LLM sampling or manual curation (Wu et al., 14 Oct 2025).
2. Pairwise Duel Evaluation: For each round, select promising prompt pairs using Bayesian or bandit criteria and query preference via LLM judge. For batch efficiency, evaluate on a small set (typically $n \leq 50$) of inputs per round (Xiang et al., 7 Feb 2025, Wu et al., 14 Oct 2025).
3. Mutation/Exploration: Regularly mutate top performers based on meta-prompts (e.g., request prompt expansion, addition of few-shot exemplars).
4. Selection and Pruning: Maintain the prompt pool size by pruning low performers and incorporating high-value mutants (Wu et al., 14 Oct 2025).
5. Partial Label Integration: Optionally incorporate known ground-truth for a subset of examples to calibrate noisy LLM judges.
6. Stopping Criteria: Halt after T rounds or convergence in Copeland/Condorcet score.

A typical cost per optimization cycle is minimal; for instance, Self-Supervised Prompt Optimization (SPO) achieves comparable or superior prompt performance vs. label-based methods at $\sim1.1\%$–$5.6\%$ of their total call budget, using as few as 3 sampled inputs per iteration (Xiang et al., 7 Feb 2025).

4. Theoretical Analysis and Sample Complexity

Dueling-based prompt optimization methods are supported by theoretical guarantees analogous to regret bounds in online learning and GP optimization:

• D-TS: Expected cumulative regret of $O(K^2 \log T)$, converging to Copeland winners (Wu et al., 14 Oct 2025).
• PBO: Posterior-inference and acquisition maximize preference scores efficiently; exploration–exploitation tradeoff is tunable via acquisition function choice (Gonzalez et al., 2017).
• COMP-GP-UCB: When direct queries are expensive and duels cheap, comparison filtering allows the number of required labels to scale with the (smaller) information gain in the filtered region, yielding lower regret and accelerated convergence (Xu et al., 2019).
• Lower Bounds: Dueling optimization with monotone adversaries requires $\Omega(d)$ iterations and $\Omega(d)$ regret in dimension $d$, even in the best case, establishing linear dependence as information-theoretically necessary (Blum et al., 2023).

5. Empirical Results and Performance Benchmarks

PDO frameworks have been systematically evaluated on challenging prompt selection tasks:

• Datasets: BIG-bench Hard (16 reasoning tasks), MS-MARCO QA (varied open-ended queries), GPQA-Diamond, AGIEval-Math, LIAR, WSC, BBH-Navigate; assessed both in closed- and open-ended settings (Wu et al., 14 Oct 2025, Xiang et al., 7 Feb 2025).
• Metrics: Prompt win rate, Copeland score, LLM-judge or true accuracy, cost (LLM API calls).
• Benchmarks:
  • PDO with D-TS and top-performer mutation outperforms label-free hill-climbing (SPO), RUCB, and random dueling approaches, delivering $+5$–$10$ percentage point accuracy gains and winning on up to 13/16 tasks.
  • On open-ended tasks, win rates over baseline prompts range from 60% to 85% (Xiang et al., 7 Feb 2025).
  • Integration of partial ground-truth labels further accelerates convergence and mitigates overfitting to LLM judge biases (Wu et al., 14 Oct 2025).
  • Typical optimization (3–10 samples per round, $\sim 10$–$30$ rounds) obtains near-optimal prompts at a fraction of the label cost required by supervised alternatives.

Method	Avg. Accuracy Gain	Cost (\$) per Run	Label-Free
APE	—	9.07	No
OPRO	—	4.51	No
PromptAgent	—	2.71	No
PromptBreeder	—	4.82	No
TextGrad	—	13.14	No
SPO	+0–2pp	0.15	Yes
PDO (D-TS)	+5–10pp vs. SPO	Similar	Yes

Note: “Label-Free” indicates no ground truth required during optimization. “+pp” = percentage points.

6. Limitations, Practical Considerations, and Future Directions

While PDO methods are robust and broadly applicable, several limitations and open questions remain:

• Judge Dependency: Final prompt selection is aligned to the LLM judge's notion of quality, which may not reflect the true task metric. Partial label fusion mitigates but does not eliminate this risk (Wu et al., 14 Oct 2025).
• Noise and Bias: Small judges (e.g., Llama-8B) exhibit higher variance and bias; answer-based duels are more reliable than reasoning-based duels, necessitating weighted update schemes (Wu et al., 14 Oct 2025).
• Binary Feedback Limitation: Current implementations use binary win/loss duels; integrating graded or scalar preference (e.g., “score on a 1-5 scale”) is a future direction (Xiang et al., 7 Feb 2025).
• Prompt Pool Scalability: Scaling beyond a few hundred prompts per round remains an open engineering and sample-complexity challenge (Wu et al., 14 Oct 2025).
• Theoretical Generalizations: The monotone adversary lower bounds (Blum et al., 2023) formalize necessary sample complexity, but extensions to more flexible or structured adversaries are an active area of research.
• Mixed-Fidelity Extensions: Hybrid approaches leveraging both comparisons and pointwise measurements can further accelerate convergence, especially when direct evaluation is occasionally feasible (Xu et al., 2019).

7. Relationship to Dueling Optimization Theory

Prompt Duel Optimizers generalize core ideas from classic dueling optimization, where the optimization agent receives only preference-based feedback between pairs of points (Blum et al., 2023). Key results include:

• Efficient randomized algorithms achieving $O(d \log^2(1/\varepsilon))$ iteration complexity and $O(d)$ total cost for smooth objectives under Polyak–Łojasiewicz conditions.
• Lower bounds of $\Omega(d)$ cost and iteration for the general case, reflecting the intrinsic dimension-of-interest.
• Extensions to nonconvex and kernelized settings via Bayesian and bandit-inspired inference (Gonzalez et al., 2017, Xu et al., 2019).

PDOs thus occupy a foundational role in scaling preference-based optimization to high-dimensional, combinatorial, or human-in-the-loop problems in prompt engineering for LLMs.