PRESTO: Preimage-Informed Instruction Optimization

• PRESTO is a framework that optimizes instructions for black-box LLMs by leveraging preimage structures to amplify data efficiency and maximize task-specific metrics.
• It formulates instruction optimization as a bandit problem, using score sharing and diversity-maximized preimage initialization to improve exploration under a fixed query budget.
• Empirical results demonstrate that PRESTO can achieve a 14× increase in effective labeled data and deliver state-of-the-art performance across multiple instruction and reasoning tasks.

PREimage-informed inSTruction Optimization (PRESTO) is a framework for efficient instruction optimization in prompting black-box LLMs. It leverages the many-to-one mapping from soft prompts to discrete instructions observed in white-box LLMs, introducing preimage-based strategies to substantially amplify data efficiency under a fixed query budget while optimizing task-specific metrics. PRESTO’s core innovation is to treat the preimage structure—sets of distinct soft prompts that decode to the same natural language instruction—not as a hindrance, but as a useful prior enabling score sharing, diversity-maximized initialization, and unsupervised score consistency regularization (Chu et al., 29 Oct 2025).

1. Formal Problem Definition and Preimage Structure

Let $f_b$ denote a deterministic black-box LLM (e.g., GPT-4), and $h(\cdot, \cdot)$ a task-specific metric such as accuracy or F₁. The goal is to find a textual instruction $v \in \Omega$ that maximizes expected score on a distribution of input-output pairs $(x, y) \sim D_{val}$:

$$v^* = \arg\max_{v \in \Omega} \mathbb{E}_{(x, y) \sim D_{val}}[h(f_b(v, x), y)].$$

Direct optimization over $\Omega$ is intractable due to its combinatorial size. Following recent practice, optimization proceeds over a continuous soft prompt $z \in \mathbb{R}^{N_z \times d}$, mapped to instructions by a white-box LLM $f_w$:

$$z^* = \arg\max_{z \in \mathcal{Z}} \mathbb{E}_{(x, y) \sim D_{val}} [h(f_b(f_w(z), x), y)].$$

A candidate pool $Z = \{z_j\}_{j=1}^N$ is constructed (via a scrambled Sobol sequence and random projection). Each $h(\cdot, \cdot)$0 is mapped through $h(\cdot, \cdot)$1 to a set of instructions $h(\cdot, \cdot)$2. The mapping $h(\cdot, \cdot)$3 is many-to-one: multiple $h(\cdot, \cdot)$4 may produce the same $h(\cdot, \cdot)$5. The preimage of $h(\cdot, \cdot)$6 is defined as:

$h(\cdot, \cdot)$7

This “preimage group” structure is central to PRESTO’s design.

2. Black-Box Instruction Optimization as Bandit Problem

Instruction optimization is formulated as a black-box bandit problem, where each $h(\cdot, \cdot)$8 is an arm with reward

$h(\cdot, \cdot)$9

Under a fixed budget $v \in \Omega$0, only $v \in \Omega$1 queries to $v \in \Omega$2 are permitted. The objective is to pick $v \in \Omega$3 to maximize the best-observed $v \in \Omega$4. Each $v \in \Omega$5 is evaluated by querying $v \in \Omega$6 on $v \in \Omega$7; due to preimage redundancy, multiple $v \in \Omega$8's may represent the same $v \in \Omega$9, but only distinct instructions yield new supervision.

3. Core Components of PRESTO

a. Score Sharing

Once $(x, y) \sim D_{val}$0 is evaluated for any $(x, y) \sim D_{val}$1, its scalar score $(x, y) \sim D_{val}$2 is distributed to all $(x, y) \sim D_{val}$3:

$(x, y) \sim D_{val}$4

This expands the set of labeled pairs from $(x, y) \sim D_{val}$5 true queries to $(x, y) \sim D_{val}$6 soft prompts.

b. Preimage-Based Initialization

To initialize the model, $(x, y) \sim D_{val}$7 preimage groups are selected to cover the embedding space. For each instruction $(x, y) \sim D_{val}$8, $(x, y) \sim D_{val}$9 is constructed, where $$v^* = \arg\max_{v \in \Omega} \mathbb{E}_{(x, y) \sim D_{val}}[h(f_b(v, x), y)].$$0 is the final token embedding from $$v^* = \arg\max_{v \in \Omega} \mathbb{E}_{(x, y) \sim D_{val}}[h(f_b(v, x), y)].$$1. Groups are greedily chosen using the coverage-score:

$$v^* = \arg\max_{v \in \Omega} \mathbb{E}_{(x, y) \sim D_{val}}[h(f_b(v, x), y)].$$2

where $$v^* = \arg\max_{v \in \Omega} \mathbb{E}_{(x, y) \sim D_{val}}[h(f_b(v, x), y)].$$3 promotes large preimages, and $$v^* = \arg\max_{v \in \Omega} \mathbb{E}_{(x, y) \sim D_{val}}[h(f_b(v, x), y)].$$4 leverages squared maximum-mean-discrepancy (MMD$$v^* = \arg\max_{v \in \Omega} \mathbb{E}_{(x, y) \sim D_{val}}[h(f_b(v, x), y)].$$5) for embedding diversity:

$$v^* = \arg\max_{v \in \Omega} \mathbb{E}_{(x, y) \sim D_{val}}[h(f_b(v, x), y)].$$6

c. Score Consistency Regularization

For unseen preimages $$v^* = \arg\max_{v \in \Omega} \mathbb{E}_{(x, y) \sim D_{val}}[h(f_b(v, x), y)].$$7, the regression model $$v^* = \arg\max_{v \in \Omega} \mathbb{E}_{(x, y) \sim D_{val}}[h(f_b(v, x), y)].$$8 is regularized to output identical predictions for all $$v^* = \arg\max_{v \in \Omega} \mathbb{E}_{(x, y) \sim D_{val}}[h(f_b(v, x), y)].$$9:

$\Omega$0

The full predictor loss is:

$\Omega$1

with $\Omega$2 linearly annealed from 0 to $\Omega$3 over a warm-up period.

4. PRESTO Optimization Algorithm

PRESTO operates in three main stages:

• Preprocessing: For each candidate $\Omega$4, compute its embedding $\Omega$5 and instruction $\Omega$6, forming all preimage groups.
• Initialization: Greedily select $\Omega$7 preimage groups maximizing $\Omega$8. For each, query $\Omega$9 on a representative $z \in \mathbb{R}^{N_z \times d}$0, then share the obtained score across the group, forming labeled set $z \in \mathbb{R}^{N_z \times d}$1.
• NeuralUCB Optimization: For $z \in \mathbb{R}^{N_z \times d}$2 to $z \in \mathbb{R}^{N_z \times d}$3:
  1. Train score predictor $z \in \mathbb{R}^{N_z \times d}$4 on $z \in \mathbb{R}^{N_z \times d}$5 with joint loss.
  2. For each unlabeled $z \in \mathbb{R}^{N_z \times d}$6, estimate $z \in \mathbb{R}^{N_z \times d}$7 and predictive uncertainty $z \in \mathbb{R}^{N_z \times d}$8.
  3. Select $z \in \mathbb{R}^{N_z \times d}$9.
  4. Query $f_w$0 on $f_w$1, share the score with its preimage, and update $f_w$2.

• Output: Return the best observed instruction $f_w$3.

This design allows PRESTO to aggregate supervision over preimages, maximizing the utility of each black-box query.

5. Analysis of Query Efficiency and Data Amplification

With query budget $f_w$4, conventional methods yield $f_w$5 labeled (soft prompt, score) pairs. PRESTO’s score sharing multiplies the number of labeled prompts by the average preimage size. Empirically, across 30 tasks, PRESTO inflates the labeled set to $f_w$62,300, a $f_w$714× increase over the budget:

$f_w$8

This suggests that preimage-based score sharing is critical for unlocking efficient regression and improved uncertainty quantification in UCB-based exploration, without additional $f_w$9 queries.

6. Experimental Setup and Baselines

Experiments span 30 instruction-induction tasks (Honovich et al., ACL ’23) and 3 math reasoning benchmarks (GSM8K, AQUA-RAT, SVAMP). Metrics include exact match, F₁, and chain-of-thought (CoT) accuracy. The black-box LLM is GPT-4.1 (37B); the white-box LLM is LLaMA 3.1-8B-Instruct. Comparison baselines are APE, InstructZero, INSTINCT, EvoPrompt, ZOPO, and OPRO. Candidate pool size is $$z^* = \arg\max_{z \in \mathcal{Z}} \mathbb{E}_{(x, y) \sim D_{val}} [h(f_b(f_w(z), x), y)].$$0 soft prompts, constructed by Sobol sampling and random projections, with grid search over intrinsic dimensions and soft-token counts. The query budget is $$z^* = \arg\max_{z \in \mathcal{Z}} \mathbb{E}_{(x, y) \sim D_{val}} [h(f_b(f_w(z), x), y)].$$1 and $$z^* = \arg\max_{z \in \mathcal{Z}} \mathbb{E}_{(x, y) \sim D_{val}} [h(f_b(f_w(z), x), y)].$$2; all experiments use a single NVIDIA A6000 GPU.

7. Empirical Results and Ablation Analysis

PRESTO achieves the following empirical performance:

• Wins 18/30 tasks (compared to ZOPO’s 8, next best) with mean rank $$z^* = \arg\max_{z \in \mathcal{Z}} \mathbb{E}_{(x, y) \sim D_{val}} [h(f_b(f_w(z), x), y)].$$31.97.
• On a 20-task subset, wins 12 tasks (ZOPO: 4), with mean rank 1.90 (vs. 3.05 for ZOPO).
• On CoT benchmarks, PRESTO meets or surpasses hand-crafted and prior soft-prompting approaches on GSM8K, AQUA-RAT, and SVAMP.
• In ablations (20 tasks):

Configuration	Avg. Accuracy	Avg. Rank
Vanilla (no preimage modules)	51.91	4.55
+Score Sharing Only	59.57	3.10
+Score Sharing + Score Consistency	61.77	2.65
+Score Sharing + Preimage Init	61.82	2.30
All PRESTO components	62.91	2.20

• Score predictor RMSE on a toy task decreases from $$z^* = \arg\max_{z \in \mathcal{Z}} \mathbb{E}_{(x, y) \sim D_{val}} [h(f_b(f_w(z), x), y)].$$40.27 (vanilla) to $$z^* = \arg\max_{z \in \mathcal{Z}} \mathbb{E}_{(x, y) \sim D_{val}} [h(f_b(f_w(z), x), y)].$$50.15 (PRESTO).
• t-SNE visualization shows preimage-based initialization achieves denser and more uniform coverage of the embedding space compared to random or sharing-only initialization.

These results substantiate the complementary gains of score sharing, diversity-maximized preimage initialization, and consistency regularization.

8. Significance and Broader Implications

PRESTO demonstrates that the many-to-one mapping from soft prompts to instructions—previously viewed as a source of inefficiency—can be systematically exploited to amplify effective data, increase sample efficiency, and improve generalization in instruction optimization for black-box LLMs. The preimage structure enables up to 14× effective labeling, state-of-the-art task performance, and scalable optimization via score sharing, informed candidate selection, and unsupervised regularization. These findings provide a foundation for further advances in black-box LLM instruction engineering and suggest that similar preimage-informed priors may benefit other combinatorial black-box optimization domains (Chu et al., 29 Oct 2025).