DistillPrompt: Non-Gradient Autoprompting

• DistillPrompt is a non-gradient autoprompting method that iteratively refines prompts using embedding, compression, and aggregation to optimize LLM task performance.
• It employs a multi-stage workflow where candidate prompts are generated, integrated with task examples, and compressed into a distilled, unified prompt.
• The method yields significant improvements in key metrics such as macro F1 and METEOR scores, outperforming standard and few-shot autoprompting benchmarks.

DistillPrompt refers to a non-gradient autoprompting method developed for efficient and robust optimization of prompts in LLMs, specifically targeting both classification and generation tasks. The method iteratively embeds, compresses, and aggregates task-specific information using only LLM inference, delivering substantial improvements in benchmark performance relative to competing autoprompting strategies. DistillPrompt operates through multi-stage integration, leveraging both training data examples and model-driven consolidation of prompt candidates to synthesize high-quality, distilled prompts.

1. Multi-Stage Distillation Methodology

DistillPrompt employs a multi-stage, iterative process for prompt refinement, designed to thoroughly explore and compress the prompt space:

• Exploration: The algorithm begins with an initial “best candidate” prompt. In each epoch, it generates $N$ new candidate prompts ($N=4$), with sampling performed using a temperature of 0.7 to ensure diversity and creativity in candidate generation.
• Example Embedding: For each candidate, $K=5$ examples are randomly selected from the training set. Instead of direct in-situ insertion of these examples (as in conventional few-shot prompting), the LLM is instructed to analyze the illustrative examples and extract their underlying task-solving principles, integrating these as more abstract, generalizable cues.
• Instruction Compression: After embedding, the LLM is prompted to compress each extended prompt candidate into a succinct, distilled representation. This removes redundant material, preserving core instructions and general principles derived from the examples.
• Candidate Aggregation: The compressed prompt candidates undergo aggregation—distinct aspects or advantages from each candidate are fused into a single distilled prompt, broadening the spectrum of embedded task knowledge.
• Iteration & Refinement: This distilled prompt then becomes the seed for further exploration; top scoring candidates (measured via the chosen target metric on the training data) are iteratively selected as the new anchor in subsequent epochs.

This methodology explicitly avoids any gradient-based updates, instead relying entirely on LLM-driven inferential steps for prompt search, example analysis, compression, and aggregation.

2. Implementation Details and Workflow

DistillPrompt is implemented with the t-lite-instruct-0.1 LLM, orchestrated in the following sequential workflow:

1. Initialization: Begin using the original human prompt as an initial seed.
2. Epoch Loop:
   • Candidate Generation: Generate $N=4$ prompt variations by querying the LLM, promoting diversity through controlled randomness (temperature).
   • Example Integration: For each candidate, randomly select $K=5$ training examples. The LLM is used to infer the general principles exemplified by these, extending and contextualizing the prompt accordingly.
   • Compression: The LLM then compresses each example-integrated candidate to its essential instructions.
   • Aggregation: All compressed prompts are aggregated into a unified distilled prompt—this can be conceptualized as a joint summarization or consensus step.
   • Candidate Selection: Each prompt variation is scored on the training data (using the relevant metric), and the best becomes the anchor for the next round.
3. Termination: After the designated epochs or early stopping based on convergence, the highest-scoring prompt becomes the output.

Symbolically, for epoch $i$: $$\begin{align*} & \text{Generate } C_i = \{c_i^1,\ldots,c_i^N\} \ & \forall\ c_i^j: \ c_i^{j'} = \operatorname{Embed}(c_i^j, \mathrm{Examples}) \ & \hspace{1.5cm}c_i^{j''} = \operatorname{Compress}(c_i^{j'}) \ & P_i = \operatorname{Aggregate}(\{c_i^{1''},\ldots,c_i^{N''}\}) \ & P_0 \leftarrow \arg\max_{c \in P_i\_\mathrm{variations}} \operatorname{Score}(c) \end{align*}$$ where $P_0$ is the anchor prompt per epoch, and $\operatorname{Score}$ is according to the evaluation metric.

3. Evaluation Protocol and Metrics

The method was thoroughly evaluated on classification and generation tasks, using a curated benchmark suite:

• Classification Datasets: SST-2, MNLI, TREC, MR, MedQA, BBH (aggregate).
• Metrics: Macro F1-score is the primary metric, as it offers sensitivity to class imbalance and measures performance across all label classes.
• Generation Datasets: GSM8K, SAMSum, plus BBH in a generation configuration.
• Metrics: METEOR, with an F1-analog calculation favoring recall but balancing both recall and precision.

Baselines include standard dataset prompts, few-shot prompts ($n=3$), and contemporary autoprompting benchmarks such as Grips and Protegi.

4. Experimental Results and Performance

DistillPrompt demonstrates consistently strong improvements in both the classification and generation settings, as evidenced by:

• Classification: Macro F1-score improvements such as 0.7606 (MNLI), 0.9392 (MR), and overall average improvements of $20.12\%$ compared to Grips across the dataset suite.
• Generation: Superior METEOR scores, e.g., 0.4579 (SAMSum) and 0.2961 (BBH), exceeding both standard and autoprompting baselines.

The multi-stage distillation and aggregation approach is shown to be superior not only in aggregate metric terms but also in the generalizability of the resulting prompts. Embedding, compressing, and then aggregating example-driven insight allow the distilled prompt to robustly encode the abstract structure of the task, minimizing overfitting while maximizing performance.

Task/Dataset	DistillPrompt (macro F1 or METEOR)	Relative Gain over Grips
MNLI	0.7606	Substantial
MR	0.9392	Substantial
SAMSum	0.4579 (METEOR)	Substantial
BBH (gen)	0.2961 (METEOR)	Substantial

The mechanism, rooted in model-guided candidate evolution rather than direct few-shot insertion or brute-force string search, sidesteps many pitfalls of overfitting or loss of generality.

5. Significance and Implications

DistillPrompt is identified as one of the most effective non-gradient autoprompting methods to date, highlighted by:

• Gradient-free operation: All optimization is handled via LLM inference, dramatically reducing infrastructure and engineering complexity compared to gradient-based prompt search.
• Multi-stage distillation: The pipeline’s sequential embedding, compression, and aggregation stages enable thorough but structured exploration of the prompt space, capturing both breadth and specificity in task understanding.
• Efficiency and generality: Results show strong improvements over both human and automated baselines, with broad applicability across both classification and generation tasks.
• Insights for prompt engineering research: By treating prompts as evolving, compressible, and aggregatable artifacts, DistillPrompt suggests methodological parallels with knowledge distillation in parameter space, but realized entirely at the prompt/instructional level.

A plausible implication is that similar multi-stage, inference-driven designs might further improve prompt optimization frameworks in LLM research, especially where resource efficiency and interpretability are paramount.

6. Limitations and Future Directions

While DistillPrompt avoids the computational expense of gradient-based search, its effectiveness is bounded by the diversity of generated examples, the coverage of scenario variations in candidate exploration, and the model’s intrinsic capabilities for extracting and compressing task principles. Future research could explore hybrid approaches, integrating prompt distillation with controlled gradient steps or adaptive candidate selection, as well as extending the method to more complex prompt modalities or multi-task regimes.

In summary, DistillPrompt advances the non-gradient autoprompting landscape by leveraging LLM-driven distillation, compression, and aggregation within a principled, iterative framework—demonstrating significant improvements in key evaluation metrics and yielding distilled prompts that generalize effectively across a variety of NLP tasks (Zhuravlev et al., 26 Aug 2025).