---
title: 'BATprompt: Robust Prompt Optimization'
url: https://www.emergentmind.com/topics/batprompt
type: topic
---

# BATprompt: Robust Prompt Optimization

Searching arXiv for BATprompt and closely related prompt-optimization work.
BATprompt, short for **“By Adversarial Training prompt,”** is a robustness-aware automatic prompt optimization method for large language models that is designed to generate prompts which remain effective when the input text is perturbed rather than perfectly clean. Its target setting includes typos, character corruption, extra meaningless characters, synonym substitutions, neutral filler words, sentence paraphrases, and syntactic restructuring. The method is inspired by adversarial training, but it operates in a black-box setting: it does not require access to model parameters or gradients, and instead uses the reasoning, language understanding, and self-reflection capabilities of an LLM to simulate gradient-like guidance for both perturbation generation and prompt revision [2412.18196].

## 1. Definition and conceptual basis

BATprompt addresses a limitation of many automatic prompt optimization methods: they optimize prompts on clean datasets and implicitly assume that clean-input performance transfers to noisy or perturbed inputs. The method starts from the observation that LLM performance depends not only on prompt quality but also on the semantic and structural integrity of the input. In realistic deployments, inputs often contain spelling mistakes, letter substitutions, added junk characters, paraphrasing, or structural variation, and prompts that work on clean text may fail under such perturbations [2412.18196].

The framework is explicitly robustness-centered. Rather than optimizing only for nominal task performance, BATprompt seeks prompts that preserve task performance under perturbation. Its high-level procedure has two phases repeated iteratively: an **adversarial perturbation phase**, which generates perturbed inputs that degrade the current prompt, and an **adversarial optimization phase**, which analyzes the differences between clean and perturbed inputs and rewrites the prompt accordingly. The paper describes this as a black-box analogue of adversarial training, with the crucial difference that the optimized object is the prompt rather than the model parameters [2412.18196].

A central misconception the method rejects is that robustness can be obtained simply by adding perturbed examples to the optimization data. BATprompt includes a data augmentation baseline based on perturbed text, but reports that this often fails to yield robust prompts and can even degrade performance, which the authors attribute to the excessive diversity of perturbations making it difficult for the LLM to focus on the relevant failure modes [2412.18196].

## 2. Optimization mechanism

BATprompt begins from a manually crafted prompt \(p\) and an unperturbed dataset. At each outer iteration, it samples a small set of clean examples, generates adversarially perturbed versions, selects the most damaging perturbations subject to similarity constraints, extracts generalized guidance from the clean–perturbed differences, and uses that guidance to rewrite the prompt. The prompt is also paraphrased to broaden the search space, and the best-performing candidate is retained for the next round [2412.18196].

The adversarial perturbation step is written as
\[
x' = x + \epsilon \cdot \arg\min \mathcal{L}_{adv}(x + g, y),
\]
where \(x\) is the original input, \(x'\) is the perturbed input, \(g\) is perturbation-specific guidance, \(\epsilon\) is the perturbation budget, and \(\mathcal{L}_{adv}\) is the adversarial loss. This is not a white-box gradient update; the “gradient” is conceptual and is simulated through LLM-mediated guided transformation rather than model derivatives [2412.18196].

For optimization, BATprompt derives a generalized guidance signal from multiple clean–perturbed pairs:
\[
g' =  \mathcal{G}(\mathcal{D}(x_0,x_0^{'}) \cup \mathcal{D}(x_1,x_1^{'})  \cup  \ldots \cup \mathcal{D}(x_n,x_n^{'})),
\]
where \(\mathcal{D}(\cdot)\) denotes difference generation and \(\mathcal{G}(\cdot)\) generates the optimization guidance. The overall prompt-optimization loss is written as
\[
\mathcal{L}_{opt}(p) = \mathbb{E}_{(x', y) \sim D} \left[ \mathcal{L}(x_1^{'}, y; p) + \mathcal{L}(x_2^{'}, y; p) + \ldots +\mathcal{L}(x_n^{'}, y; p) \right],
\]
and the prompt update is expressed in gradient notation as
\[
p' = p + \nabla_{p} \mathcal{L}_{opt}(p).
\]
The paper is explicit that this gradient is not computed from model internals; it is approximated through LLM reasoning and self-reflection [2412.18196].

Algorithmically, the outer loop receives as input an initial prompt \(p\), adversarial attack gradients \(g_{adv}\), optimization gradient \(g_{opt}\), an attack operator \(f_{adv}(\cdot)\), an optimization operator \(f_{opt}(\cdot)\), a similarity constraint \(\mathcal D(\cdot)\), and a scoring function \(\mathcal S(\cdot)\). At each iteration, BATprompt selects clean texts \(T^i\), generates perturbations \(T' \leftarrow f_{adv}(T^i, g_{adv}^k)\), keeps the perturbation with \(\mathcal D(T', T^i) < \epsilon\) and low score, generates optimization guidance from the resulting differences, updates the prompt with \(p \leftarrow f_{opt}(p, g_{opt})\), and retains the highest-scoring prompt [2412.18196].

## 3. Perturbation taxonomy and simulated gradients

BATprompt organizes perturbations into two groups. **P1 perturbations** mostly preserve sentence semantics and structure while injecting surface corruption: **C1** changes words to have typos, **C2** changes letters, **C3** adds extraneous characters, and **S1** adds a meaningless handle or string. **P2 perturbations** preserve meaning while changing lexical or structural form: **W1** changes words to synonyms, **W2** deletes meaningless words, **W3** adds neutral words, **S2** paraphrases the sentence, and **S3** changes the syntactic structure [2412.18196].

The paper distinguishes these groups because it treats them differently during adversarial example generation. For P1, BATprompt uses a **mix-mode**:
\[
x' = x_0 + \mathcal{P}(x_0,g_1) + \mathcal{P}(x_1,g_2) + \ldots +  \mathcal{P}(x_{n-1},g_n),
\]
reflecting the claim that P1 perturbations mostly do not interfere strongly with each other. For P2, it uses a **combined-mode**:
\[
\mathcal{U}(x') = \mathcal{P}(x,g_1) \cup \mathcal{P}(x,g_2) \cup \ldots \cup \mathcal{P}(x_,g_n),
\]
because semantic and structural perturbations interfere more when stacked on the same string [2412.18196].

Similarity control is perturbation-type dependent. For P1, BATprompt uses **Levenshtein distance**; for P2, it uses **semantic similarity**. The appendix reports that the resulting perturbations remain close to the originals: for XSum, both Levenshtein and semantic similarity are above 98%, while for classification and simplification datasets Levenshtein similarity is mostly above 90% and semantic similarity mostly above 80% [2412.18196].

This perturbation machinery also clarifies what BATprompt is not. It is not white-box adversarial training, not gradient-based token attack, and not standard data augmentation. Its “gradient” is a language-level search direction encoded by perturbation-specific guides and optimization summaries. The paper explicitly presents this as a black-box analogue of FGSM-style intuition rather than a literal differentiable attack pipeline [2412.18196].

## 4. Empirical performance and implementation

BATprompt is evaluated on both language understanding and language generation tasks. The language-understanding datasets are **SST-2**, **CR**, **MR**, **SST-5**, **AG’s News**, and **TREC**, evaluated with accuracy. The generation datasets are **ASSET** for simplification, evaluated with **SARI**, and **XSum** for summarization, evaluated with **ROUGE-1**, **ROUGE-2**, and **ROUGE-L**. The backbone used for adversarial training and prompt generation is **GPT-3.5-turbo**, and evaluation is also performed on **GPT-4o-mini** and **Llama2-7b** to test transferability [2412.18196].

On language understanding, BATprompt achieves the strongest reported average robustness. Under **P1 perturbations**, the average accuracy is **75.4**, compared with **74.4** for Manual Instructions, **62.9** for Natural Instructions, **51.0** for EvoPrompt, and **72.1** for BATprompt\(^*\), the ablation that removes adversarial training. Under **P2 perturbations**, BATprompt reaches **73.2**, compared with **71.5** for Data Augmentation, **71.4** for Manual Instructions, **60.2** for Natural Instructions, **49.7** for EvoPrompt, and **69.2** for BATprompt\(^*\). The paper highlights about **3% improvement on TREC for P1** and about **12% improvement on TREC for P2** [2412.18196].

On summarization, BATprompt shows larger gains. Under **C2 perturbation**, the XSum scores are **21.68 / 4.76 / 16.42** for ROUGE-1 / ROUGE-2 / ROUGE-L, compared with **17.62 / 3.16 / 14.96** for EvoPrompt and **18.31 / 3.08 / 15.50** for BATprompt\(^*\). Under **W3 perturbation**, BATprompt reaches **22.03 / 5.23 / 17.18**, compared with **17.54 / 3.09 / 14.93** for EvoPrompt and **18.42 / 3.10 / 15.82** for BATprompt\(^*\). The authors state that BATprompt beats the second-best method by about **23% under C2 perturbation** [2412.18196].

On simplification, BATprompt is consistently best or tied-best. Under **W1 perturbation**, its **SARI** score is **49.50**, compared with **45.39** for EvoPrompt. Across the reported perturbations, BATprompt ranges from **45.11** to **49.55**, while EvoPrompt ranges from **44.61** to **49.09** [2412.18196].

Transfer experiments indicate that prompts generated by BATprompt remain useful across models. On **GPT-4o-mini**, average summarization scores are **19.72 / 3.55 / 14.82** under P1 and **19.26 / 3.71 / 15.12** under P2. On **Llama2-7b**, the corresponding averages are **17.62 / 3.53 / 14.25** and **18.98 / 3.74 / 15.17** [2412.18196].

Implementation is intentionally lightweight. The method uses **5 outer iterations**, because prompt performance generally peaks around the fourth or fifth round and then declines, **3 iterative attacks** in the adversarial phase, **5 examples per iteration for P1**, and **3 examples per iteration for P2**. For attack and prompt optimization, decoding uses **top-p = 0.95** and **temperature = 1**; for testing, it uses **top-p = 1** and **temperature = 0**. The appendix reports token consumption from **0.0258M** tokens on SST-5 (P1) up to **1.0401M** tokens on XSum (P2) [2412.18196].

## 5. Position within the broader prompt-optimization landscape

BATprompt belongs to a broader family of black-box prompt methods, but its emphasis is distinct: it optimizes prompts for **robustness under perturbed inputs**, not merely for clean-input task accuracy. A nearby line, TRIPLE, formulates prompt selection under a fixed evaluation budget as **fixed-budget best arm identification**, treating each candidate prompt as an arm and using variants such as Sequential Halving, Continuously Reject, clustering over prompt embeddings, and regression over embeddings to improve budgeted selection [2402.09723]. That formulation is budget-aware and black-box, but it assumes a pre-generated prompt pool and focuses on efficient selection rather than adversarial robustness.

Another related direction studies **bandit-based strategy selection** inside prompt optimizers. OPTS treats prompt design strategies as arms, adds an inaction arm, and uses mechanisms including **Thompson sampling** to improve EvoPrompt; the Thompson-sampling variant achieves the best average performance among the tested selectors on BIG-Bench Hard [2503.01163]. This is adjacent to BATprompt because both operate without parameter access and both use iterative prompt revision, but OPTS is a strategy-selection layer over an existing optimizer rather than an adversarially trained robustness objective.

A separate neighboring literature investigates **black-box adversarial prompting** as an attack problem against generative models. "Black Box Adversarial Prompting for Foundation Models" formulates prompt search in a black-box setting using **Token Space Projection**, **Square Attack**, and **TuRBO** to induce target behaviors in image or text generation [2302.04237]. BATprompt differs in objective: it uses adversarial perturbations to improve downstream prompt robustness, whereas black-box adversarial prompting searches directly for prompts that cause the target model to fail.

Finally, some black-box prompt-related systems do not optimize prompts on the target model at all, but instead use prompting to improve auxiliary supervision. BT-Classifier, for example, uses a small prompt-finetuned auxiliary language model to pseudo-label in-domain unlabeled text, then trains a classifier on frozen black-box representations for few-shot text classification [2305.13785]. This is methodologically adjacent in its use of prompt-based signals under black-box constraints, but it is a data-augmentation pipeline rather than prompt optimization.

## 6. Limitations, interpretation, and open questions

BATprompt is explicitly heuristic in one central respect: its “gradient” is simulated through LLM reasoning rather than derived from a differentiable model objective. The paper presents this as a strength for black-box settings, but it also means the optimization process is less formally grounded than white-box adversarial training [2412.18196].

The perturbation taxonomy is predefined. BATprompt relies on a manually specified set of perturbation families—C1, C2, C3, S1, W1, W2, W3, S2, S3—rather than discovering failure modes automatically. A plausible implication is that robustness will depend on whether these perturbation families match the deployment environment. The paper’s experiments cover classification, summarization, and simplification, but not a broader range of tasks [2412.18196].

The method also appears sensitive to over-optimization. Performance tends to improve until iteration 4 or 5 and then decline, which the authors interpret as deterioration from excessive optimization. This suggests that prompt robustness is being improved within a relatively narrow search horizon rather than through a monotone iterative procedure [2412.18196].

At the same time, BATprompt should not be understood as merely trading clean performance for robustness. The paper reports that on unperturbed datasets BATprompt often achieves the best performance, and when it is not the best it remains close to the top method. Likewise, the ablation BATprompt\(^*\), which keeps iterative prompt optimization but removes adversarial training, consistently underperforms full BATprompt, indicating that the adversarial phase contributes materially rather than acting as incidental prompt rewriting [2412.18196].

In the current prompt-optimization literature, BATprompt is therefore best viewed as a black-box, adversarially inspired prompt generator specialized for the noisy-input regime. Its defining contribution is to move prompt optimization away from the clean-input assumption and toward an iterative loop in which perturbation generation, failure analysis, and prompt rewriting are coupled. That design situates it at the intersection of automatic prompt engineering, adversarial robustness, and black-box LLM control [2412.18196].

Source: https://www.emergentmind.com/topics/batprompt