Single-Prompt Evaluation

Updated 4 July 2026

Single-Prompt Evaluation is a method that measures language model performance using one fixed instruction template per task.
This approach is simple and reproducible yet often suffers from brittleness, producing misleading point estimates due to high prompt sensitivity.
Recent research advocates for multi-prompt aggregation and advanced prompt optimization frameworks to overcome inherent limitations and improve reliability.

Searching arXiv for papers on single-prompt evaluation and prompt sensitivity. Single-prompt evaluation is the practice of measuring a model’s performance on a task using exactly one fixed natural-language instruction template, or one fixed prompt, for all test instances. In contemporary LLM evaluation, this convention is widespread because it is simple to compute, reproduce, and compare, but a growing literature argues that it can be brittle, distributionally incomplete, and sometimes evaluator-dependent. Across text-only LLMs, instruction-tuned embedding models, multimodal LLMs, LLM judges, and formal mathematical reasoning systems, the central concern is consistent: a single prompt yields a point estimate, whereas actual model behavior is distributed over plausible prompt variants, prompt orderings, and evaluator logics. The topic therefore spans both measurement and control: it includes the design of prompt-sensitive benchmarks, the formal aggregation of multi-prompt results, and optimization schemes that either diagnose or exploit prompt dependence (Mizrahi et al., 2023, Kostiuk et al., 21 May 2026, Karmakar et al., 3 May 2026, Cazares, 20 Apr 2026).

1. Definition and scope

In its most basic form, single-prompt evaluation refers to measuring an LLM’s score on a task using one fixed instruction template per task. That definition is stated explicitly for general LLM benchmarks in "State of What Art? A Call for Multi-Prompt LLM Evaluation" (Mizrahi et al., 2023), for instruction-tuned embedding models in "One prompt is not enough: Instruction Sensitivity Undermines Embedding Model Evaluation" (Kostiuk et al., 21 May 2026), and for benchmark-style accuracy reporting in "What Single-Prompt Accuracy Misses: A Multi-Variant Reliability Audit of LLMs" (Karmakar et al., 3 May 2026). The same pattern appears in task-specific settings: a single prompt can mean one prompt per mathematical theorem-proving query (Cazares, 20 Apr 2026), one prompt per multimodal benchmark item (Xie et al., 2024), one prompt per candidate answer for an LLM judge (Zhang, 30 Mar 2026), or one prompt selected from a candidate set by a probability-based selector (Yang et al., 2023).

The apparent simplicity of the setup obscures several distinct interpretations of what is being measured. In benchmarking, the score is often treated as a model capability estimate. In prompt optimization, the same score may be treated as a control-signal for choosing or rewriting prompts. In interactive systems such as EvalLM, single-prompt evaluation can instead mean judging one prompt against a fixed baseline or prior revision on user-defined criteria (Kim et al., 2023). These variants share a common structural assumption: prompt choice is held fixed while model behavior is observed.

A plausible implication is that "single-prompt evaluation" is less a single methodology than a family of evaluation regimes that differ in what is frozen and what is varied. Some papers freeze the prompt and study model variance; others freeze the model and study prompt variance; still others treat the prompt itself as an object of diagnosis and optimization (Hong et al., 11 Mar 2026, Chen et al., 25 Nov 2025).

2. Why single-prompt evaluation became problematic

The main critique is brittleness. "State of What Art?" reports analyses across approximately 6.5 million prompt-completion pairs, 20 instruction-tuned LLMs, and 39 tasks, and finds large prompt-induced variance in both absolute scores and model rankings (Mizrahi et al., 2023). On 7/10 LMentry tasks, Alpaca-13B’s score on the original prompt exceeded its paraphrase mean by more than $1\,\sigma$ , Friedman tests reject identical rankings across prompts for 21/25 tasks with $p < 10^{-13}$ , and Kendall’s $\tau$ between worst- and best-prompt rankings is negative for 15/25 tasks. The same paper also reports that even single-word changes such as “excludes” $\rightarrow$ “lacks” can cause $\pm 40$ percentage-point swings (Mizrahi et al., 2023).

The same failure mode appears in embedding benchmarks. Kostiuk & Enevoldsen evaluate 6 instruction-tuned embedding models on 11 MTEB tasks with 15 task-specific prompts each, for 990 total evaluations, and show that default prompts can either understate or overstate performance (Kostiuk et al., 21 May 2026). Their reported mean absolute sensitivity is nearly $0.05$, with individual model-task pairs swinging by as much as $0.15$, and under fully adversarial prompt allocation any of the six models can be promoted to rank 1 (Kostiuk et al., 21 May 2026).

The multimodal setting exhibits analogous effects. "TP-Eval: Tap Multimodal LLMs' Potential in Evaluation by Customizing Prompts" documents prompt sensitivity and cross-model bias in MLLM evaluation, including a spot-similarity case where llava-1.5-7b answers “Yes” to all 180 samples under one wording and nearly doubles accuracy when the prompt is rewritten to “Focus on visual cues” (Xie et al., 2024). The same paper reports that detailed wording improved LLaVA from $0.65$ to $0.88$ on a helmet anomaly detection task, while the same detailed prompt reduced DeepSeek from $0.79$ to $p < 10^{-13}$ 0, directly demonstrating model-specific prompt preference (Xie et al., 2024).

These findings undermine two common assumptions. First, semantic equivalence of prompts does not imply behavioral equivalence. Second, a benchmark’s official prompt is not necessarily a neutral measurement instrument. This suggests that single-prompt evaluation can conflate model capability with prompt luck, prompt mismatch, or evaluator mismatch.

3. Formalizations, metrics, and statistical views

A central contribution of the literature is to replace a single point estimate with distributions, aggregates, and robustness metrics. In "State of What Art?" the score of model $p < 10^{-13}$ 1 on task $p < 10^{-13}$ 2 with prompt $p < 10^{-13}$ 3 is denoted $p < 10^{-13}$ 4, and multi-prompt aggregation is defined by

$p < 10^{-13}$ 5

$p < 10^{-13}$ 6

$p < 10^{-13}$ 7

The same paper further defines stakeholder-specific metrics: $p < 10^{-13}$ 8 for downstream users, $p < 10^{-13}$ 9 for developers, and

$\tau$ 0

$\tau$ 1

These metrics treat prompt sensitivity as part of the evaluation target rather than as noise to be ignored (Mizrahi et al., 2023).

For embedding models, Kostiuk & Enevoldsen define the prompt distribution $\tau$ 2 for model $\tau$ 3, dataset $\tau$ 4, and prompt $\tau$ 5, then summarize it using the mean $\tau$ 6, standard deviation $\tau$ 7, coefficient of variation

$\tau$ 8

and prompt sensitivity

$\tau$ 9

This makes explicit that the single-prompt score is one sample from a prompt-conditioned performance distribution (Kostiuk et al., 21 May 2026).

In multimodal evaluation, TP-Eval formalizes prompt sensitivity as

$\rightarrow$ 0

where $\rightarrow$ 1 is a multimodal LLM, $\rightarrow$ 2 a dataset, and $\rightarrow$ 3 a set of prompt variants (Xie et al., 2024). The same paper also introduces a combined prompt score

$\rightarrow$ 4

balancing few-shot accuracy $\rightarrow$ 5 against semantic similarity $\rightarrow$ 6 to the original prompt.

A different line of work focuses on single-prompt selection rather than averaging. "Improving Probability-based Prompt Selection Through Unified Evaluation and Analysis" defines a prompt-selection score $\rightarrow$ 7 computed from model output probabilities over an unlabeled evaluation set, and unifies several selectors under a mutual-information objective: $\rightarrow$ 8 Within that framework, oracle prompt selection effectiveness rises from $\rightarrow$ 9 to $\pm 40$ 0 under the best MI variant, and to $\pm 40$ 1 with Calibration by Marginalization (Yang et al., 2023).

A further development is evaluator-side reliability. "What Single-Prompt Accuracy Misses" argues that prompt variation is only one axis: calibration definition, parsing logic, and confidence elicitation also change conclusions. It defines prompt-perturbation spread as

$\pm 40$ 2

over prompt variants, and reports that the correlation between model size and spread ranges from $\pm 40$ 3 to $\pm 40$ 4 across benchmarks, with no robust monotonic relation (Karmakar et al., 3 May 2026).

4. Empirical findings across domains

The empirical record is strikingly consistent in showing that single-prompt results can be unstable, but the mechanisms differ by domain.

Domain	Representative finding	Source
General LLM benchmarks	Top-3 model rankings changed in over 80% of tasks when switching from original prompt to AvgP or MaxP	(Mizrahi et al., 2023)
Embedding evaluation	Any model in the study can be promoted to first place under favorable prompt selection	(Kostiuk et al., 21 May 2026)
Multimodal evaluation	Per-model prompt customization improves MMT-S scores by +2.9 to +4.0 points	(Xie et al., 2024)
Mathematical reasoning	Balanced hard accuracy saturates at approximately 60–79% under 45+ prompt variants	(Cazares, 20 Apr 2026)
Reliability auditing	Chain-of-thought plus first-character evaluation on ARC-Challenge reduced apparent accuracy by 72–88%	(Karmakar et al., 3 May 2026)
Reference-grounded judging	Holistic prompts match or exceed self-decomposing atomic prompts on ASQA and QAMPARI	(Zhang, 30 Mar 2026)

In formal mathematical reasoning, "Less Is More: Cognitive Load and the Single-Prompt Ceiling in LLM Mathematical Reasoning" offers an especially explicit account of saturation. The task is deciding whether one equational law implies another over all magmas, where the FALSE case is semi-decidable by finite counterexample search but the TRUE case is undecidable in general. Over five weeks, more than 40 prompt variants from 0 to 4,878 bytes were evaluated across four splits and three models. For gpt-oss-120B on hard3, the no-cheatsheet baseline achieved $\pm 40$ 5 accuracy, $\pm 40$ 6 TRUE recall, and $\pm 40$ 7 FALSE recall, while the best prompt AN45c achieved $\pm 40$ 8 accuracy, $\pm 40$ 9 TRUE recall, and $0.05$0 FALSE recall; yet no single-prompt evaluation exceeded approximately $0.05$1 on hard3 (Cazares, 20 Apr 2026). The paper names this regime the "single-prompt ceiling."

In LLM reliability auditing, evaluator design can be more consequential than prompt wording alone. The ARC-Challenge result in "What Single-Prompt Accuracy Misses" shows that pairing a chain-of-thought prompt with a first-character evaluator reduces apparent accuracy to $0.05$2–$0.05$3, with four out of five models falling below the $0.05$4 random baseline, while regex re-parsing recovers $0.05$5 of the gap and constrained decoding recovers $0.05$6 (Karmakar et al., 3 May 2026). Here the failure is not simply prompt sensitivity but a prompt-evaluator incompatibility.

In LLM judging, "Rethinking Atomic Decomposition for LLM Judges" studies completeness-sensitive reference-support classification on TruthfulQA, ASQA, and QAMPARI. With three pre-frozen prompt variants per design family and four model families, the holistic judge matches or exceeds the atomic judge on ASQA and QAMPARI across all four families, with the advantage concentrated in partially_supported cases, while TruthfulQA shows a small atomic edge (Zhang, 30 Mar 2026). This demonstrates that single-prompt design decisions can encode cognitive structure, not merely wording differences.

5. Prompt evaluation and optimization frameworks

The topic also includes frameworks that attempt to make single-prompt evaluation more interpretable or more actionable.

PEEM proposes a unified rubric with 9 axes: 3 prompt criteria—clarity/structure, linguistic quality, fairness—and 6 response criteria—accuracy, coherence, relevance, objectivity, clarity, conciseness. Scores are assigned on a 1–5 Likert scale, with prompt and response averages computed as

$0.05$7

Across 7 benchmarks and 5 task models, PEEM’s accuracy axis preserves model rankings with aggregate Spearman $0.05$8 about $0.05$9 and Pearson $0.15$0 about $0.15$1, while robustness under paraphrase remains approximately $0.15$2 depending on evaluator (Hong et al., 11 Mar 2026). The framework treats a single prompt not as a scalar accuracy source but as an analyzable object with prompt-level and response-level dimensions.

EvalLM takes a more interactive approach. It evaluates multiple outputs from one prompt on user-defined natural-language criteria, returning a JSON structure containing per-criterion score, explanation, and evidence highlights. In a comparative study with $0.15$3, the system helped participants examine twice as many outputs and reach satisfactory prompts with $0.15$4 fewer revisions than manual evaluation (Kim et al., 2023). Although designed mainly for iterative prompt refinement, the paper explicitly notes that the same evaluator pipeline works in single-prompt settings by comparing outputs against a fixed baseline or previous revision.

Two query-dependent optimization papers move further away from static prompt evaluation. Prompt-OIRL learns an offline reward model $0.15$5 to predict correctness for query-prompt pairs without gold answers at inference time, then uses best-of-$0.15$6 prompt selection. Across 3 LLMs and 3 arithmetic datasets, it reports $0.15$7 percentage points in a scarce-demo setting, $0.15$8 points in a rich-demo setting, and more than $0.15$9 recovery of an oracle upper bound (Sun et al., 2023). "A Unified Evaluation-Instructed Framework for Query-Dependent Prompt Optimization" similarly defines an execution-free evaluator with four metrics—Negative Log-Likelihood, Semantic Stability, Mutual Information, and Query Entropy—and reports $0.65$0 validation accuracy in predicting whether a prompt will achieve at least $0.65$1 accuracy for a query (Chen et al., 25 Nov 2025). In both cases, the premise is that one static prompt is not uniformly optimal across queries.

A plausible implication is that the field is moving from prompt-as-template toward prompt-as-latent policy. Under that view, single-prompt evaluation becomes a special case of a larger control problem.

6. Limits, controversies, and future directions

The most direct limit is that some tasks impose a ceiling on what any finite static prompt can encode. The mathematical reasoning study on equational implication over magmas makes this explicit: the TRUE case requires universal proof over all magmas, including infinite ones, and the paper argues that no finite prompt text can encode a sound and complete proof system for undecidable cases (Cazares, 20 Apr 2026). The reported empirical plateau near $0.65$2 TRUE recall, coupled with trade-offs against FALSE recall, is presented as evidence of that structural limit rather than mere engineering incompleteness (Cazares, 20 Apr 2026).

A second limit is cognitive load. In the same study, Llama 3.3 70B collapsed to $0.65$3 TRUE recall when prompt length exceeded approximately 2 KB, while the minimal 289-byte prompt AN19c was the only variant maintaining balanced recalls on Llama at approximately $0.65$4 TRUE and $0.65$5 FALSE (Cazares, 20 Apr 2026). This is described as "cognitive load collapse." PREMISE reaches a related conclusion from a different direction: long reasoning traces are often unnecessarily verbose, and prompt-level optimization can reduce reasoning tokens by up to $0.65$6 while preserving or slightly improving benchmark accuracy on GSM8K, SVAMP, and Math500 (Yu et al., 12 Jun 2025). Together these papers suggest that more prompt text is not equivalent to more usable control.

A third controversy concerns fairness and ranking. If different models prefer different prompts, then using one common prompt across models may itself introduce evaluation bias. TP-Eval argues precisely this for multimodal benchmarks and replaces one-size-fits-all prompting with per-model optimized prompts (Xie et al., 2024). By contrast, multi-prompt evaluation papers such as (Mizrahi et al., 2023) and (Kostiuk et al., 21 May 2026) do not primarily seek each model’s optimum, but rather a distribution-aware characterization of robustness. These are different evaluation philosophies: one targets latent potential, the other prompt robustness.

A fourth concern is evaluator-side opacity. "What Single-Prompt Accuracy Misses" shows that calibration definitions, answer parsing, and confidence elicitation materially affect reported reliability (Karmakar et al., 3 May 2026). This suggests that prompt sensitivity cannot be isolated from the rest of the evaluation pipeline. The recommendation to publish evaluator logic, raw generations, and prompt-perturbation spread therefore extends the notion of single-prompt evaluation beyond prompting alone (Karmakar et al., 3 May 2026).

Across the literature, several future directions recur. "State of What Art?" advocates standardized sets of paraphrases and reporting of AvgP and MaxP or CPS (Mizrahi et al., 2023). PromptSuite focuses on automatic multi-prompt generation through modular perturbation of instruction, prompt format, demonstrations, and instance content, with the total number of prompt variations formalized as

$0.65$7

for $0.65$8 prompt components and $0.65$9 perturbations per component (Habba et al., 20 Jul 2025). The mathematical reasoning ceiling paper states that escaping the static single-prompt ceiling likely requires external routing between specialized strategies, fine-tuning on large formal-graph datasets, or hybrid LLM+symbolic architectures (Cazares, 20 Apr 2026).

The cumulative view is therefore not that single-prompt evaluation is useless, but that it is a narrow and often fragile measurement regime. It remains operationally convenient and, in some settings, necessary. Yet the modern literature increasingly treats any single-prompt score as incomplete unless accompanied by prompt-distribution statistics, evaluator specifications, or an explicit account of what notion of capability the prompt is intended to measure (Mizrahi et al., 2023, Kostiuk et al., 21 May 2026, Karmakar et al., 3 May 2026).