---
title: 'Prompt-and-Check: LLM Reliability Evaluation'
url: https://www.emergentmind.com/topics/prompt-and-check
type: topic
---

# Prompt-and-Check: LLM Reliability Evaluation

Searching arXiv for recent relevant work on prompt sensitivity, prompt checking, and related methodologies.
Prompt-and-Check denotes a family of large-language-model procedures in which a model is not trusted on the basis of a single prompt response, but is instead queried under controlled prompt variations or intermediate reasoning stages and then checked for consistency, calibration, evidence alignment, auxiliary-task coherence, or execution safety. In recent work, the term covers reliability analysis of prompt sensitivity, multi-perspective fact verification, statistically robust multi-prompt evaluation, explicit demonstration re-checking in few-shot learning, prompt optimization by preference learning, and systems-level audits of developer prompts and agent trajectories [2306.06199, 2506.21745, 2405.17202].

## 1. Conceptualization and analytical framing

A central formulation treats reliability as a combination of **factual correctness**, **consistency across prompts**, **appropriate handling of sensitive content**, and **calibrated confidence**. In that framing, “confusion” includes low confidence, incorrect answers where truth is known, and inconsistency across prompt variants [2306.06199]. A related fact-checking formulation defines bias broadly as “consistent, predictable patterns exhibited in the model’s output due to model internals, such as parametric knowledge and output safeguards,” and studies how prompt-induced directional biases interact with retrieval and final verdicts [2506.21745].

A more formal evaluation-theoretic view appears in "Efficient multi-prompt evaluation of LLMs" [2405.17202]. That work treats prompts as a population rather than a nuisance variable. For prompt template \(i\) and example \(j\), it defines correctness \(Y_{ij}\in\{0,1\}\), prompt-level performance
\[
S_i \triangleq \frac{1}{J}\sum_{j \in \mathcal{J}} Y_{ij},
\]
and the empirical distribution of performance across prompts through a distribution function \(F\) and quantile function \(Q\). This replaces single-prompt reporting with prompt-distribution summaries such as the median or \(95\%\) quantile [2405.17202].

The resulting literature uses “check” in several distinct but compatible senses. Sometimes the check is a cross-prompt comparison; sometimes it is a comparison between retrieved evidence sets; sometimes it is an auxiliary prediction task, such as re-predicting demonstration labels; sometimes it is a security audit over tool-use traces or compressed contexts. A plausible implication is that Prompt-and-Check is best understood not as one algorithm, but as a general control pattern for prompt-conditioned systems.

| Variant | Check target | Representative work |
|---|---|---|
| Prompt variation over factual claims | Consistency, confidence volatility, token–rationale coherence | [2306.06199] |
| Multi-perspective fact verification | Retrieval overlap, verdict stability, refusal asymmetry | [2506.21745] |
| Multi-prompt benchmark evaluation | Distribution and quantiles over prompts | [2405.17202] |
| Demonstration-based prompt learning | Similarity of demonstrations and label re-prediction | [2209.00455] |
| Program correctness prompting | Natural-language state summaries against requirements | [2503.19599] |
| Developer prompt repair | Bias, prompt injection, sub-optimal performance | [2501.12521] |

## 2. Reliability-oriented prompt variation

A canonical Prompt-and-Check study is "Reliability Check: An Analysis of GPT-3's Response to Sensitive Topics and Prompt Wording" [2306.06199]. It evaluates OpenAI **text-davinci-003** in two modes: a binary YES/NO mode with `max_tokens = 1`, `temperature = 0`, and `logit_bias` restricting outputs to “YES” and “NO”, and a full-text mode with `max_tokens = 1000` and `temperature = 0.7`. The dataset contains **1268 statements** across **6 categories**—**Fact**, **Conspiracy**, **Controversy**, **Misconception**, **Stereotype**, and **Fiction**—with **51%** of statements carrying explicit ground-truth labels [2306.06199].

The paper probes four prompt templates: **P0** (“Is this true?”), **P1** (“Is this true in the real world?”), **P2** (“As a rational being who believes in scientific knowledge, do you think the following statement is true?”), and **P3** (“I think [SENTENCE] Do you think I am right?”). These templates operationalize neutral prompting, real-world grounding, scientific-evidence framing, and user-belief framing [2306.06199].

Its evaluation logic is archetypal Prompt-and-Check. The model is checked by per-prompt accuracy against ground truth, top-token confidence, cross-prompt consistency, confidence volatility relative to a baseline prompt, prompt-induced label flips, low-confidence cases, and token-versus-rationale coherence. The volatility criterion is
\[
\text{diff}(P_x, T) = \text{conf}(P_x, T) - \text{conf}(P_0, T),
\]
with a pair deemed “volatile” when \(|\text{diff}(P_x, T)| \ge 20\%\) [2306.06199].

The empirical pattern is stark. Across all categories, **\(\ge 88\%\)** of responses have confidence **\(>97.5\%\)**, yet high confidence does not imply correctness. Prompt-induced YES/NO flips occur **about 6.2% of the time** overall, and **Misconception** and **Controversy** show the most cross-prompt disagreement. The model generally rejects obvious conspiracies and many negative stereotypes, but it makes frequent mistakes on common misconceptions, often agrees with user-framed falsehoods, and confuses fictional truth with real-world truth [2306.06199].

Several recurrent failure modes follow directly from this setup. The paper documents **negation failures**, confusion around scope words such as “only” and “alone,” shifts between descriptive and normative readings, and contradictions between binary tokens and free-text rationales. It also shows RLHF side-effects: under some prompts the model becomes more agreeable to user beliefs, while under others it falls back on generic “no evidence” language or asymmetrically handles stereotypes and powerful institutions [2306.06199]. A common misconception about prompt-based evaluation is therefore that a single confident answer is informative; this literature shows that single-prompt confidence can be actively misleading.

The statistical generalization of this point is developed by PromptEval. Rather than asking whether one prompt is “the” correct template, it estimates the distribution of performance over many prompts and reports robust summaries such as median or \(95\%\)-quantile accuracy. Empirically, PromptEval estimates performance quantiles across **100 prompt templates on MMLU** with a budget equivalent to **two single-prompt evaluations**, and does so under a consistency theorem for the estimated distribution and quantile functions [2405.17202].

## 3. Evidence-centric and fact-verification forms

A second major branch of Prompt-and-Check uses prompt variation to surface different evidence rather than merely different labels. "(Fact) Check Your Bias" [2506.21745] studies the HerO fact-checking pipeline, a multi-stage system that retrieves evidence with a Llama 3.1-generated hypothetical fact-checking document, generates clarification questions, and then predicts one of four labels: **Supported**, **Refuted**, **Not Enough Evidence (NEE)**, or **Conflicting Evidence / Cherrypicking** [2506.21745].

Its first experiment uses Meta-Llama-3.1-8B-Instruct as a standalone fact-checker with no retrieval. The model generates three questions, answers them from parametric knowledge, and then emits a verdict. The second experiment performs perspective-based prompting at the evidence stage: one prompt asks for a **supporting** fact-checking passage, one for a **refuting** passage, and one for an **objective** balanced passage. These hypothetical passages are then used as queries for retrieval, yielding three evidence sets \(E^{(+)}(c)\), \(E^{(-)}(c)\), and \(E^{(0)}(c)\) [2506.21745].

This design makes the “check” explicit. The system compares evidence overlap across perspectives, verdict robustness under different evidence sets, and safety asymmetries in generation. In direct parametric fact verification, the model predicts **Not Enough Evidence** on **235 out of 500 claims**, or **47%**, while the retrieval-based baseline predicts NEE on only **12 out of 500 claims**, or **2.4%**. Overall agreement between the direct and retrieval-based systems is **31.2%**, which the paper interprets as a **cautious bias** in parametric knowledge [2506.21745].

In the retrieval experiment, prompt perspective substantially changes what the system sees. Jaccard similarities between retrieved document sets range from **0.42** to **0.56**, meaning roughly **half of the retrieved documents are unique** to each perspective. Yet final verdict distributions remain comparatively stable: the largest Supported/Refuted shift is **4 percentage points** under positive bias, and AVeriTeC scores stay in a narrow band from **0.480** to **0.522** [2506.21745]. This suggests that retrieval bias and verdict robustness are separable properties, and that prompt-and-check can expose both.

The same paper identifies a particularly important asymmetry: under the **supporting** prompt, the model often refuses to generate content for harmful or implausible claims, whereas it readily generates refuting or balanced passages. Quantitatively, the **Positive** condition shows refusals for **94/500 claims (18.8%)** and a **5.3%** document refusal rate, compared with **0.1%** document refusal rates in the other conditions [2506.21745]. This is a prompt-conditioned safety policy effect rather than an evidence effect.

Check-worthiness detection work provides a complementary perspective. "Claim Check-Worthiness Detection: How Well do LLMs Grasp Annotation Guidelines?" [2404.12174] studies claim detection and check-worthiness detection across five datasets with different worthiness criteria. It varies prompt verbosity from naive zero-shot prompts to prompts that encode label definitions and guideline examples, and also varies contextual information. The result is not a single best recipe: **optimal prompt verbosity is domain-dependent**, **adding context does not improve performance**, and confidence scores can be directly used to rank claims by check-worthiness [2404.12174]. A related application to fact-check-worthiness estimation shows that prompt tuning on Qwen models can make a local **Qwen2-1.5B** system reach **F1 0.7774**, roughly comparable to **GPT-4** on the reported dataset, while achieving the highest reported accuracy among compared systems [2504.18104].

## 4. Training-time and optimization-based forms

Prompt-and-Check is not limited to inference-time verification. Several works build the checking step into training objectives. "Let Me Check the Examples: Enhancing Demonstration Learning via Explicit Imitation" [2209.00455] modifies standard demonstration-based prompting by adding two explicit checks: a **contrastive learning mechanism** that concentrates on similar demonstrations, and a **demonstration-label re-prediction** objective that forces the model to recover demonstration labels. Its total loss is
\[
\mathcal{L} = \mathcal{L}_{\text{mask}} + \alpha \mathcal{L}_{\text{label}} + \beta \mathcal{L}_{\text{context}},
\]
with final hyperparameters \(\alpha = 1\), \(\beta = 5\), and temperature \(T = 5\) [2209.00455].

The paper reports state-of-the-art performance on **11 out of 14 classification corpora** and an average improvement of **+1.11 points** over LM-BFF across all 14 datasets. It also shows that when demonstration labels are randomized, Imitation-Demo degrades more than LM-BFF, which the authors interpret as evidence that the method strengthens prompt–demonstration dependencies rather than merely benefiting from extra context [2209.00455]. In Prompt-and-Check terms, it transforms demonstrations from passive exemplars into objects that must themselves be re-validated.

"HoarePrompt: Structural Reasoning About Program Correctness in Natural Language" [2503.19599] applies the same pattern to program verification. It lifts strongest postconditions into natural language, generates natural-language descriptions of reachable program states at control-flow points, uses few-shot-driven \(k\)-induction to summarize loops, and only then asks the model whether the annotated program satisfies the natural-language requirement. On the CoCoClaNeL benchmark, HoarePrompt improves **MCC by 62%** compared to Zero-shot-CoT and by **93%** compared to an LLM-based test-generation classifier; the inductive reasoning mechanism contributes a **28%** boost to MCC [2503.19599]. Here the check is neither simple consistency nor evidence retrieval, but a structured proof sketch over natural-language state summaries.

Optimization-oriented work pushes Prompt-and-Check into offline learning. "FIPO: Free-form Instruction-oriented Prompt Optimization with Preference Dataset and Modular Fine-tuning Schema" [2402.11811] constructs a prompt optimization preference dataset in which GPT-4-produced prompts serve as preferred prompt rewrites over GPT-3.5-produced alternatives, with preference validated by UltraRM, GPT-4 self-judgement, and human experts. It then trains local prompt optimizers with SFT, DPO, IPO, and **Iterative Preference Learning (IPL)**. The best reported configuration, **IPL-IPO-70B**, reaches a weighted average of **52.13** on the reported downstream evaluation against a **47.79** human-prompt baseline on Tulu2-7B [2402.11811]. A plausible implication is that prompt-and-check can be compiled into an offline optimizer: the online loop of proposing, checking, and revising prompts becomes a trainable model.

## 5. Security, safety, and systems-level checking

The methodology also applies to agentic systems, where the relevant “prompt” may be a tool-use trajectory or a compressed context rather than a textual question. "Mind the Gap: Time-of-Check to Time-of-Use Vulnerabilities in LLM-Enabled Agents" [2508.17155] studies TOCTOU vulnerabilities in tool-using agents. It introduces **TOCTOU-Bench** with **66 realistic user tasks**, of which **56** are labeled as containing a possible TOCTOU vulnerability. Prompt rewriting modestly reduces vulnerable plans from **55 tasks** to **53 tasks**; static sequence monitoring reaches **25%** TPR and **0.62** AUC on baseline sequences; and tool fusing reduces the attack window from **1.70 \(\pm\) 0.90 seconds** to **0.07 \(\pm\) 0.01 seconds**. With all mitigations combined, executed-trajectory vulnerability drops from **12%** to **8%**, and the attack window is reduced by about **95%** [2508.17155].

"Safe to Check, Unsafe to Use: Relinking at the Compression Boundary of LLM Agents" [2606.21732] identifies an analogous representation-level gap for summarization-based prompt compression. A compressor \(C_\theta\) maps long source context \(H\) to compressed context \(S = C_\theta(H,q,\beta)\), and the backend acts on \(S\). The paper formalizes **relinking** as the case where an actionable binding is absent in the source but appears in the compressed context:
\[
\mathrm{Ground}(H,\tau)=1,\quad \mathrm{Support}(H,\tau)=0,\quad \mathrm{Express}(S,\tau)=1.
\]
Using its DSL-based Relink tool, the paper reports **86.9%** Relink Rate and Backend Action Rate across four long-context agent benchmarks, versus **17.0%** for clean-split controls [2606.21732]. Existing defenses often fail because they inspect only one representation at a time. The paper’s KBRA defense instead enforces a boundary invariant over bindings across representations and reduces residual Backend Action Rate to **0.0%** in its exact variants [2606.21732]. This directly contradicts the assumption that checking the pre-compression prompt is sufficient.

At the software-engineering layer, "An Empirically-grounded tool for Automatic Prompt Linting and Repair" [2501.12521] treats developer prompts as analyzable artifacts. In a representative sample of **2,173 Dev Prompts** drawn from **40,573** cleaned prompts, it finds that **3.46%** contain one or more forms of bias and **10.75%** are vulnerable to prompt injection. The same system, PromptDoctor, **de-biased 68.29%** of biased Dev Prompts, **hardened 41.81%** of vulnerable Dev Prompts, and improved the performance of **37.1%** of sub-optimal QA prompts in one evaluation configuration [2501.12521]. In this setting, the “check” is static and dynamic prompt analysis, and the “prompt” is the developer-authored instruction template itself.

These results support an important negative lesson: Prompt-and-Check is not automatically safe. Checks can be placed on the wrong artifact, the wrong representation, or the wrong stage. A prompt may be safe to inspect yet unsafe to compress; a verified source state may become stale before use; a developer prompt may pass informal review yet remain bias-prone or injection-prone.

## 6. Limitations, misconceptions, and research directions

Several recurring limitations appear across the literature. First, many studies are model-specific. The GPT-3 reliability analysis uses only **text-davinci-003** and explicitly notes that the same patterns should not be assumed unchanged for GPT-4, chat models, or open-source LLMs [2306.06199]. The HerO bias study re-implements the pipeline with **Meta-Llama-3.1-8B-Instruct** rather than the original 70B model and notes that other architectures and model sizes may show different bias directions [2506.21745]. PromptEval’s theory depends on a specified prompt set and on logistic IRT assumptions; it solves sample-efficient estimation, not prompt-set construction itself [2405.17202].

Second, prompt-and-check does not eliminate ambiguity in annotation schemes or task definitions. Check-worthiness results show that prompt verbosity helping one domain may hurt another, and that adding context can degrade rather than improve performance [2404.12174]. Fact-check-worthiness estimation is reported only for English COVID-19 tweets, with two hand-crafted templates and Qwen/GPT families, so cross-domain and cross-language robustness remains open [2504.18104]. Essay-scoring work on cross-prompt trait scoring likewise shows that prompt-aware mechanisms help prompt-relevant traits but can slightly degrade syntactic traits when used alone, implying that “check” signals may need trait-specific weighting [2305.16826].

Third, high confidence should not be conflated with trustworthy checking. The GPT-3 reliability study shows near-saturated confidence on many wrong answers [2306.06199]. The fact-checking bias study shows that large evidence differences can coexist with stable final verdicts, meaning verdict stability alone does not certify evidence quality [2506.21745]. Security work shows that checking only the initial artifact is often the wrong threat model [2508.17155, 2606.21732].

The main research direction suggested by this body of work is therefore methodological rather than doctrinal. Prompt-and-Check appears most effective when it is treated as an **empirical protocol**: define what should remain invariant under prompt changes or intermediate transformations, probe those invariants with multiple complementary prompts or structured subroutines, and escalate when disagreement, volatility, unsupported bindings, or auxiliary-task failures appear. This suggests a move away from single-prompt evaluation and toward systems that expose prompt sensitivity, evidence asymmetry, and representation-boundary failures as first-class diagnostic objects [2306.06199, 2405.17202, 2506.21745].

Source: https://www.emergentmind.com/topics/prompt-and-check