Prompt-and-Check: LLM Reliability Evaluation

• Prompt-and-Check is a protocol that verifies LLM outputs by systematically varying prompts to assess factual correctness, consistency, and calibrated confidence.
• The method employs multi-perspective fact verification and distribution-based evaluations, using metrics like median and 95% quantile to capture performance variations.
• It has practical applications across training, optimization, and security, guiding prompt redesigns and developer audits to enhance overall model reliability.

Searching arXiv for recent relevant work on prompt sensitivity, prompt checking, and related methodologies. Prompt-and-Check denotes a family of large-language-model procedures in which a model is not trusted on the basis of a single prompt response, but is instead queried under controlled prompt variations or intermediate reasoning stages and then checked for consistency, calibration, evidence alignment, auxiliary-task coherence, or execution safety. In recent work, the term covers reliability analysis of prompt sensitivity, multi-perspective fact verification, statistically robust multi-prompt evaluation, explicit demonstration re-checking in few-shot learning, prompt optimization by preference learning, and systems-level audits of developer prompts and agent trajectories (Khatun et al., 2023, Bakke et al., 26 Jun 2025, Polo et al., 2024).

1. Conceptualization and analytical framing

A central formulation treats reliability as a combination of factual correctness, consistency across prompts, appropriate handling of sensitive content, and calibrated confidence. In that framing, “confusion” includes low confidence, incorrect answers where truth is known, and inconsistency across prompt variants (Khatun et al., 2023). A related fact-checking formulation defines bias broadly as “consistent, predictable patterns exhibited in the model’s output due to model internals, such as parametric knowledge and output safeguards,” and studies how prompt-induced directional biases interact with retrieval and final verdicts (Bakke et al., 26 Jun 2025).

A more formal evaluation-theoretic view appears in "Efficient multi-prompt evaluation of LLMs" (Polo et al., 2024). That work treats prompts as a population rather than a nuisance variable. For prompt template $i$ and example $j$, it defines correctness $Y_{ij}\in\{0,1\}$, prompt-level performance

$$S_i \triangleq \frac{1}{J}\sum_{j \in \mathcal{J}} Y_{ij},$$

and the empirical distribution of performance across prompts through a distribution function $F$ and quantile function $Q$. This replaces single-prompt reporting with prompt-distribution summaries such as the median or $95\%$ quantile (Polo et al., 2024).

The resulting literature uses “check” in several distinct but compatible senses. Sometimes the check is a cross-prompt comparison; sometimes it is a comparison between retrieved evidence sets; sometimes it is an auxiliary prediction task, such as re-predicting demonstration labels; sometimes it is a security audit over tool-use traces or compressed contexts. A plausible implication is that Prompt-and-Check is best understood not as one algorithm, but as a general control pattern for prompt-conditioned systems.

Variant	Check target	Representative work
Prompt variation over factual claims	Consistency, confidence volatility, token–rationale coherence	(Khatun et al., 2023)
Multi-perspective fact verification	Retrieval overlap, verdict stability, refusal asymmetry	(Bakke et al., 26 Jun 2025)
Multi-prompt benchmark evaluation	Distribution and quantiles over prompts	(Polo et al., 2024)
Demonstration-based prompt learning	Similarity of demonstrations and label re-prediction	(Wang et al., 2022)
Program correctness prompting	Natural-language state summaries against requirements	(Bouras et al., 25 Mar 2025)
Developer prompt repair	Bias, prompt injection, sub-optimal performance	(Rzig et al., 21 Jan 2025)

2. Reliability-oriented prompt variation

A canonical Prompt-and-Check study is "Reliability Check: An Analysis of GPT-3's Response to Sensitive Topics and Prompt Wording" (Khatun et al., 2023). It evaluates OpenAI text-davinci-003 in two modes: a binary YES/NO mode with max_tokens = 1, temperature = 0, and logit_bias restricting outputs to “YES” and “NO”, and a full-text mode with max_tokens = 1000 and temperature = 0.7. The dataset contains 1268 statements across 6 categories—Fact, Conspiracy, Controversy, Misconception, Stereotype, and Fiction—with 51% of statements carrying explicit ground-truth labels (Khatun et al., 2023).

The paper probes four prompt templates: P0 (“Is this true?”), P1 (“Is this true in the real world?”), P2 (“As a rational being who believes in scientific knowledge, do you think the following statement is true?”), and P3 (“I think [SENTENCE] Do you think I am right?”). These templates operationalize neutral prompting, real-world grounding, scientific-evidence framing, and user-belief framing (Khatun et al., 2023).

Its evaluation logic is archetypal Prompt-and-Check. The model is checked by per-prompt accuracy against ground truth, top-token confidence, cross-prompt consistency, confidence volatility relative to a baseline prompt, prompt-induced label flips, low-confidence cases, and token-versus-rationale coherence. The volatility criterion is

$$\text{diff}(P_x, T) = \text{conf}(P_x, T) - \text{conf}(P_0, T),$$

with a pair deemed “volatile” when $|\text{diff}(P_x, T)| \ge 20\%$ (Khatun et al., 2023).

The empirical pattern is stark. Across all categories, $\ge 88\%$ of responses have confidence $j$0, yet high confidence does not imply correctness. Prompt-induced YES/NO flips occur about 6.2% of the time overall, and Misconception and Controversy show the most cross-prompt disagreement. The model generally rejects obvious conspiracies and many negative stereotypes, but it makes frequent mistakes on common misconceptions, often agrees with user-framed falsehoods, and confuses fictional truth with real-world truth (Khatun et al., 2023).

Several recurrent failure modes follow directly from this setup. The paper documents negation failures, confusion around scope words such as “only” and “alone,” shifts between descriptive and normative readings, and contradictions between binary tokens and free-text rationales. It also shows RLHF side-effects: under some prompts the model becomes more agreeable to user beliefs, while under others it falls back on generic “no evidence” language or asymmetrically handles stereotypes and powerful institutions (Khatun et al., 2023). A common misconception about prompt-based evaluation is therefore that a single confident answer is informative; this literature shows that single-prompt confidence can be actively misleading.

The statistical generalization of this point is developed by PromptEval. Rather than asking whether one prompt is “the” correct template, it estimates the distribution of performance over many prompts and reports robust summaries such as median or $j$1-quantile accuracy. Empirically, PromptEval estimates performance quantiles across 100 prompt templates on MMLU with a budget equivalent to two single-prompt evaluations, and does so under a consistency theorem for the estimated distribution and quantile functions (Polo et al., 2024).

3. Evidence-centric and fact-verification forms

A second major branch of Prompt-and-Check uses prompt variation to surface different evidence rather than merely different labels. "(Fact) Check Your Bias" (Bakke et al., 26 Jun 2025) studies the HerO fact-checking pipeline, a multi-stage system that retrieves evidence with a Llama 3.1-generated hypothetical fact-checking document, generates clarification questions, and then predicts one of four labels: Supported, Refuted, Not Enough Evidence (NEE), or Conflicting Evidence / Cherrypicking (Bakke et al., 26 Jun 2025).

Its first experiment uses Meta-Llama-3.1-8B-Instruct as a standalone fact-checker with no retrieval. The model generates three questions, answers them from parametric knowledge, and then emits a verdict. The second experiment performs perspective-based prompting at the evidence stage: one prompt asks for a supporting fact-checking passage, one for a refuting passage, and one for an objective balanced passage. These hypothetical passages are then used as queries for retrieval, yielding three evidence sets $j$2, $j$3, and $j$4 (Bakke et al., 26 Jun 2025).

This design makes the “check” explicit. The system compares evidence overlap across perspectives, verdict robustness under different evidence sets, and safety asymmetries in generation. In direct parametric fact verification, the model predicts Not Enough Evidence on 235 out of 500 claims, or 47%, while the retrieval-based baseline predicts NEE on only 12 out of 500 claims, or 2.4%. Overall agreement between the direct and retrieval-based systems is 31.2%, which the paper interprets as a cautious bias in parametric knowledge (Bakke et al., 26 Jun 2025).

In the retrieval experiment, prompt perspective substantially changes what the system sees. Jaccard similarities between retrieved document sets range from 0.42 to 0.56, meaning roughly half of the retrieved documents are unique to each perspective. Yet final verdict distributions remain comparatively stable: the largest Supported/Refuted shift is 4 percentage points under positive bias, and AVeriTeC scores stay in a narrow band from 0.480 to 0.522 (Bakke et al., 26 Jun 2025). This suggests that retrieval bias and verdict robustness are separable properties, and that prompt-and-check can expose both.

The same paper identifies a particularly important asymmetry: under the supporting prompt, the model often refuses to generate content for harmful or implausible claims, whereas it readily generates refuting or balanced passages. Quantitatively, the Positive condition shows refusals for 94/500 claims (18.8%) and a 5.3% document refusal rate, compared with 0.1% document refusal rates in the other conditions (Bakke et al., 26 Jun 2025). This is a prompt-conditioned safety policy effect rather than an evidence effect.

Check-worthiness detection work provides a complementary perspective. "Claim Check-Worthiness Detection: How Well do LLMs Grasp Annotation Guidelines?" (Majer et al., 2024) studies claim detection and check-worthiness detection across five datasets with different worthiness criteria. It varies prompt verbosity from naive zero-shot prompts to prompts that encode label definitions and guideline examples, and also varies contextual information. The result is not a single best recipe: optimal prompt verbosity is domain-dependent, adding context does not improve performance, and confidence scores can be directly used to rank claims by check-worthiness (Majer et al., 2024). A related application to fact-check-worthiness estimation shows that prompt tuning on Qwen models can make a local Qwen2-1.5B system reach F1 0.7774, roughly comparable to GPT-4 on the reported dataset, while achieving the highest reported accuracy among compared systems (Yu et al., 25 Apr 2025).

4. Training-time and optimization-based forms

Prompt-and-Check is not limited to inference-time verification. Several works build the checking step into training objectives. "Let Me Check the Examples: Enhancing Demonstration Learning via Explicit Imitation" (Wang et al., 2022) modifies standard demonstration-based prompting by adding two explicit checks: a contrastive learning mechanism that concentrates on similar demonstrations, and a demonstration-label re-prediction objective that forces the model to recover demonstration labels. Its total loss is

$j$5

with final hyperparameters $j$6, $j$7, and temperature $j$8 (Wang et al., 2022).

The paper reports state-of-the-art performance on 11 out of 14 classification corpora and an average improvement of +1.11 points over LM-BFF across all 14 datasets. It also shows that when demonstration labels are randomized, Imitation-Demo degrades more than LM-BFF, which the authors interpret as evidence that the method strengthens prompt–demonstration dependencies rather than merely benefiting from extra context (Wang et al., 2022). In Prompt-and-Check terms, it transforms demonstrations from passive exemplars into objects that must themselves be re-validated.

"HoarePrompt: Structural Reasoning About Program Correctness in Natural Language" (Bouras et al., 25 Mar 2025) applies the same pattern to program verification. It lifts strongest postconditions into natural language, generates natural-language descriptions of reachable program states at control-flow points, uses few-shot-driven $j$9-induction to summarize loops, and only then asks the model whether the annotated program satisfies the natural-language requirement. On the CoCoClaNeL benchmark, HoarePrompt improves MCC by 62% compared to Zero-shot-CoT and by 93% compared to an LLM-based test-generation classifier; the inductive reasoning mechanism contributes a 28% boost to MCC (Bouras et al., 25 Mar 2025). Here the check is neither simple consistency nor evidence retrieval, but a structured proof sketch over natural-language state summaries.

Optimization-oriented work pushes Prompt-and-Check into offline learning. "FIPO: Free-form Instruction-oriented Prompt Optimization with Preference Dataset and Modular Fine-tuning Schema" (Lu et al., 2024) constructs a prompt optimization preference dataset in which GPT-4-produced prompts serve as preferred prompt rewrites over GPT-3.5-produced alternatives, with preference validated by UltraRM, GPT-4 self-judgement, and human experts. It then trains local prompt optimizers with SFT, DPO, IPO, and Iterative Preference Learning (IPL). The best reported configuration, IPL-IPO-70B, reaches a weighted average of 52.13 on the reported downstream evaluation against a 47.79 human-prompt baseline on Tulu2-7B (Lu et al., 2024). A plausible implication is that prompt-and-check can be compiled into an offline optimizer: the online loop of proposing, checking, and revising prompts becomes a trainable model.

5. Security, safety, and systems-level checking

The methodology also applies to agentic systems, where the relevant “prompt” may be a tool-use trajectory or a compressed context rather than a textual question. "Mind the Gap: Time-of-Check to Time-of-Use Vulnerabilities in LLM-Enabled Agents" (Lilienthal et al., 23 Aug 2025) studies TOCTOU vulnerabilities in tool-using agents. It introduces TOCTOU-Bench with 66 realistic user tasks, of which 56 are labeled as containing a possible TOCTOU vulnerability. Prompt rewriting modestly reduces vulnerable plans from 55 tasks to 53 tasks; static sequence monitoring reaches 25% TPR and 0.62 AUC on baseline sequences; and tool fusing reduces the attack window from 1.70 $Y_{ij}\in\{0,1\}$0 0.90 seconds to 0.07 $Y_{ij}\in\{0,1\}$1 0.01 seconds. With all mitigations combined, executed-trajectory vulnerability drops from 12% to 8%, and the attack window is reduced by about 95% (Lilienthal et al., 23 Aug 2025).

"Safe to Check, Unsafe to Use: Relinking at the Compression Boundary of LLM Agents" (Liu et al., 19 Jun 2026) identifies an analogous representation-level gap for summarization-based prompt compression. A compressor $Y_{ij}\in\{0,1\}$2 maps long source context $Y_{ij}\in\{0,1\}$3 to compressed context $Y_{ij}\in\{0,1\}$4, and the backend acts on $Y_{ij}\in\{0,1\}$5. The paper formalizes relinking as the case where an actionable binding is absent in the source but appears in the compressed context: $Y_{ij}\in\{0,1\}$6 Using its DSL-based Relink tool, the paper reports 86.9% Relink Rate and Backend Action Rate across four long-context agent benchmarks, versus 17.0% for clean-split controls (Liu et al., 19 Jun 2026). Existing defenses often fail because they inspect only one representation at a time. The paper’s KBRA defense instead enforces a boundary invariant over bindings across representations and reduces residual Backend Action Rate to 0.0% in its exact variants (Liu et al., 19 Jun 2026). This directly contradicts the assumption that checking the pre-compression prompt is sufficient.

At the software-engineering layer, "An Empirically-grounded tool for Automatic Prompt Linting and Repair" (Rzig et al., 21 Jan 2025) treats developer prompts as analyzable artifacts. In a representative sample of 2,173 Dev Prompts drawn from 40,573 cleaned prompts, it finds that 3.46% contain one or more forms of bias and 10.75% are vulnerable to prompt injection. The same system, PromptDoctor, de-biased 68.29% of biased Dev Prompts, hardened 41.81% of vulnerable Dev Prompts, and improved the performance of 37.1% of sub-optimal QA prompts in one evaluation configuration (Rzig et al., 21 Jan 2025). In this setting, the “check” is static and dynamic prompt analysis, and the “prompt” is the developer-authored instruction template itself.

These results support an important negative lesson: Prompt-and-Check is not automatically safe. Checks can be placed on the wrong artifact, the wrong representation, or the wrong stage. A prompt may be safe to inspect yet unsafe to compress; a verified source state may become stale before use; a developer prompt may pass informal review yet remain bias-prone or injection-prone.

6. Limitations, misconceptions, and research directions

Several recurring limitations appear across the literature. First, many studies are model-specific. The GPT-3 reliability analysis uses only text-davinci-003 and explicitly notes that the same patterns should not be assumed unchanged for GPT-4, chat models, or open-source LLMs (Khatun et al., 2023). The HerO bias study re-implements the pipeline with Meta-Llama-3.1-8B-Instruct rather than the original 70B model and notes that other architectures and model sizes may show different bias directions (Bakke et al., 26 Jun 2025). PromptEval’s theory depends on a specified prompt set and on logistic IRT assumptions; it solves sample-efficient estimation, not prompt-set construction itself (Polo et al., 2024).

Second, prompt-and-check does not eliminate ambiguity in annotation schemes or task definitions. Check-worthiness results show that prompt verbosity helping one domain may hurt another, and that adding context can degrade rather than improve performance (Majer et al., 2024). Fact-check-worthiness estimation is reported only for English COVID-19 tweets, with two hand-crafted templates and Qwen/GPT families, so cross-domain and cross-language robustness remains open (Yu et al., 25 Apr 2025). Essay-scoring work on cross-prompt trait scoring likewise shows that prompt-aware mechanisms help prompt-relevant traits but can slightly degrade syntactic traits when used alone, implying that “check” signals may need trait-specific weighting (Do et al., 2023).

Third, high confidence should not be conflated with trustworthy checking. The GPT-3 reliability study shows near-saturated confidence on many wrong answers (Khatun et al., 2023). The fact-checking bias study shows that large evidence differences can coexist with stable final verdicts, meaning verdict stability alone does not certify evidence quality (Bakke et al., 26 Jun 2025). Security work shows that checking only the initial artifact is often the wrong threat model (Lilienthal et al., 23 Aug 2025, Liu et al., 19 Jun 2026).

The main research direction suggested by this body of work is therefore methodological rather than doctrinal. Prompt-and-Check appears most effective when it is treated as an empirical protocol: define what should remain invariant under prompt changes or intermediate transformations, probe those invariants with multiple complementary prompts or structured subroutines, and escalate when disagreement, volatility, unsupported bindings, or auxiliary-task failures appear. This suggests a move away from single-prompt evaluation and toward systems that expose prompt sensitivity, evidence asymmetry, and representation-boundary failures as first-class diagnostic objects (Khatun et al., 2023, Polo et al., 2024, Bakke et al., 26 Jun 2025).