ReliableBench: Reliability Benchmarking Overview

Updated 5 July 2026

ReliableBench is a family of evaluation frameworks designed to assess system reliability under perturbation, distribution shifts, and real-world stress beyond conventional metrics.
It benchmarks diverse systems such as LLM judges, tool-using agents, reward models, and tabular predictors by employing metrics like judge concordance, pass@k, and RETA.
These frameworks expose hidden failure modes that traditional accuracy measures overlook, offering actionable insights to enhance operational dependability.

Searching arXiv for papers on ReliableBench / ReliabilityBench to ground the article in current literature. ReliableBench is a name used in several recent arXiv works for reliability-centered benchmarking, rather than for a single universally standardized artifact. Across these works, the term denotes evaluation frameworks that shift emphasis away from single-shot accuracy or pass@1 and toward stability under perturbation, repeated execution, distribution shift, calibrated uncertainty, or top-quantile selection. The main instantiations in current usage are a curated safety-evaluation subset for LLM judges, a production-style stress benchmark for tool-using agents, and a reward-model benchmark centered on the RETA metric; adjacent work further treats ReliableBench as a design target for judge stress testing and tabular uncertainty benchmarking (Schwinn et al., 4 Feb 2026, Gupta, 3 Jan 2026, Chen et al., 21 Apr 2025, Dev et al., 5 Mar 2026, Costa et al., 27 May 2026).

1. Terminological scope and core idea

In current literature, “ReliableBench” and “ReliabilityBench” designate benchmark programs whose common purpose is to measure whether a system remains dependable when nominal performance is stressed by realistic failure modes. The evaluated object varies by paper: one benchmark targets LLM judges in adversarial safety evaluation, one targets tool-using LLM agents under production-like perturbations and tool/API faults, and one targets reward models by asking whether their top-ranked responses are actually high-quality under an oracle. A related line of work proposes infrastructure for constructing judge-reliability suites and explicitly positions it as a practical backend for a ReliableBench-style evaluation program (Schwinn et al., 4 Feb 2026, Gupta, 3 Jan 2026, Chen et al., 21 Apr 2025, Dev et al., 5 Mar 2026).

Variant	Evaluated system	Core formalism
ReliableBench in safety evaluation	LLM judges	$R(D)=E_{x\sim D}[1\{J(x)=Y(x)\}]$
ReliabilityBench for agents	Tool-using LLM agents	$R(k,\varepsilon,\lambda)$ and $\mathrm{pass}^k$
ReliableBench for reward models	Reward models	$\mathrm{RETA}(\eta)$
ReliableBench-style tabular evaluation	Tabular models under conformal prediction	SSC and SSCS

The shared methodological claim is not that conventional metrics are useless, but that they are incomplete. High in-distribution judge agreement, high pass@1, high AUC, or strong ranking accuracy can coexist with low reliability once perturbations, distribution shift, infrastructure failures, or aggressive top-quantile selection are introduced. This suggests that the name has become associated with reliability-first benchmarking: evaluation protocols designed to expose failure modes that are invisible to optimistic single-number summaries (Schwinn et al., 4 Feb 2026, Gupta, 3 Jan 2026, Costa et al., 27 May 2026).

2. ReliableBench for LLM-judge reliability in adversarial safety

In "A Coin Flip for Safety: LLM Judges Fail to Reliably Measure Adversarial Robustness" (Schwinn et al., 4 Feb 2026), ReliableBench is a curated evaluation subset of behaviors from HarmBench chosen because they remain “more consistently judgeable” across attacks and victim models. The motivating problem is that LLM-as-a-Judge frameworks, although commonly validated on static in-distribution data, degrade under the distribution shifts inherent to red teaming. The paper identifies three shifts: Attack Shift, where adversarial prompting distorts outputs toward high-perplexity or otherwise atypical patterns; Model Shift, where judges validated on one victim model underperform on another; and Data (Semantic) Shift, where harmfulness judgments vary by semantic category. The paper formalizes judge reliability on a distribution $D$ as

$R(D)=E_{x\sim D}[1\{J(x)=Y(x)\}],$

and reports that under attack, model, and semantic shift, $R(D)$ often drops to around random chance (Schwinn et al., 4 Feb 2026).

ReliableBench in this setting is built from HarmBench. The authors start from HarmBench’s 400 harmful queries, subsample 100 queries to allocate labeling budget while retaining semantic coverage, and then sort behaviors by average cross-judge accuracy from easy to hard. ReliableBench retains the top 41 behaviors. The benchmark uses English prompts and generations spanning HarmBench categories such as cyber intrusion, harassment/bullying, chemical/biological, illegal, and misinformation/disinformation. It is evaluated across 4 open-weight victim models—Gemma-3-1B, Llama-3.1-8B, Gemma-27B, and Qwen-3-32B—5 attack families—Direct Prompting, GCG, GCG-REINFORCE, BoN, and PAIR—and 4 judges—AegisGuard, the HarmBench Llama-2-13B classifier, JailJudge, and LlamaGuard-3 (Schwinn et al., 4 Feb 2026).

The study uses 6,642 human-verified labels. Human annotation is performed on a 1–5 scale based on intent and compliance, with harmful defined as rating $\ge 3$ and benign as $<3$ . The authors labeled 2,370 samples directly; an additional 4,272 were labeled via Labelbox after a 10-example quiz; and a manual review of 90 assignments achieved 95% agreement on binary harmful versus benign. Final human label counts are 1,437 harmful and 5,205 benign. The paper does not specify train/validation/test splits for ReliableBench, emphasizing instead its role as an evaluation subset for judge reliability (Schwinn et al., 4 Feb 2026).

The principal result is that restricting evaluation to the 41 easiest-to-judge behaviors improves average judge accuracy to about 70% from roughly 57%, with reduced variance across attacks and victim models. This is paired with JudgeStressTest, a 971-example dataset constructed to isolate failure cases where judges systematically disagree with humans. The paper also introduces the Judge Concordance score, defined for a sample $x$ by first computing $R(k,\varepsilon,\lambda)$ 0, then

$R(k,\varepsilon,\lambda)$ 1

A central finding is that high concordance does not reliably imply correctness relative to human ground truth (Schwinn et al., 4 Feb 2026).

A second major contribution is the correction of inflated Attack Success Rate. If $R(k,\varepsilon,\lambda)$ 2 is judge precision, then the paper recommends

$R(k,\varepsilon,\lambda)$ 3

For sampling-heavy attacks such as BoN, the paper argues that many apparent “successes” are judge false positives rather than genuinely harmful generations. It further recommends reporting

$R(k,\varepsilon,\lambda)$ 4

to quantify the chance that at least one flagged sample is truly harmful. The paper explicitly cautions against treating judge ensembles or threshold tuning as sufficient fixes, noting that ensembles and ROC thresholding do not robustly repair distribution-shift failures (Schwinn et al., 4 Feb 2026).

3. ReliabilityBench for production-style evaluation of tool-using agents

In "ReliabilityBench: Evaluating LLM Agent Reliability Under Production-Like Stress Conditions" (Gupta, 3 Jan 2026), ReliabilityBench evaluates tool-using agents across three dimensions: consistency under repeated execution, robustness to semantically equivalent perturbations, and fault tolerance under controlled tool/API failures. The benchmark’s core claim is that pass@1 on a clean prompt overestimates production readiness because deployed agents must remain stable under stochastic sampling, user paraphrase, reordering of constraints, and realistic infrastructure failures (Gupta, 3 Jan 2026).

Agent success is verified deterministically by a state-based oracle. If an agent executes

$R(k,\varepsilon,\lambda)$ 5

then execution succeeds when

$R(k,\varepsilon,\lambda)$ 6

Consistency is measured using $R(k,\varepsilon,\lambda)$ 7,

$R(k,\varepsilon,\lambda)$ 8

with the simplifying relation $R(k,\varepsilon,\lambda)$ 9 only under independence. Robustness is parameterized by perturbation intensity

$\mathrm{pass}^k$ 0

and fault tolerance by a fault intensity $\mathrm{pass}^k$ 1 such that

$\mathrm{pass}^k$ 2

These are unified in the reliability surface

$\mathrm{pass}^k$ 3

The paper also defines a surface volume $\mathrm{pass}^k$ 4 and degradation gradient $\mathrm{pass}^k$ 5 as derived quantities (Gupta, 3 Jan 2026).

The benchmark covers four domains—scheduling, travel, customer support, and e-commerce—using synthetic tool ecosystems with deterministic end-state verification rather than LLM judges. ReliabilityBench introduces action metamorphic relations that define correctness via end-state equivalence rather than text similarity. Perturbation categories include linguistic, structural, contextual, and temporal transformations. Fault injection follows a chaos-engineering-style framework with TransientTimeout, ConnectionReset, SoftRateLimit, HardRateLimit, PartialResponse, SchemaDrift, StaleData, and EmptyResponse. The main experimental grid comprises 1,280 episodes, $\mathrm{pass}^k$ 6 independent trials, $\mathrm{pass}^k$ 7, $\mathrm{pass}^k$ 8, two models—Gemini 2.0 Flash and GPT-4o—and two agent architectures—ReAct and Reflexion (Gupta, 3 Jan 2026).

The reported degradation under perturbation is substantial. For Gemini 2.0 Flash, baseline $\mathrm{pass}^k$ 9 pass rate is 96.88%, dropping to 88.12% at $\mathrm{RETA}(\eta)$ 0, with overall $\mathrm{RETA}(\eta)$ 1 of 91.04%. GPT-4o has baseline pass 95.00%, $\mathrm{RETA}(\eta)$ 2 pass 88.75%, and overall $\mathrm{RETA}(\eta)$ 3 of 90.42%. Under combined perturbation and fault injection, Gemini at $\mathrm{RETA}(\eta)$ 4 reaches 84.0% at $\mathrm{RETA}(\eta)$ 5. ReAct is more robust than Reflexion under combined stress, with surface volume 0.900 versus 0.875 and steeper degradation for Reflexion in $\mathrm{RETA}(\eta)$ 6; the paper reports $\mathrm{RETA}(\eta)$ 7 per 0.1 increment for Reflexion versus $\mathrm{RETA}(\eta)$ 8 for ReAct (Gupta, 3 Jan 2026).

The fault ablation identifies rate limiting as the most damaging isolated fault. At $\mathrm{RETA}(\eta)$ 9, timeout-only pass is 98.75%, partial-response-only is 97.50%, mixed baseline is 96.25%, and rate-limit-only is 93.75%. Recovery behavior likewise favors ReAct: 47 faults encountered, 80.9% successful recoveries, and +1.2 additional tool calls per fault, versus 52 faults, 67.3% recoveries, and +1.8 additional calls for Reflexion. The paper also reports a strong cost asymmetry: Gemini 2.0 Flash slightly outperforms GPT-4o by +0.62% in overall pass rate while costing 82× less, with total cost $D$ 09.77 over 480 episodes and cost per 100 episodes of $D$ 12.04 (Gupta, 3 Jan 2026).

4. ReliableBench for reward-model reliability and RETA

In "Establishing Reliability Metrics for Reward Models in LLMs" (Chen et al., 21 Apr 2025), the paper states that its pipeline can be adopted as ReliableBench—the reward-model reliability benchmark centered on the Reliable at $D$ 2 (RETA) metric. The motivating problem is that high reward-model scores do not necessarily correspond to actual human preferences, especially under RLHF or best-of- $D$ 3 style overoptimization. The paper argues that classical ranking metrics and best-of- $D$ 4 curves are unstable because they focus on a single selected response, whereas RM reliability should ask whether the top- $D$ 5 quantile selected by the reward model is genuinely high-quality under an oracle (Chen et al., 21 Apr 2025).

For a prompt $D$ 6, candidate responses $D$ 7 are generated from a reference policy, the reward model assigns scores $D$ 8, and an oracle provides quality scores $D$ 9. Let $R(D)=E_{x\sim D}[1\{J(x)=Y(x)\}],$ 0 denote the $R(D)=E_{x\sim D}[1\{J(x)=Y(x)\}],$ 1-quantile threshold of reward-model scores and define the top- $R(D)=E_{x\sim D}[1\{J(x)=Y(x)\}],$ 2 set

$R(D)=E_{x\sim D}[1\{J(x)=Y(x)\}],$ 3

The population RETA metric is

$R(D)=E_{x\sim D}[1\{J(x)=Y(x)\}],$ 4

RETA $R(D)=E_{x\sim D}[1\{J(x)=Y(x)\}],$ 5 corresponds to random selection. The paper also defines a finite-sample estimator using the $R(D)=E_{x\sim D}[1\{J(x)=Y(x)\}],$ 6 highest-scoring responses per prompt and proposes a resampling-and-smoothing estimator for asymptotic unbiasedness, with recommended subset sizes $R(D)=E_{x\sim D}[1\{J(x)=Y(x)\}],$ 7 and averaging over multiple resamples (Chen et al., 21 Apr 2025).

The benchmark pipeline performs prompt sampling with k-DPP, response generation from a reference policy, oracle labeling once, and then repeated reward-model evaluation without additional oracle cost. The helpfulness benchmark uses prompts from the Anthropic Helpful test split, k-DPP sampling with $R(D)=E_{x\sim D}[1\{J(x)=Y(x)\}],$ 8, $R(D)=E_{x\sim D}[1\{J(x)=Y(x)\}],$ 9 responses per prompt from Llama2-7B-Chat at $R(D)$ 0, and oracle scoring with GPT-4-Turbo-2024-04-09 using 10 averaged outputs, for a total labeling cost of approximately $R(D)$ 1k=20 $R(D)$ 264.0 (Chen et al., 21 Apr 2025).

The principal results are expressed as RETA-versus- $R(D)$ 3 curves and benchmark rankings. On the helpfulness benchmark, reported RETA $R(D)$ 4 values are 1.1840 for the oracle upper-bound reference, 1.0626 for Starling-7B, 1.0608 for Deberta-0.3B, 1.0567 for RMv1-7B, 1.0563 for RM5H-7B, 1.0555 for RMEns-3x7B, 1.0545 for RMv3-7B, 1.0541 for RMv2-7B, 1.0390 for Ziya-7B, 1.0363 for RAFT-3B, and 1.0322 for Pythia-1.4B. Starling-7B is strong across $R(D)$ 5; Deberta-0.3B surpasses it at large $R(D)$ 6 but degrades for small $R(D)$ 7; and RMEns-3x7B mitigates hacking relative to its constituents. On the multi-turn benchmark, RETA converges similarly, Starling-7B is best, Ziya-7B is second, and many reward models are near or below the random baseline, indicating that multi-turn reliability is harder (Chen et al., 21 Apr 2025).

The benchmark’s substantive claim is that RETA is more stable than best-of- $R(D)$ 8 style evaluation because it averages oracle quality over the top $R(D)$ 9 fraction rather than a single response. The paper further argues that per-prompt normalization reduces prompt-selection bias, and recommends using RETA curves to choose an operational $\ge 3$ 0 that avoids regions where reliability declines at very small quantiles. This is a materially different notion of reliability from judge accuracy or agent fault tolerance: it treats reliability as top-quantile selection fidelity under a fixed response distribution (Chen et al., 21 Apr 2025).

Two additional lines of work provide methodology that is explicitly presented as useful for a ReliableBench-style program. First, "Judge Reliability Harness: Stress Testing the Reliability of LLM Judges" (Dev et al., 5 Mar 2026) introduces an open-source library for constructing validation suites for LLM judges across free-response and agentic tasks. Its four-stage workflow consists of loading and normalizing a seed dataset, generating perturbed items through synthetic pipelines, performing human-in-the-loop review, and evaluating judges with standardized reporting. The perturbation families include label flip, format invariance, semantic paraphrase, verbosity bias, stochastic stability, synthetic ordinal generation, and agentic transcript editing with planning, editing, summarization, and optional verification components. Metrics include accuracy, false-positive and false-negative rates for binary tasks, and MAE, Pearson’s correlation, Spearman’s rank correlation, and concordance correlation coefficient for ordinal grading. The paper reports that no evaluated judge is uniformly reliable across benchmarks and perturbation types, and specifically emphasizes formatting brittleness and task-dependent failures (Dev et al., 5 Mar 2026).

Second, "High Performance, Low Reliability: Uncertainty Benchmarking for Tabular Foundation Models" (Costa et al., 27 May 2026) does not title itself ReliableBench, but its technical summary is framed as integration guidance for ReliableBench. Here, reliability refers to uncertainty quantification under conformal prediction rather than to judge behavior, agent execution, or reward-model ranking. The evaluation uses split conformal prediction with Least Ambiguous Class nonconformity,

$\ge 3$ 1

prediction sets

$\ge 3$ 2

and conditional-coverage diagnostics via Size-Stratified Coverage and the Size-Stratified Coverage Score,

$\ge 3$ 3

Across 112 TALENT datasets, TFMs attain the highest mean AUC—TabICL 0.890 ± 0.019 and PFN-v2 0.889 ± 0.019—but lower SSCS, approximately 0.494–0.517, than GBDTs, approximately 0.532–0.544. On synthetic high-noise, low-separation tasks, the trade-off intensifies: foundation models reach AUC 0.924 ± 0.005 with SSCS 0.614 ± 0.081, whereas GBDTs attain AUC 0.889 ± 0.007 with SSCS 0.840 ± 0.020 (Costa et al., 27 May 2026).

A further antecedent is "SuperBench: Improving Cloud AI Infrastructure Reliability with Proactive Validation" (Xiong et al., 2024). Although not named ReliableBench, it broadens the reliability-benchmarking idiom to AI infrastructure. SuperBench combines a comprehensive benchmark suite, a Validator that learns benchmark criteria using empirical-CDF similarity, and a Selector that balances validation time against incident penalties using survival modeling. In Azure production, the paper reports up to 22.61× increase in mean time between incidents and deployment over two years validating hundreds of thousands of GPUs. Its relevance is conceptual: it treats reliability as proactive validation under realistic gray-failure modes rather than as peak benchmark performance (Xiong et al., 2024).

6. Interpretation, limitations, and research trajectory

The current literature does not define a single canonical ReliableBench. Instead, it presents a family of reliability-first benchmark constructions with different targets, metrics, and threat models. One misconception corrected across these works is that strong nominal performance implies operational reliability. In the judge setting, near-random performance can arise under attack, model, and semantic shift despite prior in-distribution validation (Schwinn et al., 4 Feb 2026). In the agent setting, perturbations alone reduce success from 96.9% at $\ge 3$ 4 to 88.1% at $\ge 3$ 5, and rate limiting causes the largest isolated degradation (Gupta, 3 Jan 2026). In the tabular setting, state-of-the-art AUC coexists with lower conditional coverage under conformal prediction (Costa et al., 27 May 2026). In the reward-model setting, a model can rank highly by conventional criteria yet remain unreliable at extreme top quantiles, which RETA exposes (Chen et al., 21 Apr 2025).

The limitations are correspondingly domain-specific. The judge-centric ReliableBench is English-only, single-turn, limited to open-weight victim models, and reports a 95% spot-check agreement rather than full $\ge 3$ 6, $\ge 3$ 7, or confidence intervals for the human-labeling pipeline (Schwinn et al., 4 Feb 2026). ReliabilityBench for agents uses simulated tool ecosystems, only two proprietary models, and evaluates $\ge 3$ 8 rather than larger repeated-run regimes (Gupta, 3 Jan 2026). RETA-based ReliableBench depends on the reference-policy distribution and on GPT-4-Turbo-2024-04-09 as oracle, which the paper treats as strong but not perfect (Chen et al., 21 Apr 2025). The judge-harness work relies on down-sampled datasets and validator models that may themselves introduce noise (Dev et al., 5 Mar 2026). The tabular uncertainty work notes that split conformal prediction guarantees marginal validity but not conditional coverage, so SSCS diagnoses rather than solves the problem (Costa et al., 27 May 2026).

Taken together, these works define a research trajectory in which reliability is measured not by a single aggregate score but by structured stress surfaces, stratified error analyses, conditional-coverage diagnostics, and reusable human-verified evaluation sets. A plausible implication is that “ReliableBench” is evolving into an umbrella label for benchmark designs that make hidden failure modes measurable: adversarial judge failures, stochastic inconsistency in agents, reward-model misselection at the top of the ranking, and uncertainty under distributional difficulty. Under that reading, the central contribution of the ReliableBench line is methodological rather than nominal: it recasts evaluation from best-case capability measurement to explicit reliability characterization under realistic stress (Schwinn et al., 4 Feb 2026, Gupta, 3 Jan 2026, Chen et al., 21 Apr 2025, Dev et al., 5 Mar 2026, Costa et al., 27 May 2026, Xiong et al., 2024).