Reliability Paradox in Model Evaluation

• Reliability Paradox is a recurring issue where metrics indicating model reliability are mistakenly taken as validation of true structural or causal fidelity.
• It demonstrates that high calibration and accuracy can coexist with shortcut-based, non-generalizable decision rules and unstable internal computations.
• Empirical studies across XAI, language models, reasoning benchmarks, and cognitive tasks reveal that favorable reliability signals require external corroboration to guarantee trustworthiness.

The Reliability Paradox denotes a family of paradoxical-looking but principled failures of common inferences from a narrow notion of reliability to a stronger claim of trustworthiness. In scientific XAI, a reliable model and a faithful post-hoc explanation do not justify claims about the true structure of the phenomenon; in language-model evaluation, lower calibration error can coexist with more shortcut-based, non-generalizable decision rules; in reasoning and agent benchmarks, correctness can mask unstable internal computation or stochastic drift; and in cognitive measurement, tasks with strong group-level effects can still exhibit poor between-individual reliability (Oh et al., 28 Jun 2026, Bihani et al., 2024, Sahoo et al., 3 Mar 2026, Westrin, 24 May 2026). The term is therefore used across several literatures, but the recurring pattern is the same: a favorable reliability signal is mistaken for a stronger epistemic or operational warrant.

1. A recurrent inferential pattern

Across domains, the paradox arises when one reliability criterion answers a narrower question than the one practitioners actually care about. In scientific XAI, reliability concerns whether a model’s output matches the phenomenon’s outcomes, whereas the desired conclusion is about causal, mechanistic, or counterfactual structure. In calibration work, Expected Calibration Error measures alignment between confidence and observed correctness, whereas the stronger target is robust, generalizable decision rules. In reasoning benchmarks, benchmark accuracy records whether a model was correct on a run, whereas the stronger target is stable, faithful computation across runs or paraphrases. In cognitive psychometrics, classical reliability indices address test–retest stability across individuals, whereas many tasks were optimized for strong within-subject effects rather than individual-differences measurement (Oh et al., 28 Jun 2026, Bihani et al., 2024, Flouro et al., 11 Jan 2026, Westrin, 24 May 2026).

This suggests a common structure: a metric or diagnostic that is well-defined for one inferential role is pressed into service for a stronger role that it does not, by itself, license. Different literatures formulate the stronger target differently—how-actually explanation, decision-rule reliability, semantic stability, canonical path adherence, or between-individual reliability—but each case turns on a gap between surface adequacy and the property of interest (Lee, 22 Feb 2026, Martin, 2023).

2. Scientific XAI: reliability, faithfulness, and the structural gap

The most explicit formalization appears in "Reliability, Faithfulness, and the Limits of Post-hoc Explanations of Opaque Scientific Models" (Oh et al., 28 Jun 2026). That work defines Reliability as the question whether the model’s output matches the phenomenon’s outcomes, and Faithfulness as the question whether the explanation accurately reflects what the model does. The stronger scientific claim, however, is whether the model explains the structure of the phenomenon itself—its causal, mechanistic, or counterfactual organization. The paper’s thesis is that reliability + faithfulness do not compose into a warrant for structure claims.

The formal chain is written as

$f \prec h^* \prec g^*$

where $f$ is the true function representing the natural phenomenon, $h^*$ is the trained predictive model, and $g^* = E(h^*)$ is the post-hoc explanation produced by an explanation method $E$. The paper distinguishes the model-world link between $h^*$ and $f$ from the model-of-model link between $g^*$ and $h^*$. A key terminological insistence is that $g^*$ is not a candidate model of the world on the same level as $f$0; it is a model of a model.

The paper also separates two verdicts of different epistemic kinds. The faithfulness verdict is descriptive: it tells what the model computes, as a fact about the artifact in $f$1, accessed via $f$2. The reliability verdict is justificatory, but only for prediction: it tells that the model tracks the phenomenon’s outcomes well enough to justify using the model’s predictions as evidence about $f$3. Neither verdict says whether the model works as the phenomenon works. That missing warrant is called the structural gap (Oh et al., 28 Jun 2026).

Crucially, the failure is described as structural, not merely practical. It persists even with perfect reliability and perfect faithfulness, because neither check ever asks whether the model’s internal organization matches the phenomenon’s organization. A model may be perfectly reliable while using completely different internal machinery from the phenomenon, and two equally predictive models can admit equally faithful explanations even if only one mirrors the phenomenon’s real structure. The appendix sharpens the argument with three cases: a confound that fails outside the training distribution, a confound that persists across all tested populations, and a perfect tracker whose internal structure still differs from the phenomenon’s structure. For that reason, post-hoc explanations can support only weaker claims unless supplemented by external corroboration such as independent mechanistic theory, evidence from similar systems, experimental intervention, theoretical or empirical constraints on the phenomenon, or prior knowledge about confounds (Oh et al., 28 Jun 2026).

3. Calibration and shortcut learning in LLMs

A second major use of the term concerns calibration. "The Reliability Paradox: Exploring How Shortcut Learning Undermines LLM Calibration" argues that a model can appear better calibrated according to standard metrics such as Expected Calibration Error (ECE) while being less reliable in the sense that it relies more heavily on shortcut-based, non-generalizable decision rules (Bihani et al., 2024). The paper uses the standard binned form

$f$4

with

$f$5

The central criticism is that ECE is blind to the reason a prediction is correct. If shortcut-based predictions happen to be accurate on the evaluation set, ECE may improve even though the decision rule is fragile.

To identify shortcuts, the paper combines Integrated Gradients (IG) for token attribution with Local Mutual Information (LMI) for dataset-level token-label association. A prediction is classified as shortcut-cued if the model’s top-3 attributed tokens also appear in the top 5% of LMI-scored tokens for the predicted label. It further distinguishes Lexicon-cued from Grammar-cued shortcuts. Across 8 text classification datasets and five transformer PLMs—BERT, RoBERTa, DeBERTa, ALBERT, and BART—the study reports that shortcut learning is above 50% for every task and model, and that models considered more calibrated in terms of ECE also tend to rely more on shortcuts. A concrete example is BERT on COLA, with $f$6, $f$7, and ECE $f$8, versus DeBERTa on COLA, with $f$9, $h^*$0, and ECE $h^*$1: BERT appears better calibrated but is more shortcut-dependent (Bihani et al., 2024).

"Calibration vs Decision Making: Revisiting the Reliability Paradox in Unlearned LLMs" extends the same logic to decoder-only generative LMs in the machine unlearning setting (Shukla et al., 20 May 2026). On TOFU, under the RELU multiple-choice protocol, the study evaluates probabilistic reliability with ECE, MCE, and Brier score, and decision-rule reliability with IG and LMI. Fine-tuned models achieve low calibration error, with ECE around 0.04 on retain splits, compared with ECE > 0.5 for pretrained models; models after unlearning retain similarly low calibration despite reduced accuracy on the forget split. Yet attribution analysis shows continued or increased reliance on correlation-based tokens, with $h^*$2 often above 85%. The broader conclusion in both papers is the same: low calibration error does not guarantee reliable decision rules (Shukla et al., 20 May 2026).

4. Benchmark correctness versus computational reliability

Recent work on reasoning models and agents reformulates the paradox as a gap between observed correctness and internal computational reliability. "When Shallow Wins: Silent Failures and the Depth-Accuracy Paradox in Latent Reasoning" studies Qwen2.5-Math-7B on 500 GSM8K problems and reports 61% accuracy, but only 18.4% of correct predictions use stable, faithful reasoning, while 81.6% of correct predictions emerge through computationally inconsistent pathways. The paper also identifies 8.8% of all predictions as silent failures, defined as confident yet incorrect outputs (Sahoo et al., 3 Mar 2026).

The paper introduces a composite faithfulness metric

$h^*$3

with strict-faithfulness thresholds $h^*$4, $h^*$5, and $h^*$6. It reports a weak negative correlation between reasoning quality and binary correctness, $h^*$7, $h^*$8, but interprets this as a binary threshold artifact rather than a monotonic inverse relationship. Continuous analysis yields AUROC = 0.78, and the paper argues that one-shot accuracy can conceal unstable or shallow pathways. This is the reliability paradox in latent reasoning: benchmark accuracy can be decent even when the model is internally unreliable (Sahoo et al., 3 Mar 2026).

"Hallucinations Live in Variance" recasts the issue as variance-driven instability under semantic perturbation (Flouro et al., 11 Jan 2026). It distinguishes hallucination from bias or missing knowledge and from calibration failure, and defines Paraphrase Consistency by

$h^*$9

Using greedy decoding to remove sampling noise, the paper reports that a dense Qwen3-0.6B agrees with itself only 23.8% of the time across paraphrases, while at 32.1% sparsity (R4) agreement rises to 55.9%. Its central warning is that stability and correctness are orthogonal: a model can be correct but unreliable, or reliable but wrong (Flouro et al., 11 Jan 2026).

"Capable but Unreliable: Canonical Path Deviation as a Causal Mechanism of Agent Failure in Long-Horizon Tasks" applies the same distinction to tool-using agents (Lee, 22 Feb 2026). The paper separates Capability failure from Reliability failure and studies mixed-outcome units in Toolathlon where the same model succeeds on some runs and fails on others solely because of LLM sampling stochasticity. For task $g^* = E(h^*)$0, the canonical tool set is defined as

$g^* = E(h^*)$1

and adherence by Jaccard similarity,

$g^* = E(h^*)$2

Within the headline CF-LOO sample, successful runs are more adherent by +0.060 Jaccard, $g^* = E(h^*)$3, $g^* = E(h^*)$4, and each off-canonical tool call raises the probability that the next call is also off-canonical by 22.7 percentage points. A runtime intervention that restarts the bottom tercile of runs based on mid-trajectory canonical adherence yields +8.8 percentage points success lift among intervened runs. Here the paradox is that the agent is capable of solving the task, yet does not reliably stay within the task’s operating envelope (Lee, 22 Feb 2026).

5. Cognitive tasks, information-theoretic reliability, and statistical reliability

In cognitive psychology, the phrase refers to the observation that tasks can produce robust group-level effects while showing poor test–retest reliability across individuals. "Information-Theoretic Reliability is Robust to Analytic Choice: A 24-Specification Multiverse on Public Cognitive Test-Retest Data" follows the formulation associated with Hedge et al. (2018): cognitive tasks can show strong within-subject contrasts, often with Cohen’s $g^* = E(h^*)$5, yet still produce poor between-individual reliability (Westrin, 24 May 2026). The paper introduces an information-theoretic complement to ICC, the difference between the empirical mutual information estimate and the Gaussian baseline implied by the observed correlation: $g^* = E(h^*)$6

$g^* = E(h^*)$7

$g^* = E(h^*)$8

Across 50 estimable primary measures from Flanker, Stroop, Stop-Signal, Go/No-Go, and Posner, the median $g^* = E(h^*)$9 is $E$0 nats, with interquartile range $E$1, and 0 of 50 primary measures exceed the headline rule. The companion ICC(2,1) analysis reproduces the classical pattern, and the 24-specification multiverse yields 0 of 1,200 estimable cells passing the headline rule. The conclusion is negative but methodologically important: replacing or augmenting ICC with an information-theoretic reliability measure does not rescue these tasks from the reliability paradox, and the null is robust to analytic choice (Westrin, 24 May 2026).

A distinct but related ambiguity appears in "Fisher's underworld and the behavioral-statistical reliability balance in scientific inference" (Martin, 2023). That paper argues that reliability can mean either behavioral reliability, as coherence in a betting system, or statistical reliability, as frequentist-style validity in repeated use. Its “reliability paradox” is that methods can be behaviorally safe but statistically conservative / uninformative, or statistically useful but behaviorally vulnerable / incoherent. The paper introduces invulnerability and establishes the chain

$E$2

Its broader conclusion is that scientific inference requires a balance between behavioral and statistical priorities rather than a single undifferentiated notion of reliability (Martin, 2023).

6. Other formulations: source trust, forecasting, path-finding, and software systems

The term also appears in source aggregation and belief merging. "Merging with unknown reliability" argues that unknown reliability is not the same as equal reliability (Liberatore, 2021). With source 1 assigning $E$3 and source 2 assigning $E$4 to scenarios $E$5, equal weighting uniquely prefers $E$6, but learning that source 1 is twice as reliable as source 2 can make $E$7 and $E$8 tie. The paradox-like feature is that adding information about reliability can increase uncertainty in the merged result. The paper resolves this by modeling ignorance over reliability profiles: when every reliability profile is possible, the result leads to maxcons-based merging under drastic distance; when exactly one source is fully reliable but unknown, it leads to arbitration (Liberatore, 2021).

In ensemble forecasting, the paradox takes the form of the signal-to-noise paradox. "Ensemble reliability and the signal-to-noise paradox in large-ensemble subseasonal forecasts" studies cases where the correlation between the forecast ensemble mean and observations exceeds the correlation between the forecast ensemble mean and its own members, that is, $E$9 (Roberts et al., 2024). The paper argues that a perfectly reliable ensemble with sufficiently large $h^*$0 and $h^*$1 will not exhibit an SNP, but that finite $h^*$2 can create an apparent SNP even in a reliable system. In the ECMWF IFS 100-member subseasonal reforecasts, NAO shows symptoms of the SNP, yet the authors do not find robust evidence for an underestimation of predictable signals and do not exclude the possibility that the apparent paradox is a consequence of observational sampling uncertainties. An unbiased reliability calibration can remove the apparent SNP, but the paper warns that this may overfit and produce unphysical lead-time variations (Roberts et al., 2024).

In network path-finding, "Fault-Tolerant, but Paradoxical Path-Finding in Physical and Conceptual Systems" shows that when risk increases uniformly, the most reliable path can switch from wider and longer to shorter and narrower (Knowles et al., 2014). The models operate on directed acyclic graphs with edge-failure probabilities, and the paradoxical implication is that a novice, represented by higher edge-failure probability, may be better served by instructions with fewer back-up plans, whereas an expert may benefit from richer alternative routes.

A contemporary organizational version is the Productivity-Reliability Paradox in AI-assisted software development. "The Productivity-Reliability Paradox: Specification-Driven Governance for AI-Augmented Software Development" defines this as the coexistence of statistically significant improvements in individual-level productivity with statistically significant degradation in system-level dependability metrics (Farrag, 1 May 2026). The paper attributes the pattern to non-deterministic code generation and insufficient specification discipline, and proposes the Specification Governance Model as a response. A plausible implication across these otherwise distant literatures is that reliability is never a free-standing property: it is always relative to a target claim, a decision context, and the external premises needed to connect performance to trust.