---
title: Reliability Paradox in Model Evaluation
url: https://www.emergentmind.com/topics/reliability-paradox
type: topic
---

# Reliability Paradox in Model Evaluation

The **Reliability Paradox** denotes a family of paradoxical-looking but principled failures of common inferences from a narrow notion of reliability to a stronger claim of trustworthiness. In scientific XAI, a **reliable** model and a **faithful** post-hoc explanation do not justify claims about the **true structure of the phenomenon**; in language-model evaluation, lower calibration error can coexist with more shortcut-based, non-generalizable decision rules; in reasoning and agent benchmarks, correctness can mask unstable internal computation or stochastic drift; and in cognitive measurement, tasks with strong group-level effects can still exhibit poor between-individual reliability [2606.29346] [2412.15269] [2603.03475] [2605.24995]. The term is therefore used across several literatures, but the recurring pattern is the same: a favorable reliability signal is mistaken for a stronger epistemic or operational warrant.

## 1. A recurrent inferential pattern

Across domains, the paradox arises when one reliability criterion answers a narrower question than the one practitioners actually care about. In scientific XAI, reliability concerns whether a model’s output matches the phenomenon’s outcomes, whereas the desired conclusion is about causal, mechanistic, or counterfactual structure. In calibration work, Expected Calibration Error measures alignment between confidence and observed correctness, whereas the stronger target is robust, generalizable decision rules. In reasoning benchmarks, benchmark accuracy records whether a model was correct on a run, whereas the stronger target is stable, faithful computation across runs or paraphrases. In cognitive psychometrics, classical reliability indices address test–retest stability across individuals, whereas many tasks were optimized for strong within-subject effects rather than individual-differences measurement [2606.29346] [2412.15269] [2601.07058] [2605.24995].

This suggests a common structure: a metric or diagnostic that is well-defined for one inferential role is pressed into service for a stronger role that it does not, by itself, license. Different literatures formulate the stronger target differently—**how-actually** explanation, **decision-rule reliability**, **semantic stability**, **canonical path adherence**, or **between-individual reliability**—but each case turns on a gap between surface adequacy and the property of interest [2602.19008] [2312.14912].

## 2. Scientific XAI: reliability, faithfulness, and the structural gap

The most explicit formalization appears in "Reliability, Faithfulness, and the Limits of Post-hoc Explanations of Opaque Scientific Models" [2606.29346]. That work defines **Reliability** as the question whether the model’s output matches the phenomenon’s outcomes, and **Faithfulness** as the question whether the explanation accurately reflects what the model does. The stronger scientific claim, however, is whether the model explains the **structure of the phenomenon itself**—its causal, mechanistic, or counterfactual organization. The paper’s thesis is that **reliability + faithfulness do not compose into a warrant for structure claims**.

The formal chain is written as
\[
f \prec h^* \prec g^*
\]
where \(f\) is the true function representing the natural phenomenon, \(h^*\) is the trained predictive model, and \(g^* = E(h^*)\) is the post-hoc explanation produced by an explanation method \(E\). The paper distinguishes the **model-world link** between \(h^*\) and \(f\) from the **model-of-model link** between \(g^*\) and \(h^*\). A key terminological insistence is that \(g^*\) is not a candidate model of the world on the same level as \(h^*\); it is a **model of a model**.

The paper also separates two verdicts of different epistemic kinds. The **faithfulness verdict** is descriptive: it tells what the model computes, as a fact about the artifact in \(H\), accessed via \(E : H \to P\). The **reliability verdict** is justificatory, but only for prediction: it tells that the model tracks the phenomenon’s outcomes well enough to justify using the model’s predictions as evidence about \(f\). Neither verdict says whether the model works as the phenomenon works. That missing warrant is called the **structural gap** [2606.29346].

Crucially, the failure is described as **structural, not merely practical**. It persists even with perfect reliability and perfect faithfulness, because neither check ever asks whether the model’s internal organization matches the phenomenon’s organization. A model may be perfectly reliable while using completely different internal machinery from the phenomenon, and two equally predictive models can admit equally faithful explanations even if only one mirrors the phenomenon’s real structure. The appendix sharpens the argument with three cases: a confound that fails outside the training distribution, a confound that persists across all tested populations, and a **perfect tracker** whose internal structure still differs from the phenomenon’s structure. For that reason, post-hoc explanations can support only weaker claims unless supplemented by **external corroboration** such as independent mechanistic theory, evidence from similar systems, experimental intervention, theoretical or empirical constraints on the phenomenon, or prior knowledge about confounds [2606.29346].

## 3. Calibration and shortcut learning in language models

A second major use of the term concerns calibration. "The Reliability Paradox: Exploring How Shortcut Learning Undermines Language Model Calibration" argues that a model can appear better calibrated according to standard metrics such as **Expected Calibration Error (ECE)** while being less reliable in the sense that it relies more heavily on shortcut-based, non-generalizable decision rules [2412.15269]. The paper uses the standard binned form
\[
\mathrm{ECE}=\sum_{m=1}^{M} \frac{|B_m|}{n}\left|\mathrm{acc}(B_m)-\mathrm{conf}(B_m)\right|
\]
with
\[
\operatorname{acc}(B_m)=\frac{1}{|B_m|}\sum_{i\in B_m}\mathbf{1}(\hat{y}_i=y_i),
\qquad
\operatorname{conf}(B_m)=\frac{1}{|B_m|}\sum_{i\in B_m}\hat{p}_i.
\]
The central criticism is that ECE is blind to the **reason** a prediction is correct. If shortcut-based predictions happen to be accurate on the evaluation set, ECE may improve even though the decision rule is fragile.

To identify shortcuts, the paper combines **Integrated Gradients (IG)** for token attribution with **Local Mutual Information (LMI)** for dataset-level token-label association. A prediction is classified as shortcut-cued if the model’s top-3 attributed tokens also appear in the top 5% of LMI-scored tokens for the predicted label. It further distinguishes **Lexicon-cued** from **Grammar-cued** shortcuts. Across 8 text classification datasets and five transformer PLMs—BERT, RoBERTa, DeBERTa, ALBERT, and BART—the study reports that shortcut learning is above 50% for every task and model, and that models considered more calibrated in terms of ECE also tend to rely more on shortcuts. A concrete example is **BERT on COLA**, with \(P_{sc}=66.73\), \(T_{sc}=1.28\), and ECE \(=0.11\), versus **DeBERTa on COLA**, with \(P_{sc}=54.07\), \(T_{sc}=1.63\), and ECE \(=0.22\): BERT appears better calibrated but is more shortcut-dependent [2412.15269].

"Calibration vs Decision Making: Revisiting the Reliability Paradox in Unlearned Language Models" extends the same logic to decoder-only generative LMs in the machine unlearning setting [2605.20915]. On TOFU, under the RELU multiple-choice protocol, the study evaluates **probabilistic reliability** with ECE, MCE, and Brier score, and **decision-rule reliability** with IG and LMI. Fine-tuned models achieve low calibration error, with **ECE around 0.04** on retain splits, compared with **ECE > 0.5** for pretrained models; models after unlearning retain similarly low calibration despite reduced accuracy on the forget split. Yet attribution analysis shows continued or increased reliance on correlation-based tokens, with \(P_{SC}\) often above **85%**. The broader conclusion in both papers is the same: **low calibration error does not guarantee reliable decision rules** [2605.20915].

## 4. Benchmark correctness versus computational reliability

Recent work on reasoning models and agents reformulates the paradox as a gap between observed correctness and internal computational reliability. "When Shallow Wins: Silent Failures and the Depth-Accuracy Paradox in Latent Reasoning" studies **Qwen2.5-Math-7B** on **500 GSM8K problems** and reports **61% accuracy**, but only **18.4% of correct predictions** use **stable, faithful reasoning**, while **81.6% of correct predictions** emerge through **computationally inconsistent pathways**. The paper also identifies **8.8% of all predictions** as **silent failures**, defined as confident yet incorrect outputs [2603.03475].

The paper introduces a composite faithfulness metric
\[
F(q) = 0.35 \cdot S(q) +0.35 \cdot A(q) +0.30 \cdot \epsilon(q),
\]
with strict-faithfulness thresholds \(F \ge 0.60\), \(S \ge 0.65\), and \(E \ge 0.60\). It reports a weak negative correlation between reasoning quality and binary correctness, \(r=-0.21\), \(p=0.002\), but interprets this as a **binary threshold artifact** rather than a monotonic inverse relationship. Continuous analysis yields **AUROC = 0.78**, and the paper argues that one-shot accuracy can conceal unstable or shallow pathways. This is the reliability paradox in latent reasoning: benchmark accuracy can be decent even when the model is internally unreliable [2603.03475].

"Hallucinations Live in Variance" recasts the issue as **variance-driven instability under semantic perturbation** [2601.07058]. It distinguishes hallucination from **bias or missing knowledge** and from **calibration failure**, and defines **Paraphrase Consistency** by
\[
\mathrm{PC}@k(x) = \frac{\max_a |\{i : a_i = a\}|}{k},
\qquad
\mathrm{SS} = \mathbb{E}_x[\mathrm{PC}@k(x)].
\]
Using greedy decoding to remove sampling noise, the paper reports that a dense **Qwen3-0.6B** agrees with itself only **23.8%** of the time across paraphrases, while at **32.1% sparsity (R4)** agreement rises to **55.9%**. Its central warning is that **stability and correctness are orthogonal**: a model can be correct but unreliable, or reliable but wrong [2601.07058].

"Capable but Unreliable: Canonical Path Deviation as a Causal Mechanism of Agent Failure in Long-Horizon Tasks" applies the same distinction to tool-using agents [2602.19008]. The paper separates **Capability failure** from **Reliability failure** and studies **mixed-outcome units** in Toolathlon where the same model succeeds on some runs and fails on others solely because of LLM sampling stochasticity. For task \(j\), the canonical tool set is defined as
\[
C_j = \left\{ t \;\middle|\; \frac{\sum_{\tau \in \mathcal{S}_j} \mathbf{1}[t \in \mathcal{T}(\tau)]}{|\mathcal{S}_j|} > 0.5 \right\},
\]
and adherence by Jaccard similarity,
\[
A(\tau, j) = \frac{|\mathcal{T}(\tau) \cap C_j|}{|\mathcal{T}(\tau) \cup C_j|}.
\]
Within the headline CF-LOO sample, successful runs are more adherent by **+0.060 Jaccard**, \(p<0.0001\), \(n=488\), and each off-canonical tool call raises the probability that the next call is also off-canonical by **22.7 percentage points**. A runtime intervention that restarts the **bottom tercile** of runs based on mid-trajectory canonical adherence yields **+8.8 percentage points** success lift among intervened runs. Here the paradox is that the agent is capable of solving the task, yet does not reliably stay within the task’s operating envelope [2602.19008].

## 5. Cognitive tasks, information-theoretic reliability, and statistical reliability

In cognitive psychology, the phrase refers to the observation that tasks can produce robust group-level effects while showing poor test–retest reliability across individuals. "Information-Theoretic Reliability is Robust to Analytic Choice: A 24-Specification Multiverse on Public Cognitive Test-Retest Data" follows the formulation associated with Hedge et al. (2018): cognitive tasks can show strong within-subject contrasts, often with **Cohen’s \(d > 1\)**, yet still produce poor between-individual reliability [2605.24995]. The paper introduces an information-theoretic complement to ICC, the difference between the empirical mutual information estimate and the Gaussian baseline implied by the observed correlation:
\[
MI(X_1; X_2) = H(X_1) + H(X_2) - H(X_1, X_2),
\]
\[
\widehat{MI}_{\mathrm{Gauss}} = -\tfrac{1}{2}\,\log\!\bigl(1 - \hat\rho^{\,2}\bigr),
\]
\[
NLR \equiv \widehat{MI}_{\mathrm{KSG}} - \widehat{MI}_{\mathrm{Gauss}} \quad \text{(nats)}.
\]

Across **50 estimable primary measures** from **Flanker, Stroop, Stop-Signal, Go/No-Go, and Posner**, the median \(NLR\) is **\(-0.138\) nats**, with interquartile range **\([-0.257,\,-0.034]\)**, and **0 of 50** primary measures exceed the headline rule. The companion **ICC(2,1)** analysis reproduces the classical pattern, and the **24-specification multiverse** yields **0 of 1,200** estimable cells passing the headline rule. The conclusion is negative but methodologically important: replacing or augmenting ICC with an information-theoretic reliability measure does **not** rescue these tasks from the reliability paradox, and the null is robust to analytic choice [2605.24995].

A distinct but related ambiguity appears in "Fisher's underworld and the behavioral-statistical reliability balance in scientific inference" [2312.14912]. That paper argues that reliability can mean either **behavioral reliability**, as coherence in a betting system, or **statistical reliability**, as frequentist-style validity in repeated use. Its “reliability paradox” is that methods can be **behaviorally safe but statistically conservative / uninformative**, or **statistically useful but behaviorally vulnerable / incoherent**. The paper introduces **invulnerability** and establishes the chain
\[
\text{generalized Bayes IM} \in \{\text{invulnerable IMs}\} \subseteq \{\text{valid IMs}\} \subseteq \{\text{no-sure-loss IMs}\}.
\]
Its broader conclusion is that scientific inference requires a balance between behavioral and statistical priorities rather than a single undifferentiated notion of reliability [2312.14912].

## 6. Other formulations: source trust, forecasting, path-finding, and software systems

The term also appears in source aggregation and belief merging. "Merging with unknown reliability" argues that **unknown reliability is not the same as equal reliability** [2101.02516]. With source 1 assigning \([3,1,0]\) and source 2 assigning \([0,1,3]\) to scenarios \(A,B,C\), equal weighting uniquely prefers \(B\), but learning that source 1 is twice as reliable as source 2 can make \(B\) and \(C\) tie. The paradox-like feature is that adding information about reliability can increase uncertainty in the merged result. The paper resolves this by modeling ignorance over reliability profiles: when every reliability profile is possible, the result leads to **maxcons-based merging** under drastic distance; when exactly one source is fully reliable but unknown, it leads to **arbitration** [2101.02516].

In ensemble forecasting, the paradox takes the form of the **signal-to-noise paradox**. "Ensemble reliability and the signal-to-noise paradox in large-ensemble subseasonal forecasts" studies cases where the correlation between the forecast ensemble mean and observations exceeds the correlation between the forecast ensemble mean and its own members, that is, \(r_{fo} > r_{ff}\) [2411.17694]. The paper argues that a perfectly reliable ensemble with sufficiently large \(N\) and \(M\) will not exhibit an SNP, but that finite \(M\) can create an apparent SNP even in a reliable system. In the ECMWF IFS 100-member subseasonal reforecasts, **NAO** shows symptoms of the SNP, yet the authors do not find robust evidence for an underestimation of predictable signals and do not exclude the possibility that the apparent paradox is a consequence of observational sampling uncertainties. An unbiased reliability calibration can remove the apparent SNP, but the paper warns that this may overfit and produce unphysical lead-time variations [2411.17694].

In network path-finding, "Fault-Tolerant, but Paradoxical Path-Finding in Physical and Conceptual Systems" shows that when risk increases uniformly, the most reliable path can switch from **wider and longer** to **shorter and narrower** [1406.2234]. The models operate on directed acyclic graphs with edge-failure probabilities, and the paradoxical implication is that a novice, represented by higher edge-failure probability, may be better served by instructions with fewer back-up plans, whereas an expert may benefit from richer alternative routes.

A contemporary organizational version is the **Productivity-Reliability Paradox** in AI-assisted software development. "The Productivity-Reliability Paradox: Specification-Driven Governance for AI-Augmented Software Development" defines this as the coexistence of statistically significant improvements in individual-level productivity with statistically significant degradation in system-level dependability metrics [2605.01160]. The paper attributes the pattern to non-deterministic code generation and insufficient specification discipline, and proposes the **Specification Governance Model** as a response. A plausible implication across these otherwise distant literatures is that reliability is never a free-standing property: it is always relative to a target claim, a decision context, and the external premises needed to connect performance to trust.

Source: https://www.emergentmind.com/topics/reliability-paradox