Error Consistency (EC) in Classifier Evaluation

• Error Consistency (EC) is a metric that measures the trial-by-trial agreement of classifiers by comparing their binary correctness indicators using Cohen’s kappa adjusted for chance.
• It captures shared error patterns beyond accuracy by quantifying how often classifiers are jointly correct or incorrect on individual stimuli.
• Bootstrap resampling and generative copy models are employed to estimate uncertainty and guide the design of experiments using EC.

Searching arXiv for exact and related uses of “Error Consistency (EC)” to ground the article and disambiguate the term. {"query":"\"Error Consistency\" arXiv", "max_results": 10} Error Consistency (EC) is a behavioral agreement metric for comparing two classifiers at the level of individual stimuli rather than aggregate accuracy. In the usage established in model–human benchmarking and re-examined in "Quantifying Uncertainty in Error Consistency: Towards Reliable Behavioral Comparison of Classifiers" (Klein et al., 9 Jul 2025), EC measures whether two systems tend to be correct or incorrect on the same items, normalized for chance agreement given their marginal accuracies. Operationally, EC is Cohen’s kappa applied not to raw class labels but to binary correctness indicators. This makes EC more sensitive than accuracy to shared structure in errors, but also couples its interpretation to the accuracies of the compared systems (Klein et al., 9 Jul 2025).

1. Definition and conceptual scope

For two systems indexed by $j \in \{1,2\}$, each stimulus $x_i$ has a ground-truth label $y_i \in \{1,\dots,K\}$ and each system outputs a class prediction $\hat{y}_i^{(j)}$. EC is computed after converting each prediction sequence into a binary correctness sequence,

$$r_i^{(j)} = \begin{cases} 1 & \text{if } \hat{y}_i^{(j)} = y_i,\ 0 & \text{otherwise.} \end{cases}$$

The key object is therefore the pair of binary vectors $r^{(1)}, r^{(2)}$, not the original categorical responses (Klein et al., 9 Jul 2025).

This construction makes EC a metric of trial-by-trial behavioral alignment. Accuracy $p_j$ is only a marginal statistic: it records how often a system is correct, but not which stimuli were correct or incorrect. EC instead quantifies joint behavior beyond chance. In NeuroAI settings, this distinction is central because two models can have similar accuracies while differing substantially in which stimuli they fail on, or conversely can have different accuracies while sharing a substantial portion of their errors (Klein et al., 9 Jul 2025).

The same source also emphasizes a structural limitation: EC depends on the marginal accuracies. If one system is perfect, then $\kappa = 0$ regardless of the other system’s accuracy, because there are no errors to be “consistent” about; if both systems are perfect, EC is undefined. More generally, mismatched accuracies impose bounds that can make high EC impossible. Accordingly, EC must be interpreted together with accuracies rather than as a standalone behavioral statistic (Klein et al., 9 Jul 2025).

2. Formalization as Cohen’s kappa on correctness

Let

• $a = \#(r^{(1)}=1, r^{(2)}=1)$,
• $b = \#(r^{(1)}=1, r^{(2)}=0)$,
• $x_i$0,
• $x_i$1,

with $x_i$2. The marginal accuracies are

$x_i$3

Observed agreement and chance agreement are

$x_i$4

EC is then

$x_i$5

This is exactly Cohen’s kappa on the binary categories $x_i$6 (Klein et al., 9 Jul 2025).

The normalization by $x_i$7 removes the agreement expected under independence given the two systems’ marginal accuracies. Hence $x_i$8 corresponds to identical correctness patterns, $x_i$9 to independence, and $y_i \in \{1,\dots,K\}$0 to disagreement beyond chance (Klein et al., 9 Jul 2025).

The same formalism can be written in categorical form as

$y_i \in \{1,\dots,K\}$1

applied to the binary categories $y_i \in \{1,\dots,K\}$2. In the binary-correctness case,

$y_i \in \{1,\dots,K\}$3

and

$y_i \in \{1,\dots,K\}$4

This dependence on $y_i \in \{1,\dots,K\}$5 and $y_i \in \{1,\dots,K\}$6 explains why EC is bounded and can become unstable near floor or ceiling accuracies (Klein et al., 9 Jul 2025).

A further practical distinction concerns benchmark normalization. In Brain-Score, EC is ceiling-normalized by dividing model EC by human-vs-human EC. For uncertainty estimation, however, the reanalysis in (Klein et al., 9 Jul 2025) reverts to raw EC, bootstraps that quantity, and only then notes the relevance of ceiling-normalization.

3. Uncertainty quantification and statistical inference

A central contribution of (Klein et al., 9 Jul 2025) is that empirically measured EC values are typically noisy, so reporting point estimates without uncertainty makes benchmarking conclusions problematic. The proposed remedy is nonparametric bootstrap over paired trial outcomes.

In the single-condition case, one resamples with replacement from the paired correctness outcomes $y_i \in \{1,\dots,K\}$7, recomputes $y_i \in \{1,\dots,K\}$8 on each bootstrap sample, and forms a confidence interval from the empirical quantiles of the bootstrap distribution. The paper uses percentile intervals,

$y_i \in \{1,\dots,K\}$9

with the Highest Density (Posterior) Interval as an alternative. For CI size simulations the authors use $\hat{y}_i^{(j)}$0 replicates, and for ranking stability analyses $\hat{y}_i^{(j)}$1 replicates; the practical recommendation is that $\hat{y}_i^{(j)}$2–$\hat{y}_i^{(j)}$3 replicates are adequate (Klein et al., 9 Jul 2025).

In hierarchical settings, such as model-vs-human comparisons aggregated across observers, conditions, and corruptions, the bootstrap must pass through the same hierarchy as the score definition. Trials are resampled within each observer-condition cell, ECs are recomputed, then averaged across observers to condition-level EC, across conditions, and finally across corruption families if applicable. This hierarchical bootstrap is necessary because the score being reported is itself a hierarchical average (Klein et al., 9 Jul 2025).

The paper also formulates significance testing against independence. Under the null hypothesis $\hat{y}_i^{(j)}$4, the method estimates accuracies $\hat{y}_i^{(j)}$5 from the observed data, optionally samples them from Beta posteriors, then simulates independent Bernoulli correctness sequences and recomputes $\hat{y}_i^{(j)}$6. A two-sided $\hat{y}_i^{(j)}$7-value is

$\hat{y}_i^{(j)}$8

The method assumes independence under $\hat{y}_i^{(j)}$9, binomial trials, and Beta posteriors for accuracies given uniform priors. Because the null is simulated, the resolution of the resulting $$r_i^{(j)} = \begin{cases} 1 & \text{if } \hat{y}_i^{(j)} = y_i,\ 0 & \text{otherwise.} \end{cases}$$0-value is limited by $$r_i^{(j)} = \begin{cases} 1 & \text{if } \hat{y}_i^{(j)} = y_i,\ 0 & \text{otherwise.} \end{cases}$$1 (Klein et al., 9 Jul 2025).

The same source notes that kappa is a biased estimator, especially with skewed marginals. Near floor or ceiling accuracies, bootstrap resamples can contain edge cases such as apparent perfect accuracy, which forces $$r_i^{(j)} = \begin{cases} 1 & \text{if } \hat{y}_i^{(j)} = y_i,\ 0 & \text{otherwise.} \end{cases}$$2 toward $$r_i^{(j)} = \begin{cases} 1 & \text{if } \hat{y}_i^{(j)} = y_i,\ 0 & \text{otherwise.} \end{cases}$$3 or makes it undefined. This produces wide confidence intervals and a slight negative bias at low $$r_i^{(j)} = \begin{cases} 1 & \text{if } \hat{y}_i^{(j)} = y_i,\ 0 & \text{otherwise.} \end{cases}$$4 and extreme accuracies (Klein et al., 9 Jul 2025).

4. Generative interpretation via the copying model

Beyond interval estimation, (Klein et al., 9 Jul 2025) proposes a generative model that gives EC an interpretable latent parameter. In this model, one system’s response is copied by the other with probability $$r_i^{(j)} = \begin{cases} 1 & \text{if } \hat{y}_i^{(j)} = y_i,\ 0 & \text{otherwise.} \end{cases}$$5; otherwise the second system draws independently from its own marginal distribution. Formally, if $$r_i^{(j)} = \begin{cases} 1 & \text{if } \hat{y}_i^{(j)} = y_i,\ 0 & \text{otherwise.} \end{cases}$$6 is drawn from $$r_i^{(j)} = \begin{cases} 1 & \text{if } \hat{y}_i^{(j)} = y_i,\ 0 & \text{otherwise.} \end{cases}$$7, then

$$r_i^{(j)} = \begin{cases} 1 & \text{if } \hat{y}_i^{(j)} = y_i,\ 0 & \text{otherwise.} \end{cases}$$8

The observed marginals satisfy

$$r_i^{(j)} = \begin{cases} 1 & \text{if } \hat{y}_i^{(j)} = y_i,\ 0 & \text{otherwise.} \end{cases}$$9

This yields a direct bridge between agreement and a latent copying parameter (Klein et al., 9 Jul 2025).

When the marginals match, $r^{(1)}, r^{(2)}$0, Cohen’s kappa equals the copy probability:

$r^{(1)}, r^{(2)}$1

For EC, this condition holds whenever the two systems have equal correctness marginals, that is, equal accuracies. In the general mismatched case,

$r^{(1)}, r^{(2)}$2

where

$r^{(1)}, r^{(2)}$3

For binary correctness categories, this becomes

$r^{(1)}, r^{(2)}$4

Hence $r^{(1)}, r^{(2)}$5 conflates copying with marginal mismatch unless $r^{(1)}, r^{(2)}$6 (Klein et al., 9 Jul 2025).

This leads to an estimator

$r^{(1)}, r^{(2)}$7

truncated to $r^{(1)}, r^{(2)}$8, with a bootstrap CI obtained by recomputing both $r^{(1)}, r^{(2)}$9 and $p_j$0 on each resample. The paper also uses the same model for planning experiments with a target EC and target accuracies: given desired marginals and target EC, one solves for $p_j$1, constructs $p_j$2, and simulates trial sequences accordingly (Klein et al., 9 Jul 2025).

A notable edge case is identifiability. If one system is perfect or always wrong, EC is $p_j$3 or undefined, and $p_j$4 is not identifiable from $p_j$5. The paper therefore recommends designing experiments to avoid floor and ceiling regimes. One practical suggestion is to design the reference marginal to be uniform, so that $p_j$6 and $p_j$7 regardless of the other system’s accuracy. This suggests that task difficulty can be used not merely to regulate accuracy, but to simplify the interpretation of EC itself (Klein et al., 9 Jul 2025).

5. Power analysis, benchmark reanalysis, and reporting practice

The copy model in (Klein et al., 9 Jul 2025) is also used for sample-size planning. Because no closed-form analytic CI is provided for hierarchical EC, the recommended procedure is simulation-based: choose target accuracies and target EC, generate synthetic trial sequences under the copy model, bootstrap each synthetic dataset, and increase $p_j$8 until the CI width or testing power meets the design target. The paper reports that with ground-truth EC $p_j$9 and equal accuracies, 95% CI widths are large at typical human-study sizes; at $\kappa = 0$0, the CI spans roughly $\kappa = 0$1 kappa units, and widths shrink slowly with $\kappa = 0$2 (Klein et al., 9 Jul 2025).

From these simulations, the paper gives a practical rule of thumb: collect at least $\kappa = 0$3 trials per classifier to obtain reasonably narrow confidence intervals, and target accuracies around $\kappa = 0$4 while avoiding $\kappa = 0$5 or $\kappa = 0$6 to mitigate ceiling and floor effects. Power calculations for detecting a difference $\kappa = 0$7 are likewise simulation-based and must be carried out at the intended operating point $\kappa = 0$8, since the variance of $\kappa = 0$9 depends strongly on both accuracies and EC (Klein et al., 9 Jul 2025).

These methods were applied to popular NeuroAI benchmarks. In Model-VS-Human, EC was bootstrapped through the full hierarchy of observers, conditions, and corruptions. The main conclusion survives: inter-human EC is higher than model-vs-human EC, and the best model remains a CLIP-trained ViT. However, many pairwise differences between models are not statistically resolved, and ranking stability measured by Kendall’s $a = \#(r^{(1)}=1, r^{(2)}=1)$0 over $a = \#(r^{(1)}=1, r^{(2)}=1)$1 bootstraps is approximately $a = \#(r^{(1)}=1, r^{(2)}=1)$2, indicating substantial but imperfect ranking robustness (Klein et al., 9 Jul 2025).

The Brain-Score behavioral analysis reaches a similar conclusion. Using raw EC together with conservative confidence intervals, the study finds that among the top-30 models, most of the top ten are statistically indistinguishable from each other; CLIP ViT still stands out, but the remainder of the ranking is unreliable within uncertainty. A plausible implication is that benchmark leaderboards built from EC should be reported together with score uncertainty and ranking stability, rather than only with point estimates (Klein et al., 9 Jul 2025).

6. Terminological ambiguity and other uses of “EC”

The acronym “EC” is not unique across arXiv literature, and in some domains it denotes concepts unrelated to behavioral Error Consistency. This ambiguity is substantive enough that precise disambiguation is often necessary.

Usage of “EC”	Meaning	Representative paper
Error Consistency	Cohen’s kappa on binary correctness indicators for classifier comparison	(Klein et al., 9 Jul 2025)
Expectation Consistency	Approximate inference framework with moment consistency constraints; used for probabilistic MIMO detection and neural-network calibration	(Cépedes et al., 2019, Clarté et al., 2023)
EC = rt-consistency	In requirements engineering, “inevitability of definitive errors should be anticipated” in discrete-time timed automata	(Jéron et al., 2020)

In probabilistic MIMO detection, “EC” refers to Expectation Consistency, an approximate inference framework that optimizes a non-convex free-energy-like objective under moment consistency constraints. In that setting, EC generalizes Belief Propagation and Expectation Propagation and is specialized to soft symbol detection in high-dimensional MIMO systems (Cépedes et al., 2019). In neural-network calibration, “EC” again denotes Expectation Consistency, now as a one-parameter post-hoc rescaling of the last-layer logits chosen so that average validation confidence equals empirical accuracy (Clarté et al., 2023).

A different usage appears in formal verification. In "Incremental methods for checking real-time consistency" (Jéron et al., 2020), EC corresponds exactly to the paper’s notion of rt-consistency: if all infinite continuations of a finite execution inevitably violate some requirement, then the finite execution itself should already be recognized as violating. There, EC is formulated over products of complete deterministic timed automata and checked through CTL model checking or bounded SMT-based algorithms (Jéron et al., 2020).

For work on classifier behavior, however, the dominant meaning is the kappa-based notion described above. In that sense, EC is not a synonym for accuracy, confidence calibration, or formal consistency of requirements. It is a chance-corrected statistic of shared correctness structure, whose reliable use depends on uncertainty quantification, attention to marginal accuracies, and adequately powered experimental design (Klein et al., 9 Jul 2025).