Evaluation as Inference

• Evaluation as inference is a methodological stance that treats evaluation as a process of statistical, causal, or decision-theoretic reasoning, emphasizing explicit assumptions about calibration and uncertainty.
• It integrates experimental design with inferential techniques by defining evaluation conditions, target estimands, and protocols that yield quantifiable and reliable outcomes.
• This approach is applied across diverse domains—from causal inference to large language model evaluation—to enable rigorous benchmarking through methods such as sequential testing and credible intervals.

Searching arXiv for recent and foundational papers on “evaluation as inference” and closely related formulations across causal evaluation, benchmarking, LLM evaluation, and inference-method assessment. Evaluation as inference denotes a family of methodologies in which evaluation is treated as a statistical, causal, logical, or decision-theoretic inference problem rather than as the mere reporting of a score. In this view, evaluators specify an evaluation condition or data-generating process, define target quantities such as treatment effects, size, coverage, bias, win rates, success probabilities, or protection rates, and then infer those quantities from measurements, simulations, verifier outputs, or adversarial probes under explicit assumptions about calibration, equivalency, sampling, and causality (Zhan et al., 2024, Ferman, 2019, Amaranath et al., 2 Sep 2025). Across these formulations, evaluation becomes an experiment plus inference: outcomes are measured, uncertainty is quantified, and decisions are tied to confidence intervals, credible intervals, posterior distributions, or sequential tests rather than to ad hoc summaries alone (Zhan et al., 2024, Sadhuka et al., 2 Dec 2025).

1. Ontology and general formulation

A general formulation appears in “Evaluatology: The Science and Engineering of Evaluation” (Zhan et al., 2024). There, evaluation is organized around subjects, evaluation conditions (ECs), evaluation models (EMs), measurement and testing, and evaluation results. The EC is formalized as $C = E' \times E \times A' \times A \times S$, where $E'$ is the problem/task space, $E$ the space of problem/task instances, $A'$ the algorithm-like mechanism space, $A$ the implementation space, and $S$ the support system space. The EM is then $M = C \times U$, with $U$ the subject space, and observed outcomes are written as $y = (\rho(m), \tau(m))$, where $\rho(\cdot)$ denotes measurement and $E'$0 testing. Composite evaluation results are obtained through a value function $E'$1 (Zhan et al., 2024).

This formalization makes explicit that evaluation depends on what is held fixed, what varies, and what inferential quantity is extracted from observations. The same paper introduces five axioms: the Axiom of the Essence of Composite Evaluation Metrics, the Axiom of True Evaluation Outcomes, the Axiom of Evaluation Traceability, the Axiom of Comparable Evaluation Outcomes, and the Axiom of Consistent Evaluation Outcomes (Zhan et al., 2024). It also distinguishes Equivalent Evaluation Conditions and Least Equivalent Evaluation Conditions, thereby making comparability itself an inferential assumption rather than a default property of benchmark scores.

A related framing appears in Ferman’s “Assessing Inference Methods” (Ferman, 2019). There, evaluation is explicitly described as the estimation of performance functionals—size, coverage, bias, variance, and power—under a specified data-generating process. The paper emphasizes that these quantities are not known; they are estimated with uncertainty from finite Monte Carlo replicates. The Monte Carlo estimate of expected loss is

$E'$2

with Monte Carlo standard error

$E'$3

This shifts attention from single benchmark numbers to the inferential status of the benchmark itself (Ferman, 2019).

Taken together, these formulations suggest that “evaluation as inference” is not a single method but a methodological stance: evaluation requires a target, a model of uncertainty, a controlled comparison structure, and an explicit account of what conclusions are warranted under the chosen design.

2. Estimands, uncertainty, and inferential machinery

Several papers instantiate this stance by defining evaluation targets as estimands with explicit inferential operators. In “Statistical Inference for Score Decompositions,” expected forecast scores are decomposed into miscalibration, discrimination, and uncertainty,

$E'$4

and inference is performed on these components rather than only on aggregate predictive scores (Dimitriadis et al., 4 Mar 2026). This decomposition connects score-based evaluation to classical calibration tests such as Mincer–Zarnowitz regression and financial backtests, and enables tests for equal calibration or equal discrimination under both smooth and non-smooth scoring functions (Dimitriadis et al., 4 Mar 2026).

In “Efficient Inference for Noisy LLM-as-a-Judge Evaluation,” the estimand is typically a mean parameter such as average benchmark score or pairwise win rate, but the observed judge output $E'$5 is treated as a noisy surrogate for latent ground truth $E'$6 (Chen et al., 8 Jan 2026). The paper studies two corrective families—measurement-error correction and prediction-powered inference—and derives efficient influence function based estimators. Under the surrogate-outcome formulation, with $E'$7, the efficient estimator is

$E'$8

which makes inference depend on both abundant noisy evaluations and a small labeled calibration set (Chen et al., 8 Jan 2026). The paper’s central claim is that evaluation with LLM judges should be treated as inference with noisy surrogates, not as direct observation.

A third formulation appears in “Prediction-Powered Inference Across Many Tasks for AI Evaluation & Social Science Research,” where the per-task estimand is the finite-population mean

$E'$9

and cross-task recalibration exploits shared structure in the proxy–ground-truth relationship to improve per-task confidence intervals while preserving task-specific inference (Emmenegger et al., 28 May 2026). The paper proves that efficiency gains beyond power-tuned PPI are only possible when the proxy–ground-truth relationship contains nonlinear structure; affine cross-task recalibrations are asymptotically equivalent to using the original proxy (Emmenegger et al., 28 May 2026).

These works differ in domain and mathematical apparatus, but all treat evaluation metrics as estimators or decompositions of latent quantities whose uncertainty must be modeled explicitly.

3. Causal evaluation and benchmarking

A major strand of evaluation-as-inference arises in causal evaluation, where the objective is to infer treatment effects or estimator performance under counterfactual uncertainty. In “Bayesian causal inference in automotive software engineering and online evaluation,” randomized field experiments remain the gold standard, but the paper argues that in the automotive domain randomization is “not always desired, possible, or even ethical” (Liu et al., 2022). It introduces BOAT, “Bayesian causal modelling for ObvservAtional Testing,” and presents three Bayesian causal inference models for three common challenges: Bayesian propensity score matching, Bayesian regression discontinuity design, and Bayesian difference-in-differences (Liu et al., 2022). The abstract’s core claim is that online software evaluation can be enabled “without the need of a fully randomised experiment” when causal assumptions are modeled explicitly (Liu et al., 2022).

“Improving Generative Methods for Causal Evaluation via Simulation-Based Inference” moves the inferential burden to the benchmark itself (Amaranath et al., 2 Sep 2025). SBICE treats the evaluation of causal estimators as a posterior predictive problem over uncertain data-generating-process parameters. Rather than fixing the average treatment effect or confounding strength to point values, it places priors over generative parameters, infers their posterior with SMC-ABC, and evaluates estimators on posterior predictive synthetic datasets filtered to be close to the source distribution (Amaranath et al., 2 Sep 2025). In this formulation, estimator performance is integrated over the posterior,

$E$0

so that benchmarking itself becomes posterior inference rather than fixed-scenario comparison (Amaranath et al., 2 Sep 2025).

Benchmark construction papers make the same move from different directions. The IBM “Benchmarking Framework for Performance-Evaluation of Causal Inference Analysis” simulates full counterfactual outcomes on real-world covariates and evaluates submissions using ENoRMSE, RMSE, Bias, Coverage, CIC, and ENCIS, with separate scaling and censoring tracks (Shimoni et al., 2018). “An evaluation framework for comparing causal inference models” complements mean ATE error and PEHE with performance profiles of Dolan and Moré, the Friedman test, and the Bergmann–Hommel procedure, explicitly to reduce the influence of a small number of simulations dominating the benchmarking process (Kiriakidou et al., 2022).

This literature treats evaluation as inference at two levels simultaneously: first, in estimating causal effects from observational data; second, in inferring comparative properties of causal estimators from synthetic or semi-synthetic benchmarks.

4. Language, inference, and adversarial evaluation

In natural-language evaluation, the phrase acquires a more literal meaning: quality is evaluated through inferential relations rather than surface similarity. “MENLI: Robust Evaluation Metrics from Natural Language Inference” argues that many BERT-based metrics are models of semantic similarity and therefore do not directly capture information correctness (Chen et al., 2022). MENLI instead uses NLI posteriors over entailment, contradiction, and neutral labels, with task-dependent directional choices and pooling functions, thereby treating evaluation as inference over whether candidate claims follow from a source or reference (Chen et al., 2022). The paper’s broader claim is that correctness is an entailment problem, not a similarity problem.

“Stress Test Evaluation for Natural Language Inference” makes a related argument from the model-evaluation side (Naik et al., 2018). It proposes automatically constructed stress tests that isolate inferential phenomena—word overlap, negation, length mismatch, antonymy, numerical reasoning, and spelling errors—and examines whether NLI systems can make “real inferential decisions” under label-preserving perturbations (Naik et al., 2018). Here evaluation becomes a probe of logical invariance and targeted reasoning competence, not merely aggregate accuracy on SNLI or MultiNLI (Naik et al., 2018).

“Ordinal Common-sense Inference” extends the idea further by replacing binary entailment with ordinal plausibility judgments (Zhang et al., 2016). Given a context $E$1 and a hypothesis $E$2, the task is to predict one of five likelihood levels—very likely, likely, plausible, technically possible, or impossible—thereby evaluating a model’s estimate of graded plausibility rather than its assignment to a categorical truth state (Zhang et al., 2016). This explicitly reframes recognition of textual entailment as inference over ordinal human responses.

The same inferential turn appears in privacy evaluation. “Subject-level Inference for Realistic Text Anonymization Evaluation” argues that span masking does not measure what an adversary can infer after anonymization (Oh et al., 23 Apr 2026). SPIA therefore shifts the unit of evaluation from text spans to identifiable individuals and defines subject-level protection metrics: $E$3 where $E$4 is the adversarially inferred score for subject $E$5 and $E$6 the number of ground-truth PIIs for that subject (Oh et al., 23 Apr 2026). The benchmark’s central finding is that even when over 90% of PII spans are masked, subject-level inference protection can remain low because contextual inference is still possible (Oh et al., 23 Apr 2026).

5. Interactive, human-centered, and sequential inference

Evaluation-as-inference also appears in settings where the evaluator is a human reasoner or an online monitor rather than an offline benchmark script. “Inferential Tasks as an Evaluation Technique for Visualization” introduces inferential tasks as a structured task family in which participants generate or inspect a visualization, infer a candidate observation, explore the dataset to find a different subset exhibiting the same characteristic, and validate the finding (Suh et al., 2022). The method is explicitly positioned between low-level tasks and open-ended insight studies, with observable outcomes such as accuracy, speed, search breadth, and validated findings (Suh et al., 2022). The paper’s thesis is that visualization should be assessed by observing and measuring the user’s inferential reasoning with data.

“InFerActive: Towards Scalable Human Evaluation of LLMs through Interactive Inference” treats human evaluation not as inspection of isolated outputs but as interactive exploration of an LLM’s deterministic probability space induced by autoregressive generation (Hwangbo et al., 11 Dec 2025). The system represents the output space as a branching probability tree, where a response corresponds to a root-to-leaf path with probability

$E$7

and evaluators mark branches as good or bad while the interface sums their probabilities to estimate success and failure mass (Hwangbo et al., 11 Dec 2025). This makes evaluation a process of inferring model behavior from a structured probability space rather than from repeated independent samples.

“E-valuator: Reliable Agent Verifiers with Sequential Hypothesis Testing” formalizes online trajectory evaluation as sequential inference (Sadhuka et al., 2 Dec 2025). Given verifier scores $E$8 along an agent trajectory, it defines an e-process

$E$9

and uses the stopping rule

$A'$0

Via Ville’s inequality, this yields anytime-valid control of false alarm rates under adaptive stopping (Sadhuka et al., 2 Dec 2025). The paper’s conceptual contribution is to convert black-box verifier heuristics into statistically valid online decision rules.

These settings share an important feature: evaluation unfolds over time, and inference must remain valid under exploration, interaction, or sequential stopping.

6. Protocol dependence, compute, and epistemic limits

A recurrent conclusion in this literature is that evaluation results are protocol-dependent. “How Inference Compute Shapes Frontier LLM Evaluation” makes this explicit for frontier-model benchmarks (McFadyen et al., 16 Jun 2026). The paper defines trajectory curves

$A'$1

and proposes a capability function $A'$2, where $A'$3 encodes protocol choices such as feedback, submission rules, context compaction, and width–depth allocation (McFadyen et al., 16 Jun 2026). Its core claim is that benchmark scores are protocol-dependent: low scores may reflect restrictive budgets rather than underlying capability, and evaluations should therefore report capability as a function of inference-time compute rather than at a single point (McFadyen et al., 16 Jun 2026).

Ferman’s analysis reaches an analogous conclusion for simulation-based assessment of inference methods (Ferman, 2019). “Natural” simulation practices can be misleading; design-based placebo permutations can confound treatment effects with dependence, residual-based methods can create sequential-testing distortions, and more realistic simulations may detect failures that iid-normal screens do not, but they also introduce cost, tuning choices, and finite-sample pathologies (Ferman, 2019). This makes evaluation design itself an object of inference and sensitivity analysis.

A similar diagnostic perspective appears in score decomposition and noisy-surrogate correction. Score decomposition shows that equal overall predictive ability can coexist with different discrimination or miscalibration components (Dimitriadis et al., 4 Mar 2026). Noisy LLM-as-a-judge correction shows that raw judge outputs, Rogan–Gladen correction, prediction-powered inference, and efficient influence function estimators correspond to different inferential assumptions and can have sharply different efficiency and calibration properties (Chen et al., 8 Jan 2026). In both cases, the central issue is not only what score is reported, but what inferential quantity that score identifies.

Taken together, these works suggest that evaluation as inference imposes a demanding standard. One must specify the estimand, the protocol, the noise model, the comparison class, and the uncertainty quantification scheme; identify assumptions such as equivalency, overlap, continuity, parallel trends, MCAR labeling, or judge calibration; and report how conclusions change when these assumptions or protocols change. On this view, evaluation is credible only when the path from observed signals to substantive claims is itself an explicit inferential object.