Causal Judge Evaluation (CJE)

• Causal Judge Evaluation is a framework that rigorously assesses automated and human judges in causal reasoning, model selection, and decision-making through well-calibrated surrogate metrics.
• It employs methods like mean-preserving isotonic regression, importance weight stabilization, and DAG-based scoring to ensure statistically sound and interpretable comparisons.
• CJE has wide applications including LLM evaluation, physics problem solving, and benchmark testing in causal inference, offering robust guarantees at reduced labeling costs.

Causal Judge Evaluation (CJE) encompasses a family of frameworks, algorithms, and theoretical formalisms for evaluating the performance of automated or human "judges" in causal reasoning, model selection, decision-making, and model performance assessment. CJE extends from process-level LLM assessment and causal effect benchmarking in observational data to rigorous specification testing in instrumental-variable designs and risk-based comparison of structure learning algorithms. Across these domains, CJE systems formalize and operationalize the standards by which causal inferences, policy scores, intermediate solution steps, or model predictions are credited, benchmarked, and compared, often under strong process-level, distributional, or statistical constraints.

1. Statistical Foundations and Motivations

The central tenet of CJE is the rigorous alignment of surrogate or automated scoring mechanisms with well-specified, stakeholder-relevant causal estimands, with theoretical guarantees on calibration, admissibility, and discrimination. The CJE framework is prompted by substantial empirical and theoretical evidence that naïve "judge" models—such as LLM-as-judge scoring, outcome-based success signals, or ad hoc error metrics—are statistically unsound and may yield arbitrarily poor or inverted rankings in both human and automated assessment regimes (Landesberg, 11 Dec 2025). Specifically, naive reliance on uncalibrated surrogate scores, importance sampling under limited overlap, or stepwise heuristics commonly leads to preference inversions, near-zero confidence interval coverage, and unreliable credit assignment.

CJE solutions thus integrate

• surrogate-to-oracle calibration with monotonicity and mean-preservation,
• sample and intervention-based diagnostics for distributional coverage,
• robust partial credit functions grounded in causal process graphs or Bayesian model-selection priors,
• non-parametric statistical comparison mechanisms for fair and interpretable benchmarking,
• and uncertainty propagation techniques accounting for both finite-sample evaluation variance and calibration estimation error.

2. Surrogate Calibration and Weight Stabilization in LLM Evaluation

In large-scale LLM evaluation, CJE provides a statistically defendable alternative to conventional "LLM-as-judge" scoring (Landesberg, 11 Dec 2025). The data consists of tuples (Xᵢ, Aᵢ, Sᵢ, Lᵢ, Yᵢ), where S is a cheap, automated judge score (e.g., an LLM-based reward), and Y is an expensive oracle label (e.g., human or high-fidelity model output). The goal is to estimate each policy's true oracle value V(π′) = E_{X,A∼π′}[Y] at minimal oracle-labeling cost, with confidence intervals that account for calibration and reward uncertainty.

Core CJE components in this context include:

• AutoCal-R: Mean-preserving isotonic regression, learning a monotone calibration f̂ mapping S to expected Y on the oracle slice, such that E[f̂(S)] = E[Y]. A two-stage variant corrects for covariate-induced judge score bias via smoothed indices (Landesberg, 11 Dec 2025).
• SIMCal-W: Surrogate-indexed monotone calibration of importance weights. Stacking and monotone projections (PAVA up and down) are used to stabilize the SNIPS weights W^{m1}=π′/π₀. OOF stacking minimizes the influence function covariance across candidate projections, with a finite-sample variance guard to cap weight dispersion and guarantee ESS lower bounds.
• Oracle-Uncertainty-Aware (OUA) Inference: Total estimator variance is decomposed into variance conditional on the fixed calibration and the uncertainty in fitting f̂, with delete-one-fold jackknife providing an honest estimator of calibration variance.

In combination, this CJE pipeline reliably achieves policy ranking accuracy (94%–99%) at one-twelfth the oracle-labeling cost, with valid confidence intervals (95–96% coverage) even when overlap and target-typicality coverage substantially constrain off-policy identification (Landesberg, 11 Dec 2025). Coverage-Limited Efficiency (CLE) diagnostics further explain why high ESS is insufficient: if TTC (target-typicality coverage) < 0.7, CJE gates off IPS/SNIPS-based estimation.

3. Process-Level Evaluation in Scientific and Mathematical Reasoning

CJE finds critical application in process-level evaluation of scientific reasoning, most notably in PRISM-Physics for physics problem solving (Zhao et al., 3 Oct 2025). Here, the solution to a complex task is encoded as a directed acyclic graph (DAG) of canonicalized formula-nodes:

• DAG Representation: Each node in G=(V,E) is a minimal, canonical formula φ(v); E encodes "justification" edges (u→v), acyclic and ordered by derivation.
• Ancestor-Closure Scoring: Given a set of stepwise matches M, the score S=|Ach(M)|/|F| is computed where Ach(M) is the ancestor-closure containing M and all reference nodes on directed paths to M. This policy is the unique admissible scoring rule consistent with matched inclusion, ancestor closure, and soundness (Theorem 2.2).
• Rule-Based Formula Equivalence: Deterministic two-stage matching combines constant substitution and randomized algebraic solution-set equivalence to match formulas up to rearrangement, unit, and constant changes; no LLM-judgments are involved.
• Comparative Experimental Results: PRISM-Physics DAG scoring aligns best with expert human grading (Kendall τ_b=0.346), outperforms both LLM-judge scoring (τ_b=0.294) and linear stepwise metrics (τ_b=0.213), gives diagnostic insight into persistent physics reasoning errors (e.g., condition, derivation, and modeling errors), and enables granular, interpretable reward signals for downstream learning (Zhao et al., 3 Oct 2025).

4. Benchmarking Causal Inference and Structure Learning Models

CJE provides robust, problem-level assessment protocols for benchmarking causal effect estimation models and structure learning algorithms under both randomized and observational data regimes (Kiriakidou et al., 2022, Eigenmann et al., 2020).

• Core Metrics (Causal Effect Estimation): Benchmarking is conducted via ATE error (absolute difference between estimated and true average treatment effect) and PEHE (root mean squared individual effect estimation error). Performance profiles (Dolan–Moré) compare solvers using the relative error ratio and cumulative best-fraction curve ρ_s(α); non-parametric Friedman and post-hoc tests assign significance to ranking differences, controlling for outlier instances and multiple comparisons (Kiriakidou et al., 2022).
• Decision-Theoretic Risk (Structure Learning): Evaluation is formalized as minimizing (empirical) causal risk: e.g., node-wise Jaccard-descendant loss between estimated and oracle graphs. The combined risk estimator \hat R(𝒜;X) blends naive in-distribution and cross-validation (leave-one-intervention-out) OOD risk. Simulation studies confirm high power for ranking algorithms even with modest sample/intervention size, provided robust two-sample descendant estimation is used (Eigenmann et al., 2020).

CJE platforms for these tasks implement plug-in APIs for benchmark data/model ingestion, automated error and risk metrics, performance-profile computation, and statistical-test-driven leaderboards, supporting extensibility to user-defined custom metrics and fair, statistically-interpretable comparisons.

5. Specification Testing in Instrumental Variable and Judge Leniency Designs

For causal identification in legal or policy settings, CJE provides sharp, exhaustive specification tests for IV-based identification strategies such as the judge leniency design (Coulibaly et al., 10 May 2024). The setup involves random judge assignment acting as an instrument for treatment D, with outcome Y.

• Sharp Testable Implications: Necessary and sufficient inequalities in outcome distribution—quantified over observable "cubes" in propensity and outcome space—completely characterize model-predicted monotonicity, random assignment, and exclusion restrictions. The test statistic T_n aggregates maximal normalized moment violations. If any inequality is violated, the identifying assumptions fail on the observed data; if all are satisfied, IV is justified for causal effect estimation.
• No Discordant Recommendations: The sharp test cannot yield conflicting design-validity verdicts, a property not shared by prior non-sharp or ad-hoc-bounded approaches (e.g., Frandsen–Lefgren–Leslie), which may be miscalibrated (Coulibaly et al., 10 May 2024).
• Partial Assumption Salvage: When global exclusion or monotonicity fails, the framework yields conditional (partial) variants, identifying MTE(x,u)=∂E[Y|X=x,P=p]/∂p within covariate strata. Simulation studies and empirical applications (Philadelphia bail data) confirm the power and validity control advantages of the sharp CJE test.

6. CJE in Human and Human-AI Causal Judgment

CJE is also used to benchmark and diagnose human or human-in-the-loop causal inference in both visual analytics and algorithm-assisted decision-making (Kale et al., 2021, Imai et al., 2020).

• Bayesian Causal Support as Benchmark: Causal support—log-posterior odds comparing alternative DAGs given data—serves as normative Bayesian ground truth. Human responses (e.g., vote allocations across hypotheses given visualized contingency tables) are mapped to log-odds and compared to causal support via a linear-in-log-odds (LLO) slope/intercept; invariably, chart users under-react to evidence (LLO slopes ≈0.3–0.4) and reveal strong biases, often being insensitive to sample size and misinterpreting interaction in visualizations (Kale et al., 2021).
• RCTs of Algorithmic Recommendations: In the judicial context, randomized controlled trials of algorithm-assisted recommendations (e.g. Pretrial PSA) are analyzed within the potential outcomes framework. CJE here quantifies ITT, CATE/CATE_g, principal stratification (APCE_p), operational utility, and fairness (difference in decision rates within risk strata). Study results reveal nuanced subgroup effects, lack of global impact, and shifts in fairness properties due to the algorithmic aid (Imai et al., 2020).

7. Theoretical Guarantees, Best Practices, and Limitations

The mathematical architecture of CJE methods is grounded in influence function theory, mean-sufficiency projections, and variance decomposition, ensuring that estimators attain sharp efficiency bounds within justified model classes subject to monotonicity, mean sufficiency, and overlap (Landesberg, 11 Dec 2025). Key best practices include:

• rigorous calibration gatekeeping and distributional diagnostics (e.g., TTC, Bhattacharyya affinity),
• statistical specification tests before causal or instrumental-variable conclusions,
• fair error/risk comparison across model families and problem types,
• propagation of uncertainty from all nuisance estimation and calibration sources.

Assumptions must be explicitly checked: oracle alignment, model/score transportability, support overlap, and calibration range must all be validated, with fallback or partial-identification measures in cases of violation (e.g., REFUSE-LEVEL mode under severe calibration extrapolation (Landesberg, 11 Dec 2025)). Small sample diagnostic tools (e.g., jackknife, bootstrap, cross-validation) and subgroup fairness reporting are essential for robustness and equity.

Summary Table: CJE Methodological Archetypes

Domain	CJE Mechanism	Key Properties/Guarantees
LLM-as-judge evaluation	AutoCal-R, SIMCal-W, OUA-inference	Monotone, mean-preserving calibration, robust stacking/stabilization, valid CIs, CLE gate (Landesberg, 11 Dec 2025)
Physics reasoning	PRISM-Physics DAG, ancestor-closure scoring	Causality-respecting minimal credit, exact symbolic matching, maximal human alignment (Zhao et al., 3 Oct 2025)
Effect/model benchmarking	ATE/PEHE, performance profiles, Friedman/Nemenyi-Holm	Robust, outlier-insensitive rankings, non-parametric tests (Kiriakidou et al., 2022)
Structure learning	Combined in-sample/OOD risk, Jaccard loss	Black-box, intervention-based robust selection, practical for large graphs (Eigenmann et al., 2020)
Legal/IV designs	Sharp testable implications, full-distribution specification test	Necessary and sufficient for identification, no discordant results, salvage via partial monotonicity (Coulibaly et al., 10 May 2024)
Human causal inference	Causal support (Bayesian), LLO benchmarking	Quantification of human biases, normative misalignment analytics (Kale et al., 2021)

Causal Judge Evaluation thus constitutes a unified class of process- and criteria-based methods furnishing theoretical rigor, empirical effectiveness, and diagnostic transparency for causal model benchmarking, process-level evaluation, and algorithmic or human "judge" assessment across diverse scientific, machine learning, and policy domains.