---
title: Counterfactual Cross-Validation Metric
url: https://www.emergentmind.com/topics/counterfactual-cross-validation-metric
type: topic
---

# Counterfactual Cross-Validation Metric

Searching arXiv for recent and foundational papers on counterfactual cross-validation and related evaluation metrics.
The **counterfactual cross-validation metric** is a model-selection criterion for **conditional average treatment effect** (CATE) prediction designed for settings in which the target label is inherently unobserved. In ordinary supervised learning, cross-validation estimates predictive loss against observed labels. In CATE prediction, however, the relevant target is
\[
\tau(x)=\mathbb{E}[Y(1)-Y(0)\mid X=x],
\]
and this quantity is never directly observed for any individual. The metric introduced in "Counterfactual Cross-Validation: Stable Model Selection Procedure for Causal Inference Models" addresses this problem by replacing unavailable CATE labels with a **doubly robust pseudo-label** and by optimizing not merely unbiasedness of risk estimation but preservation of the **rank order** of competing CATE predictors, so that model selection and hyperparameter tuning remain stable [1909.05299].

## 1. Problem setting and motivation

The metric is defined for observational validation sets of the form
\[
\mathcal{V}=\{(X_i,T_i,Y_i)\}_{i=1}^n,
\]
where \(X_i\) are covariates, \(T_i\) is treatment, and \(Y_i\) is the observed factual outcome. The central evaluation target is the **PEHE / CATE MSE**
\[
R(\hat\tau)=\mathbb{E}_X\left[(\tau(X)-\hat\tau(X))^2\right],
\]
but this risk cannot be computed directly because \(\tau(X)\) is counterfactual [1909.05299].

The motivating claim of the metric is that **standard cross-validation is inadequate for CATE model selection**. In supervised learning, validation proceeds against observed labels; in CATE prediction, the relevant label is unavailable, observed outcomes are factual rather than causal labels, naive plug-in evaluation can be noisy and unstable, and for model selection the decisive requirement is often not perfect risk estimation but preservation of the correct **ordering** of candidate models. The paper therefore formulates model selection as a **ranking problem** over a candidate set \(\mathcal{M}=\{\hat\tau_1,\ldots,\hat\tau_{|\mathcal{M}|}\}\) [1909.05299].

This ranking perspective is expressed by the implication
\[
R(\hat\tau)\le R(\hat\tau') \;\Rightarrow\; \hat R(\hat\tau)\le \hat R(\hat\tau') ,
\quad \forall \hat\tau,\hat\tau'\in\mathcal{M},
\]
which states that an estimated validation metric should preserve the true risk ordering. The metric is thus not presented merely as a surrogate loss, but as a **stable model-selection procedure** whose primary objective is to identify the best candidate by its estimated score [1909.05299].

## 2. Ranking-preservation principle

The theoretical construction begins with two guidelines for the plug-in estimator used inside the evaluation score. First, the plug-in estimate should be **unbiased for CATE**. Second, it should have **low finite-sample uncertainty**. These conditions arise from an analysis of how estimated validation scores deviate from true CATE risk in finite samples [1909.05299].

If the plug-in estimator \(\tilde\tau\) is unbiased for \(\tau(X)\), then the expected validation score decomposes as
\[
\mathbb{E}\!\left[\hat R(\hat\tau)\right]
=R(\hat\tau)+
\mathbb{E}\left[\left(\tau(X)-\tilde\tau(X,T,Y)\right)^2\right].
\]
The second term does not depend on the candidate predictor \(\hat\tau\), so the expected ranking is preserved:
\[
\mathbb{E}[\hat R(\hat\tau_1)]-\mathbb{E}[\hat R(\hat\tau_2)]
=
R(\hat\tau_1)-R(\hat\tau_2).
\]
This is the central ranking guarantee of the method [1909.05299].

Finite-sample instability is traced to a stochastic term in the empirical decomposition
\[
\hat R(\hat\tau)
=
\frac{1}{n}\sum_{i=1}^n(\tau(X_i)-\hat\tau(X_i))^2
-
\frac{2}{n}\sum_{i=1}^n (\hat\tau(X_i)-\tau(X_i))(\tilde\tau_i-\tau(X_i))
+
\frac{1}{n}\sum_{i=1}^n(\tau(X_i)-\tilde\tau_i)^2.
\]
The middle term, denoted \(\mathcal W\), is the source of ranking instability. Under independence across instances and unbiasedness, its variance is upper bounded by
\[
\operatorname{Var}(\mathcal W)
\le
4n^{-1}\,\mathbb{E}_X\left[\operatorname{Var}(\tilde\tau(X,T,Y))\right].
\]
Accordingly, the paper advocates a plug-in estimator that satisfies
\[
\min_{\tilde\tau\in\Theta}\; \mathbb{E}_X\!\left[\operatorname{Var}\bigl(\tilde\tau(X,T,Y)\bigr)\right]
\quad \text{s.t.} \quad
\mathbb{E}[\tilde\tau(X,T,Y)]=\tau(X),
\]
so that model selection is both rank-preserving in expectation and less sensitive to finite-sample noise [1909.05299].

## 3. Metric construction and doubly robust pseudo-labels

The generic validation metric has the same form as ordinary squared-error validation, except that the unavailable target \(\tau(X_i)\) is replaced by a pseudo-label:
\[
\hat R(\hat\tau)
=
\frac{1}{n}\sum_{i=1}^n
\left(\tilde\tau(X_i,T_i,Y_i)-\hat\tau(X_i)\right)^2.
\]
The novelty lies in the construction of \(\tilde\tau\) [1909.05299].

The proposed pseudo-label is a **doubly robust CATE estimator**
\[
\tilde\tau_{DR}(X,T,Y;f_t)
=
\frac{T-e(X)}{e(X)\bigl(1-e(X)\bigr)}\bigl(Y-f_T(X)\bigr)
+
f_1(X)-f_0(X),
\]
where \(e(X)\) is the propensity score and \(f_t\) are regression functions. The paper also gives the equivalent decomposition
\[
\tilde\tau_{DR}=\tilde\tau_{DR_1}-\tilde\tau_{DR_0},
\]
with
\[
\tilde\tau_{DR_1}(X,T,Y)
=
\frac{T}{e(X)}\bigl(Y-f_1(X)\bigr)+f_1(X),
\]
\[
\tilde\tau_{DR_0}(X,T,Y)
=
\frac{1-T}{1-e(X)}\bigl(Y-f_0(X)\bigr)+f_0(X).
\]
Given the true propensity score \(e(X)\), this pseudo-label is unbiased:
\[
\mathbb{E}\left[\tilde\tau_{DR}(X,T,Y;f_t)\right]=\tau(X)
\]
for any choice of \(f_t\) [1909.05299].

This pseudo-label is used **only for evaluation and model selection**, not as a deployed predictor, because it depends on observed treatment \(T\) and outcome \(Y\). The paper explicitly places the method within the standard causal assumptions of **unconfoundedness**, **overlap**, and **consistency**, and notes that performance depends on the quality of the estimated propensity score \(e(x)\). Hidden confounding is not addressed [1909.05299].

## 4. Counterfactual regression regularization

Although the doubly robust pseudo-label is unbiased for any \(f_t\), its **conditional variance** depends on how well the regression functions approximate the outcome regressions \(m_t(X)=\mathbb{E}[Y(t)\mid X]\). The paper derives
\[
\mathbb{E}_X\left[\operatorname{Var}\bigl(\tilde\tau_{DR}(X,T,Y;f_t)\bigr)\right]
=
\zeta +
\mathbb{E}_X\left[
\left\{
\sum_{t\in\{0,1\}}
\sqrt{w_t(X)}\bigl(f_t(X)-m_t(X)\bigr)
\right\}^2
\right],
\]
where
\[
w_t(X)=\frac{t(1-2e(X))+e(X)^2}{e(X)(1-e(X))},
\]
and \(\zeta\) is independent of \(f\) [1909.05299].

Because \(m_0\) and \(m_1\) are themselves counterfactual objects, the paper does not optimize this expression directly. Instead, it derives an upper bound in terms of weighted factual risks and an **integral probability metric** (IPM) over learned representations. For a representation \(\Phi\) and hypothesis \(h\) such that \(f_t(x)=h(\Phi(x),t)\), the paper states
\[
\mathbb{E}_X\left[
\left\{
\sum_{t\in\{0,1\}}
\sqrt{w_t(X)}\bigl(f_t(X)-m_t(X)\bigr)
\right\}^2
\right]
\le
2\Bigl(
\epsilon_{F_1}^{w_1}(h,\Phi)
+
\epsilon_{F_0}^{w_0}(h,\Phi)
+
IPM(t,1-t)
-
2\sigma^2
\Bigr).
\]
This connects the evaluation metric to **counterfactual regression** (CFR) [1909.05299].

The practical objective used to train the pseudo-label model is
\[
\min_{h,\Phi}
\;
\sum_{i=1}^n \frac{w'_t(x_i)}{n}
\,L\!\left(h(\Phi(x_i),t_i),y_i\right)
+
\alpha\, IPM\!\left( \{\Phi(x_i)\}_{i:t_i=0}, \{\Phi(x_i)\}_{i:t_i=1} \right),
\]
with
\[
w'_t(x_i)
=
\frac{w_t(x_i)}{2}
\left(
\frac{t_i}{\hat\pi_1}
+
\frac{1-t_i}{\hat\pi_0}
\right),
\qquad
\hat\pi_t
=
\frac{1}{n}\sum_{i=1}^n \mathbf 1\{t_i=t\}.
\]
In the experiments, the paper uses deep neural networks for \(h\) and \(\Phi\), the Adam optimizer, and Wasserstein distance as the IPM. For CF-CV and IPW, propensity is estimated with logistic regression; for plug-in and CF-CV training, a **\(\mu\)-risk heuristic** is used to tune the regression function inside the pseudo-outcome model [1909.05299].

## 5. Procedure, empirical performance, and limitations

Algorithmically, the method proceeds in five steps. First, it trains \(f(x,t)=h(\Phi(x),t)\) on the validation data by minimizing the weighted CFR-style objective. Second, it estimates the propensity score \(e(x)\) if it is not known. Third, it computes the doubly robust pseudo-outcomes \(\tilde\tau_{DR}\) for the validation samples. Fourth, it evaluates each candidate \(\hat\tau\in\mathcal M\) using
\[
\hat R(\hat\tau)
=
\frac{1}{n}\sum_{i=1}^n
\left(\tilde\tau_{DR}(X_i,T_i,Y_i)-\hat\tau(X_i)\right)^2.
\]
Fifth, it selects
\[
\hat\tau^*=\arg\min_{\hat\tau\in\mathcal M}\hat R(\hat\tau).
\]
The metric is therefore a **cross-validation score for causal model selection**, but one whose internal target is counterfactual rather than factual [1909.05299].

Empirical evaluation is conducted on the standard semi-synthetic **IHDP** dataset with 747 children, 25 features, synthetic outcomes with known ground-truth CATE, and induced confounding by removing a biased subset of treated units. Against IPW validation, plug-in validation, and \(\tau\)-risk, the paper reports the following mean performance over 100 runs:

| Metric | Rank corr | Regret | NRMSE |
|---|---:|---:|---:|
| IPW | 0.195 | 1.032 | 0.336 |
| \(\tau\)-risk | 0.312 | 1.392 | 0.324 |
| Plug-in | 0.914 | 0.073 | 0.257 |
| CF-CV | 0.921 | 0.066 | 0.256 |

The paper emphasizes that **worst-case performance** is especially important and reports that CF-CV achieves the best worst-case rank correlation, regret, and NRMSE among the compared methods. In hyperparameter tuning of a DAL + GBR CATE model using Optuna, CF-CV again achieves the best worst-case tuning performance, with mean/worst-case NRMSE of \(0.256/0.483\), compared with \(0.257/0.490\) for plug-in validation, \(0.324/0.700\) for \(\tau\)-risk, and \(0.336/0.737\) for IPW [1909.05299].

The paper also studies the trade-off hyperparameter \(\alpha\) and finds that CF-CV generally outperforms plug-in validation for small \(\alpha\), remains strong across a range of \(\alpha\), and consistently improves regret compared with plug-in validation. At the same time, several limitations are explicit: the method assumes unconfoundedness, overlap, and consistency; it depends on propensity estimation; it does not address hidden confounding; and the pseudo-outcome is not a predictor usable at test time for new individuals [1909.05299].

## 6. Related and distinct uses of “counterfactual validation” and adjacent metrics

The expression **counterfactual cross-validation metric** is not used uniformly across the literature. Several later works define adjacent validation or evaluation objects that address different questions, even when they resemble cross-validation conceptually.

| Paper | Object | Main role |
|---|---|---|
| "Rethinking Distance Metrics for Counterfactual Explainability" [2410.14522] | Mahalanobis-style counterfactual similarity metric centered at \((1-\alpha)\mu+\alpha x\) | Counterfactual proximity, not a cross-validation score |
| "Longitudinal Counterfactuals: Constraints and Opportunities" [2403.00105] | \(L(x,e;D,s)\), the average of the \(s\) nearest observed longitudinal changes | Plausibility scoring and generation |
| "Designing User-Centric Metrics for Evaluation of Counterfactual Explanations" [2507.15162] | Acceptability Weighted Proximity (AWP) | User-centric evaluation model |
| "Can We Validate Counterfactual Estimations in the Presence of General Network Interference?" [2502.01106] | Batch-level MSE over time-block cross-validation | Counterfactual estimation under interference |
| "The Digital Twin Counterfactual Framework" [2604.01325] | Five-level validation architecture with calibration and treatment-effect discrepancy tests | Validation architecture rather than a single metric |

In **counterfactual explanation**, later work uses “metric” primarily to mean **proximity**, **plausibility**, or **acceptability**, not a cross-validation score for model selection. The distance metric of [2410.14522] is explicitly described as a **probabilistically derived counterfactual similarity metric** rather than a cross-validation procedure; it replaces plain \(\ell_2\) proximity with a Mahalanobis-style quadratic form centered at \((1-\alpha)\mu+\alpha x\). The longitudinal metric of [2403.00105] scores a counterfactual by comparing its implied change \((e-x)\) to the average of the \(s\) nearest observed longitudinal differences, thereby using historical transitions as a proxy for plausibility. The AWP model of [2507.15162] combines feasibility filtering with personalized weighted proximity and is presented as a user-centric evaluation model whose reported 84.37% accuracy is explicitly characterized as retrospective and potentially overfit [2410.14522] [2403.00105] [2507.15162].

In **counterfactual validation of simulators and interference models**, the term broadens further. The Digital Twin Counterfactual Framework introduces a five-level validation architecture with marginal calibration, conditional calibration, individual-level calibration, treatment-effect calibration, and distributional stress testing rather than a single literal cross-validation metric [2604.01325]. By contrast, the network-interference paper introduces a genuine cross-validation methodology: time blocks serve as folds, validation batches are exposure-stratified, and the validation loss is
\[
\MSE_{\estimator,\batchSize,\batchCount}
=
\frac{1}{b_v (T+1)}
\sum_{j=1}^{b_v}\sum_{t=0}^T
\Big[
\CFETest{t}{\batch_j^v}
-
\ECFTrain{t}{\batch_j^v}{\estimator,\batchSize,\batchCount}
\Big]^2,
\]
with distribution-preserving network bootstrap used to manufacture valid pseudo-replicates under interference [2502.01106].

This broader usage suggests a terminological distinction. In the narrow and historically specific sense, the **Counterfactual Cross-Validation metric** denotes the CATE model-selection criterion of [1909.05299]. In a wider sense, later literature uses related language for any metric or validation architecture that evaluates counterfactual objects without direct access to ground-truth counterfactual labels. The common thread is the same: ordinary validation against factual observations is misaligned with the inferential target, so evaluation must be reconstructed through pseudo-labels, reweighting, historical transitions, user judgments, simulator fidelity checks, or structured pseudo-replication [1909.05299] [2604.01325] [2502.01106].

Source: https://www.emergentmind.com/topics/counterfactual-cross-validation-metric