---
title: Counterfactual Baselines
url: https://www.emergentmind.com/topics/counterfactual-baselines
type: topic
---

# Counterfactual Baselines

Counterfactual baselines are reference objects used to define what a factual prediction, explanation, policy, or outcome is being contrasted against. Across the literature, the term denotes several distinct but related constructions: an input reference for path attribution, a target-class exemplar for one-vs-one saliency, a comparator policy in reinforcement learning, a control variate for counterfactual values in extensive-form games, an endpoint assumption about cross-world dependence in retrospective causal prediction, or a meaning-preserving prompt perturbation used to estimate generic model sensitivity. What unifies these uses is that the baseline fixes the counterfactual question being asked; it is therefore not merely an implementation detail but part of the semantics of the analysis itself [2508.14482, 2606.23206, 2603.27320, 2605.01048].

## 1. Conceptual scope and taxonomy

In the most general sense, a counterfactual baseline specifies the alternative state relative to which a factual quantity is interpreted. In some settings this state is an explicit object, such as an image \(\hat{x}\) or a policy \(\pi_{cf}\). In others it is an implicit modeling choice, such as assuming \(\rho=0\) or \(\rho=1\) in retrospective counterfactual prediction. The same word therefore spans reference inputs, comparator distributions, control variates, null perturbations, and benchmark methods.

| Setting | Baseline object | Primary role |
|---|---|---|
| Path attribution | input \(\hat{x}\) or \(p(\hat{x})\) | defines the contrastive notion of “missingness” [2508.14482] |
| One-vs-one saliency | target-class sample \(B_{c_t}(x)\) | isolates discriminative rather than shared features [2011.06015] |
| Policy optimization | counterfactual policy \(\pi_{cf}\) | reference policy for causal grounding regularization [2606.23206] |
| MCCFR | state-action baseline \(b_i(I,a)\) | variance-reducing control variate for counterfactual values [1809.03057] |
| Retrospective causal prediction | endpoint choice \(\rho\in\{0,1\}\) | encodes cross-world dependence assumptions [2603.27320] |
| Counterfactual prompting | benign paraphrase baseline | estimates general prompt sensitivity [2605.01048] |

This taxonomy also clarifies an important terminological ambiguity. In reinforcement learning, “baseline” often means a variance-reduction term such as a value function. In CFPO, by contrast, the counterfactual baseline is a comparator policy induced by intervention, and the paper explicitly states that this is not a baseline in the classic variance-reduction sense [2606.23206]. Conversely, in VR-MCCFR, counterfactual baselines are precisely control variates for sampled counterfactual values [1809.03057]. The literature therefore uses the same word for distinct mathematical roles, even when all of them are counterfactual in spirit.

## 2. Semantic baselines in attribution and explanation

In attribution methods, the baseline is often the most direct embodiment of counterfactual contrast. For Integrated Gradients, given classifier \(\mathcal{C}\), input \(x\), and baseline \(\hat{x}\), attribution is defined by
\[
\mathrm{IG}_i(x) = (x_i-\hat{x}_i)\int_0^1 \frac{\partial \mathcal{C}(\hat{x}+\alpha(x-\hat{x}))}{\partial x_i}\,d\alpha.
\]
Expected Gradients extends this by averaging over a baseline distribution \(p(\hat{x})\). In medical settings, the central critique is that IG and EG are only as meaningful as the semantics of \(\hat{x}\): a zero image, blur, or random noise may define a mathematical contrast while failing to define a clinically meaningful absence of pathology. The proposed remedy is an input-specific, clinically normal, input-close counterfactual baseline, constructed in the paper with a VAE and latent optimization but presented as generative-model-agnostic [2508.14482].

This medical perspective reframes missingness itself. The paper argues that in domains such as manometry and pneumothorax detection, absence can be pathognomonic, so “all-zero” may encode disease rather than the absence of disease. The baseline should therefore represent absence of pathological evidence, not absence of signal. A good baseline must be both normal and close: if it is normal but anatomically unrelated, interpolation becomes unrealistic; if it is close but not normal, the contrast does not isolate pathology [2508.14482].

A related but class-contrastive formulation appears in GANMEX, where the desired baseline for one-vs-one attribution is the closest realistic sample from a specified target class:
\[
B_{c_t}(x)=\arg\min_{\tilde{x}\in G_{c_t}}\|x-\tilde{x}\|.
\]
Because \(G_{c_t}\) is not directly tractable, the paper relaxes this to an objective combining distance, realism, and target-class probability. The generator is coupled to the classifier to be explained, and an explicit similarity loss is added so that the counterfactual baseline preserves non-essential factors while changing class-discriminative content. This produces target-class-specific references for IG, DeepLIFT, DeepSHAP, Occlusion, and EG, thereby operationalizing a genuine “why class \(A\) instead of class \(B\)?” explanation [2011.06015].

Latent-CF takes a different route. It uses an autoencoder, encodes \(x_0\) to \(z_0=E(x_0)\), optimizes \(z\) until \(f(D(z))\) reaches the desired target probability, and decodes \(x_{cf}=D(z)\). The paper’s baseline claim is methodological: a simple latent-space search can already balance speed, sparsity, and in-distribution quality relative to more complex feature-space methods [2012.09301]. DeDUCE, in turn, shows that input-space saliency updates can remain close to the original while maintaining realism if they are guided by target-class density in feature space rather than by a separate generator; empirically it is much closer than JSMA, VLK, or REVISE while keeping comparable realism [2111.15639]. In forecasting, the same issue appears in another form: nearest-neighbor retrieval and naive global shifts are used as counterfactual baselines, but gradient-based optimization against the forecasting model is needed to satisfy trajectory-wide validity constraints [2310.08137].

## 3. Construction principles: locality, manifold adherence, and group geometry

Across these works, three recurring design principles govern counterfactual baseline construction. The first is locality: the baseline should remain close to the factual instance in whatever representation is relevant. The second is manifold adherence: the baseline should represent a realistic state rather than an arbitrary perturbation. The third is geometry preservation when the object of explanation is a group rather than an individual.

The medical counterfactual-baseline paper makes locality and normality explicit through the latent optimization
\[
z^*=\arg\min_z \Big[\mathcal{L}_{CE}(\mathcal{C}(\mathcal{D}(z)),y_{\text{normal}})+\lambda_{\text{sim}}\|x-\mathcal{D}(z)\|_2^2\Big],
\]
then implements a simplified version by starting from \(z_0=\mathcal{E}(x)\) and optimizing only the normal-class cross-entropy with early stopping. This uses the initialization and stopping rule as implicit proximity control [2508.14482]. GANMEX imposes analogous structure via classifier-coupled realism and explicit \(L_1\) similarity [2011.06015]. Latent-CF relies on the decoder image of the latent space as an implicit manifold constraint [2012.09301].

For group counterfactuals, geometry preservation becomes the distinctive criterion. Optimal Transport Group Counterfactual Explanations replaces fixed point allocations with a transport map \(\hat g\) minimizing empirical squared transport cost while satisfying target-label constraints and a bi-Lipschitz condition:
\[
\frac1k\|\mathbf{x}^{(i)}-\mathbf{x}^{(j)}\|_2 \le \|\hat g(\mathbf{x}^{(i)})-\hat g(\mathbf{x}^{(j)})\|_2 \le K\|\mathbf{x}^{(i)}-\mathbf{x}^{(j)}\|_2.
\]
The paper positions three baselines against this: Independent, Group w/ Lipschitz, and Group w/ bi-Lipschitz. All optimize \(nd\) variables for a fixed group and do not generalize to unseen group members. The OT map uses fewer parameters and can be applied without re-optimization, while convex subclasses such as PSD affine or scaled Gaussian maps provide tractable geometry-preserving transport [2601.20692]. This suggests that, for group explanations, a baseline is not only a reference state but also a choice about whether explanation is allocation-based or map-based.

A similar shift from explicit density baselines to implicit generative modeling appears in time-varying treatment settings. There, KDE and weighted Plugin+KDE serve as counterfactual density baselines, while CVAE and diffusion are unweighted generative ablations. The proposed marginal structural generative models add inverse-probability weighting to match the counterfactual distribution under sequential confounding, and outperform the density-estimation baselines in high-dimensional outcomes [2305.15742]. The common pattern is that baseline construction increasingly trades explicit tractability for implicit but better-aligned generative structure when dimensionality or sequentiality becomes large.

## 4. Counterfactual baselines in optimization, control, and games

In multimodal reinforcement learning, CFPO defines a counterfactual baseline at the level of policy comparison. Critical visual cues are identified through cross-modal attention, then suppressed in latent attention/value space to produce a counterfactual state \(Z_{cf}\) and a counterfactual policy
\[
\pi_{cf}(o\mid q,I)=\pi_\theta(o\mid do(Z=Z_{cf})).
\]
The key contrast is
\[
\Delta_{cf}=\log\pi_\theta(o\mid Z)-\log\pi_\theta(o\mid do(Z=Z_{cf})),
\]
and the corresponding divergence \(KL_{cf}=\mathbb{D}_{KL}(\pi_\theta\|\pi_{cf})\) is added to GRPO or DAPO. The paper is explicit that this is not a standard RL baseline like a value function; it is a counterfactual reference policy used as a regularization target [2606.23206].

In Monte Carlo Counterfactual Regret Minimization, by contrast, the baseline is exactly a control variate. Given an unbiased estimator \(\hat v_i(\sigma,I,a)\) of an action-dependent counterfactual value, VR-MCCFR defines
\[
\hat v_i^b(\sigma,I,a)=\hat v_i(\sigma,I,a)-\hat b_i(\sigma,I,a)+b_i(\sigma,I,a),
\]
where \(b_i(I,a)\) is a state-action baseline and \(\hat b_i\) is an unbiased sampled estimator of it. The method then bootstraps baseline-corrected estimates recursively along the sampled trajectory, and the paper proves that a perfect baseline can reduce the variance of updated value estimates to zero [1809.03057]. Here the baseline is not a semantic counterfactual state but a variance-reduction device attached to counterfactual values.

Counterfactual realizability extends the idea further by asking when counterfactual distributions themselves can be sampled physically rather than merely identified symbolically. Under the paper’s fundamental constraint of experimentation, some \(\mathcal{L}_3\) distributions are realizable and others are not. The CTF-REALIZE algorithm is complete, and under maximal feasible counterfactual randomization a distribution \(P(\*W_\star)\) is realizable iff \(An(\*W_\star)\) does not contain the same variable under different regimes. In motivating examples, counterfactual strategies then dominate observational and interventional baselines: in fairness they optimize the actual counterfactual target rather than a proxy, and in bandit-style decision problems the optimal realizable \(\mathcal{L}_3\) strategy provably dominates observational and interventional baselines in expected reward [2503.11870].

## 5. Baselines as modeling assumptions in causal and fair prediction

In retrospective counterfactual prediction, the baseline is an explicit cross-world assumption. The paper studies
\[
\mu(x,y)=\mathbb{E}[Y(1)\mid X=x,Y(0)=y]
\]
and shows that common heuristics correspond to endpoint assumptions on the unobserved cross-world correlation
\[
\rho(x)=\operatorname{cor}(Y(1),Y(0)\mid X=x).
\]
Under a Gaussian cross-world model,
\[
\mu(x,y)=\mu_1(x)+\rho(x)\frac{\sigma_1(x)}{\sigma_0(x)}(y-\mu_0(x)).
\]
When \(\rho=0\), the factual outcome is ignored, reproducing direct-outcome or matching-style baselines. When \(\rho=1\) and \(\lambda(x)=1\), the counterfactual becomes a shifted factual outcome, \(\hat Y(1)=Y(0)+\hat\tau(X)\). The paper’s main point is that these are not innocuous defaults; they are endpoint assumptions about cross-world dependence [2603.27320].

In conformal fairness, the situation is different. \(C^3F\) does not benchmark against a standalone counterfactual-fairness algorithm; its experimental baselines are weighted and fairness-oriented conformal methods, while the counterfactual component enters as a path-specific regularizer. The most faithful counterfactual baseline is therefore the \(\lambda=0\) ablation: weighted group-conditional conformal calibration without counterfactual regularization. The paper explicitly states that the named baselines are not themselves counterfactual fairness baselines, and that the causal references mainly motivate the regularizer [2509.25295].

The same distinction between conceptual and implemented baselines appears in fair prediction with GCFN. The paper positions latent-variable methods such as CFAN, CFUA, mCEVAE, DCEVAE, and ADVAE as the main counterfactual-fairness baselines, and contrasts them with CFGAN as a GAN-based interventional fairness method rather than a true counterfactual baseline. GCFN instead learns the counterfactual distribution of mediators directly and regularizes
\[
\mathcal{R}_{cm}(h)=\mathbb{E}\big[\|h(X,M)-h(X,\hat M_{A'})\|_2^2\big].
\]
Its contribution is thus a new baseline family for counterfactual fairness: direct counterfactual descendant generation rather than latent-background inference [2310.17687].

## 6. Null baselines, benchmarking practice, and recurring misconceptions

A distinct line of work treats the baseline as a null reference needed to interpret observed counterfactual effects. In counterfactual prompting, the central argument is that a targeted textual edit is always a compound treatment, since it changes both the intended factor and the linguistic carrier. The paper therefore compares targeted interventions against meaning-preserving paraphrases matched in token-edit magnitude. On MedQA, patient-gender edits produced a 14.9% flip rate, while benign paraphrases produced 14.1%; in the MedPerturb re-analysis, only 5 of 120 tests remained significant after baseline correction. The paper also shows that per-sample JSD and KL are much more powerful than aggregate flip-rate or MI-style metrics [2605.01048]. In this literature, the baseline is not the counterfactual itself but the null comparator required to attribute an effect to the target factor.

Benchmarking work in counterfactual text generation uses “baseline” in yet another way: as representative comparator methods. CEval standardizes MICE, GBDA, CREST, and LLAMA-2, and shows that no perfect method exists: target-aware methods achieve better validity, while LLM prompting produces better text quality but weaker counterfactual performance [2404.17475]. In tabular augmentation, CCRAL distinguishes Standard from Counterfactual, where the latter augments all training points with treatment-flipped counterfactuals; CCRAL’s contribution is selective inclusion based on uncertainty, not a different counterfactual model [2207.12086]. In forecasting, BaseNN and BaseShift play analogous roles as retrieval and naive-shift baselines [2310.08137].

Several misconceptions recur across these domains. First, a baseline is often treated as harmless infrastructure; the attribution literature argues the opposite, namely that it is a semantic commitment about what counts as missing evidence [2508.14482]. Second, “counterfactual baseline” does not always mean a value-function baseline; in CFPO it denotes a counterfactual comparator policy, whereas in VR-MCCFR it is a control variate [2606.23206, 1809.03057]. Third, empirical studies may invoke counterfactual fairness or counterfactual prompting without a proper null or implemented causal comparator; some papers therefore rely on conceptual counterfactual baselines or ablations rather than direct benchmark algorithms [2509.25295, 2605.01048].

Taken together, this literature shows that counterfactual baselines are not a single technical device but a family of reference constructions that determine what is being contrasted, what is being regularized, what is being held fixed, and what counts as a meaningful alternative. This suggests that progress on counterfactual methods depends as much on baseline design as on the headline optimization algorithm itself.

Source: https://www.emergentmind.com/topics/counterfactual-baselines