Counterfactual Baselines

• Counterfactual baselines are reference states that define what is being contrasted in factual versus alternative scenarios, crucial for semantic interpretation.
• They are applied across methods including path attribution, reinforcement learning, and causal inference, often using latent optimization and generative models.
• Their careful design influences variance reduction, model regularization, and fairness assessments, underscoring their role in robust counterfactual analysis.

Counterfactual baselines are reference objects used to define what a factual prediction, explanation, policy, or outcome is being contrasted against. Across the literature, the term denotes several distinct but related constructions: an input reference for path attribution, a target-class exemplar for one-vs-one saliency, a comparator policy in reinforcement learning, a control variate for counterfactual values in extensive-form games, an endpoint assumption about cross-world dependence in retrospective causal prediction, or a meaning-preserving prompt perturbation used to estimate generic model sensitivity. What unifies these uses is that the baseline fixes the counterfactual question being asked; it is therefore not merely an implementation detail but part of the semantics of the analysis itself (Geiger et al., 20 Aug 2025, Yu et al., 22 Jun 2026, Bodik, 28 Mar 2026, Yang et al., 1 May 2026).

1. Conceptual scope and taxonomy

In the most general sense, a counterfactual baseline specifies the alternative state relative to which a factual quantity is interpreted. In some settings this state is an explicit object, such as an image $\hat{x}$ or a policy $\pi_{cf}$. In others it is an implicit modeling choice, such as assuming $\rho=0$ or $\rho=1$ in retrospective counterfactual prediction. The same word therefore spans reference inputs, comparator distributions, control variates, null perturbations, and benchmark methods.

Setting	Baseline object	Primary role
Path attribution	input $\hat{x}$ or $p(\hat{x})$	defines the contrastive notion of “missingness” (Geiger et al., 20 Aug 2025)
One-vs-one saliency	target-class sample $B_{c_t}(x)$	isolates discriminative rather than shared features (Shih et al., 2020)
Policy optimization	counterfactual policy $\pi_{cf}$	reference policy for causal grounding regularization (Yu et al., 22 Jun 2026)
MCCFR	state-action baseline $b_i(I,a)$	variance-reducing control variate for counterfactual values (Schmid et al., 2018)
Retrospective causal prediction	endpoint choice $\rho\in\{0,1\}$	encodes cross-world dependence assumptions (Bodik, 28 Mar 2026)
Counterfactual prompting	benign paraphrase baseline	estimates general prompt sensitivity (Yang et al., 1 May 2026)

This taxonomy also clarifies an important terminological ambiguity. In reinforcement learning, “baseline” often means a variance-reduction term such as a value function. In CFPO, by contrast, the counterfactual baseline is a comparator policy induced by intervention, and the paper explicitly states that this is not a baseline in the classic variance-reduction sense (Yu et al., 22 Jun 2026). Conversely, in VR-MCCFR, counterfactual baselines are precisely control variates for sampled counterfactual values (Schmid et al., 2018). The literature therefore uses the same word for distinct mathematical roles, even when all of them are counterfactual in spirit.

2. Semantic baselines in attribution and explanation

In attribution methods, the baseline is often the most direct embodiment of counterfactual contrast. For Integrated Gradients, given classifier $\pi_{cf}$0, input $\pi_{cf}$1, and baseline $\pi_{cf}$2, attribution is defined by

$\pi_{cf}$3

Expected Gradients extends this by averaging over a baseline distribution $\pi_{cf}$4. In medical settings, the central critique is that IG and EG are only as meaningful as the semantics of $\pi_{cf}$5: a zero image, blur, or random noise may define a mathematical contrast while failing to define a clinically meaningful absence of pathology. The proposed remedy is an input-specific, clinically normal, input-close counterfactual baseline, constructed in the paper with a VAE and latent optimization but presented as generative-model-agnostic (Geiger et al., 20 Aug 2025).

This medical perspective reframes missingness itself. The paper argues that in domains such as manometry and pneumothorax detection, absence can be pathognomonic, so “all-zero” may encode disease rather than the absence of disease. The baseline should therefore represent absence of pathological evidence, not absence of signal. A good baseline must be both normal and close: if it is normal but anatomically unrelated, interpolation becomes unrealistic; if it is close but not normal, the contrast does not isolate pathology (Geiger et al., 20 Aug 2025).

A related but class-contrastive formulation appears in GANMEX, where the desired baseline for one-vs-one attribution is the closest realistic sample from a specified target class: $\pi_{cf}$6 Because $\pi_{cf}$7 is not directly tractable, the paper relaxes this to an objective combining distance, realism, and target-class probability. The generator is coupled to the classifier to be explained, and an explicit similarity loss is added so that the counterfactual baseline preserves non-essential factors while changing class-discriminative content. This produces target-class-specific references for IG, DeepLIFT, DeepSHAP, Occlusion, and EG, thereby operationalizing a genuine “why class $\pi_{cf}$8 instead of class $\pi_{cf}$9?” explanation (Shih et al., 2020).

Latent-CF takes a different route. It uses an autoencoder, encodes $\rho=0$0 to $\rho=0$1, optimizes $\rho=0$2 until $\rho=0$3 reaches the desired target probability, and decodes $\rho=0$4. The paper’s baseline claim is methodological: a simple latent-space search can already balance speed, sparsity, and in-distribution quality relative to more complex feature-space methods (Balasubramanian et al., 2020). DeDUCE, in turn, shows that input-space saliency updates can remain close to the original while maintaining realism if they are guided by target-class density in feature space rather than by a separate generator; empirically it is much closer than JSMA, VLK, or REVISE while keeping comparable realism (Höltgen et al., 2021). In forecasting, the same issue appears in another form: nearest-neighbor retrieval and naive global shifts are used as counterfactual baselines, but gradient-based optimization against the forecasting model is needed to satisfy trajectory-wide validity constraints (Wang et al., 2023).

3. Construction principles: locality, manifold adherence, and group geometry

Across these works, three recurring design principles govern counterfactual baseline construction. The first is locality: the baseline should remain close to the factual instance in whatever representation is relevant. The second is manifold adherence: the baseline should represent a realistic state rather than an arbitrary perturbation. The third is geometry preservation when the object of explanation is a group rather than an individual.

The medical counterfactual-baseline paper makes locality and normality explicit through the latent optimization

$\rho=0$5

then implements a simplified version by starting from $\rho=0$6 and optimizing only the normal-class cross-entropy with early stopping. This uses the initialization and stopping rule as implicit proximity control (Geiger et al., 20 Aug 2025). GANMEX imposes analogous structure via classifier-coupled realism and explicit $\rho=0$7 similarity (Shih et al., 2020). Latent-CF relies on the decoder image of the latent space as an implicit manifold constraint (Balasubramanian et al., 2020).

For group counterfactuals, geometry preservation becomes the distinctive criterion. Optimal Transport Group Counterfactual Explanations replaces fixed point allocations with a transport map $\rho=0$8 minimizing empirical squared transport cost while satisfying target-label constraints and a bi-Lipschitz condition: $\rho=0$9 The paper positions three baselines against this: Independent, Group w/ Lipschitz, and Group w/ bi-Lipschitz. All optimize $\rho=1$0 variables for a fixed group and do not generalize to unseen group members. The OT map uses fewer parameters and can be applied without re-optimization, while convex subclasses such as PSD affine or scaled Gaussian maps provide tractable geometry-preserving transport (Valero-Leal et al., 28 Jan 2026). This suggests that, for group explanations, a baseline is not only a reference state but also a choice about whether explanation is allocation-based or map-based.

A similar shift from explicit density baselines to implicit generative modeling appears in time-varying treatment settings. There, KDE and weighted Plugin+KDE serve as counterfactual density baselines, while CVAE and diffusion are unweighted generative ablations. The proposed marginal structural generative models add inverse-probability weighting to match the counterfactual distribution under sequential confounding, and outperform the density-estimation baselines in high-dimensional outcomes (Wu et al., 2023). The common pattern is that baseline construction increasingly trades explicit tractability for implicit but better-aligned generative structure when dimensionality or sequentiality becomes large.

4. Counterfactual baselines in optimization, control, and games

In multimodal reinforcement learning, CFPO defines a counterfactual baseline at the level of policy comparison. Critical visual cues are identified through cross-modal attention, then suppressed in latent attention/value space to produce a counterfactual state $\rho=1$1 and a counterfactual policy

$\rho=1$2

The key contrast is

$\rho=1$3

and the corresponding divergence $\rho=1$4 is added to GRPO or DAPO. The paper is explicit that this is not a standard RL baseline like a value function; it is a counterfactual reference policy used as a regularization target (Yu et al., 22 Jun 2026).

In Monte Carlo Counterfactual Regret Minimization, by contrast, the baseline is exactly a control variate. Given an unbiased estimator $\rho=1$5 of an action-dependent counterfactual value, VR-MCCFR defines

$\rho=1$6

where $\rho=1$7 is a state-action baseline and $\rho=1$8 is an unbiased sampled estimator of it. The method then bootstraps baseline-corrected estimates recursively along the sampled trajectory, and the paper proves that a perfect baseline can reduce the variance of updated value estimates to zero (Schmid et al., 2018). Here the baseline is not a semantic counterfactual state but a variance-reduction device attached to counterfactual values.

Counterfactual realizability extends the idea further by asking when counterfactual distributions themselves can be sampled physically rather than merely identified symbolically. Under the paper’s fundamental constraint of experimentation, some $\rho=1$9 distributions are realizable and others are not. The CTF-REALIZE algorithm is complete, and under maximal feasible counterfactual randomization a distribution $\hat{x}$0 is realizable iff $\hat{x}$1 does not contain the same variable under different regimes. In motivating examples, counterfactual strategies then dominate observational and interventional baselines: in fairness they optimize the actual counterfactual target rather than a proxy, and in bandit-style decision problems the optimal realizable $\hat{x}$2 strategy provably dominates observational and interventional baselines in expected reward (Raghavan et al., 14 Mar 2025).

5. Baselines as modeling assumptions in causal and fair prediction

In retrospective counterfactual prediction, the baseline is an explicit cross-world assumption. The paper studies

$\hat{x}$3

and shows that common heuristics correspond to endpoint assumptions on the unobserved cross-world correlation

$\hat{x}$4

Under a Gaussian cross-world model,

$\hat{x}$5

When $\hat{x}$6, the factual outcome is ignored, reproducing direct-outcome or matching-style baselines. When $\hat{x}$7 and $\hat{x}$8, the counterfactual becomes a shifted factual outcome, $\hat{x}$9. The paper’s main point is that these are not innocuous defaults; they are endpoint assumptions about cross-world dependence (Bodik, 28 Mar 2026).

In conformal fairness, the situation is different. $p(\hat{x})$0 does not benchmark against a standalone counterfactual-fairness algorithm; its experimental baselines are weighted and fairness-oriented conformal methods, while the counterfactual component enters as a path-specific regularizer. The most faithful counterfactual baseline is therefore the $p(\hat{x})$1 ablation: weighted group-conditional conformal calibration without counterfactual regularization. The paper explicitly states that the named baselines are not themselves counterfactual fairness baselines, and that the causal references mainly motivate the regularizer (Alpay et al., 29 Sep 2025).

The same distinction between conceptual and implemented baselines appears in fair prediction with GCFN. The paper positions latent-variable methods such as CFAN, CFUA, mCEVAE, DCEVAE, and ADVAE as the main counterfactual-fairness baselines, and contrasts them with CFGAN as a GAN-based interventional fairness method rather than a true counterfactual baseline. GCFN instead learns the counterfactual distribution of mediators directly and regularizes

$p(\hat{x})$2

Its contribution is thus a new baseline family for counterfactual fairness: direct counterfactual descendant generation rather than latent-background inference (Ma et al., 2023).

6. Null baselines, benchmarking practice, and recurring misconceptions

A distinct line of work treats the baseline as a null reference needed to interpret observed counterfactual effects. In counterfactual prompting, the central argument is that a targeted textual edit is always a compound treatment, since it changes both the intended factor and the linguistic carrier. The paper therefore compares targeted interventions against meaning-preserving paraphrases matched in token-edit magnitude. On MedQA, patient-gender edits produced a 14.9% flip rate, while benign paraphrases produced 14.1%; in the MedPerturb re-analysis, only 5 of 120 tests remained significant after baseline correction. The paper also shows that per-sample JSD and KL are much more powerful than aggregate flip-rate or MI-style metrics (Yang et al., 1 May 2026). In this literature, the baseline is not the counterfactual itself but the null comparator required to attribute an effect to the target factor.

Benchmarking work in counterfactual text generation uses “baseline” in yet another way: as representative comparator methods. CEval standardizes MICE, GBDA, CREST, and LLAMA-2, and shows that no perfect method exists: target-aware methods achieve better validity, while LLM prompting produces better text quality but weaker counterfactual performance (Nguyen et al., 2024). In tabular augmentation, CCRAL distinguishes Standard from Counterfactual, where the latter augments all training points with treatment-flipped counterfactuals; CCRAL’s contribution is selective inclusion based on uncertainty, not a different counterfactual model (Mohammed et al., 2022). In forecasting, BaseNN and BaseShift play analogous roles as retrieval and naive-shift baselines (Wang et al., 2023).

Several misconceptions recur across these domains. First, a baseline is often treated as harmless infrastructure; the attribution literature argues the opposite, namely that it is a semantic commitment about what counts as missing evidence (Geiger et al., 20 Aug 2025). Second, “counterfactual baseline” does not always mean a value-function baseline; in CFPO it denotes a counterfactual comparator policy, whereas in VR-MCCFR it is a control variate (Yu et al., 22 Jun 2026, Schmid et al., 2018). Third, empirical studies may invoke counterfactual fairness or counterfactual prompting without a proper null or implemented causal comparator; some papers therefore rely on conceptual counterfactual baselines or ablations rather than direct benchmark algorithms (Alpay et al., 29 Sep 2025, Yang et al., 1 May 2026).

Taken together, this literature shows that counterfactual baselines are not a single technical device but a family of reference constructions that determine what is being contrasted, what is being regularized, what is being held fixed, and what counts as a meaningful alternative. This suggests that progress on counterfactual methods depends as much on baseline design as on the headline optimization algorithm itself.