Counterfactual Relabeling in Causal Inference

• Counterfactual Relabeling is a process of reassigning observed outcomes with hypothetical labels to simulate alternative scenarios, enhancing causal inference and decision-making.
• It enables learning from augmented data by reducing observational bias and approximating randomized trial conditions using techniques like local regression and self-training.
• Methods such as COCOA, CST, HInt, and CAST leverage counterfactual relabeling to improve treatment effect estimation, reward structuring, and model audit for reliable predictions.

Counterfactual relabeling denotes, across several research programs, the reassignment, imputation, or synthesis of labels, outcomes, actions, goals, or related annotations so that a learning system is trained or evaluated against hypothetical alternatives rather than only realized observations. In causal effect estimation, the operation targets missing potential outcomes; in counterfactual classification, unobserved action outcomes are pseudolabeled to simulate a randomized trial; in goal-conditioned reinforcement learning, achieved states are relabeled as alternative goals; in instruction-conditioned robotics, the same observation is paired with multiple plausible language–action labels; and in model audit settings, a minimal subset of training labels is altered to determine whether a prediction can be flipped (Aloui et al., 2023, Gao et al., 2021, Chuck et al., 6 May 2025, Glossop et al., 19 Aug 2025, Yang et al., 2023). The unifying technical problem is that only one factual realization is observed per unit, while the desired relabel usually depends on counterfactual structure that may be estimated, approximated, or explicitly chosen.

1. Formal objects and recurring problem formulations

A central formal object is the potential-outcome representation used in conditional treatment effect estimation. For covariates $X$ and potential outcomes $Y_1, Y_0$, the conditional average treatment effect is

$\tau(x) = \mathbb{E}[Y_1 - Y_0 \mid X = x].$

Here, counterfactual relabeling appears as the imputation of the missing potential outcome for each individual, with the relabeled sample appended to the observed dataset (Aloui et al., 2023).

In statistical decision theory, the same idea is expressed at the level of loss functions. For treatment set $D$, a counterfactual loss is written as

$\ell(d; \{Y(d')\}_{d' \in D}),$

and the associated counterfactual risk of a policy $\pi$ is

$$R(\pi; \ell) = \mathbb{E}\big[\ell(\pi(X); \{Y(d)\}_{d \in D})\big].$$

This formulation evaluates a decision using all potential outcomes rather than only the observed outcome under the chosen action (Koch et al., 13 May 2025).

In explainability, counterfactual instances are defined differently: they are modified inputs that alter a model prediction toward a desired target while remaining close to the original input. A deep reinforcement learning formulation casts the generation problem as a one-step Markov Decision Process with state $s = (x, y_M, y_T, c)$, where $x$ is the input instance, $y_M$ the model prediction, $Y_1, Y_0$0 the desired target, and $Y_1, Y_0$1 an optional feature-conditioning vector (Samoilescu et al., 2021).

Goal-conditioned reinforcement learning introduces another usage. Hindsight experience replay relabels failed trajectories with achieved goals, treating them as successful if they match the new goal. The limitation emphasized in object-centric domains is that standard hindsight relabeling can overproduce trivial goals corresponding to trajectories that never interacted with the relevant object (Chuck et al., 6 May 2025).

These formulations differ in surface syntax, but they share a common structure: an observed tuple is supplemented with a counterfactual label or target whose validity depends on assumptions about similarity, interaction, causal structure, or policy invariance. This suggests that “counterfactual relabeling” is best viewed as a family of augmentation and reinterpretation operations rather than a single algorithmic primitive.

2. Potential-outcome imputation and self-training under observational bias

A direct form of counterfactual relabeling is developed in COCOA, COntrastive COunterfactual Augmentation, a model-agnostic data augmentation method for CATE estimation. The method first learns a similarity measure via contrastive learning. Positive pairs are individuals with the same treatment and similar observed outcomes within a tolerance $Y_1, Y_0$2; negative pairs are individuals with the same treatment but dissimilar outcomes. A classifier $Y_1, Y_0$3 is trained to predict whether two individuals are similar (Aloui et al., 2023).

For an individual $Y_1, Y_0$4 observed under treatment $Y_1, Y_0$5, COCOA scans the alternative treatment group $Y_1, Y_0$6 and forms the neighbor set

$Y_1, Y_0$7

Imputation proceeds only if $Y_1, Y_0$8 for a threshold $Y_1, Y_0$9 such as $\tau(x) = \mathbb{E}[Y_1 - Y_0 \mid X = x].$0. The missing potential outcome is then inferred by local regression over the selected neighbors, using either local linear regression or Gaussian process regression. In the Gaussian-process case, the imputed counterfactual is

$\tau(x) = \mathbb{E}[Y_1 - Y_0 \mid X = x].$1

The relabeled triple $\tau(x) = \mathbb{E}[Y_1 - Y_0 \mid X = x].$2 is added to the dataset (Aloui et al., 2023).

The stated trade-off is between disparity reduction and imputation error. Augmenting the data reduces the statistical difference between treatment groups, making CATE estimation more like an RCT, while restricting imputation to cases with enough similar neighbors is intended to keep imputation error low. The theoretical analysis gives regret bounds indicating that accurate CATE estimation may require small imputation error, and the asymptotic argument shows that, under positivity, the augmented data distribution approaches the RCT distribution as data size grows (Aloui et al., 2023).

Empirically, COCOA is reported to reduce $\tau(x) = \mathbb{E}[Y_1 - Y_0 \mid X = x].$3 across benchmark CATE models including TARNet, CFR-Wass, BART, and Causal Forests on IHDP, Twins, News, and synthetic Linear and Nonlinear datasets. The method is described as less prone to overfitting, superior to Perfect Match, superior to propensity scores or Euclidean distance as a similarity measure, and robust to choices of $\tau(x) = \mathbb{E}[Y_1 - Y_0 \mid X = x].$4, embedding radius $\tau(x) = \mathbb{E}[Y_1 - Y_0 \mid X = x].$5, and similarity sensitivity parameter $\tau(x) = \mathbb{E}[Y_1 - Y_0 \mid X = x].$6 (Aloui et al., 2023).

Counterfactual Self-Training (CST) addresses a related problem in counterfactual classification, where only partial feedback is observed. CST treats observational data as a source domain and a notional randomized trial as the target domain. A base classifier is trained on factual data; pseudolabels are then imputed for every unobserved action $\tau(x) = \mathbb{E}[Y_1 - Y_0 \mid X = x].$7 for each context $\tau(x) = \mathbb{E}[Y_1 - Y_0 \mid X = x].$8; the dataset is augmented with these counterfactual labels; and the model is retrained iteratively until convergence (Gao et al., 2021).

Its core objective combines cross-entropy on factual outcomes and cross-entropy on imputed counterfactual outcomes,

$\tau(x) = \mathbb{E}[Y_1 - Y_0 \mid X = x].$9

and can be augmented with Counterfactual VAT through the consistency regularizer

$D$0

yielding

$D$1

The paper reports improved negative log-likelihood and Hamming loss over Direct Method, HSIC, and UDM on synthetic and real datasets, with additional gains from input consistency (Gao et al., 2021).

Taken together, these two lines of work illustrate two distinct counterfactual relabeling regimes. COCOA relabels only a selected subset where local approximation is considered reliable; CST imputes all unseen actions and relies on iterative self-training plus consistency regularization. The difference is methodological rather than conceptual: both aim to reduce observational bias by constructing a dataset that more closely resembles randomized exposure.

3. Hindsight relabeling, null counterfactuals, and object-centric reinforcement learning

In goal-conditioned reinforcement learning, hindsight relabeling is a standard response to reward sparsity. The difficulty identified in object-centric domains is that standard hindsight relabeling can assign high rewards to trajectories that never interact with the target object, because such trajectories are relabeled as successful for goals corresponding to the object’s initial position. These behaviors are only useful when the object is already at the goal, which is described as an extremely rare case in practice (Chuck et al., 6 May 2025).

HInt, Hindsight Relabeling using Interactions, addresses this failure mode by filtering relabelings through inferred interaction structure. The key theoretical move is to define interaction through a null counterfactual. A cause object $D$2 is interacting with a target object $D$3 if the target’s transition dynamics would differ in a world where the cause object did not exist: $D$4 The operation $D$5 denotes the state with $D$6 “nulled out” (Chuck et al., 6 May 2025).

Because such counterfactual transitions are not directly observable, Null Counterfactual Interaction Inference (NCII) learns a masked forward dynamics model

$D$7

where a mask matrix simulates nulling out specific factors. Interaction is detected by comparing predicted log-likelihoods with and without the candidate cause object. The null test is

$D$8

To avoid combinatorial querying, a neural interaction classifier $D$9 is distilled from the null test so that online inference is $\ell(d; \{Y(d')\}_{d' \in D}),$0 rather than $\ell(d; \{Y(d')\}_{d' \in D}),$1 (Chuck et al., 6 May 2025).

HInt then relabels only those trajectory segments for which the target object’s achieved state can be reached from the agent action through the inferred temporal interaction graph, possibly with path length limited to $\ell(d; \{Y(d')\}_{d' \in D}),$2 for robustness. This removes trivial goals from the replay buffer and concentrates learning on outcomes the agent actually controlled (Chuck et al., 6 May 2025).

The reported empirical outcome is that NCII achieves the lowest misprediction rates relative to strong baselines across simple linear dynamics domains and dynamic robotic domains in Spriteworld, Robosuite, Robot Air Hockey, and Franka Kitchen, while HInt improves sample efficiency by up to $\ell(d; \{Y(d')\}_{d' \in D}),$3 (Chuck et al., 6 May 2025). A common misconception addressed here is that hindsight relabeling is uniformly beneficial in sparse-reward settings; the object-centric analysis shows that indiscriminate relabeling can be dominated by relabeling conditioned on inferred causal interaction.

4. Synthetic counterfactual labels for instruction-conditioned robotics

In vision-language-action learning, counterfactual relabeling is used to increase semantic diversity and language grounding rather than to estimate missing potential outcomes. CAST, Counterfactual Augmentation with Synthetic Trajectories, starts from the observation that if each visual observation in a robot dataset is paired with only one instruction and action, the model may ignore the language command because observation alone predicts the action. The paper identifies this as posterior collapse (Glossop et al., 19 Aug 2025).

CAST leverages vision-LLMs to generate counterfactual language and action labels for existing trajectories without any additional data collection. The pipeline begins with atomic segmentation of trajectories into primitive actions such as “turn left” or “go forward,” automated via yaw change thresholds. Vision-LLMs then generate hindsight language labels for executed trajectories and a second VLM pass filters for relevance and correctness. At selected decision points, the VLM is prompted to produce alternative instructions the robot could take and the corresponding atomic command. The counterfactual action sequence is synthesized by rolling out a policy trained to follow simple atomic commands (Glossop et al., 19 Aug 2025).

The training set is thereby expanded from single $\ell(d; \{Y(d')\}_{d' \in D}),$4 tuples to multiple plausible $\ell(d; \{Y(d')\}_{d' \in D}),$5 tuples for the same observation $\ell(d; \{Y(d')\}_{d' \in D}),$6. The downstream VLA policy is trained by behavioral cloning,

$\ell(d; \{Y(d')\}_{d' \in D}),$7

The theoretical account is given in terms of conditional mutual information. The paper states that the model should attend to language if and only if language adds information about action beyond what is available in the observation, quantified by $\ell(d; \{Y(d')\}_{d' \in D}),$8, and provides the lower bound

$\ell(d; \{Y(d')\}_{d' \in D}),$9

where $\pi$0 is the atomic command (Glossop et al., 19 Aug 2025).

Evaluation is reported on $\pi$1 real-world navigation tasks in $\pi$2 environments—office hallways, a kitchen, and an outdoor park—covering object navigation, referential navigation, and continuous navigation. With a $\pi$3B-parameter PaliGemma backbone, CounterfactualVLA trained on CAST achieves a $\pi$4 average success rate, compared with $\pi$5 for a standard VLA trained without counterfactuals, corresponding to a $\pi$6-percentage-point improvement. The gains are described as especially strong on referential and continuous tasks (Glossop et al., 19 Aug 2025).

This usage broadens the meaning of counterfactual relabeling. The relabeled target is not an unobserved treatment outcome but an alternative instruction–action pair compatible with the same observation. The objective is not identification of a latent causal estimand, but forcing the policy to condition on language by increasing the entropy of valid labels at fixed observation.

5. Counterfactual relabeling for explanations, contestability, and data auditing

A nearby but distinct research direction uses counterfactual generation to explain model predictions. A deep reinforcement learning approach formulates counterfactual instance generation as a one-step MDP over latent representations. An autoencoder maps the input into latent space, an actor network produces a counterfactual latent code conditioned on the original prediction, desired target, and feature constraints, and a decoder reconstructs the counterfactual instance. The method relies only on model predictions, not gradients or internal structure, and supports numerical and categorical features, immutability constraints, monotonicity constraints, value-range constraints, and other modalities such as images (Samoilescu et al., 2021).

The reward for classification is

$\pi$7

and the actor objective combines critic maximization, sparsity, and consistency,

$\pi$8

A principal operational claim is that, once trained, the generator can produce a batch of counterfactuals in a single forward pass rather than solving a separate optimization problem for each instance (Samoilescu et al., 2021). Although this is not label relabeling in the narrow sense, it occupies the same conceptual space: the system produces an alternative target-consistent annotation of a factual example.

Relabeling becomes literal in the problem of flipping a test prediction by changing training labels. For binary classification models with convex loss, the goal is to identify the smallest training subset $\pi$9 such that relabeling those points and retraining flips the prediction on a test point $$R(\pi; \ell) = \mathbb{E}\big[\ell(\pi(X); \{Y(d)\}_{d \in D})\big].$$0. The training objective is

$$R(\pi; \ell) = \mathbb{E}\big[\ell(\pi(X); \{Y(d)\}_{d \in D})\big].$$1

and the effect of relabeling is approximated with an extended influence function (Yang et al., 2023).

If $$R(\pi; \ell) = \mathbb{E}\big[\ell(\pi(X); \{Y(d)\}_{d \in D})\big].$$2 is relabeled, the parameter perturbation is approximated by

$$R(\pi; \ell) = \mathbb{E}\big[\ell(\pi(X); \{Y(d)\}_{d \in D})\big].$$3

and the induced change in the test prediction is

$$R(\pi; \ell) = \mathbb{E}\big[\ell(\pi(X); \{Y(d)\}_{d \in D})\big].$$4

The algorithm computes the influence of flipping each training label, sorts points in descending order, and accumulates them until the predicted change crosses the classification threshold. Its total complexity per test point is $$R(\pi; \ell) = \mathbb{E}\big[\ell(\pi(X); \{Y(d)\}_{d \in D})\big].$$5 (Yang et al., 2023).

The empirical claims are notable: relabeling fewer than $$R(\pi; \ell) = \mathbb{E}\big[\ell(\pi(X); \{Y(d)\}_{d \in D})\big].$$6 of the training points can always flip a prediction; the cardinality $$R(\pi; \ell) = \mathbb{E}\big[\ell(\pi(X); \{Y(d)\}_{d \in D})\big].$$7 is highly related to the noise ratio in the training set; $$R(\pi; \ell) = \mathbb{E}\big[\ell(\pi(X); \{Y(d)\}_{d \in D})\big].$$8 is correlated with but complementary to predicted probabilities; and the identified subsets reveal training points leading to group attribution bias (Yang et al., 2023). This establishes counterfactual relabeling as a mechanism for contestability and data auditing, not merely augmentation.

6. Identifiability, realizability, and the status of counterfactual labels

The major theoretical question is whether a counterfactual relabel is identifiable from available data. In statistical decision theory with counterfactual loss, the answer is sharply characterized. Under strong ignorability, the difference in counterfactual risk between any two policies is identifiable if and only if the loss is additive in the potential outcomes: $$R(\pi; \ell) = \mathbb{E}\big[\ell(\pi(X); \{Y(d)\}_{d \in D})\big].$$9 Absolute risk is exactly identifiable if and only if $s = (x, y_M, y_T, c)$0. The paper further shows that with two treatment options, every additive counterfactual loss is equivalent, up to a constant, to a standard factual-only loss, whereas with three or more treatments additive counterfactual losses can yield different optimal decisions, including policies that penalize overtreatment (Koch et al., 13 May 2025).

A second question is whether the desired counterfactual quantity can be physically sampled at all. The theory of counterfactual realizability introduces the Fundamental Constraint of Experimentation, under which each mechanism can be enacted only once per unit, and formalizes counterfactual randomization as a physical action that fixes the value of a variable as input to mechanisms generating a subset of its children without erasing the natural value of that variable. The CTF-REALIZE algorithm determines whether a counterfactual distribution $s = (x, y_M, y_T, c)$1 is realizable given a causal diagram and available actions (Raghavan et al., 14 Mar 2025).

The correctness result states that an $s = (x, y_M, y_T, c)$2-distribution is realizable if and only if CTF-REALIZE returns a sample. The graphical criterion says that $s = (x, y_M, y_T, c)$3 is realizable iff the ancestor set $s = (x, y_M, y_T, c)$4 does not contain a pair of the same variable under different regimes. Thus $s = (x, y_M, y_T, c)$5 is never realizable, while $s = (x, y_M, y_T, c)$6 or $s = (x, y_M, y_T, c)$7 can be realizable (Raghavan et al., 14 Mar 2025). This directly limits which counterfactual labels can be collected rather than merely inferred.

The same tension appears in high-dimensional generative settings. Counterfactual image editing formalizes image counterfactuals using augmented structural causal models (ASCMs) but proves two impossibility results: counterfactual editing is impossible from i.i.d. image samples and their corresponding labels alone, and even when the causal relationships between latent generative factors and images are available, no guarantees regarding the output of the model can be provided. The proposed relaxation is a family of counterfactual-consistent estimators defined relative to a care set of features, together with an efficient neural-causal algorithm for generation (Pan et al., 2024). For relabeling, the implication is that exact counterfactual labels may be unattainable even when observational fit and graph structure are correct.

Canonical representations of Markovian structural causal models push this point further by separating observational and interventional constraints from counterfactual conceptions. A counterfactual model is written as

$s = (x, y_M, y_T, c)$8

where $s = (x, y_M, y_T, c)$9 fixes the causal graphical model and $x$0 is a collection of one-step-ahead counterfactual process measures. The normalization procedure transports a standard normal process through monotone maps $x$1 to produce outcome-space process measures,

$x$2

This permits comonotonic, countermonotonic, deterministic, or stochastic cross-world couplings without altering observational or interventional constraints (Lara, 22 Jul 2025).

The resulting perspective is that many counterfactual labels are not “estimated” in the same sense as ordinary supervised targets. Some are identified from assumptions such as strong ignorability and additivity; some are physically realizable through specialized experimental procedures; some are non-identifiable and must be approximated within bounds; and some depend on an explicit choice of counterfactual conception. Counterfactual relabeling is therefore both an algorithmic technique and a statement about which contrary-to-fact annotations one is willing, or able, to treat as meaningful.