Counterfactual Samples Synthesizing (CCSS)

• CCSS is a family of methods that construct hypothetical 'what-if' samples while preserving essential observed instance properties.
• It is applied across domains such as visual question answering, language understanding, causal estimation, and tissue imaging for explanation and robustness.
• CCSS methods are categorized into direct and support-oriented approaches, emphasizing outcome generation or auxiliary calibration for improved model performance.

Counterfactual Samples Synthesizing (CCSS) denotes a family of methods that construct hypothetical samples answering a “what-if” query while preserving, to the extent required by the application, salient properties of an observed instance. In the literature, the name appears explicitly in Visual Question Answering as “Counterfactual Samples Synthesizing” or “Counterfactual Samples Synthesizing and Training,” but closely related mechanisms also appear under other names in language understanding, semantic image editing, causal generative modeling, tissue-image translation, program synthesis for recourse, and synthetic-label generation for conformal inference (Chen et al., 2020, Chen et al., 2021, Feng et al., 2021, Jacob et al., 2021, Wu et al., 2023, Zhu et al., 2024, Farzaneh et al., 4 Sep 2025). Across these settings, CCSS is used to generate target-class images, latent counterfactual text representations, masked multimodal training pairs, artificial tissue samples for opposite outcome groups, boundary-aware synthetic points, or synthetic counterfactual labels, depending on whether the goal is explanation, robustness, recourse, causal estimation, or uncertainty quantification (Paulikat et al., 2023, Luo et al., 2020, Zhao et al., 11 Dec 2025).

1. Scope and conceptual boundaries

The term is not universal. Some works use the exact label “Counterfactual Samples Synthesizing” for VQA debiasing, some use “Counterfactual Samples Synthesizing and Training,” and others explicitly state that they do not use the term CCSS while still containing a clear counterfactual sample synthesis mechanism, such as the generation module in the Counterfactual Reasoning Model for language understanding (Chen et al., 2020, Chen et al., 2021, Feng et al., 2021). In practice, the common denominator is not the name but the operation: start from a factual object, specify an alternative label, intervention, or decision, and synthesize a sample that is relevant to that alternative while remaining close to the factual instance in a task-appropriate sense.

The synthesized object varies substantially by domain. In VQA, the counterfactual is a modified image-question pair formed by masking critical objects or words and assigning pseudo ground-truth answers (Chen et al., 2020, Chen et al., 2021). In high-resolution vision explanation, STEEX fixes the semantic layout of an image and optimizes semantic latent codes so that the classifier flips while overall scene structure is preserved (Jacob et al., 2021). In language understanding, CRM generates one latent counterfactual representation per alternative class, without decoding back to text (Feng et al., 2021). In causal generative modeling for time-varying treatments, the target is a sample from a counterfactual outcome distribution under a user-specified treatment history (Wu et al., 2023). In tissue imaging, CF-HistoGAN generates an artificial paired sample that resembles the original tissue sample as closely as possible but captures the characteristics of a different patient outcome group (Paulikat et al., 2023).

A useful distinction is between methods that synthesize the counterfactual object itself and methods that synthesize auxiliary objects for downstream use. This suggests two helpful shorthands: “direct CCSS” (Editor’s term) for methods that directly generate target counterfactual outcomes or samples, and “support-oriented CCSS” (Editor’s term) for methods that generate synthetic counterfactual labels, rationales, support points, or calibration data for another procedure. Direct CCSS includes semantic image counterfactuals, time-varying-treatment outcome generation, diffusion-based high-dimensional counterfactual images, and cross-domain structural counterfactual generation (Jacob et al., 2021, Wu et al., 2023, Zhu et al., 2024, Kher et al., 17 Feb 2025). Support-oriented CCSS includes VQA masking-based training, commonsense counterfactual sentence construction, synthetic counterfactual labels for conformal inference, counterfactual rationales for preference optimization, and CTGAN-based augmentation for matched Difference-in-Differences (Chen et al., 2021, Liu et al., 2024, Farzaneh et al., 4 Sep 2025, Liang et al., 9 Jul 2025, Grassi et al., 27 Mar 2026).

Setting	Synthesized object	Primary role
VQA CSS/CSST	Masked image-question pair with pseudo answers	Robust training
STEEX	Semantic-layout-preserving image	Counterfactual explanation
CRM	Latent counterfactual text representation	Retrospection at test time
CF-HistoGAN	Artificial paired tissue sample	Exploratory outcome comparison
Time-varying treatment models	Counterfactual outcome sample under treatment history	Causal generation
SP-CCI / CheXPO	Synthetic counterfactual labels or rationales	Calibration or preference learning

2. Core formulations

A recurring formulation in CCSS is to search for a sample that changes the target model’s decision while staying close to the factual sample. In STEEX, the generic image-space counterfactual objective is written as

$$\argmin\nolimits_{x \in \mathcal{X}} \; L_{\text{decision}(M(x)), y} + \lambda L_{\text{dist}(x^I,x)},$$

and, with a generator prior $G:z\mapsto x$, becomes

$$\argmin\nolimits_{z \in \mathcal{Z}} \; L_{\text{decision}(M(G(z))), y} + \lambda L_{\text{dist}(z^I,z)}.$$

The paper’s semantic version further fixes the semantic mask $S^I$ and optimizes only semantic latent codes $z$, so the synthesized counterfactual is generated as $G(S^I,z)$ under a fixed scene layout (Jacob et al., 2021).

This structure reappears in other settings with different semantics. The classifier-reconstruction work recalls the standard counterfactual form

$$\min \ \mathcal{L}(f(x'), y') + \lambda \cdot d(x,x'),$$

and then argues that such counterfactuals should be treated as soft boundary samples rather than ordinary class examples, assigning them label $0.5$ in a structured label space $\mathcal{Y}=\{0,0.5,1\}$ (Zhao et al., 11 Dec 2025). The oversampling framework for imbalanced classification similarly defines a loss combining target-class fit and proximity to a majority-class factual sample, then selects the perturbation satisfying “minimum inversion,” so the new minority sample is the closest version of a majority sample that the fixed classifier labels as minority (Luo et al., 2020).

In causal generative models for time-varying treatment, the formal object is the counterfactual outcome distribution $f_a$ associated with a treatment history $G:z\mapsto x$0, and the objective is to match a learned conditional generator $G:z\mapsto x$1 to that distribution: $G:z\mapsto x$2 Because samples from $G:z\mapsto x$3 are not observed, the method replaces direct likelihood with an inverse-probability-weighted objective derived from the longitudinal g-formula (Wu et al., 2023).

Where direct optimization in observation space is unsafe or implausible, CCSS usually inserts a structural prior. STEEX relies on a semantic-to-image generator and a latent-distance penalty rather than inference-time realism losses (Jacob et al., 2021). DiffusionCounterfactuals combines a diffusion generator, a causal projector, and a neural structural causal model, modifying reverse diffusion with a guidance term based on the gradient of mismatch between desired intervened factors and factors predicted from the current noisy image (Zhu et al., 2024). Cross-domain structural counterfactual generation instead defines the target counterfactual through a joint causal graph and the expression

$G:z\mapsto x$4

so the target-domain counterfactual preserves source-domain effect-intrinsic latent content while resampling domain-intrinsic latent factors in the target domain (Kher et al., 17 Feb 2025).

3. Synthesis mechanisms and controllability

The main design question in CCSS is not only how to generate a counterfactual, but also which variables are allowed to change. STEEX answers this by decomposing the image into a fixed semantic layout $G:z\mapsto x$5 and semantic latent codes $G:z\mapsto x$6. Only the latent codes are optimized, and a user may further restrict optimization to a subset $G:z\mapsto x$7, producing “region-targeted counterfactual explanations.” In that setting, the question changes from “what small change would flip the decision?” to “how should these selected semantic regions change to flip the decision?” (Jacob et al., 2021). This is a particularly explicit form of controllability.

A different form of controllability appears in diffusion-based high-dimensional counterfactual generation. DiffusionCounterfactuals infers causal factors from the factual image, applies an intervention in factor space through a neural structural causal model, and then guides denoising toward images whose predicted factors match the intervened target. The paper also supports multiple intervention steps through iterative autoregressive application of the same procedure (Zhu et al., 2024). This makes the synthesis mechanism an encode–intervene–generate pipeline rather than direct local editing.

In latent language CCSS, CRM learns class-specific decomposition functions $G:z\mapsto x$8 that extract label-irrelevant content and synthesizes a target-class counterfactual representation by

$G:z\mapsto x$9

The counterfactual is therefore generated by class-conditioned geometric injection in hidden space, supervised by paired factual/counterfactual examples (Feng et al., 2021). The method is explicit that it is not sequence-to-sequence rewriting or lexical editing.

Graph CCSS uses yet another mechanism. CGC produces hard negative graphs by perturbing adjacency or masking node features while maximizing KL divergence between the original and generated graph-class distributions and minimizing perturbation magnitude. It therefore operationalizes a counterfactual negative as a graph that is similar to the original in structure or features but pushed toward a different semantic prediction (Yang et al., 2022). In imbalanced classification, the oversampling method perturbs majority samples featurewise under a truncated normal distribution and retains only perturbations that flip the classifier from majority to minority while minimizing MAD-normalized distance (Luo et al., 2020).

Program-synthesis-based recourse shifts from sample generation to sequential intervention generation. Rather than directly outputting a single counterfactual sample, the method learns a program that emits a sequence of actions $$\argmin\nolimits_{z \in \mathcal{Z}} \; L_{\text{decision}(M(G(z))), y} + \lambda L_{\text{dist}(z^I,z)}.$$0 so that the final state $$\argmin\nolimits_{z \in \mathcal{Z}} \; L_{\text{decision}(M(G(z))), y} + \lambda L_{\text{dist}(z^I,z)}.$$1 flips a black-box classifier. Search is guided by Monte Carlo Tree Search and a cost-aware score, while symbolic distillation turns successful traces into an automaton with decision-tree rules at each node (Toni et al., 2022). This is CCSS at the level of intervention policies rather than isolated samples.

The cross-domain structural model is distinctive because it disambiguates exogenous causes into effect-intrinsic variables $$\argmin\nolimits_{z \in \mathcal{Z}} \; L_{\text{decision}(M(G(z))), y} + \lambda L_{\text{dist}(z^I,z)}.$$2 and domain-intrinsic variables $$\argmin\nolimits_{z \in \mathcal{Z}} \; L_{\text{decision}(M(G(z))), y} + \lambda L_{\text{dist}(z^I,z)}.$$3. The factual source-domain posterior over $$\argmin\nolimits_{z \in \mathcal{Z}} \; L_{\text{decision}(M(G(z))), y} + \lambda L_{\text{dist}(z^I,z)}.$$4 is combined with the target-domain prior over $$\argmin\nolimits_{z \in \mathcal{Z}} \; L_{\text{decision}(M(G(z))), y} + \lambda L_{\text{dist}(z^I,z)}.$$5, so the synthesized target sample is intended to preserve cross-domain identity/content while changing domain-specific realization (Kher et al., 17 Feb 2025). This suggests that controllability in CCSS is often achieved by deciding which latent factors are preserved, which are intervened on, and which are resampled.

4. Training-time uses and downstream integration

A large part of CCSS is training-time rather than inference-time. In VQA, CSS generates counterfactual samples by masking critical image objects or question words and assigning pseudo ground-truth answers; CSST adds a second stage in which the model is trained on both original and synthesized samples and is further urged to distinguish original samples from superficially similar counterfactual ones through supervised contrastive losses (Chen et al., 2020, Chen et al., 2021). The point is not merely augmentation but forcing the model to focus on all critical evidence.

A similar logic appears in commonsense statement plausibility estimation. CCSG identifies critical words using gradient-based contribution scores, constructs negative counterfactuals by replacing those words with high-cosine-similarity alternatives from a vector knowledge base, and constructs positive counterfactuals through low-level dropout. The final model is trained with binary classification loss plus sentence-level supervised contrastive loss (Liu et al., 2024). In both VQA and commonsense settings, the synthetic counterfactuals are designed to expose whether the model is right for the right reasons.

Other work uses CCSS to improve downstream estimation rather than classification robustness. In SP-CCI, synthetic counterfactual labels $$\argmin\nolimits_{z \in \mathcal{Z}} \; L_{\text{decision}(M(G(z))), y} + \lambda L_{\text{dist}(z^I,z)}.$$6 are generated for control-arm covariates $$\argmin\nolimits_{z \in \mathcal{Z}} \; L_{\text{decision}(M(G(z))), y} + \lambda L_{\text{dist}(z^I,z)}.$$7, forming a synthetic calibration set

$$\argmin\nolimits_{z \in \mathcal{Z}} \; L_{\text{decision}(M(G(z))), y} + \lambda L_{\text{dist}(z^I,z)}.$$8

These pseudo-labels are not trusted directly; instead, they are inserted into a debiased risk estimator inspired by prediction-powered inference and then used within risk-controlling prediction sets to choose interval widening while preserving marginal coverage in a high-probability sense (Farzaneh et al., 4 Sep 2025). The synthetic object is therefore a calibration label, not the final output.

CheXPO uses counterfactual rationale as a rejected response in preference learning for chest X-ray VLMs. A correct but low-confidence answer is turned into a clinically coherent but intentionally incorrect alternative by replacing the answer along dimensions such as anatomy, abnormality, or severity and retrieving the most similar rationale from the training set. These chosen/rejected pairs are then used in DPO-style optimization (Liang et al., 9 Jul 2025). This is a text-level, preference-oriented form of CCSS.

Classifier reconstruction offers another downstream use. The method assumes access to one-sided counterfactuals for negatively predicted samples and argues that these counterfactuals are informative because they lie close to the decision boundary but are not fully representative of either class distribution. Instead of naive augmentation, it integrates them through Wasserstein barycenter prototypes $$\argmin\nolimits_{z \in \mathcal{Z}} \; L_{\text{decision}(M(G(z))), y} + \lambda L_{\text{dist}(z^I,z)}.$$9, with an additional symmetry regularizer to keep the counterfactual distribution geometrically central (Zhao et al., 11 Dec 2025).

In small-$S^I$0 causal panels, CTGAN-based augmentation does not synthesize each unit’s missing potential outcome directly. Instead, it generates treatment-arm-specific longitudinal trajectories conditioned on empirical baseline skeletons, producing datasets of the form $S^I$1. These synthetic samples are then used to improve overlap, enable stricter matching, and stabilize event-study Difference-in-Differences estimation (Grassi et al., 27 Mar 2026). This is support-oriented CCSS in the sense that the synthetic samples improve the comparison set rather than instantiate identified unit-level counterfactuals.

5. Empirical criteria and reported findings

The empirical evaluation of CCSS is highly task-specific, but several recurrent criteria appear: validity of the counterfactual condition, proximity or minimality, realism or plausibility, preservation of non-target content, and usefulness for downstream analysis. STEEX evaluates validity by success rate, realism by Fréchet Inception Distance, identity/content preservation by Face Verification Accuracy, and semantic minimality by Mean Number of Attributes Changed. With ADAM, learning rate $S^I$2, 100 steps, and $S^I$3, it reports above $S^I$4 success rate for all five classifiers, with markedly better FID and FVA than Progressive Exaggeration and DiVE on CelebA, CelebAMask-HQ, and BDD100k (Jacob et al., 2021). The ablation $S^I$5 substantially worsens FID and FVA, showing that the latent-distance term is the main mechanism preserving realism and original content.

For sample-level causal generation under time-varying treatment, evaluation focuses on how closely the generated distribution matches the true or benchmark counterfactual distribution. The time-varying-treatment paper uses mean difference and 1-Wasserstein distance on fully synthetic data and FID$S^I$6 on high-dimensional semi-synthetic data, reporting that the inverse-probability-weighted CVAE and diffusion variants outperform unweighted generative baselines and plugin density-estimation baselines (Wu et al., 2023). DiffusionCounterfactuals evaluates realism with FID and sFID, fidelity with PSNR, and intervention faithfulness with ACM, where ACM measures the discrepancy between desired generative-factor values and those predicted from the generated counterfactual. The reported results show generally best or near-best FID/sFID, much better PSNR on synthetic datasets, and clearly best ACM across six image datasets (Zhu et al., 2024).

In tissue imaging, the emphasis shifts from perceptual realism to biological interpretability and statistical sensitivity. CF-HistoGAN analyzes mean channel variation and absolute channel variation of difference maps and compares unpaired Student’s $S^I$7-tests with artificially paired paired $S^I$8-tests constructed through synthetic counterfactuals. For several CRC and CTCL markers, paired tests yield much smaller $S^I$9-values than unpaired tests; for example, in CRC, Vimentin changes from unpaired $z$0 to paired $z$1 in the CLR $z$2 DII direction (Paulikat et al., 2023). The paper interprets this as greater sensitivity for detecting group-associated protein differences.

VQA and commonsense work evaluate not only accuracy but properties attributed to the synthesized counterfactuals. The original CSS paper reports that LMH rises from $z$3 to $z$4 on VQA-CP v2, and the later CSST paper reports $z$5 for LMH-CSST and $z$6 for LMH-CSST+SAR on the LXMERT backbone, together with gains in Average Importance, Confidence Improvement, and rephrasing robustness (Chen et al., 2020, Chen et al., 2021). Commonsense CCSG reports $z$7 average accuracy across nine plausibility-estimation datasets for CCSG+T5(5B), versus $z$8 for VERA+T5(5B), and an ablation from $z$9 without CCSG to $G(S^I,z)$0 with CCSG (Liu et al., 2024).

Support-oriented CCSS also shows measurable efficiency gains. The classifier-reconstruction paper evaluates fidelity between target and surrogate predictions and reports, for example, Adult Income fidelity of $G(S^I,z)$1 versus $G(S^I,z)$2 and $G(S^I,z)$3 at 500 queries for the proposed method, Baseline 2, and Baseline 1 respectively (Zhao et al., 11 Dec 2025). SP-CCI reports the same target coverage with narrower intervals than standard CCI; on IHDP, APIW decreases from $G(S^I,z)$4 for CCI to $G(S^I,z)$5, $G(S^I,z)$6, and $G(S^I,z)$7 for low-, medium-, and high-quality synthetic labels (Farzaneh et al., 4 Sep 2025). CheXPO reports $G(S^I,z)$8 for CheX-Phi3.5V(30k) against $G(S^I,z)$9 for the SFT baseline, with an ablation showing DPO (Ours) at $$\min \ \mathcal{L}(f(x'), y') + \lambda \cdot d(x,x'),$$0 versus $$\min \ \mathcal{L}(f(x'), y') + \lambda \cdot d(x,x'),$$1 for DPO without counterfactual rationale (Liang et al., 9 Jul 2025).

6. Limitations, ambiguities, and open directions

Several limitations recur across the literature. First, many methods depend on a strong representation or generator assumption. STEEX requires that the image can be decomposed into a semantic mask and semantic style codes; preserving layout also means it cannot naturally handle object insertion, deletion, movement, or geometry changes, and the paper explicitly notes future work on shifting, removing, or adding objects (Jacob et al., 2021). DiffusionCounterfactuals assumes access to ground-truth generative factors during training and does not provide a formal identifiability theorem for the overall pipeline (Zhu et al., 2024). Cross-domain structural counterfactual generation assumes known causal graphs in both domains, shared effect-intrinsic variables, and Markovianity, while empirical validation is limited to a synthetic dataset (Kher et al., 17 Feb 2025).

Second, several methods generate counterfactuals only in latent or auxiliary space. CRM does not decode latent counterfactual representations back to natural language, so direct human inspection is unavailable, and it requires paired factual/counterfactual supervision for training the generator (Feng et al., 2021). CheXPO synthesizes counterfactual rationales at the response level rather than counterfactual images, and the faithfulness of rationales remains difficult to evaluate (Liang et al., 9 Jul 2025). SP-CCI synthesizes labels for calibration and preserves validity only through debiasing and RCPS-style high-probability bounds; synthetic label quality still materially affects interval width (Farzaneh et al., 4 Sep 2025).

Third, many CCSS systems are expensive or operationally constrained. STEEX performs iterative latent optimization for each query (Jacob et al., 2021). Program-synthesis recourse requires a causal graph, a handcrafted action DSL, and repeated black-box queries during training; the distilled program removes inference-time queries, but the policy may still sacrifice local optimality for amortized efficiency (Toni et al., 2022). The Wasserstein-prototype reconstruction method does not study scalability to very large datasets or high-dimensional spaces, and its performance depends strongly on the quality of externally generated counterfactuals (Zhao et al., 11 Dec 2025).

Fourth, causal validity is often weaker than counterfactual plausibility. CF-HistoGAN explicitly states that its synthesized tissue samples are not guaranteed to be causally valid counterfactuals and should be understood as plausible outcome-conditional counterfactuals within the observed data manifold (Paulikat et al., 2023). The CTGAN-based palliative-care study likewise states that synthetic data “cannot create new information ex nihilo,” does not directly synthesize true missing potential outcomes, and may reproduce or amplify bias already present in the data (Grassi et al., 27 Mar 2026). The time-varying-treatment generator depends on sequential ignorability and positivity, both of which become difficult as treatment-history length grows (Wu et al., 2023).

These limitations suggest a common research agenda. One direction is stronger structure: better disentanglement of invariant and intervention-sensitive factors, finer causal constraints, and explicit modeling of geometry or object-level interventions. Another is tighter integration between generation quality and downstream validity, as already seen in RCPS/PPI-style calibration and prototype-based integration of soft boundary samples (Farzaneh et al., 4 Sep 2025, Zhao et al., 11 Dec 2025). A third is broader modality coverage with controllability comparable to region-targeted STEEX or skeleton-conditioned panel synthesis, but without requiring privileged supervision such as semantic masks, labeled generative factors, or carefully handcrafted rejection pools (Jacob et al., 2021, Grassi et al., 27 Mar 2026, Liang et al., 9 Jul 2025).

Taken together, the literature portrays CCSS not as a single algorithmic template but as a design pattern for constructing hypothetical samples under constraints of validity, proximity, plausibility, and task usefulness. Some methods search in semantic latent space, some generate full causal outcomes, some create auxiliary pseudo-labels or rationales, and some synthesize intervention policies instead of points. What unifies them is the use of synthetic counterfactual objects to expose decision mechanisms, regularize models, recover missing trajectories, densify boundary information, or improve uncertainty quantification under missing counterfactual data (Jacob et al., 2021, Wu et al., 2023, Zhu et al., 2024, Zhao et al., 11 Dec 2025, Farzaneh et al., 4 Sep 2025).