CounterFactual Policy Optimization

• CFPO is a framework that evaluates alternative actions by simulating counterfactual outcomes to guide policy updates.
• It integrates offline and sequential methods, using estimators like IPW, DR, and density ratios to correct for support mismatch and distribution shifts.
• Recent variants incorporate safety constraints, inference-aware adjustments, and credit assignments to optimize robust policy performance in complex environments.

Searching arXiv for papers directly using the term “CounterFactual Policy Optimization (CFPO)” and closely related variants. CounterFactual Policy Optimization (CFPO) denotes a class of policy-learning approaches in which policy parameters are selected or updated using counterfactual quantities rather than only observed factual rewards. In the broadest research sense, the term covers methods that optimize a target policy from logged data, simulator-based interventions, latent counterfactual worlds, or counterfactually perturbed internal states; in a narrower and explicit sense, it also names a recent reinforcement-learning framework for large vision-LLMs that regularizes policy updates by comparing factual predictions with predictions under a counterfactual state where critical visual cues are suppressed (Yu et al., 22 Jun 2026). Across these usages, the common idea is that policy improvement should depend on “what would have happened under an alternative action, policy, trajectory, or representation,” although the underlying formalism ranges from contextual bandits and off-policy evaluation to continuing RL, simulator-based interventions, structural causal models, and multimodal policy regularization (Zhang et al., 2019).

1. Conceptual scope and historical placement

CFPO is not a single uniform formalism. A plausible interpretation is that the literature uses the phrase to denote several closely related families of methods: offline contextual policy optimization based on counterfactual estimators, off-policy policy-gradient optimization of a counterfactual objective in continuing RL, simulator-based counterfactual rollout evaluation in multi-agent systems, and explicit latent-state counterfactual regularization in multimodal reasoning (Guha et al., 9 Jan 2025). Some papers are directly about policy optimization, while others are better read as counterfactual policy evaluation or execution-gating methods that supply reusable components for policy optimization.

In contextual or one-shot decision problems, CFPO is closest to counterfactual risk minimization and offline policy selection from logs. In this setting, logged data consist of triples $\mathcal D=\{(x_i,a_i,r_i)\}_{i=1}^n$, and the target is the value of an evaluation policy $\pi_e$. The dynamic-auction study formalizes the policy as a context-to-action mapping,

$\pi:\mathcal C \rightarrow \mathcal A,$

and then uses off-policy estimators to compare candidate policies and to optimize a new policy against an estimated counterfactual objective (Guha et al., 9 Jan 2025). The continuous-action CRM literature makes the same move in a more explicit optimization form, defining expected risk

$L(\pi) = \mathbb E_{(x,a,y)\sim P_\pi}[y]$

and solving

$$\hat \pi \in \arg\min_{\pi\in\Pi} \left\{ \hat L(\pi) + \Omega(\pi)\right\}$$

from logged continuous-action bandit data (Zenati et al., 2020).

In continuing RL, the term becomes tied to optimization under a deployment-relevant state distribution rather than the behavior-policy distribution. “Generalized Off-Policy Actor-Critic” introduces a counterfactual objective that interpolates between the excursion objective and the alternative-life objective, and then derives the policy gradient needed to optimize it (Zhang et al., 2019). In simulator-based or SCM-based work, the counterfactual object is a modified world or trajectory. “Woulda, Coulda, Shoulda: Counterfactually-Guided Policy Search” evaluates and improves policies by inferring latent exogenous variables from logged episodes and then rolling out alternative actions under the same inferred scenario (Buesing et al., 2018). In multimodal reasoning, “CFPO: Counterfactual Policy Optimization for Multimodal Reasoning” uses internal latent interventions rather than environment-level rollouts, constructing a counterfactual state in which high-saliency visual cues are suppressed and then maximizing the discrepancy between factual and counterfactual predictions (Yu et al., 22 Jun 2026).

2. Core estimators and optimization primitives

A recurring CFPO pattern is to optimize a policy against an estimated counterfactual value functional. In discrete-action off-policy evaluation, the standard estimators are Inverse Probability Weighting (IPW), Self-Normalized IPW (SNIPW), Direct Method (DM), Doubly Robust (DR), and Self-Normalized Doubly Robust (SNDR) (Guha et al., 9 Jan 2025). For IPW, the policy-value estimator is

$\hat{V}_{\mathrm{IPW} (\pi_e; \mathcal{D}) := \mathbb{E}_{n} [ w(x_i,a_i) \cdot r_i]$

with importance weight

$w(x,a):=\pi_e (a|x)/\pi_b (a|x),$

and clipped weight

$\hat{w}(x,a) := \min \{ \lambda, w(x,a) \}.$

The same study states that when the behavior policy is known, IPW is unbiased and consistent for the true policy value, but it has high variance when the evaluation policy differs substantially from the behavior policy (Guha et al., 9 Jan 2025).

Self-normalization yields

$\hat{V}_{\mathrm{SNIPW} (\pi_e; \mathcal{D}) := \frac{\mathbb{E}_{n} [w(x_i,a_i) \cdot r_i]}{ \mathbb{E}_{n} [w(x_i,a_i)]},$

which is not unbiased even when the behavior policy is known, but is still consistent and more stable (Guha et al., 9 Jan 2025). The model-based alternative is the Direct Method, using

$q(x,a)=\mathbb{E}[r|x,a]$

and

$\pi_e$0

DR combines both components: $\pi_e$1 and the paper states that consistency holds if either the importance weights are correct or the reward model is correct (Guha et al., 9 Jan 2025).

Continuous-action CFPO requires density ratios or continuous-treatment estimators rather than exact action matching. The dynamic-auction study uses the kernel estimator

$\pi_e$2

where $\pi_e$3, and emphasizes the bias-variance role of bandwidth $\pi_e$4 (Guha et al., 9 Jan 2025). The continuous-action CRM work instead uses density-ratio estimators such as

$\pi_e$5

self-normalized variants,

$\pi_e$6

and a variance-regularized soft-clipped objective

$\pi_e$7

for learning stochastic continuous policies from logged bandit data (Zenati et al., 2020).

The optimization step that turns evaluation into CFPO appears explicitly in the dynamic-auction paper’s “New Policy Learning” section. There the procedure is: learn continuous OPE models for outcomes such as cost and returns, parameterize a policy $\pi_e$8 via an MLP, and optimize the MLP by gradient descent through a loss function defined on OPE-estimated outcomes. The concrete objective is

$\pi_e$9

This is an explicit offline counterfactual policy optimization procedure over continuous payment actions (Guha et al., 9 Jan 2025).

3. Sequential RL formulations of the counterfactual objective

In continuing RL, CFPO is often about correcting the mismatch between the behavior-policy state distribution and the distribution induced by the target policy when deployed. “Generalized Off-Policy Actor-Critic” defines the excursion objective

$\pi:\mathcal C \rightarrow \mathcal A,$0

and the alternative-life objective

$\pi:\mathcal C \rightarrow \mathcal A,$1

then introduces a counterfactual objective $\pi:\mathcal C \rightarrow \mathcal A,$2 that interpolates between them through an interpolated stationary distribution $\pi:\mathcal C \rightarrow \mathcal A,$3 (Zhang et al., 2019). At $\pi:\mathcal C \rightarrow \mathcal A,$4, the objective reduces to $\pi:\mathcal C \rightarrow \mathcal A,$5; at $\pi:\mathcal C \rightarrow \mathcal A,$6, it becomes $\pi:\mathcal C \rightarrow \mathcal A,$7. The paper’s point is that optimizing $\pi:\mathcal C \rightarrow \mathcal A,$8 can be misleading about deployment performance because it evaluates $\pi:\mathcal C \rightarrow \mathcal A,$9 under $L(\pi) = \mathbb E_{(x,a,y)\sim P_\pi}[y]$0, not under the states the target policy would actually induce (Zhang et al., 2019).

This requires a density ratio

$L(\pi) = \mathbb E_{(x,a,y)\sim P_\pi}[y]$1

and a gradient theorem that explicitly differentiates the stationary-distribution term. The Generalized Off-Policy Policy Gradient Theorem adds a correction term beyond standard off-policy policy-gradient formulas because the objective depends on $L(\pi) = \mathbb E_{(x,a,y)\sim P_\pi}[y]$2, and $L(\pi) = \mathbb E_{(x,a,y)\sim P_\pi}[y]$3 depends on $L(\pi) = \mathbb E_{(x,a,y)\sim P_\pi}[y]$4 (Zhang et al., 2019). The resulting Geoff-PAC algorithm combines an actor, a critic, a density-ratio estimator $L(\pi) = \mathbb E_{(x,a,y)\sim P_\pi}[y]$5, and emphatic traces. The paper presents this as a direct optimization method for a counterfactual objective in continuing RL, making it one of the clearest theoretical antecedents of CFPO in the policy-gradient setting (Zhang et al., 2019).

A distinct sequential formulation appears in “Woulda, Coulda, Shoulda: Counterfactually-Guided Policy Search,” which treats a POMDP as an SCM with exogenous scenario variables and performs policy search from off-policy experience (Buesing et al., 2018). The key subroutine is counterfactual inference: given observed data $L(\pi) = \mathbb E_{(x,a,y)\sim P_\pi}[y]$6, SCM $L(\pi) = \mathbb E_{(x,a,y)\sim P_\pi}[y]$7, intervention $L(\pi) = \mathbb E_{(x,a,y)\sim P_\pi}[y]$8, and query variable $L(\pi) = \mathbb E_{(x,a,y)\sim P_\pi}[y]$9, sample $$\hat \pi \in \arg\min_{\pi\in\Pi} \left\{ \hat L(\pi) + \Omega(\pi)\right\}$$0, replace the prior with a point mass at $$\hat \pi \in \arg\min_{\pi\in\Pi} \left\{ \hat L(\pi) + \Omega(\pi)\right\}$$1, intervene on the policy mechanism, and simulate the counterfactual query. The paper proves the identity

$$\hat \pi \in \arg\min_{\pi\in\Pi} \left\{ \hat L(\pi) + \Omega(\pi)\right\}$$2

which justifies counterfactual rollout as an unbiased estimator under a correct SCM (Buesing et al., 2018).

CF-GPS then optimizes a policy using return-weighted regression on these posterior-grounded counterfactual trajectories: $$\hat \pi \in \arg\min_{\pi\in\Pi} \left\{ \hat L(\pi) + \Omega(\pi)\right\}$$3 Its distinguishing feature is that the trajectories are generated by counterfactual inference conditioned on logged episodes rather than from the model prior, which reduces bias when the latent scenario prior is hard to model accurately (Buesing et al., 2018).

4. Safety, support, and inference-aware variants

A major theme in CFPO is that counterfactual objectives are useful only to the extent that they remain identifiable, stable, and aligned with the downstream decision rule. Support mismatch is one central difficulty. The dynamic-auction paper notes that logged data come from prior A/B tests among a few deployed policies rather than broad randomized exploration, so candidate policies may place action mass in poorly represented regions of the logs (Guha et al., 9 Jan 2025). The continuous-action CRM paper makes the same point more formally through the density-ratio objective, effective sample size diagnostics, and model-selection protocol based on rejecting low-ESS evaluations (Zenati et al., 2020).

The deterministic-logging setting sharpens this problem. “Counterfactual Learning with General Data-generating Policies” introduces the Approximate Propensity Score

$$\hat \pi \in \arg\min_{\pi\in\Pi} \left\{ \hat L(\pi) + \Omega(\pi)\right\}$$4

which relaxes global positivity by using weak local support near decision boundaries (Narita et al., 2022). Under Assumption 2, the paper writes the policy value as

$$\hat \pi \in \arg\min_{\pi\in\Pi} \left\{ \hat L(\pi) + \Omega(\pi)\right\}$$5

and estimates the pairwise constants $$\hat \pi \in \arg\min_{\pi\in\Pi} \left\{ \hat L(\pi) + \Omega(\pi)\right\}$$6 from locally overlapping regions (Narita et al., 2022). The method is useful for deficient-support or deterministic logging, but the paper also makes clear that without the constant conditional mean differences assumption, the estimator identifies only a boundary-local effect rather than the global value of arbitrary policies (Narita et al., 2022). For CFPO, this means actions never taken are not globally optimizable from logs alone.

A different limitation arises when the objective itself exploits prediction error rather than true policy improvement. “Beating the Winner’s Curse via Inference-Aware Policy Optimization” studies the standard plug-in approach in which one predicts counterfactual outcomes and then maximizes estimated policy value, and argues that this creates a winner’s curse because the selected policy can exploit prediction error and then fail downstream evaluation (Bastani et al., 20 Oct 2025). The paper defines

$$\hat \pi \in \arg\min_{\pi\in\Pi} \left\{ \hat L(\pi) + \Omega(\pi)\right\}$$7

and the IPW evaluation statistic

$$\hat \pi \in \arg\min_{\pi\in\Pi} \left\{ \hat L(\pi) + \Omega(\pi)\right\}$$8

then optimizes not only expected improvement but also the expected $$\hat \pi \in \arg\min_{\pi\in\Pi} \left\{ \hat L(\pi) + \Omega(\pi)\right\}$$9-score $\hat{V}_{\mathrm{IPW} (\pi_e; \mathcal{D}) := \mathbb{E}_{n} [ w(x_i,a_i) \cdot r_i]$0, where $\hat{V}_{\mathrm{IPW} (\pi_e; \mathcal{D}) := \mathbb{E}_{n} [ w(x_i,a_i) \cdot r_i]$1 is the standard deviation of the downstream IPW estimator (Bastani et al., 20 Oct 2025). Its policy family $\hat{V}_{\mathrm{IPW} (\pi_e; \mathcal{D}) := \mathbb{E}_{n} [ w(x_i,a_i) \cdot r_i]$2 traces a Pareto frontier between expected value and inferential success, making it an inference-aware CFPO variant rather than a pure value-maximizer.

Safety-constrained variants also appear in ranking. “Proximal Ranking Policy Optimization for Practical Safety in Counterfactual Learning to Rank” is a ranking-specific form of counterfactual policy optimization in which policy updates are clipped relative to a safe logging policy through the ratio $\hat{V}_{\mathrm{IPW} (\pi_e; \mathcal{D}) := \mathbb{E}_{n} [ w(x_i,a_i) \cdot r_i]$3 (Gupta et al., 2024). The estimator

$\hat{V}_{\mathrm{IPW} (\pi_e; \mathcal{D}) := \mathbb{E}_{n} [ w(x_i,a_i) \cdot r_i]$4

imposes a bounded deviation from the logging model without relying on user-behavior assumptions, which the paper describes as “unconditional safety in deployment” (Gupta et al., 2024). Although ranking-specific, the underlying principle is recognizably CFPO: optimize a counterfactual objective while structurally limiting the optimizer’s incentive to exploit unreliable regions.

5. Counterfactual worlds, diagnostics, and interpretability

Not all CFPO-relevant work performs direct policy optimization. Some papers instead define counterfactual worlds or trajectories and then use them for execution gating, robust selection, or explanation. “Counterfactual Policy Evaluation for Decision-Making in Autonomous Driving” constructs a family of counterfactual worlds $\hat{V}_{\mathrm{IPW} (\pi_e; \mathcal{D}) := \mathbb{E}_{n} [ w(x_i,a_i) \cdot r_i]$5 by replacing the policy of a nearby non-ego vehicle with an independent alternative from a predefined pool $\hat{V}_{\mathrm{IPW} (\pi_e; \mathcal{D}) := \mathbb{E}_{n} [ w(x_i,a_i) \cdot r_i]$6, simulates each world for horizon $\hat{V}_{\mathrm{IPW} (\pi_e; \mathcal{D}) := \mathbb{E}_{n} [ w(x_i,a_i) \cdot r_i]$7, and computes empirical collision probabilities (Hart et al., 2020). For a replaced vehicle $\hat{V}_{\mathrm{IPW} (\pi_e; \mathcal{D}) := \mathbb{E}_{n} [ w(x_i,a_i) \cdot r_i]$8, the paper defines

$\hat{V}_{\mathrm{IPW} (\pi_e; \mathcal{D}) := \mathbb{E}_{n} [ w(x_i,a_i) \cdot r_i]$9

and then averages collision risk across $w(x,a):=\pi_e (a|x)/\pi_b (a|x),$0 nearby vehicles: $w(x,a):=\pi_e (a|x)/\pi_b (a|x),$1 The deployment rule is to execute the learned ego policy only if $w(x,a):=\pi_e (a|x)/\pi_b (a|x),$2, otherwise hand control to a conventional lane-following fallback (Hart et al., 2020). This is not policy optimization in the paper itself, but it provides an explicit counterfactual risk functional that a CFPO method could turn into a constraint or penalty.

A related interpretability-oriented use of counterfactuals appears in “Counterfactual Explanation Policies in RL,” which frames counterfactual explanation as an optimization over nearby policies (Deshmukh et al., 2023). The paper defines

$w(x,a):=\pi_e (a|x)/\pi_b (a|x),$3

or in parameterized form

$w(x,a):=\pi_e (a|x)/\pi_b (a|x),$4

When $w(x,a):=\pi_e (a|x)/\pi_b (a|x),$5 and $w(x,a):=\pi_e (a|x)/\pi_b (a|x),$6 loss is used, this reduces to the trust-region penalty form

$w(x,a):=\pi_e (a|x)/\pi_b (a|x),$7

which the paper presents as an equivalence to trust-region policy optimization in that special case (Deshmukh et al., 2023). In encyclopedia terms, this places explanation-driven nearby-policy editing inside the CFPO family: the method optimizes a counterfactual policy for a desired target outcome while remaining close to a reference policy.

Model-based worst-case trajectory generation provides another interpretability route. “Constraint-Generation Policy Optimization (CGPO)” optimizes structured policies in mixed discrete-continuous MDPs by repeatedly finding the most violated trajectory-level constraint,

$w(x,a):=\pi_e (a|x)/\pi_b (a|x),$8

and then adding it to the master policy optimization problem (Gimelfarb et al., 2024). The paper explicitly states that these worst-case trajectories “provide counterfactual explanations of optimal actions.” This is not off-policy counterfactual estimation, but it is counterfactual-by-construction policy optimization: the policy is optimized against alternative action sequences and disturbance realizations that expose its current deficiencies (Gimelfarb et al., 2024).

6. Explicit recent uses of the acronym CFPO

A recent and explicit use of the acronym appears in “CFPO: Counterfactual Policy Optimization for Multimodal Reasoning,” which treats causal grounding failure in large vision-LLMs as a policy-optimization problem (Yu et al., 22 Jun 2026). The policy is written as

$w(x,a):=\pi_e (a|x)/\pi_b (a|x),$9

where $\hat{w}(x,a) := \min \{ \lambda, w(x,a) \}.$0 is the decoder attention output. The method decomposes the attention matrix into low- and high-saliency parts and constructs a counterfactual state in which the high-saliency visual cues are intervened on: $\hat{w}(x,a) := \min \{ \lambda, w(x,a) \}.$1 The corresponding counterfactual policy is

$\hat{w}(x,a) := \min \{ \lambda, w(x,a) \}.$2

and the counterfactual discrepancy is

$\hat{w}(x,a) := \min \{ \lambda, w(x,a) \}.$3

The paper then defines a counterfactual KL-style term

$\hat{w}(x,a) := \min \{ \lambda, w(x,a) \}.$4

and adds it to a GRPO- or DAPO-style objective: $\hat{w}(x,a) := \min \{ \lambda, w(x,a) \}.$5 Its stated effect is to regularize the policy so that suppressing critical visual cues materially changes the output distribution, thereby enforcing causal consistency between visual perception and textual reasoning (Yu et al., 22 Jun 2026).

A second explicit acronym variant is “Group Causal Counterfactual Policy Optimization” for LLM reasoning, written as GC$\hat{w}(x,a) := \min \{ \lambda, w(x,a) \}.$6PO (Wang et al., 6 Feb 2026). The method interprets multiple reasoning trajectories for the same question as a family of counterfactual experiments, decomposes each trajectory into episodes $\hat{w}(x,a) := \min \{ \lambda, w(x,a) \}.$7, and defines an episodic causal counterfactual reward

$\hat{w}(x,a) := \min \{ \lambda, w(x,a) \}.$8

where

$\hat{w}(x,a) := \min \{ \lambda, w(x,a) \}.$9

encodes robustness and

$\hat{V}_{\mathrm{SNIPW} (\pi_e; \mathcal{D}) := \frac{\mathbb{E}_{n} [w(x_i,a_i) \cdot r_i]}{ \mathbb{E}_{n} [w(x_i,a_i)]},$0

encodes effectiveness (Wang et al., 6 Feb 2026). These episode-level rewards are converted into token-level advantages and optimized with a GRPO-style clipped objective. This explicit recent usage narrows CFPO from a general counterfactual optimization idea to a concrete family of group-based causal reward-shaping methods for reasoning models.

Another recent specialization is “Counterfactual Credit Policy Optimization” in collaborative multi-agent LLM training (Li et al., 23 Mar 2026). There the main object is an agent-specific marginal contribution

$\hat{V}_{\mathrm{SNIPW} (\pi_e; \mathcal{D}) := \frac{\mathbb{E}_{n} [w(x_i,a_i) \cdot r_i]}{ \mathbb{E}_{n} [w(x_i,a_i)]},$1

obtained by constructing a counterfactual trajectory with agent $\hat{V}_{\mathrm{SNIPW} (\pi_e; \mathcal{D}) := \frac{\mathbb{E}_{n} [w(x_i,a_i) \cdot r_i]}{ \mathbb{E}_{n} [w(x_i,a_i)]},$2’s contribution removed. After global-history-aware normalization and within-prompt standardization, the final agent-specific advantage feeds a clipped GRPO-style objective

$\hat{V}_{\mathrm{SNIPW} (\pi_e; \mathcal{D}) := \frac{\mathbb{E}_{n} [w(x_i,a_i) \cdot r_i]}{ \mathbb{E}_{n} [w(x_i,a_i)]},$3

This is a counterfactual policy optimization method specialized to credit assignment rather than single-agent policy value estimation (Li et al., 23 Mar 2026).

7. Open issues and recurrent misconceptions

A common misconception is that any method using the word “counterfactual” is automatically doing the same thing. The literature instead supports a more differentiated view. In some papers, counterfactual means SCM-style abduction–action–prediction over latent scenarios (Buesing et al., 2018). In some, it means off-policy value estimation under a behavior-policy correction (Guha et al., 9 Jan 2025). In others, it means simulator-based interventions on neighboring agents’ policies (Hart et al., 2020), latent-state perturbations inside an LVLM (Yu et al., 22 Jun 2026), or principal-strata utilities depending on the joint set of potential outcomes (Ben-Michael et al., 2022). This suggests that CFPO is better treated as a family resemblance term than as a single estimator or theorem.

A second misconception is that counterfactual policy optimization always implies full RL. The dynamic-auction work is highly relevant to CFPO but is formally much closer to contextual bandits or continuous-treatment OPE than to trajectory-based RL, with no explicit transition model, Bellman equations, or trajectory-level importance sampling (Guha et al., 9 Jan 2025). Conversely, Geoff-PAC and CF-GPS are explicitly sequential, but they address different forms of counterfactualness: deployment-state-distribution correction in the former and SCM-grounded alternative trajectories in the latter (Zhang et al., 2019, Buesing et al., 2018).

A third misconception is that counterfactual objectives automatically yield safe or identifiable policies. The literature repeatedly warns otherwise. Deficient-support logging can reduce global policy learning to local boundary improvement (Narita et al., 2022). Directly optimizing a noisy counterfactual surrogate can produce “gaming the estimator,” as the dynamic-auction paper notes in discussing possible overfit to the OPE surrogate (Guha et al., 9 Jan 2025). Joint potential-outcome utilities can make even the population target nonidentifiable, requiring partial identification and minimax criteria (Ben-Michael et al., 2022). Inference-aware work shows that maximizing predicted value alone can produce policies that fail downstream significance tests, even when final evaluation is honest (Bastani et al., 20 Oct 2025).

Taken together, the literature portrays CFPO as a technically heterogeneous but increasingly coherent research area. Its unifying concern is not simply off-policy evaluation, causal inference, or policy gradients in isolation, but the design of policy objectives and updates that depend on explicit counterfactual contrasts—between actions, policies, state occupancies, latent worlds, rankings, reasoning episodes, or agent contributions—and that therefore aim to optimize not only observed success, but the underlying decision mechanism that would matter under alternative choices (Yu et al., 22 Jun 2026).