---
title: Counterfactually-Guided Policy Search (CF-GPS)
url: https://www.emergentmind.com/topics/counterfactually-guided-policy-search-cf-gps
type: topic
---

# Counterfactually-Guided Policy Search (CF-GPS)

Counterfactually-Guided Policy Search (CF-GPS) is a framework for reinforcement learning in partially-observable Markov decision processes (POMDPs) that leverages structural causal models (SCMs) to perform policy evaluation and search via counterfactual inference. Key to the method is conditioning model-based rollouts on real, logged data to reduce bias and variance, contrasting with de novo synthetic rollouts or importance sampling approaches. CF-GPS generalizes prior methods such as Guided Policy Search and can be related to value gradient approaches through the lens of counterfactual structural inference [1811.06272].

## 1. Formal Foundations: POMDPs and Structural Causal Models

CF-GPS operates within the episodic POMDP setting. Let $\mathcal{S}$ be the hidden state space, $\mathcal{A}$ the action space, and $\mathcal{O}$ the observation space. The agent trajectory is $\tau = (S_1, O_1, A_1, ..., S_T, O_T)$ with the stochastic policy $\pi(a_t|h_t)$ over the full action-observation history $h_t$. Transition and observation kernels are $P_{S_{t+1}|S_t,A_t}$ and $P_{O_t|S_t, A_{t-1}}$ respectively, and the undiscounted return is $G(\tau) = \sum_{t=1}^T R_t$, where $R_t$ is included in $O_t$.

The entire generative process of the POMDP plus policy $\pi$ is encoded as a structural causal model $M^\pi$, with scenario (noise) variables $U = \{U_{s1}, U_{o1}, U_{a1}, U_{s2}, ..., U_{oT}, U_{aT}\}$ encoding randomness in initial state, transitions, observations, and policy actions. Deterministic mechanisms $f_i$ map scenario variables to process variables; for instance, $S_1 = f_{s1}(U_{s1})$ and $A_t = f_\pi(h_t, U_{at})$.

Interventions—substituting the policy mechanism $\pi$ by another policy $\mu$—correspond to the standard SCM operation $\mathrm{do}(\pi \rightarrow \mu)$, yielding the SCM $M^\mu$ and corresponding trajectory distribution $P^\mu$.

## 2. Counterfactual Evaluation in SCMs

Given observed variables $X_o = \hat{x}_o$, an intervention $I$ (e.g., $\mathrm{do}(\mu \to \pi)$), and query variables $X_q$, the counterfactual distribution $p^{\mathrm{do}(I)|\hat{x}_o}(X_q)$ is obtained by:
1. Inferring scenario variables $U$ from $p(U | X_o = \hat{x}_o)$.
2. Fixing $U$ to the sampled value, applying the intervention, and forward-simulating the new mechanism.
3. Returning the queried variables.

Algorithmically, this involves: sampling $\hat{u} \sim p(U| \hat{x}_o)$, replacing $p(U)$ with a delta function at $\hat{u}$, altering mechanisms per $I$, and simulating $X_q$. Averaging over real data $(\hat{x}_o)$ ensures unbiased estimation: $\mathbb{E}_{\hat{x}_o \sim p}[p^{\mathrm{do}(I)|\hat{x}_o}(x)] = p^{\mathrm{do}(I)}(x)$.

For policy evaluation (CF-PE), with off-policy dataset $D = \{\hat{h}_T^i \sim \mathbb{P}^\mu\}$, the counterfactual value estimator for candidate $\pi$ is
$$
\hat V_{CF}(\pi) = \frac{1}{N} \sum_{i=1}^N G^{(i)},\quad G^{(i)} = \mathrm{CFI}(\hat h^i_T,M,I(\mu\to\pi),G).
$$
Under a correct model, $E[\hat V_{CF}(\pi)] = V(\pi)$.

## 3. The CF-GPS Algorithm and Policy Improvement

CF-GPS alternates between logging real experience and model-based planning, with simulated rollouts "grounded" in inferred scenarios from observed episodes. Each logged episode $\hat{h}_T^i$ is used to derive a posterior over scenario variables $U_{CF}^i$. Multiple counterfactual rollouts are generated for each $U_{CF}^i$ under a planner or candidate policy $\lambda$.

Policy improvement then proceeds using a return-weighted regression objective, with rollouts from the counterfactual distribution,
$$
\pi^{k+1}
= \arg\max_\pi
\sum_{i=1}^{N}
\exp(G(\tau^i))
\frac{p^{\pi^k}(\tau^i)}{p^{\lambda}(\tau^i)}
\log p^{\pi}(\tau^i),\quad \tau^i \sim p^{\lambda|\hat{h}^i_T}(\cdot).
$$
Here, each trajectory $\tau^i$ is simulated conditioned on the inferred $U_{CF}^i$ ("posterior grounding") instead of starting from the prior, thereby reducing model mismatch.

### CF-GPS Algorithmic Outline

```python
# Pseudocode summary
Procedure CF-GPS(M, π⁰, N):
    k←0
    repeat:
        Possibly update behavior μ←π^k
        For i=1 to N:
            Gather ĥ^i_T ∼ real env under μ
            τ^i ← CFI(ĥ^i_T, M, I(μ→λ), full trajectory T)
        π^{k+1} ← policy‐improvement using {τ^i}
        k←k+1
    until convergence
```

CF-GPS can incorporate a "planner" $\lambda$ that uses an expert policy or current policy, allowing for annealed mixtures.

## 4. Methodological Comparisons and Theoretical Insights

CF-GPS situates itself among multiple strands of reinforcement learning:

- **Vanilla model-based RL (MB-PE, MB-PS):** These approaches draw scenario variables from the model prior $p(U)$, leading to biased results under model mismatch between $p(U)$ and the true environment.

- **Importance-Sampling off-policy methods:** Reweight logged returns according to trajectory likelihood ratios $w^i \propto p^\pi(\hat h_T^i)/p^\mu(\hat h_T^i)$. While unbiased, these methods suffer from high variance if $\pi$ and $\mu$ differ significantly or lack mutual support.

CF-GPS achieves bias reduction by sampling from the posterior $p(U| \hat{h}_T)$, grounding rollouts in real experience. Variance is reduced by generating counterfactual trajectories under fixed scenario noise, randomizing only action noise if needed, and dispensing with trajectory-level importance weighting.

### Relations to Other RL Methods

- **Guided Policy Search (GPS):** GPS, as in Levine & Koltun (2014), is a special case of CF-GPS for fully-observed MDPs with linear-Gaussian SCMs and LQR planners. Here, linear dynamics parameters inferred from real data act as counterfactual scenario inference.

- **Stochastic Value Gradient (SVG):** SVG interprets policy and model computation as deterministic functions of shared noise $U$, making inference over $U_{st}$ and freezing $U_{at}$ equivalent to counterfactual trajectory analysis and gradient estimation.

## 5. Empirical Evaluation and Benchmarking

CF-GPS performance is evaluated on the PO-Sokoban grid-world benchmark:
- **Environment:** $10 \times 10$ grid, 3 boxes, 3 targets, strong partial observability (each tile flipped to empty with $0.9$ probability in observation), episode length $T=50$, structured rewards.
- **Scenario variables:** The initial level layout $U_{s1}$, inferred for each episode.

### Off-policy Evaluation (CF-PE)
- Model $M$ is learned only over initial state distribution $p(U_{s1})$; transitions/observations are assumed known.
- Three policies $\pi_0, \pi_1, \pi_2$ are trained under a uniform behavior $\mu$.
- Value estimates using MB-PE ($t=0$) incur high error ($>50\%$). As more transitions are conditioned on ($t \to 50$), error drops to near zero. Importance-sampling approaches perform poorly (error $>80\%$).

### Policy Search (CF-GPS)
- Logged data are collected by 64 actors under the behavior policy $\mu \approx \pi$.
- Scenario inference $p(U_{s1}|\hat{h}_T)$ utilizes DRAW.
- The planner $\lambda$ is an annealed mixture of the current policy and an expert.
- Each scenario supports 10 counterfactual rollouts, used for return-weighted regression policy updates.

### Key Outcomes
- Counterfactual policy evaluation error approaches zero with full smoothing, versus large errors for purely prior-based simulations.
- CF-GPS learning curves surpass both vanilla model-based planning and GPS-like baselines that do not fully condition on observations.
- Given the same real data budget, CF-GPS outperforms model-free IMPALA.

## 6. Context, Limitations, and Generalizations

CF-GPS formalizes model-based policy learning as counterfactual inference with SCMs, centering the role of posterior scenario inference in reducing bias otherwise endemic to model-simulator approaches. The methodology assumes correct or well-calibrated models for scenario variable inference; model errors can propagate into policy search steps. The generalization to fully-observed MDPs and incorporation of planners or experts in $\lambda$ broadens applicability, subsuming Guided Policy Search and linking to Stochastic Value Gradient methods ("reparameterization-based algorithms can be interpreted as counterfactual methods"). This suggests potential for further theoretical unification and practical extensions in RL.

## 7. References

- Buesing, L., Weber, T., Zwols, Y., Racanière, S., Guez, A., Lespiau, J.-B., & Heess, N. (2019). "Woulda, Coulda, Shoulda: Counterfactually-Guided Policy Search" [1811.06272].
- Pearl, J. (2009). *Causality*. Cambridge University Press.
- Levine, S., & Koltun, V. (2014). "Guided policy search." ICML.
- Heess, N., et al. (2015). "Learning continuous control policies by stochastic value gradients." NIPS.

Source: https://www.emergentmind.com/topics/counterfactually-guided-policy-search-cf-gps