Counterfactual Rollouts: Methods & Applications

• Counterfactual rollouts are simulated trajectories in sequential decision-making that alter key intervention factors to assess outcome variations.
• They integrate causal inference, robust policy evaluation, and explainable AI across applications such as healthcare, market design, and reinforcement learning.
• Methodologies include equilibrium constraints, MIQCQP optimization, and generative models, offering quantifiable bounds and improved policy adaptation.

Counterfactual rollouts are simulated trajectories in sequential decision-making or multi-agent systems that evaluate outcomes under alternative interventions, policies, or structural assumptions. The central objective is to quantify, bound, or explain how system behavior would change if key factors—such as rules, dynamics, treatments, or initial states—were modified, all while maintaining fidelity to observed data. This technique connects causal inference, robust policy evaluation, and explainable AI, and it is implemented across domains ranging from robust market design and reinforcement learning to temporal point processes and healthcare simulations.

1. Formal Definitions and Fundamental Principles

Counterfactual rollouts generalize classical counterfactual reasoning to multi-step or multi-agent settings, where sequential dependencies and strategic interactions are critical. A canonical setup involves an observed (factual) trajectory dataset $\mathcal{D} = \{\tau^i\}$, which is collected under certain environmental rules or a behavior policy. The analyst seeks to estimate, bound, or simulate what would happen under an altered intervention—such as changed game rules $G'$, a new MDP strategy $\sigma'$, or a shift in underlying structural parameters $\Phi'$.

In multi-agent games, the robust multi-agent counterfactual prediction (RMAC) framework replaces the one-to-one mapping $${\text{actions}} \leftrightarrow {\text{type-distribution}} \leftrightarrow {\text{counterfactual equilibrium}}$$ with set-valued (“robust-identified”) maps induced by $\epsilon$-approximate equilibrium: the set $\Lambda_\epsilon$ of type distributions consistent with the data and an $\epsilon$-BNE in both original and counterfactual settings, yielding worst-case and best-case counterfactual outcomes $Y^-_\epsilon$, $Y^+_\epsilon$ (Peysakhovich et al., 2019).

In MDPs, counterfactual strategies are defined as the minimal perturbation $\sigma^*$ of a baseline strategy $\sigma$ such that the probability of an undesired event drops below a threshold $\gamma$; $\sigma^*$ is the solution to a nonlinear optimization encoding that trades off proximity (via $\ell_0/\ell_1/\ell_\infty$ norms) and success probability constraints (Kobialka et al., 14 May 2025).

In offline RL and policy evaluation, counterfactual rollouts may represent trajectories sampled by re-running agent behavior under alternative state initializations, alternative actions, or intervened transition dynamics, often aided by importance sampling or simulation (Tang et al., 2023, Frost et al., 2022).

2. Methodologies for Counterfactual Rollout Generation

The generation of counterfactual rollouts is highly context-dependent:

• Multi-agent Strategic Environments (RMAC):

1. Discretize unobserved type space $\Theta$ and introduce weights $w_k$ over candidate types.
2. Impose equilibrium constraints matching observed actions and allow $\epsilon$-deviations.
3. For each weight vector, solve for $\epsilon$-BNE in counterfactual game $G'$, yielding $\sigma'$.
4. Optimize target functional $V$ over feasible profiles to obtain extremal counterfactual bounds. Computational procedures include mixed-integer programming (MPECs), smoothing/penalty approximations, and Revelation Game Fictitious Play (RFP) (Peysakhovich et al., 2019).

• Markov Decision Processes:

The MIQCQP formulation encodes reachability and proximity constraints across state-action probabilities, capable of synthesizing diverse counterfactual strategies via pairwise diversity constraints or determinant-based regularization (Kobialka et al., 14 May 2025).

• Offline RL / Policy Transfer:

Counterfactual rollouts are simulated by invoking alternate environment parameters (e.g., $\Phi_{cf}$), and replaying identical action trajectories under new dynamics. Such data are pooled with factual rollouts and used to distill new policies in architectures such as Decision Transformer (DT), with loss functions weighted by estimated average treatment effects (ATE) for efficient policy adaptation (Boustati et al., 2021).

• Semi-Offline Policy Evaluation:

Human annotations provide estimated future returns for unobserved actions, forming counterfactual rollouts that contribute to policy value estimates. Importance sampling (IS) and per-decision IS (PDIS) estimators are augmented to integrate these annotations without introducing bias, utilizing a corrected weighting mechanism and augmented behavior policy (Tang et al., 2023).

• Temporal Point Processes:

Counterfactual realizations arise by re-applying the Gumbel-Max thinning SCM to both observed and reconstructed rejected candidate times, under alternative intensity functions. Monotonicity conditions guarantee identifiability; sampling uses explicit pseudocode invoking Poisson superposition and SCM-based event acceptance (Noorbakhsh et al., 2021).

• Time-varying Treatments (Causal Generative Models):

Counterfactual rollouts under alternate treatment sequences are generated via conditional generative models (e.g., guided diffusion, conditional VAE) trained with inverse probability weighting to account for distribution mismatch and ensure counterfactual fidelity (Wu et al., 2023).

3. Algorithmic Details and Computational Approaches

Algorithmic implementations vary:

• RMAC Rollouts: Discretize types, setup action moment-matching, solve for candidate equilibria in both original and counterfactual environments via fixed-point iteration or variational inequality solvers; optimize evaluation functionals under constraints (Peysakhovich et al., 2019).
• MDP Counterfactual Strategies: Encode optimization in MIQCQP, precompute relevant state sets, use Gurobi or similar solvers, iterate for diversity, and sample trajectories in the induced MDP to obtain counterfactual rollouts (Kobialka et al., 14 May 2025).
• Decision Transformer with Counterfactuals:

1. Simulate factual and counterfactual episodes via interventions on $\Phi$;
2. Encode trajectories as embedded token sequences;
3. Train via maximum likelihood over pooled rollout sets, potentially with ATE weighting (Boustati et al., 2021).

• IS-based OPE with Counterfactuals:

Algorithmic recursion incorporates both factual returns and counterfactual annotations in the backward step, utilizing per-timestep weights and corrected policy likelihood ratios (Tang et al., 2023).

• Temporal Point Process Counterfactuals:

Explicit pseudocode details candidate reconstruction and SCM-based event acceptance, with complexity scaling linearly in the total number of events and noise samples (Noorbakhsh et al., 2021).

• Causal Generative Rollouts:

Sampling proceeds via diffusion or VAE inference steps conditioned on alternate treatment histories, with inverse-propensity weighting applied during training to correct for observed/counterfactual distribution mismatch (Wu et al., 2023).

4. Modeling Assumptions and Robustness Considerations

Modeling assumptions are explicit and critically impact the validity of counterfactual rollouts:

• Equilibrium Rationality:

RMAC relaxes standard BNE rationality to $\epsilon$-BNE, explicitly expanding the set of consistent type distributions $\Lambda_\epsilon$ and exposing sensitivity to bounded rationality, payoff misspecification, and multiple equilibria (Peysakhovich et al., 2019).

• Support and Annotation Quality:

In semi-offline evaluation, unbiasedness of counterfactual IS estimators critically requires annotation coverage of all evaluated actions and high annotation quality. Bias correction and regression-based imputation are recommended for practical annotation shortcomings (Tang et al., 2023).

• Structural Assumptions:

Temporal point process models require SCM monotonicity and an explicit, tractable mapping between observed and candidate event times. Counterfactual thinning is identified only if the monotonicity holds (Noorbakhsh et al., 2021).

• Distribution Shift, Positivity, and Ignorability:

Generative models for treatment counterfactuals are trained and weighted according to established longitudinal causal inference assumptions, including positivity and sequential ignorability, with propensity networks learned for IPW correction (Wu et al., 2023).

5. Practical Applications and Empirical Findings

Applied domains exploit counterfactual rollouts for robustness analysis, interpretability, transfer learning, policy evaluation, and targeted intervention:

• Market Design and Auctions:

RMAC calculates lower and upper revenue bounds for auction format changes. Even small $\epsilon$ values yield wide confidence intervals, demonstrating the fragility of point-identification-based counterfactual claims in strategic settings (Peysakhovich et al., 2019).

• Healthcare and Sequential Decision Making:

Counterfactual strategies in MDPs enable process improvement by quantifying and minimizing the probability of undesired events through minimal strategy adaptation; practical feasibility is demonstrated for models with thousands of states (Kobialka et al., 14 May 2025).

• RL Policy Transfer:

Decision Transformer models trained on combined factual and counterfactual rollouts with ATE weighting outperform baselines under substantial environment shifts, attaining higher average rewards and goal-attainment rates (Boustati et al., 2021).

• Policy Explanation under Distribution Shift:

Counterfactual rollouts generated by steering RL agents into states outside their training distribution yield improved user understanding of agent capabilities under test-time shifts (e.g., novel initial states in MiniGrid), as validated by user paper accuracy improvements (Frost et al., 2022).

• Causal Inference in Time-varying Treatments:

Guided diffusion and CVAE models trained with IPW generate high-quality counterfactual samples, outperforming baselines in mean-absolute-error and Wasserstein distance metrics, and offering insights into the impact of treatment policies over high-dimensional outcomes (Wu et al., 2023).

6. Theoretical Guarantees, Sensitivity, and Interpretability

Theoretical analysis provides performance bounds and interpretability:

• RMAC Sensitivity:

The gap $Y^+_\epsilon - Y^-_\epsilon$ quantifies robustness to rationality, payoff specification, and identification; scanning over $\epsilon$ yields a sensitivity plot for counterfactual claims (Peysakhovich et al., 2019).

• OPE with Counterfactuals:

Theoretical bias and variance analyses show that C*‐IS/C‐PDIS estimators are unbiased under annotation support and reduce variance relative to pure IS; robustness to annotation noise and missingness is empirically observed (Tang et al., 2023).

• Temporal Logic Satisfaction:

In LTL-constrained RL, eventual discounting guarantees that, as $\gamma \to 1$, the surrogate-optimal policy converges to the true maximum satisfaction probability. Counterfactual replay exponentially increases data yield without requiring additional MDP knowledge (Voloshin et al., 2023).

• Structural Causal Identification:

Monotonicity conditions in SCM thinning guarantee identifiability for temporal point process counterfactual simulations, with explicit construction of the event acceptance dynamics under alternative intensities (Noorbakhsh et al., 2021).

Counterfactual rollouts, formalized across these domains, are a foundational component for policy robustness, sensitivity analysis, causal inference, and interpretability in high-dimensional, sequential, and multi-agent systems.