An Introduction to Causal Reinforcement Learning

Abstract: Causal inference provides a set of principles and tools that allow one to combine data and knowledge about an environment to reason with questions of counterfactual nature, i.e., what would have happened had reality been different, even when no data of this unrealized reality is currently available. Reinforcement learning provides methods to learn a policy that optimizes a specific measure (e.g., reward, regret) when the agent is deployed in an environment and pursues an exploratory, trial-and-error approach. These two disciplines have evolved independently and with virtually no interaction between them. We note that they operate over different aspects of the same building block, counterfactual relations, which makes them umbilically connected. Based on these observations, novel learning opportunities arise when this connection is explicitly acknowledged and mathematized. To realize this potential, we note that any environment where the RL agent is deployed can be decomposed as a collection of autonomous mechanisms with different causal invariances, parsimoniously modeled as a structural causal model; any standard RL setting implicitly encodes such a model. This formalization allows us to put under a unifying treatment different modes of learning, including online, off-policy, and causal calculus learning, which appear unrelated in the literature. However, these modalities are not exhaustive: we introduce several natural and pervasive classes of learning settings that entail novel dimensions of analysis. Specifically, we introduce and discuss through causal lenses generalized policy learning, where to intervene, imitation learning, and counterfactual learning. These tasks lead to a broader view of counterfactual learning and suggest great potential for studying causal inference and reinforcement learning side by side, which we call causal reinforcement learning (CRL).

• The paper introduces a unified framework that leverages structural causal models to link causal inference and reinforcement learning for robust policy optimization.
• It presents methodologies including off-policy evaluation, online learning, and counterfactual inference to overcome sample inefficiencies in standard RL.
• It provides empirical evidence and simulation analyses demonstrating improved performance and explainability in complex environments with unmeasured confounding.

Causal Reinforcement Learning: Integrating CI and RL

Introduction and Unifying Framework

"An Introduction to Causal Reinforcement Learning" (2606.24160) systematically bridges the historically independent disciplines of causal inference (CI) and reinforcement learning (RL), revealing their deep foundational connections grounded in counterfactual reasoning. The paper establishes structural causal models (SCMs) as the formal substrate for representing environments, enabling a unified treatment of diverse learning modalities—online, off-policy, and causal identification—under the broader rubric of causal reinforcement learning (CRL).

The authors argue that standard RL approaches, while powerful in structured, well-specified problem settings, remain brittle and sample-inefficient in real-world domains due to their disregard for the causal mechanisms underlying observed dynamics. By contrast, the CRL framework makes explicit both the environment's generative process ($M^*$) and the agent's working causal knowledge ($G$), allowing the design and analysis of agents that are robust, generalizable, and explainable.

Structural Foundation: SCMs and the Pearl Causal Hierarchy

The paper’s technical foundation leverages SCMs as expressive, non-parametric generative models capable of encapsulating both standard RL environments (e.g., MABs, MDPs) and settings with unobserved confounding. Under the SCM formalism, every environment induces not just a joint distribution over endogenous variables but also delineates the three layers of the Pearl Causal Hierarchy (PCH):

• Associational ($L_1$): Passive observations, $P(y|x)$
• Interventional ($L_2$): Effects of actions, $P(y|do(x))$
• Counterfactual ($L_3$): Hypotheticals, $P(y_x|x',y')$

This decomposition is central: it demonstrates that while RL traditionally concerns itself with $L_2$ distributions, correct inference about interventions—and hence robust policy optimization—demands explicit attention to the underlying causal structure, especially in the presence of unmeasured confounding or when only observational data is available.

Policy Optimization Through Causal Decision Models

The formulation of the causal decision model (CDM) is the main abstraction for policy optimization in CRL. Each CDM is defined by:

• an environment SCM $M^*$
• a policy space $G$0 (the agent’s permissible interventions)
• a reward function $G$1

The optimal policy is given by

$G$2

where $G$3 specifies mappings from observed variables to actions, and the expectation is evaluated in the SCM subject to intervening with policy $G$4.

The framework supports general policy spaces, such as full histories, Markov, and stationary policies, allowing for the representation of both classical and more general decision problems (e.g., dynamic treatment regimes, POMDPs). This generalization is critical for the analysis and transfer of results between RL and CI.

Causal Reinforcement Learning Tasks: Learning Regimes and Structural Assumptions

A core contribution is the formalization of CRL tasks via a quadruple $G$5, where $G$6 specifies the agent’s interaction regime (seeing/doing), $G$7 encodes structural assumptions (e.g., a causal diagram $G$8 or the “No Unmeasured Confounder”—NUC—assumption), and $G$9 are as above.

Distinct learning scenarios—such as off-policy evaluation, online learning, and broader causal identification—are instantiated as particular combinations of $L_1$0. This taxonomy helps clarify when classic RL algorithms can be justified, and when they can fail.

Limits of Standard RL: The Need for Causal Structure

A detailed analysis shows that, even in environments where the Markov property is satisfied both observationally and interventionally, knowledge of only one layer (e.g., the observational distribution) is generically insufficient for policy optimization when unobserved confounders may be present. This is formalized via the Causal Hierarchy Theorem, which establishes the strict separation between layers of the PCH. The paper provides constructive counterexamples—in MDPs—where different SCMs yield the same observational (or interventional) distributions but have divergent policy implications.

Figure 2: The regret of RCT with varying trial budget, illustrating the exploration–exploitation trade-off and the effect of $L_1$1.

Policy Learning Methods: Algorithms and Identifiability

Off-Policy Evaluation

When the NUC assumption holds, off-policy evaluation methods such as inverse propensity weighting (IPW) and dynamic programming (DP) are proven correct. The paper rigorously formalizes conditions under which these methods are valid and demonstrates their equivalence for general policy spaces:

$L_1$2

Violations of NUC, however, can induce strong bias, as shown both analytically and via simulation.

Online Learning

Online learning relaxes these structural requirements by leveraging randomization; RCTs and UCB are analyzed from the CRL perspective. The regret analysis recapitulates classic bandit results but emphasizes that online experimentation operationally induces the NUC property in the collected data.

Figure 4: Simulation comparing performance of UCB and RCT algorithms in MABs—UCB achieves near-optimal log-regret adaptively without tuning $L_1$3.

Causal Identification and Sequential Backdoor

The sequential backdoor criterion is developed as a graphical condition ensuring identifiability of policy effects via adjustment, generalizing the NUC. The paper shows that even when NUC fails, sufficient control for confounding can often be achieved via appropriate covariate adjustment, which can be checked graphically.

When adjustment is impossible, the do-calculus is invoked, and the front-door criterion is used to provide identifiability in diagrams with unobserved confounding blocking direct adjustment.

Strong Claims and Simulation Evidence

Empirical results throughout confirm that UCB, properly instantiated, achieves sublinear regret matching the theoretical bounds, while untuned RCT or naive off-policy learning can perform significantly worse.

Figure 6: Simulation comparing UCB with direct observational data transfer (UCB$L_1$4)—showing limitations of naive transfer.

Figure 8: Results with enhanced confounding-robust transfer (UCB$L_1$5), exhibiting improved performance even under hidden confounding.

Figure 10: UCB agent optimizing a two-stage dynamic treatment regime: contrast with RCT which randomizes stages, highlighting sample efficiency.

Figure 12: Sensitivity analysis of UCB$L_1$6 under adversarial confounding.

Implications, Practical Applications, and Future Directions

The CRL framework paves the way for RL methodologies that are robust to dataset shift, capable of sample-efficient transfer, and equipped to integrate multiple sources of data (observational, interventional). This is especially critical in domains—healthcare, economics, social systems—where experimental interaction is limited or impossible, and unmeasured confounding is ubiquitous.

The formal identification results and algorithms extend naturally to composite settings: offline-to-online transfer learning, counterfactual randomization, policy imitation under unobserved rewards, and mixed-policy exploration.

From a theoretical standpoint, the general taxonomy and explicit mapping between RL protocols and layers/identifiability in SCMs open the door for a principled analysis of generalization, transportability, and invariance in RL deployments.

On the applied side, the ability to declare when standard data-driven RL is guaranteed to fail—due to non-identifiability—and to quantify the sufficiency of available domain knowledge for safe policy optimization, is foundational for the deployment of aligned, safe, and efficient AI.

Conclusion

"An Introduction to Causal Reinforcement Learning" (2606.24160) establishes a comprehensive formalism unifying RL and CI, anchored in SCMs and the PCH. By classifying and analyzing learning tasks according to regime, agent knowledge, and causal structure, the paper delivers technical results delineating when and how policy learning is possible, and when structural or experimental limitations render classical RL tools insufficient. The CRL framework not only systematizes many disparate results across RL and CI, but also enables principled exploration of new tasks—offline-to-online transfer, causal imitation, and beyond—under clear identifiability and efficiency guarantees. The implications for theory and practice in AI, especially as systems are increasingly embedded in high-stakes, causally complex domains, are substantial.