Hierarchical Reinforcement Learning with Causal Interventions

• Hierarchical reinforcement learning with causal interventions is a framework that decomposes complex tasks into interdependent subgoals using causal graphs to guide efficient exploration.
• It leverages AND-OR structural causal models and sparsity-penalized loss functions to accurately recover true causal dependencies among subgoals.
• Targeted interventions based on causal effect and shortest-path ranking reduce training costs and accelerate performance in long-horizon, sparse-reward domains.

Hierarchical reinforcement learning with targeted causal interventions is an approach that combines hierarchical reinforcement learning (HRL) with causal structure discovery and exploitation to enhance data efficiency and control in long-horizon, sparse-reward tasks. By representing subgoal dependencies as a causal graph, this methodology enables not only the identification of a meaningful task decomposition, but also directs exploration and training toward the subgoals most likely to accelerate the achievement of the final goal.

1. Causal Structure Discovery in Subgoal Space

The central innovation is the representation of the subgoal space as a causal graph, where each node denotes a subgoal and edges encode the causal dependence (i.e., achieving one subgoal facilitates the achievement of another). The generative process for subgoal variables is formalized via an AND-OR Structural Causal Model (A–SCM), expressed as:

$$X_i^{t+1} = \theta_i(X^t) \oplus \epsilon_i^{t+1},\quad 1\le i\le n,$$

where the deterministic mechanism function θ_i depends on the subgoal’s parent set $\mathcal{P}(g_i)$:

$$\theta_i(X^t)= \begin{cases} \bigwedge_{g_j\in \mathcal{P}(g_i)} X_j^t, & \text{if }g_i\text{ is AND-subgoal} \ \bigvee_{g_j\in \mathcal{P}(g_i)} X_j^t, & \text{if }g_i\text{ is OR-subgoal} \end{cases}$$

and $\epsilon_i^{t+1}$ is a Bernoulli noise term.

The discovery procedure identifies "discoverable parents": if toggling a parent variable with all others held fixed reliably flips the child subgoal, it is statistically discoverable. The discovery algorithm minimizes a loss function over candidate parent sets, incorporating an $l_0$ sparsity penalty, as in

$$\mathcal{L}(\beta) = \mathbb{E}[(\hat{X}_i^{t+1}-X_i^{t+1})^2] + \lambda\|\beta\|_0,$$

with the minimizer $\beta^*$ guaranteed (under conditions in Theorem 1) to select only the true parents, up to statistically undiscoverable edges. This construction ensures that the causal structure is learnable and aligns with the actual interdependence of subgoals.

2. Targeted Causal Interventions for Efficient Exploration

Whereas traditional HRL explores the subgoal space through random or uniform interventions, this framework employs the learned causal graph to focus interventions on subgoals that are most causally influential for task completion. Two ranking rules are used:

• Causal Effect Ranking Rule: For each controllable subgoal $g_i$, the algorithm estimates the expected causal effect (ECE) of setting $g_i$ on versus off (using do-calculus notation) on the final subgoal $g_n$:

$$ECE_{t^*}^{\Delta}(A,B,g_n) = \mathbb{E}[X_n^{t^*+\Delta} \mid \text{do}(X_A^{t^*}=1), \text{do}(X_B^{t^*}=0)] - \mathbb{E}[X_n^{t^*+\Delta} \mid \text{do}(X_A^{t^*}=0), \text{do}(X_B^{t^*}=0)]$$

The next intervention is targeted to the subgoal with the largest causal effect on the main goal.

• Shortest Path Ranking Rule: Inspired by A* search, this rule combines the cost-to-reach $g(g_i)$ from root subgoals and an admissible heuristic $h(g_i)$ estimating remaining cost to the final subgoal:

$f(g_i) = g(g_i) + h(g_i)$

The algorithm picks the subgoal minimizing this total cost function for the next intervention.

These targeted approaches are shown, both theoretically and empirically, to reduce the sample complexity of HRL compared to random baselines.

3. Theoretical Analysis of Training Cost

The sample efficiency of the targeted strategies is rigorously quantified:

• In tree-structured subgoal graphs $G(n, b)$ (n nodes, branching factor b), the expected cost of reaching the final goal through targeted interventions is

$O\bigl(\log^2(n)\,b\bigr)$

while a random intervention strategy incurs a cost of

$\Omega\bigl(n^2\,b\bigr)$

• For "semi-Erdős–Rényi" random graphs $G(n, p)$ with edge probability $p=\frac{c\log(n)}{n-1}$, the targeted intervention’s cost is

$O\left(n^{\frac{4}{3}+\frac{2}{3}c}\log(n)\right)$

with the random baseline remaining at $\Omega(n^2)$.

These results imply exponential or near-exponential improvements in training cost as the number of subgoals grows, provided causal dependencies are explicitly exploited.

4. Empirical Performance and Benchmark Comparison

Extensive experiments conducted on synthetic graphs (tree and semi-Erdős–Rényi) support the theoretical sample complexity improvements. Empirical plots illustrate that targeted strategies require orders of magnitude fewer interventions (or system probes) to reach high success ratios as the subgoal graph scales.

In the 2D-Minecraft domain—representative of complex, long-horizon RL with sparse rewards—the causal-HRL methods using targeted interventions both achieve higher success rates and reach success thresholds (e.g., 50% solved) far more rapidly than state-of-the-art baselines (CDHRL, HAC, HER, OHRL, PPO). For instance, achieving 50% success required 33–38 minutes for the new method versus over 180 minutes for HAC and nearly 250 minutes for HER.

Additionally, causal structure recovery quality (e.g., measured by Structural Hamming Distance to the ground-truth subgoal graph) is improved over previous algorithms that use non-adapted causal discovery tools.

5. Distinction from Prior Causal HRL Approaches

Previous causal HRL works (e.g., Hu et al. 2022; Nguyen et al. 2024) typically applied generic causal discovery algorithms without theoretical analysis or HRL-specific adaptation and selected subgoal interventions at random. In contrast, the present approach provides:

• An HRL-tailored causal discovery algorithm, with formal recovery guarantees on discoverable parents.
• The use of causal effect and cost-based ranking rules for targeted, quantitatively justified interventions.
• Theoretical bounds on training cost and sample complexity under various structural assumptions about the subgoal graph.
• Empirical demonstration of both faster convergence and better structure recovery in comparison to prior methods.

6. Broader Implications and Potential Extensions

The integration of causal reasoning into HRL, as instantiated in this methodology, provides an intrinsic mechanism for hierarchical decomposition and adaptive exploration. This framework is applicable to any domain where a complex task can be decomposed into interdependent subtasks, including:

• Robotics with sequential tool use or part assembly,
• Long-horizon planning in simulated or real environments,
• Game agents and multi-stage decision processes,
• Real-world systems in which sparse rewards present a core challenge.

A plausible implication is that HRL agents equipped with a causal structure discovery and targeted exploration module will scale better and require less task-specific engineering as task complexity increases. Future research directions include extending these techniques to continuous or multi-valued subgoal spaces, automatic extraction of disentangled environment factors, and hybridizing with meta-learning schemes for rapid transfer across structurally related tasks.

---

In summary, hierarchical reinforcement learning with targeted causal interventions leverages causal structure discovery among subgoals and employs principled intervention strategies to dramatically improve the efficiency of learning in long-horizon, sparse-reward domains. This approach, substantiated by both theoretical analysis and empirical demonstration, establishes a robust bridge between causal modeling and scalable HRL (Khorasani et al., 6 Jul 2025).