Causal Discovery in Subgoal Space for Hierarchical RL

• Causal structure discovery in subgoal space is a method for uncovering key causal relationships to facilitate hierarchical policy decomposition and efficient exploration in RL.
• Algorithms employ targeted interventions and sparse regression to construct causal graphs that model interdependencies among semantically meaningful subgoals.
• Empirical evaluations indicate significant speedups and higher success rates in complex tasks by leveraging structured subgoal selection and optimal intervention strategies.

Causal structure discovery in subgoal space is a foundational methodology for improving sample efficiency, directed exploration, and hierarchy learning in reinforcement learning (RL), particularly for long-horizon, sparse-reward environments. The approach centers on uncovering and leveraging the underlying causal relationships between environment variables or state transitions that correspond to semantically meaningful subgoals, thereby facilitating policy decomposition, effective curriculum generation, and optimal intervention strategies. This paradigm is instantiated in recent frameworks such as Hierarchical RL with Targeted Causal Interventions (HRC) (Khorasani et al., 6 Jul 2025), Causality-Driven Hierarchical RL (CDHRL) (Peng et al., 2022), and Goal Discovery with Causal Capacity (GDCC) (Yu et al., 13 Aug 2025), each providing distinct formalizations and empirical validations. Their unifying principle is to explicitly model subgoal dependencies via directed acyclic graphs (DAGs) or causal graphs, discover these structures through targeted interventions or information-theoretic measures, and integrate the learned hierarchies into multi-level RL policy architectures.

1. Formalization of Subgoal Space and Causal Models

Subgoal space is typically constructed from disentangled environment variables (EVs), each representing a trackable entity or resource relevant to the agent's progression (e.g., "wood collected," "agent satiety") (Peng et al., 2022). Formally, the environment at time $t$ is described by $X^t=(X^t_1,\dots,X^t_l)$, where the first $n$ variables are binary indicators associated with subgoals $g_i$. In HRC, the structural causal model (SCM) is expressed as:

$$X_i^{t+1} = f_i\bigl(\mathrm{pa}(X_i^{t+1}), A^t, \epsilon_i^{t+1}\bigr)$$

with $A^t$ denoting agent actions and $\mathrm{pa}(\cdot)$ the immediate causal parents. In CDHRL, the SCM over $M$ EVs is given by:

$$X_{i,t+1} = f_i\bigl(X_{pa(i)}; N_i\bigr), \qquad pa(i) = \{j : c_{ij}=1\}$$

These causal graphs can encode logical mechanisms such as AND, OR relations for subgoal success (i.e., $X_i^{t+1}=1$ iff $\bigwedge_{j\in pa(g_i)}X_j^t$ for AND nodes) (Khorasani et al., 6 Jul 2025).

GDCC generalizes causal structure via "causal capacity," an information-theoretic measure of how much control an agent has over future transitions from a given state. Here, the focus shifts from environment-variable causality to state-action transition entropy:

$$\mathcal{C}(s) = \mathcal{H}(S'|S = s) = -\sum_{s'} p(s'|s) \log p(s'|s)$$

Critical states with high causal capacity correspond to influential subgoal candidates (Yu et al., 13 Aug 2025).

2. Algorithmic Causal Structure Discovery

Targeted discovery of causal subgoal graphs relies on interventional data collection and sparse regression or statistical learning. In HRC, the Boolean SEM structure is inferred via sparse logistic regression:

$$\hat X_i^{t+1} = \mathbf{1}\left\{ \beta_0 + \sum_{j=1}^n \beta_j X_j^t > 0 \right\}$$

with a regularized loss

$$\mathcal{L}(\beta) = \mathbb E_{D}\left[ (\hat X_i^{t+1} - X_i^{t+1})^2 \right] + \lambda \|\beta\|_0,$$

and in practice, $\|\beta\|_0$ is replaced by $\ell_1$ regularization. Correct parent recovery is guaranteed under assumptions of persistent interventions, AND/OR structure, and subgaussian noise $\rho<1/2$ (Theorem 5.1) (Khorasani et al., 6 Jul 2025).

CDHRL adopts a two-phase structure-function learning loop. Structure parameters $\eta$ are updated via an intervention-driven REINFORCE gradient, alternating with functional model updates using multi-layer perceptrons, thereby progressively refining the causal DAG (Peng et al., 2022). Successful subgoals are determined by policy training success (>60% within $10^4$ trials).

GDCC identifies critical subgoal states using Monte Carlo estimation of $\mathcal{C}(s)$ over observed transitions, employing count estimators in discrete domains and clustering-based approaches in continuous state spaces:

$$\hat{\mathcal{C}}(s) = -\sum_{s'} \frac{N(s,s')}{N(s)} \log \frac{N(s,s')}{N(s)}$$

In high dimensions, clustering partitions next-state clouds to estimate causal capacity without density modelling (Yu et al., 13 Aug 2025).

3. Subgoal Selection, Ordering, and Intervention Strategies

Efficient hierarchy discovery exploits the learned causal structure to determine both subgoal ordering and targeted intervention points. In HRC, subgoal selection is guided by:

• Causal-Effect Ranking: Maximizes expected causal impact on the final goal via

$$\widehat{ECE}(\{g_i\}, \emptyset; g_n) = \widehat{E}[X_n | \mathrm{do}(X_i=1)] - \widehat{E}[X_n | \mathrm{do}(X_i=0)]$$

• Shortest-Path Ranking: Uses A* search over weighted DAG edges to prioritize interventions that minimize cumulative training cost (Khorasani et al., 6 Jul 2025).

In CDHRL, candidate subgoals are restricted to those whose parents are mastered, imposing a topological ordering on the DAG. This reduces the action space per policy to parent primitives and allows progressive exploration (Peng et al., 2022).

In GDCC, the subgoal set $\mathcal{G}$ comprises all states exceeding a causal capacity threshold, with the next subgoal predicted from latent state embeddings. Hierarchical policies are trained to achieve these critical states, yielding Voronoi-type partitions and facilitating progression (Yu et al., 13 Aug 2025).

4. Integration with Reinforcement Learning Architectures

Subgoal hierarchies are incorporated into multi-level RL policies. HRC builds a policy library $\pi_h = \{\pi_0, \pi_1, \dots\}$ that exploits the causal hierarchy for masking and recursive decomposition, maximizing the goal-conditioned value:

$$V^{\pi_h}(s, g_n) = \mathbb{E}\left[ \sum_{t=0}^{\infty} \gamma^t 1\{g_n \text{ achieved}\} \right]$$

No explicit causal regularizer is needed; the causal graph itself orchestrates policy training and exploration order (Khorasani et al., 6 Jul 2025).

CDHRL applies DQN with HER for each subgoal level. Subgoal mastery and addition to the hierarchy depend on empirical training success, with action sets limited by causal graph structure. The iterative loop alternates causal discovery and subgoal training until coverage is complete (Peng et al., 2022).

GDCC couples the causal-capacity subgoal partitioning with encoder–predictor architectures. Low-level RL policies (PPO, TD3) are trained conditionally over discovered subgoals, benefiting from potential-based shaping to accelerate convergence without policy distortion (Yu et al., 13 Aug 2025).

5. Theoretical Analysis of Sample Complexity and Efficiency

Rigorous sample complexity and training cost analyses highlight dramatic efficiency gains from causal structure-guided exploration. In HRC, theoretical bounds for two common graph topologies are established:

• $b$-ary Tree: Targeted causal interventions yield expected cost $O(D^2 b)$ versus $\Omega(n^2 b)$ for random exploration.
• Sparse Erdős–Rényi Graphs: Targeted cost is $O(n^{4/3+2c/3}\log n)$; random is $\Omega(n^2)$.

These bounds are achieved by restricting exploration to ancestors in the causal graph and using optimal intervention rankings (Theorem 4.1) (Khorasani et al., 6 Jul 2025).

GDCC’s entropy-based causal capacity estimator is unbiased under random action sampling and admits standard high-probability error bounds after $N(s)$ state visits. By decomposing tasks into maximally controllable subgoals, GDCC restores near-optimal exploration guarantees reminiscent of Hindsight Experience Replay (Yu et al., 13 Aug 2025).

CDHRL’s intervention-based causal DAG learning halves the structural interventional distance (SID) to ground truth ahead of purely random data collection, validating the impact of targeted causal exploration (Peng et al., 2022).

6. Empirical Results and Practical Impact

Empirical evaluations demonstrate substantial advantages for causal structure discovery in subgoal space. HRC achieves a 3× speedup and 30 point higher final success on 2D-Minecraft compared to baselines like HAC and CDHRL, closely matching theoretical cost bounds on synthetic graphs (Khorasani et al., 6 Jul 2025). GDCC attains 25–40% higher success than the best alternative in MuJoCo and Habitat benchmark tasks, with causal-capacity maps reliably pinpointing semantically valid subgoals (e.g., room intersections, staircases) (Yu et al., 13 Aug 2025). In CDHRL, the agent consistently solves complex crafting and survival tasks more rapidly and stably than curriculum or feature-based heuristics; progressive addition of learned subgoal levels markedly increases exploration quality and downstream success (Peng et al., 2022).

Table: Empirical Results Summary

Framework	Key Environment	Speedup vs Baseline	Final Success Rate
HRC (Khorasani et al., 6 Jul 2025)	2D-Minecraft	~3×	90% (vs 60%)
GDCC (Yu et al., 13 Aug 2025)	MuJoCo Maze/Large	~2×	80% (vs 55%)
CDHRL (Peng et al., 2022)	Eden, Minecraft	~5× milestone attainment	80% (vs 0–20%)

7. Limitations and Future Directions

Limitations of current methods include dependencies on discrete, disentangled environment variables (CDHRL, HRC), scalability challenges in causal graph learning proportional to $O(M^2)$ for large variable sets, and the necessity for preliminary representation learning in fully image-based domains (Peng et al., 2022). GDCC addresses continuous state spaces but relies on local clustering heuristics. Future research is poised to enhance scalability via gradient–DAG methodologies, extend causal discovery to richer observation modalities, and generalize intervention models for continuous and partially observed environments (Yu et al., 13 Aug 2025).

A plausible implication is that unified frameworks combining causal structure discovery, latent variable learning, and optimal intervention policies may deliver further advances in solving real-world, multi-stage RL tasks with extreme reward sparsity and environment complexity.