---
title: Causal Discovery in Subgoal Space for Hierarchical RL
url: https://www.emergentmind.com/topics/causal-structure-discovery-in-subgoal-space
type: topic
---

# Causal Discovery in Subgoal Space for Hierarchical RL

Causal structure discovery in subgoal space is a foundational methodology for improving sample efficiency, directed exploration, and hierarchy learning in reinforcement learning (RL), particularly for long-horizon, sparse-reward environments. The approach centers on uncovering and leveraging the underlying causal relationships between environment variables or state transitions that correspond to semantically meaningful subgoals, thereby facilitating policy decomposition, effective curriculum generation, and optimal intervention strategies. This paradigm is instantiated in recent frameworks such as Hierarchical RL with Targeted Causal Interventions (HRC) [2507.04373], Causality-Driven Hierarchical RL (CDHRL) [2210.06964], and Goal Discovery with Causal Capacity (GDCC) [2508.09624], each providing distinct formalizations and empirical validations. Their unifying principle is to explicitly model subgoal dependencies via directed acyclic graphs (DAGs) or causal graphs, discover these structures through targeted interventions or information-theoretic measures, and integrate the learned hierarchies into multi-level RL policy architectures.

## 1. Formalization of Subgoal Space and Causal Models

Subgoal space is typically constructed from disentangled environment variables (EVs), each representing a trackable entity or resource relevant to the agent's progression (e.g., "wood collected," "agent satiety") [2210.06964]. Formally, the environment at time $t$ is described by $X^t=(X^t_1,\dots,X^t_l)$, where the first $n$ variables are binary indicators associated with subgoals $g_i$. In HRC, the structural causal model (SCM) is expressed as:
$$
X_i^{t+1} = f_i\bigl(\mathrm{pa}(X_i^{t+1}), A^t, \epsilon_i^{t+1}\bigr)
$$
with $A^t$ denoting agent actions and $\mathrm{pa}(\cdot)$ the immediate causal parents. In CDHRL, the SCM over $M$ EVs is given by:
$$
X_{i,t+1} = f_i\bigl(X_{pa(i)}; N_i\bigr), \qquad pa(i) = \{j : c_{ij}=1\}
$$
These causal graphs can encode logical mechanisms such as AND, OR relations for subgoal success (i.e., $X_i^{t+1}=1$ iff $\bigwedge_{j\in pa(g_i)}X_j^t$ for AND nodes) [2507.04373].

GDCC generalizes causal structure via "causal capacity," an information-theoretic measure of how much control an agent has over future transitions from a given state. Here, the focus shifts from environment-variable causality to state-action transition entropy:
$$
\mathcal{C}(s) = \mathcal{H}(S'|S = s) = -\sum_{s'} p(s'|s) \log p(s'|s)
$$
Critical states with high causal capacity correspond to influential subgoal candidates [2508.09624].

## 2. Algorithmic Causal Structure Discovery

Targeted discovery of causal subgoal graphs relies on interventional data collection and sparse regression or statistical learning. In HRC, the Boolean SEM structure is inferred via sparse logistic regression:
$$
\hat X_i^{t+1} = \mathbf{1}\left\{ \beta_0 + \sum_{j=1}^n \beta_j X_j^t > 0 \right\}
$$
with a regularized loss
$$
\mathcal{L}(\beta) = \mathbb E_{D}\left[ (\hat X_i^{t+1} - X_i^{t+1})^2 \right] + \lambda \|\beta\|_0,
$$
and in practice, $\|\beta\|_0$ is replaced by $\ell_1$ regularization. Correct parent recovery is guaranteed under assumptions of persistent interventions, AND/OR structure, and subgaussian noise $\rho<1/2$ (Theorem 5.1) [2507.04373].

CDHRL adopts a two-phase structure-function learning loop. Structure parameters $\eta$ are updated via an intervention-driven REINFORCE gradient, alternating with functional model updates using multi-layer perceptrons, thereby progressively refining the causal DAG [2210.06964]. Successful subgoals are determined by policy training success (>60% within $10^4$ trials).

GDCC identifies critical subgoal states using Monte Carlo estimation of $\mathcal{C}(s)$ over observed transitions, employing count estimators in discrete domains and clustering-based approaches in continuous state spaces:
$$
\hat{\mathcal{C}}(s) = -\sum_{s'} \frac{N(s,s')}{N(s)} \log \frac{N(s,s')}{N(s)}
$$
In high dimensions, clustering partitions next-state clouds to estimate causal capacity without density modelling [2508.09624].

## 3. Subgoal Selection, Ordering, and Intervention Strategies

Efficient hierarchy discovery exploits the learned causal structure to determine both subgoal ordering and targeted intervention points. In HRC, subgoal selection is guided by:

- **Causal-Effect Ranking:** Maximizes expected causal impact on the final goal via
  $$
  \widehat{ECE}(\{g_i\}, \emptyset; g_n) = \widehat{E}[X_n | \mathrm{do}(X_i=1)] - \widehat{E}[X_n | \mathrm{do}(X_i=0)]
  $$
- **Shortest-Path Ranking:** Uses A* search over weighted DAG edges to prioritize interventions that minimize cumulative training cost [2507.04373].

In CDHRL, candidate subgoals are restricted to those whose parents are mastered, imposing a topological ordering on the DAG. This reduces the action space per policy to parent primitives and allows progressive exploration [2210.06964].

In GDCC, the subgoal set $\mathcal{G}$ comprises all states exceeding a causal capacity threshold, with the next subgoal predicted from latent state embeddings. Hierarchical policies are trained to achieve these critical states, yielding Voronoi-type partitions and facilitating progression [2508.09624].

## 4. Integration with Reinforcement Learning Architectures

Subgoal hierarchies are incorporated into multi-level RL policies. HRC builds a policy library $\pi_h = \{\pi_0, \pi_1, \dots\}$ that exploits the causal hierarchy for masking and recursive decomposition, maximizing the goal-conditioned value:
$$
V^{\pi_h}(s, g_n) = \mathbb{E}\left[ \sum_{t=0}^{\infty} \gamma^t 1\{g_n \text{ achieved}\} \right]
$$
No explicit causal regularizer is needed; the causal graph itself orchestrates policy training and exploration order [2507.04373].

CDHRL applies DQN with HER for each subgoal level. Subgoal mastery and addition to the hierarchy depend on empirical training success, with action sets limited by causal graph structure. The iterative loop alternates causal discovery and subgoal training until coverage is complete [2210.06964].

GDCC couples the causal-capacity subgoal partitioning with encoder–predictor architectures. Low-level RL policies (PPO, TD3) are trained conditionally over discovered subgoals, benefiting from potential-based shaping to accelerate convergence without policy distortion [2508.09624].

## 5. Theoretical Analysis of Sample Complexity and Efficiency

Rigorous sample complexity and training cost analyses highlight dramatic efficiency gains from causal structure-guided exploration. In HRC, theoretical bounds for two common graph topologies are established:

- **$b$-ary Tree:** Targeted causal interventions yield expected cost $O(D^2 b)$ versus $\Omega(n^2 b)$ for random exploration.
- **Sparse Erdős–Rényi Graphs:** Targeted cost is $O(n^{4/3+2c/3}\log n)$; random is $\Omega(n^2)$.

These bounds are achieved by restricting exploration to ancestors in the causal graph and using optimal intervention rankings (Theorem 4.1) [2507.04373].

GDCC’s entropy-based causal capacity estimator is unbiased under random action sampling and admits standard high-probability error bounds after $N(s)$ state visits. By decomposing tasks into maximally controllable subgoals, GDCC restores near-optimal exploration guarantees reminiscent of Hindsight Experience Replay [2508.09624].

CDHRL’s intervention-based causal DAG learning halves the structural interventional distance (SID) to ground truth ahead of purely random data collection, validating the impact of targeted causal exploration [2210.06964].

## 6. Empirical Results and Practical Impact

Empirical evaluations demonstrate substantial advantages for causal structure discovery in subgoal space. HRC achieves a 3× speedup and 30 point higher final success on 2D-Minecraft compared to baselines like HAC and CDHRL, closely matching theoretical cost bounds on synthetic graphs [2507.04373]. GDCC attains 25–40% higher success than the best alternative in MuJoCo and Habitat benchmark tasks, with causal-capacity maps reliably pinpointing semantically valid subgoals (e.g., room intersections, staircases) [2508.09624]. In CDHRL, the agent consistently solves complex crafting and survival tasks more rapidly and stably than curriculum or feature-based heuristics; progressive addition of learned subgoal levels markedly increases exploration quality and downstream success [2210.06964].

Table: Empirical Results Summary

| Framework | Key Environment | Speedup vs Baseline | Final Success Rate |
|-----------|-----------------|--------------------|-------------------|
| HRC [2507.04373] | 2D-Minecraft | ~3× | 90% (vs 60%) |
| GDCC [2508.09624] | MuJoCo Maze/Large | ~2× | 80% (vs 55%) |
| CDHRL [2210.06964]| Eden, Minecraft | ~5× milestone attainment | 80% (vs 0–20%) |

## 7. Limitations and Future Directions

Limitations of current methods include dependencies on discrete, disentangled environment variables (CDHRL, HRC), scalability challenges in causal graph learning proportional to $O(M^2)$ for large variable sets, and the necessity for preliminary representation learning in fully image-based domains [2210.06964]. GDCC addresses continuous state spaces but relies on local clustering heuristics. Future research is poised to enhance scalability via gradient–DAG methodologies, extend causal discovery to richer observation modalities, and generalize intervention models for continuous and partially observed environments [2508.09624].

A plausible implication is that unified frameworks combining causal structure discovery, latent variable learning, and optimal intervention policies may deliver further advances in solving real-world, multi-stage RL tasks with extreme reward sparsity and environment complexity.

Source: https://www.emergentmind.com/topics/causal-structure-discovery-in-subgoal-space