Hierarchical Subgoal Tree Formalism

• Hierarchical subgoal tree formalism is defined as a recursive tree structure where nodes represent goals and subgoals with explicit decomposition relations.
• It employs diverse algorithms—model-based, language model synthesis, and statistical causal discovery—to recursively expand and terminate task hierarchies.
• The approach enhances planning efficiency and interpretability in applications like hierarchical reinforcement learning, embodied robotics, and modular policy synthesis.

A hierarchical subgoal tree formalism is a family of data structures and associated recursive algorithms for representing, discovering, and exploiting hierarchical decompositions of complex, goal-directed tasks. In this paradigm, an agent, planner, or learning system explicitly structures the search or planning space as a directed tree in which nodes correspond to goals or subgoals, and edges encode relations of decomposition, causality, or composition. Subgoal trees are central in hierarchical reinforcement learning, interpretable policy synthesis, causal task discovery, modular design, and embodied long-horizon planning. The formalism generalizes classical sequential planning by supporting divide-and-conquer, parallel execution, abstraction, modularity, and targeted exploration. Leading research on arXiv offers multiple complementary formalizations spanning dynamic programming, causal graph discovery, natural language decomposition, and evolutionary modularity (Sharma et al., 4 Feb 2025, Tianxing et al., 26 Jun 2025, Demin et al., 2022, Czégel, 30 Apr 2025, Choi et al., 4 Nov 2025, Jurgenson et al., 2019, Khorasani et al., 6 Jul 2025).

1. Formal Definitions and General Structure

A hierarchical subgoal tree is typically formalized as a rooted, directed tree $T = (V, E)$, where:

• $V$ is the set of nodes, each encoding an abstract goal, subgoal, or decision variable.
• $E \subseteq V \times V$ is the set of edges, with $(u \to v) \in E$ signifying $u$ is a subgoal or child component of $v$.
• A level or resolution function $\ell: V \to \mathbb{N}$ may assign abstraction depth.

Specializations include:

• Binary or $k$-ary trees: DHP (Sharma et al., 4 Feb 2025) and Sub-Goal Trees (Jurgenson et al., 2019) use binary splitting; goal assembly (Czégel, 30 Apr 2025) supports arbitrary fan-out.
• Predicate or Policy Trees: Nodes can be state-pair transitions (Sharma et al., 4 Feb 2025, Jurgenson et al., 2019), sensor predicates (Demin et al., 2022), or natural-language instructions (Tianxing et al., 26 Jun 2025, Choi et al., 4 Nov 2025).
• Causal Subgoal Trees: Nodes correspond to binary achievement variables $X_i^t$ for subgoal $g_i$, and tree edges encode causal dependencies (arborescence) (Khorasani et al., 6 Jul 2025).

This structure accommodates task decomposition (from root to leaves) and plan synthesis or credit assignment (leaves to root). Edges may be labeled by composition operators $\theta$ for modular aggregation (Czégel, 30 Apr 2025) or control-flow types in execution trees (Choi et al., 4 Nov 2025).

2. Subgoal Discovery and Decomposition Algorithms

Discovery and construction of hierarchical subgoal trees proceed recursively through task decomposition or abstraction mechanisms:

• Model-based Decomposition: DHP recursively applies a learned manager policy $\pi_\theta(s_t, s_g)$, splitting $(s_t, s_g)$ into $(s_t, s_{\text{mid}})$ and $(s_{\text{mid}}, s_g)$ until terminal conditions (max depth or reachability) (Sharma et al., 4 Feb 2025).
• LLM-based Synthesis: The STEP Planner uses a subgoal decomposition operator $f_{dec}: G \times S \rightarrow 2^G$, instantiated via LLMs, recursively generating child subgoals until a termination test $f_{term}(g, s)$ affirms primitive executability (Tianxing et al., 26 Jun 2025).
• Statistical Causal Discovery (SSD): Hierarchical RL with Targeted Causal Interventions fits a linear-threshold model to environment variables, using LASSO/L0-penalized regression to recover a minimal parent set for each subgoal, constructing the minimal arborescence (Khorasani et al., 6 Jul 2025).
• Frequentist Policy Discovery: Interpretable RL with multilevel subgoal discovery boosts policy fitness by inventing predicates representing intermediate states occurring in high-fitness trajectories, updating the subgoal tree accordingly (Demin et al., 2022).

In all frameworks, recursive expansion is bounded by maximal depth, abstraction level, or explicit termination criteria. Generated trees are guaranteed to terminate (maximum level $L$ or leaves correspond to primitive skills).

3. Dynamic Programming, Value Estimation, and Complexity Analysis

Hierarchical subgoal tree formalisms induce unique dynamic programming or value-backpropagation principles:

• Sub-Goal Tree Dynamic Programming (STDP): The cost-to-go $V_k(s, g)$ for horizon $2^k$ satisfies $$V_k(s, g) = \min_m \left[V_{k-1}(s, m) + V_{k-1}(m, g)\right]$$, contrasting with one-step Bellman updates and enabling divide-and-conquer DP (Jurgenson et al., 2019).
• Tree-Return Min-Discount Recursion: DHP defines $$G_i = \min (R_{2i+1}+\gamma G_{2i+1}, R_{2i+2}+\gamma G_{2i+2})$$, establishing a Bellman-like contraction mapping and supporting $\lambda$-bootstrapped value propagation across tree depth (Sharma et al., 4 Feb 2025).
• Goal-State Gradient Backpropagation: Goal assembly propagates fitness gradients through hierarchical compositions $\theta_{u \to v}$, enabling differentiable credit assignment over modular trees (Czégel, 30 Apr 2025).
• Branching Complexity: For binary trees of depth $D$, total nodes scale as $2^{D+1}-1$; with task horizon $N$, planning complexity is $O(\log N)$ (Sharma et al., 4 Feb 2025). General $k$-ary trees yield at most $\prod_{i=0}^{L-1} b_i$ nodes at depth $L$ (Tianxing et al., 26 Jun 2025).
• Causal Tree Training Cost: Targeted interventions in a tree-structured causal subgoal graph yield $O(\log^2 n \cdot b)$ training cost versus $\Omega(n^2 b)$ for random subgoal selection (Khorasani et al., 6 Jul 2025).

4. Control Flow, Execution, and Memory-Augmented Trees

Execution over subgoal trees involves combining agent policies or skills under various control-flow semantics:

• Hierarchical LLM Agent Trees: ReAcTree introduces agent nodes (each planning a subgoal) and typed control-flow nodes (sequence, fallback, parallel) (Choi et al., 4 Nov 2025). Nodes may expand themselves by emitting decompositions; control-flow types dictate execution strategy.
• Memory Systems: Episodic memory at subgoal-level enables in-context retrieval of previous trajectories with sentence embeddings; working memory persists persistent environment facts (e.g., object locations) (Choi et al., 4 Nov 2025).
• Leaf-Node Termination and Soundness: STEP uses a Boolean function $f_{term}$ to ascertain if a node’s subgoal maps to a primitive action, ensuring that concatenation of leaves forms a valid action sequence (Tianxing et al., 26 Jun 2025).

These hybrid control/data structures support both depth-first and breadth-first execution, as well as parallel or majority-vote aggregation of child subgoal results.

5. Causal Structure, Targeted Interventions, and Modular Design

Subgoal tree formalism admits a causal interpretation:

• Tree-Structured Causal Graphs: Subgoal spaces $\mathcal{V}$ and edges $\mathcal{E}$ define a Directed Acyclic Graph (DAG), with a unique root and in-degree at most one (arborescence). Each subgoal is a binary variable $X_i^t$ whose achievement is a deterministic or stochastic AND/OR function of its parents and agent actions (Khorasani et al., 6 Jul 2025).
• Sparse Structure Discovery (SSD): Solving for minimal parent sets via LASSO or L0 minimization robustly recovers the tree skeleton and causal dependencies between subgoals.
• Targeted Causal Interventions: Ranking subgoals by causal effect or A*-style shortest path ensures only ancestors of the final goal are explored, leading to provable reductions in sample and training complexity (Khorasani et al., 6 Jul 2025).
• Recombination and Evolvability: Goal assembly defines explicit subtree-swap operations, enabling modular recombination and robust inheritance of competencies, supporting claims for biological and engineering evolvability (Czégel, 30 Apr 2025).

A plausible implication is that the interplay of causal structure discovery and modular recombinability provides the foundations for sampling-efficient, robust, and generalizable HRL architectures.

6. Applications, Empirical Evidence, and Limitations

Hierarchical subgoal tree methods are evaluated in diverse domains:

• Long-Horizon Planning: DHP and Sub-Goal Trees show $100\%$ and significant improvements over baselines in sequential navigation and trajectory planning, with dramatic reductions in step count (Sharma et al., 4 Feb 2025, Jurgenson et al., 2019).
• Embodied Task Decomposition: STEP and ReAcTree yield state-of-the-art long-horizon completion rates in VirtualHome WAH-NL, ALFRED, and real-robot settings (e.g., $61\%$ goal success rate for ReAcTree with Qwen 2.5 72B) (Tianxing et al., 26 Jun 2025, Choi et al., 4 Nov 2025).
• Efficient Subgoal Discovery: Targeted causal interventions in HRL achieve order-of-magnitude lower training cost than prior approaches (e.g., 10x speed-up vs. CDHRL) (Khorasani et al., 6 Jul 2025).
• Interpretable RL and Policy Synthesis: Multilevel subgoal tree discovery yields incrementally interpretable, high-fitness composed policies by iteratively inventing intermediate state predicates (Demin et al., 2022).
• Power-Law Modular Biases: Goal assembly demonstrates that tree-structured composition induces heavy-tailed representational degeneracy, a key property for evolvable engineering systems (Czégel, 30 Apr 2025).

Limitations include challenges in subgoal predictor learning for high-dimensional spaces (Jurgenson et al., 2019), binary tree rigidity (mitigated by $k$-ary trees), and stochastic propagation variance. Open issues also remain concerning automated structural discovery in continuous domains and integrated actor-critic frameworks.

7. Comparison of Major Formalisms

Paper	Tree Structure	Node Type	Core Decomposition Principle
DHP (Sharma et al., 4 Feb 2025)	Rooted binary tree, S × S nodes	State-pair	Manager recursively splits by reachability
STEP (Tianxing et al., 26 Jun 2025)	Directed tree, depth-indexed	NL subgoal/action	LLM decomposition, f_dec/f_term
InterpRL (Demin et al., 2022)	Partial order, predicate chain	Predicate/state	Invention of intermediate predicates
Goal Assembly (Czégel, 30 Apr 2025)	Labeled directed tree, levels	Goal variable set	Compositional operators, aggregation
ReAcTree (Choi et al., 4 Nov 2025)	Ordered tree with control/agent nodes	Goal+ControlFlow	Dynamic LLM expansion, flow types
Sub-Goal Trees (Jurgenson et al., 2019)	Binary tree of state-pairs	State sub-trajectory	Recursive midpoint prediction
HRC (Khorasani et al., 6 Jul 2025)	Causal/arborescence	Subgoal variable	SSD (sparse regression), effect/shortest-path intervention

Each formalism emphasizes mechanisms (recursion, abstraction, composition, causal effect, episodic recall) adapted to the target application and theoretical context.

---

In summary, hierarchical subgoal tree formalism provides an explicit, recursively constructed tree structure for compositional task decomposition, value propagation, causal discovery, and modular action selection in goal-driven learning and planning systems. Its strengths lie in its expressiveness, efficiency, and close alignment with the modular and hierarchical organization of both engineered and biological control systems.