Structured Goal-guided Reinforcement Learning (SGRL)

• SGRL is a branch of goal-conditioned reinforcement learning that endows goals with explicit structure, such as recursive subgoal trees, automata, and latent factors.
• It employs techniques like probabilistic curricula, symbolic specifications, and graph-based planning to improve exploration, sample efficiency, and long-horizon control.
• By decomposing complex tasks into structured subproblems, SGRL unifies planning with reinforcement learning, enabling more effective and compositional behavior.

Searching arXiv for the cited papers and topic to ground the article in current literature. Structured Goal-guided Reinforcement Learning can be understood as an umbrella term for goal-conditioned reinforcement learning in which the goal variable is endowed with explicit internal structure rather than treated as a flat target vector. In the literature considered here, that structure appears as recursive sub-goal trees, temporal-logic and automaton states, graph or landmark abstractions, probabilistic curricula over goal space, structured latent factors, adaptive skill distributions, and stage-wise keyframes extracted from demonstrations. The common substrate remains the goal-augmented MDP with goal space $\mathcal{G}$, goal distribution $p_g$, a state-to-goal mapping $\phi:\mathcal{S}\to\mathcal{G}$, and a goal-conditioned policy $\pi(a\mid s,g)$, but SGRL adds explicit organization over goals, trajectories, or task progress to improve long-horizon control, exploration, and sample efficiency (Liu et al., 2022, Jurgenson et al., 2020, Zhao et al., 2021, Salt et al., 2 Apr 2025).

1. Formal scope and foundational abstractions

A useful starting point is the goal-augmented MDP formalism surveyed in goal-conditioned RL. Standard RL is posed on an MDP $$\mathcal{M}=\langle \mathcal{S},\mathcal{A},\mathcal{T},r,\gamma,\rho_0\rangle$$, while goal-conditioned RL augments it with $\langle \mathcal{G},p_g,\phi\rangle$, where $\mathcal{G}$ is a goal space, $p_g$ is a goal distribution, and $\phi:\mathcal{S}\to\mathcal{G}$ maps states to achieved-goal representations. The resulting policy is $\pi(a\mid s,g)$, and the reward becomes goal-dependent, $p_g$0 (Liu et al., 2022).

The same survey distinguishes desired goals, achieved goals, and behavioral goals. That distinction is especially important for SGRL because structured systems often replace a single desired goal with intermediate behavioral goals such as subgoals, waypoints, automaton states, or latent landmarks. In the standard sparse formulation, success is frequently written as a binary criterion such as

$p_g$1

but the broader literature considered here repeatedly replaces this flat threshold with structured satisfaction tests: recursive path decompositions, logical predicates, graph transitions, latent-space similarity, or curriculum-defined attainability bands (Liu et al., 2022).

This suggests a concise characterization of SGRL: it is not a single algorithmic family but a design pattern in which the goal space is organized by additional semantics. Those semantics may be temporal, geometric, symbolic, topological, probabilistic, or representational. What changes across formulations is not the existence of a goal, but the way goals are decomposed, ordered, or embedded into the learning process.

2. Principal structural regimes

Across the cited literature, SGRL spans several recurrent structural regimes. The table summarizes the main ones.

Structural regime	Representative work	Structural object
Recursive decomposition	"Sub-Goal Trees -- a Framework for Goal-Based Reinforcement Learning" (Jurgenson et al., 2020)	Intermediate-state binary tree
Symbolic or automaton-guided control	"Reinforcement Learning Agent Training with Goals for Real World Tasks" (Zhao et al., 2021); LSTS (Shukla et al., 2024)	Temporal-logic formula, automaton, DAG
Probabilistic curriculum	"Probabilistic Curriculum Learning for Goal-Based Reinforcement Learning" (Salt et al., 2 Apr 2025)	Density-ranked goal space
Graph and landmark planning	SG-RL (Zeng et al., 2018); SFL (Hoang et al., 2021); ProQ (Kobanda et al., 23 Jun 2025)	Subgoal graph, landmark graph, keypoint graph
Structured latent goals	DR-GRL (Qian et al., 2022)	Disentangled latent factors
Hierarchical future-state subgoals	NF-HIQL (Garg et al., 11 Feb 2026)	Two-level future-state hierarchy
Stage-structured rewards from demonstrations	Keyframe-guided rewards (Qiu et al., 28 Feb 2026)	Ordered keyframe sequence

These regimes are not mutually exclusive. A graph method may also be hierarchical; a logic-guided method may induce a curriculum; a latent representation may provide the geometry used by a planner. What unifies them is that they do more than condition a flat policy on a target vector. They introduce an explicit progression variable, subgoal object, or structural prior that shapes what is learned, what is sampled, and how long-horizon behavior is composed.

A plausible synthesis is that SGRL occupies the space between plain universal value-function approximation and full task-specific planning systems. It inherits the multi-goal generalization ambitions of goal-conditioned RL, but adds enough structure to make sparse, long-horizon, or compositional tasks tractable.

3. Recursive and hierarchical decompositions

One of the clearest formalizations of structured goal guidance is the sub-goal tree. In "Sub-Goal Trees -- a Framework for Goal-Based Reinforcement Learning" (Jurgenson et al., 2020), the basic control object is not the next action or next state, but an intermediate state $p_g$2 that splits a trajectory segment $p_g$3 into two subproblems. The associated dynamic program is

$p_g$4

where $p_g$5 is the shortest-path cost from $p_g$6 to $p_g$7 in $p_g$8 steps or less. The recovered trajectory is therefore a binary recursive partition rather than a stepwise chain, and the paper proves approximation-error bounds for this recursion in approximate dynamic programming (Jurgenson et al., 2020).

The same paper extends policy-gradient methodology to recursively predicted subgoals. A stochastic sub-goal-tree policy models $p_g$9, a distribution over intermediate states conditioned on segment endpoints, and the score-function estimator is indexed by tree depth and segment rather than by environment timestep. This is a particularly explicit SGRL formulation because the goal determines the decomposition of the whole trajectory, not merely the reward.

A different hierarchical design appears in "Combining Subgoal Graphs with Reinforcement Learning to Build a Rational Pathfinder" (Zeng et al., 2018). There, a high-level planner uses Simple Subgoal Graphs to compute an abstract path, and a low-level LSPI controller realizes feasible motion between adjacent subgoals under robot kinematics and local obstacle uncertainty. The high level supplies a subgoal sequence; the low level solves only local approach and obstacle-avoidance MDPs. This is structurally simpler than the recursive APSP formulation, but it makes the same core move: long-horizon navigation is converted into a sequence of shorter goal-reaching problems.

A modern offline hierarchical variant is "Data-Efficient Hierarchical Goal-Conditioned Reinforcement Learning via Normalizing Flows" (Garg et al., 11 Feb 2026). NF-HIQL uses a two-level hierarchy in which the high-level policy proposes a future state $\phi:\mathcal{S}\to\mathcal{G}$0 as subgoal conditioned on $\phi:\mathcal{S}\to\mathcal{G}$1, while the low-level policy chooses $\phi:\mathcal{S}\to\mathcal{G}$2 conditioned on $\phi:\mathcal{S}\to\mathcal{G}$3. The shared goal-conditioned value $\phi:\mathcal{S}\to\mathcal{G}$4 supervises both levels via

$\phi:\mathcal{S}\to\mathcal{G}$5

$\phi:\mathcal{S}\to\mathcal{G}$6

and both actor policies are parameterized as conditional normalizing flows rather than Gaussians. This suggests a continuous-state version of structured goal decomposition in which future states themselves act as subgoals, but the subgoal distribution is multimodal and learned from offline data.

Taken together, these works show that “hierarchy” in SGRL is not one thing. It may mean recursive path factorization, explicit planner-controller separation, or coarse-to-fine future-state conditioning. In every case, however, the structural gain comes from reducing the effective horizon seen by the low-level learner.

4. Goal-space structuring through curricula, latent factors, and adaptive skills

Not all SGRL methods build explicit hierarchies. "Probabilistic Curriculum Learning for Goal-Based Reinforcement Learning" (Salt et al., 2 Apr 2025) organizes learning directly over the goal space. The paper assumes a mapping $\phi:\mathcal{S}\to\mathcal{G}$7, so goals may be the full state or a projection of state coordinates, and defines a sparse goal reward

$\phi:\mathcal{S}\to\mathcal{G}$8

Its central approximation is

$\phi:\mathcal{S}\to\mathcal{G}$9

so future-state density becomes a proxy for goal attainability. A Mixture Density Network estimates a conditional density over candidate goals, and goals are filtered by density quantiles so that training focuses on neither trivially easy nor implausibly hard targets. This does not create temporally extended options or subpolicies, but it does impose a probabilistic ordering over the goal space.

A representationally structured variant appears in "Weakly Supervised Disentangled Representation for Goal-conditioned Reinforcement Learning" (Qian et al., 2022). DR-GRL learns a latent factorization

$\pi(a\mid s,g)$0

where $\pi(a\mid s,g)$1, $\pi(a\mid s,g)$2, and $\pi(a\mid s,g)$3 encode shape, color, and position respectively. The downstream reward uses only the position subspace,

$\pi(a\mid s,g)$4

and the representation can be recombined to synthesize unseen goals. This is not symbolic or relational structure; rather, it is attribute-structured latent guidance. The paper’s claim is that isolating the controllable factor $\pi(a\mid s,g)$5 improves both sample efficiency and goal generalization.

A third form of structuring is behavior-centric rather than state-centric. "Goal Exploration via Adaptive Skill Distribution for Goal-Conditioned Reinforcement Learning" (Wu et al., 2024) learns a history-conditioned skill distribution $\pi(a\mid s,g)$6 that maximizes local achieved-goal entropy in a contextual horizon. The skill values $\pi(a\mid s,g)$7 encode which temporally extended behavior is useful in the current local structure, and the adaptive distribution is

$\pi(a\mid s,g)$8

Here the “structure” is neither a symbolic graph nor a latent factorization, but reusable local environmental patterns captured through a context-conditioned distribution over skills.

These methods share a family resemblance. They all organize learning over a structured goal manifold, but the organizing principle differs: probability density in PCL, semantic factorization in DR-GRL, and local entropy gain in GEASD. This suggests that SGRL is as much about how goals are ordered and represented as about explicit subgoal hierarchies.

5. Symbolic specifications, automata, and stage-structured rewards

A more explicit symbolic branch of SGRL treats tasks as logical objects. "Reinforcement Learning Agent Training with Goals for Real World Tasks" (Zhao et al., 2021) introduces Inkling Goal Specification, ETLTL, and the Semi-Finite State Predicate Automaton (SFSPA). High-level goals such as reach, drive, avoid, minimize, and maximize are translated into temporal logic, including forms such as

$\pi(a\mid s,g)$9

The resulting SFSPA augments the MDP state with automaton state, and reward is generated from predicate robustness and automaton transitions rather than from hand-coded scalar shaping. This yields a structured goal-guided pipeline in which task semantics are explicit, temporally extended, and compositional.

LSTS pushes this symbolic view toward adaptive curriculum selection. "Logical Specifications-guided Dynamic Task Sampling for Reinforcement Learning Agents" (Shukla et al., 2024) begins from a SPECTRL specification $$\mathcal{M}=\langle \mathcal{S},\mathcal{A},\mathcal{T},r,\gamma,\rho_0\rangle$$0, translates it into a DAG $$\mathcal{M}=\langle \mathcal{S},\mathcal{A},\mathcal{T},r,\gamma,\rho_0\rangle$$1, and turns edges into subtasks. For edge $$\mathcal{M}=\langle \mathcal{S},\mathcal{A},\mathcal{T},r,\gamma,\rho_0\rangle$$2, the subtask is

$$\mathcal{M}=\langle \mathcal{S},\mathcal{A},\mathcal{T},r,\gamma,\rho_0\rangle$$3

A Teacher then dynamically samples among active subtasks using $$\mathcal{M}=\langle \mathcal{S},\mathcal{A},\mathcal{T},r,\gamma,\rho_0\rangle$$4-greedy selection over moving-average task values

$$\mathcal{M}=\langle \mathcal{S},\mathcal{A},\mathcal{T},r,\gamma,\rho_0\rangle$$5

where $$\mathcal{M}=\langle \mathcal{S},\mathcal{A},\mathcal{T},r,\gamma,\rho_0\rangle$$6 is the Student’s average return on task $$\mathcal{M}=\langle \mathcal{S},\mathcal{A},\mathcal{T},r,\gamma,\rho_0\rangle$$7. The result is an automaton-guided subpolicy curriculum that prunes structurally redundant branches.

A visually grounded but still stage-structured formulation appears in "Keyframe-Guided Structured Rewards for Reinforcement Learning in Long-Horizon Laboratory Robotics" (Qiu et al., 28 Feb 2026). Demonstrations are converted into latent turning points by smoothing encoded features and computing

$$\mathcal{M}=\langle \mathcal{S},\mathcal{A},\mathcal{T},r,\gamma,\rho_0\rangle$$8

A progress variable $$\mathcal{M}=\langle \mathcal{S},\mathcal{A},\mathcal{T},r,\gamma,\rho_0\rangle$$9 indexes the current stage target, and stage advancement is triggered when weighted cosine similarity to the current latent target exceeds threshold: $\langle \mathcal{G},p_g,\phi\rangle$0 The reward is then

$\langle \mathcal{G},p_g,\phi\rangle$1

with

$\langle \mathcal{G},p_g,\phi\rangle$2

This is not an automaton in formal language, but it operationalizes sequential subgoal satisfaction through stage-aware reward gating.

These works show the symbolic and stage-structured side of SGRL. Goals are not merely end states; they are executable specifications, progress monitors, or ordered latent milestones. The resulting advantage is not only denser reward, but alignment between learning dynamics and procedural task logic.

6. Graphs, landmarks, and latent-geometric planning

Another major branch of SGRL constructs explicit spatial or topological structure for long-horizon planning. "Successor Feature Landmarks for Long-Horizon Goal-Conditioned Reinforcement Learning" (Hoang et al., 2021) uses successor features

$\langle \mathcal{G},p_g,\phi\rangle$3

to define Successor Feature Similarity,

$\langle \mathcal{G},p_g,\phi\rangle$4

States that are insufficiently similar to existing landmarks are added as new landmarks; repeated transitions between localized landmarks induce graph edges; frontier landmarks with low visitation counts are sampled for exploration; and local goal-conditioned control is derived directly from

$\langle \mathcal{G},p_g,\phi\rangle$5

The graph is therefore not only a planner scaffold but also an exploration scaffold.

"Offline Goal-Conditioned Reinforcement Learning with Projective Quasimetric Planning" (Kobanda et al., 23 Jun 2025) builds a related but more geometric structure. An encoder $\langle \mathcal{G},p_g,\phi\rangle$6 maps states to latent vectors, an asymmetric quasimetric $\langle \mathcal{G},p_g,\phi\rangle$7 is learned via IQE and the QRL objective, and a sparse set of keypoints is optimized by repulsive energy

$\langle \mathcal{G},p_g,\phi\rangle$8

plus an OOD barrier

$\langle \mathcal{G},p_g,\phi\rangle$9

At inference, keypoints and the final goal define a directed graph; Floyd–Warshall yields shortest-path distances; and the next subgoal is chosen by

$\mathcal{G}$0

This is an especially explicit example of SGRL as latent-geometric waypoint planning.

A conceptually related but morphology-focused use of graph structure appears in "Subequivariant Graph Reinforcement Learning in 3D Environments" (Chen et al., 2023). There, the graph is not a topological abstraction of the environment but the agent’s own body graph, and the structured goal is a target-direction vector $\mathcal{G}$1 in 3D locomotion. The SubEquivariant Transformer is designed to respect the subgroup $\mathcal{G}$2 that preserves gravity, thereby using geometric structure to generalize locomotion toward arbitrary directions. In this setting, “structure” primarily means morphological and symmetry structure, but it still functions as goal guidance because the policy is built to steer toward $\mathcal{G}$3 under gravity-aware equivariance.

The common theme is explicit intermediate organization. Whether landmarks are discovered from successor features, learned as quasimetric keypoints, or embedded in a morphology graph, SGRL here acts by inserting a planner-relevant structure between raw state and primitive action.

7. Limits, misconceptions, and terminological ambiguities

A recurrent misconception is to equate all structured goal guidance with hierarchical RL. Several works explicitly resist that identification. PCL is goal-conditioned and curriculum-based but has “no explicit hierarchy, no options, no manager-worker architecture” (Salt et al., 2 Apr 2025). The Inkling–ETLTL–SFSPA framework is structured and temporally explicit, yet it “is not hierarchical RL in the classic options/subpolicy sense” (Zhao et al., 2021). DR-GRL strongly structures the latent goal representation, but it does not provide symbolic task decomposition or multi-step subgoal reasoning (Qian et al., 2022). SGRL therefore includes hierarchy, but is not exhausted by it.

Another recurring issue is the strength of the assumptions behind different forms of structure. Sub-goal Trees assume deterministic systems and leave the stochastic extension open (Jurgenson et al., 2020). ProQ relies on latent geometry, linear interpolation and extrapolation for OOD negatives, and performs less strongly in the teleport setting with stochastic non-local transitions (Kobanda et al., 23 Jun 2025). Keyframe-guided rewards depend on visual-latent similarity and a strictly linear stage order, which may be brittle under reflections, transparency, or branching procedures (Qiu et al., 28 Feb 2026). These are not incidental details; they delimit the regimes in which specific structural priors are effective.

The link between proxy structure and true task difficulty is also often heuristic rather than exact. In PCL, density under a learned conditional future-state model is used as a surrogate for goal attainability and difficulty, but the paper itself notes that the approximation

$\mathcal{G}$4

is intuitive rather than rigorously justified (Salt et al., 2 Apr 2025). Similar caution applies when robustness, novelty, or local entropy are used as stand-ins for true long-horizon competence.

Finally, the acronym SGRL is itself ambiguous in the current literature. "Semifactual Explanations for Reinforcement Learning" uses SGRL to mean Semifactuals Generator for Reinforcement Learning, an explainable-RL framework rather than a goal-guided control paradigm (Gajcin et al., 2024). "Subequivariant Graph Reinforcement Learning in 3D Environments" uses SGRL to denote Subequivariant Graph Reinforcement Learning, a specific morphology-agnostic 3D locomotion framework (Chen et al., 2023). This suggests that “Structured Goal-guided Reinforcement Learning” is best treated as an editorial umbrella over a family of methods rather than as a universally standardized acronym.

In aggregate, the literature supports a broad but technically coherent view. Structured Goal-guided Reinforcement Learning is the part of goal-conditioned RL that injects explicit organization into goals, subgoals, progress states, or trajectory factorizations. Its strongest forms replace flat end-to-end goal pursuit with decompositions that are recursive, logical, graph-based, probabilistic, latent, or stage-wise. Its open problems remain equally clear: stochasticity, compositional generalization, support-aware subgoal generation in high-dimensional spaces, scalable symbolic–continuous integration, and structure discovery that is both expressive and robust.