Hierarchical Policy Learning

• Hierarchical policy learning is a framework that decomposes decision making into high-level selectors and low-level controllers, enabling efficient handling of complex tasks.
• It leverages structured interfaces between abstraction levels to address sparse rewards, multi-modal returns, and transfer across domains such as robotics and dialogue systems.
• Recent advances focus on techniques like option discovery, mutual information maximization, and temporal grammars to align policies with structured control objectives.

Searching arXiv for recent and foundational papers on hierarchical policy learning to ground the article with current literature. Hierarchical policy learning denotes a family of policy-learning methods in which decision making is decomposed across multiple abstraction levels, temporal resolutions, or latent control variables. In the surveyed formulations, a high-level policy selects an option, subgoal, skill, latent intention, or procedure, and a low-level policy realizes that choice by emitting primitive actions, continuous controls, or shorter-horizon trajectories. This decomposition appears in hierarchical reinforcement learning, hierarchical imitation learning, multimodal sensor fusion, dialogue management, and long-horizon robot manipulation, where it is used to address sparse rewards, multi-modal return structure, contact-rich feedback, transfer across tasks, and interpretable control (Osa et al., 2019, Ha et al., 2020, Fox et al., 2019).

1. Formal structure and canonical decompositions

A recurring formal pattern is a high-level selector coupled to one or more low-level controllers. In advantage-weighted option learning, a discrete latent option variable $z\in\{1,\dots,O\}$ yields a factorization

$\pi(a\mid s)=\sum_{z=1}^{O}\pi(z\mid s)\mu(s,z),$

where $\pi(z\mid s)$ is a gating policy and $\mu(s,z)$ is a deterministic option policy (Osa et al., 2019). In master–subgoal formulations, the hierarchy is expressed through a goal space $\mathcal{G}$: a master policy emits a subgoal, and a sub-policy acts for a fixed horizon of primitive steps conditioned on that subgoal (Dwiel et al., 2019). In multi-objective settings, the same structural idea appears as

$$\pi(a\mid s)=\sum_{\eta\in U}\pi^H(\eta\mid s)\pi^L(a\mid s,\eta),$$

where $U$ is a generic skill set and the high-level policy chooses which skill to invoke (Abdelfattah et al., 2023).

Other formulations make the interface between levels more explicit. DISH uses a two-level hierarchy in which a planning module reasons over low-dimensional latent intentions and a shared feedback control policy executes the inferred intention at a finer time scale (Ha et al., 2020). Parameterized Hierarchical Procedures model control as a program-like stack of procedures, where each procedure can call a sub-procedure, take an environment action, or terminate (Fox et al., 2019). In multi-task skill acquisition, the hierarchy comprises a switch policy, an instruction policy over previously learned tasks, an augmented flat policy, and a base policy, so that the agent can either reuse a prior skill or act directly (Shu et al., 2017).

These variants differ in the representation of the high-level action, but they share the same functional purpose: to reduce a difficult control problem into a sequence of shorter-horizon decisions with structured interfaces between levels.

Formulation	High-level output	Representative work
Gating–option policy	Discrete option $z$	adInfoHRL (Osa et al., 2019)
Master–subgoal policy	Goal in $\mathcal{G}$	Goal-space HAC analysis (Dwiel et al., 2019)
Planner–controller split	Latent intention / command	DISH (Ha et al., 2020)
Program-like hierarchy	Procedure call / terminate	HVIL (Fox et al., 2019)
Language-mediated reuse	Instruction to prior skill	Multi-task skill acquisition (Shu et al., 2017)

A plausible implication is that “hierarchical policy learning” is best understood not as a single algorithmic family, but as a design principle for specifying interfaces between decision layers.

2. Learning objectives, credit assignment, and optimization

The literature uses heterogeneous optimization principles, reflecting the fact that different hierarchy levels often solve different problems. In adInfoHRL, the objective is to learn option policies that correspond to modes of the advantage function by maximizing mutual information between the latent option and the state-action pair under an advantage-weighted sampling distribution (Osa et al., 2019). The gating policy is selected by a softmax over option-values, while option policies are optimized by deterministic policy gradient. This makes the hierarchy explicitly responsive to multi-modal advantage structure.

In imitation learning, Directed-Info GAIL replaces ordinary mutual information with directed information from trajectories to latent sub-tasks. The resulting saddle-point objective augments the GAIL loss with a variational lower bound on directed information, enabling unsupervised segmentation of unsegmented demonstrations and learning of sub-task policies linked to the Options framework (Sharma et al., 2018). HVIL instead maximizes an ELBO over latent PHP step sequences, using a bidirectional RNN in the inference network so that inference can exploit acausal information from the full demonstration trace (Fox et al., 2019). In that setting, future observations help infer when higher-level procedures begin or end.

For joint optimization of manager and sub-policies, HiPPO derives a hierarchical policy gradient with an unbiased latent-dependent baseline and combines it with a clipped PPO-style surrogate at both hierarchy levels (Li et al., 2019). The same work introduces randomized time abstractions, where the manager’s period is sampled from a categorical distribution, so that skills become more robust to timing changes. In cluttered manipulation, HCLM uses Q-learning but supplements it with a Spatially Extended Q-update for the push option and a Two-Stage Update Scheme to alleviate the non-stationary transition problem in updating the high-level policy (Wang et al., 2023).

A separate line addresses sparse external rewards by separating intrinsic and extrinsic optimization targets. HIEM defines an extrinsic reward for finding the specified target object and an intrinsic reward for reaching a subgoal such as approaching a visible object; a proxy low-level policy is first driven by the intrinsic reward, and the hierarchy is then learned from that exploration experience toward the extrinsic goal (Ye et al., 2020). This establishes a common pattern in hierarchical learning: the lower level is often optimized with denser or more local objectives than the upper level.

3. Option discovery, subgoal discovery, and latent structure

A central question is how to identify the hierarchical variables themselves. One approach treats useful hierarchy as a latent decomposition of a multi-modal return or advantage landscape. HPSDE casts option discovery as return-weighted density estimation, fits a mixture model to weighted samples, and automatically determines the number and location of option policies via variational Bayes EM (Osa et al., 2017). adInfoHRL similarly seeks options aligned with modes of the advantage function through advantage-weighted information maximization (Osa et al., 2019). Both approaches are explicitly motivated by the failure of unimodal policies to represent distinct high-return strategies.

Another approach discovers task structure from demonstrations. The Subgoal Discovery Network divides successful dialogue trajectories into segments by identifying “hub” states, then uses the discovered subgoals in a two-level reinforcement-learning agent (Tang et al., 2018). Directed-Info GAIL discovers time-varying latent sub-tasks from unsegmented demonstrations and yields segmentations that can later be composed in novel orders (Sharma et al., 2018). HVIL discovers latent sequences of procedure calls and terminations in observation-action traces of teacher demonstrations, and samples from the learned posterior guide training of the hierarchical control policy (Fox et al., 2019).

A third approach learns low-dimensional latent representations that make high-level planning tractable. DISH uses latent variable models to represent high-dimensional observations with low-dimensional latent variables and reports 3- and 1-dimensional latent states and commands for a humanoid with 197- and 36-dimensional state features and actions (Ha et al., 2020). Because planning is then performed in low-dimensional latent space, the policy can be applied to new tasks without additional training. In multi-task skill acquisition, temporal structure is not inferred solely from trajectories; it is also regularized by a Stochastic Temporal Grammar learned from successful episodes, which injects temporal priors such as “find before pick up” into high-level decisions (Shu et al., 2017).

Taken together, these results suggest that hierarchy can be discovered from at least three signals: return multi-modality, segmentation regularities in demonstrations, and low-dimensional latent dynamics.

4. Multimodal control and domain-specific instantiations

Robotic manipulation provides several distinct instantiations of hierarchical policy learning. In contact-rich assembly, hierarchical integration is used specifically for multimodal fusion: a self-supervised encoder extracts multi-view visual features, a hybrid motion/force controller regulates force behaviors, and reinforcement learning integrates vision, force, and proprioceptive data hierarchically (Jin et al., 2022). In simulation, the reported system assembled objects with 0.25mm clearance, generalized to widely varied initial configurations and new shapes, and transferred robustly to reality without fine-tuning.

In mechanical search, the task is formulated as a hierarchical POMDP with a search policy, an action-selection policy, and low-level grasp and push policies. Replacing hand-designed components with learned sub-policies increased the success rate of retrieving the target object from less than 32% to nearly 80%, while reducing the computation time for push actions from multiple seconds to less than 10 milliseconds (Zenkri et al., 2022). In densely cluttered long-horizon manipulation, HCLM employs a high-level policy together with three parameterized action primitives—push, pick, and place. Pick and place are trained by behavior cloning; the high-level policy and push option are then trained by hierarchical reinforcement learning. On the reported benchmark, HCLM achieved 87% success on cluttered-stack, 95% on cluttered-align, and 97% on cluttered-block-insertion (Wang et al., 2023).

Hierarchical decomposition is also effective in navigation and search. HIEM separates object search into a high-level subgoal policy and a low-level action policy, with subgoals of the form “approach obj_i” for visible objects or “random.” On a single-house evaluation, the method achieved SR=1.00, AS=41.2, SPL=0.72, and AR=0.70, compared with SR=0.74, AS=182.2, SPL=0.17, and AR=0.23 for h-DQN (Ye et al., 2020). A later navigation framework decomposes planning into a sparse global connectivity graph and local neural time fields trained with physics-informed losses derived from the Eikonal PDE, reporting planning times of 0.06–0.15 s and success rates greater than 90% in scenes with 6–42 rooms (Chen et al., 1 Oct 2025).

Dialogue and recommendation systems instantiate the same principle with different action semantics. In task-oriented visual dialog, a high-level Double DQN decides whether to ask a question or make a guess, while a low-level DRRN chooses the question conditioned on a multimodal state representation; the best reported configuration reaches a 76.3% win rate (Zhang et al., 2018). In conversational recommendation, CoCHPL treats “ask” and “recommend” as options, with intra-option policies generating chains of attributes or items. On LastFM, the reported Success Rate SR@15 improves from 0.942 to 0.991, Average Turns decreases to 3.75 from 5.44, and hierarchical DCG reaches 0.493 (Fan et al., 2023).

5. Transfer, robustness, and inductive bias

A major justification for hierarchy is transfer across tasks, preferences, or environmental shifts. RHPO makes this bias explicit by constraining low-level controllers to be task-agnostic while only the high-level switching policy sees the task identity. In sequential transfer, reusing low-level components and training only a new high-level policy yields up to 5× fewer episodes than training from scratch, and in a week-long physical Sawyer stacking experiment RHPO obtains up to 3× higher final success rate than the best baseline (Wulfmeier et al., 2019). DISH reaches transfer through a different mechanism: because the planner operates in a learned low-dimensional latent space, solving a new task amounts to replanning with a new reward rather than retraining the policy (Ha et al., 2020).

The design of the interface between hierarchy levels is also decisive. A controlled study of goal-space design shows that rotation of ground-truth goal spaces and noise had no effect, while having additional unnecessary factors significantly impaired learning in hierarchical models (Dwiel et al., 2019). In the Fetch Reach task, baseline HAC converged in approximately 10 epochs, whereas HAC with one extra zero dimension failed to converge in 100 epochs in all trials. This directly challenges the common assumption that any richer latent goal representation is beneficial.

Recent work strengthens the interface further with symmetry-aware design. HEP introduces a frame transfer interface in which the high-level output is used as a coordinate frame for the low-level agent and imposes equivariance under the group $G=T(3)\times SO(2)$ (Zhao et al., 9 Feb 2025). In RLBench, HEP outperforms Chained Diffuser in 28/30 open-loop tasks with an average +10% absolute gain, and in a one-shot generalization test it succeeds 80% versus 5% for Chained Diffuser. A plausible implication is that transfer in hierarchical systems depends not only on temporal abstraction, but also on whether the inter-level representation preserves task-relevant invariances.

Robustness under non-stationarity has also been studied in multi-objective settings. A dual-phase intrinsically motivated method first learns a generic skill set and then uses it to bootstrap policy coverage sets after shifts in environment dynamics; in non-stationary scenarios, the reported gains are 30–65% in median return and 40–55% in hypervolume relative to the baselines OLS, TLO, and IM-MORL (Abdelfattah et al., 2023). Here the hierarchy serves as a mechanism for continuous adaptation rather than merely faster learning in a stationary MDP.

6. Interpretability, misconceptions, and open directions

Hierarchical policy learning is frequently associated with interpretability, but the form of interpretability varies. In multi-task skill acquisition, every task and subtask is encoded as a human instruction such as “Find blue” or “Get red,” so the instruction policy can literally reveal which previously learned skill is being invoked (Shu et al., 2017). In dialogue, SDN-discovered subgoals are reported to be often human comprehensible, and in object search the subgoal space “approach any visible object” yields trajectories naturally partitioned into interpretable segments (Tang et al., 2018, Ye et al., 2020). In continuous control, option activations can align with distinct motion phases such as stance versus swing in Walker2d (Osa et al., 2019).

Several misconceptions are corrected by the empirical record. Perfect disentanglement is not necessary for effective hierarchy: rotational entanglement of the goal space is harmless in the reported HAC experiments, and small to moderate noise is tolerated, but irrelevant dimensions are highly detrimental (Dwiel et al., 2019). Hierarchy is also not uniformly advantageous. In adInfoHRL, improvements over TD3 and PPO are pronounced on Walker2d and Ant, but smaller on HalfCheetah and Hopper, which exhibit simpler, nearly unimodal advantage structure (Osa et al., 2019). In RHPO, single-task gains over flat control appear only when low-level mixture components are initialized with distinct means (Wulfmeier et al., 2019). This suggests that hierarchy is most useful when the task genuinely contains reusable temporal, modal, or structural subproblems.

The limitations identified across the literature are similarly diverse. Some methods rely on a hand-specified curriculum and human-provided task descriptions, or on a simplistic two-token instruction template and first-order temporal grammar (Shu et al., 2017). Others depend on the quality of a VAE pre-training phase or on the choice of the number of latent codes $\pi(a\mid s)=\sum_{z=1}^{O}\pi(z\mid s)\mu(s,z),$0 (Sharma et al., 2018). HPSDE’s GP-based gating scales poorly to high-dimensional contexts, while HiPPO still uses fixed or randomly sampled durations rather than learned terminations and assumes a discrete latent space (Osa et al., 2017, Li et al., 2019). In learned goal spaces, retaining spurious latent dimensions can prevent convergence altogether (Dwiel et al., 2019).

The open directions stated in the cited work are correspondingly concrete: automatically discovering task-sequencing curricula; inducing richer temporal grammars; learning termination policies or duration distributions end-to-end; extending to continuous latent codes or deeper hierarchies; replacing fixed upper bounds on option counts with nonparametric mixtures; extending frame transfer from translations to full $\pi(a\mid s)=\sum_{z=1}^{O}\pi(z\mid s)\mu(s,z),$1 subgoals; and incorporating memory for tasks that require history (Shu et al., 2017, Li et al., 2019, Osa et al., 2017, Zhao et al., 9 Feb 2025). The aggregate trajectory of the field points toward hierarchies whose interfaces are simultaneously learned, structurally constrained, and matched to the geometry of the underlying domain.