Hierarchical Policy Networks (HPN)

• Hierarchical Policy Networks are reinforcement learning architectures that decompose overall policies into a high-level gating policy and specialized low-level option controllers for complex tasks.
• Return-weighted density estimation combined with variational Bayesian EM clustering automatically identifies modes and structures option policies, reducing hyperparameter tuning.
• HPNs enhance real-world applications in high-dimensional motion planning, facilitating efficient, scalable, and interpretable control in robotic manipulation and similar complex domains.

A Hierarchical Policy Network (HPN) refers to an architectural approach in reinforcement learning wherein the overall control policy is decomposed into two or more levels, typically featuring a high-level (gating) policy and multiple low-level (option) policies. Each option policy specializes in a distinct behavioral strategy or mode for complex tasks, while the gating policy chooses among these options according to context. This decomposition is especially valuable in domains characterized by multi-modal reward functions, permitting robust and efficient exploration, specialization, and improved credit assignment across the decision horizon.

1. Formal Structure of Hierarchical Policy Networks

Within the HPN framework, the policy decomposition is mathematically expressed as: $$\pi(\tau | s) = \sum_{o \in \mathcal{O}} \pi(o | s) \;\pi(\tau | s, o)$$ Here:

• $s$ is the context or state,
• $\tau$ is the trajectory,
• $\mathcal{O}$ is the set of option indices,
• $\pi(o | s)$ is the gating policy (high-level controller),
• $\pi(\tau | s, o)$ is the option policy associated with option $o$.

The hierarchical policy $\pi$ thus operates in two stages: the gating policy selects an option $o$ given the state $s$, then the chosen option policy generates trajectories according to its specialization. The objective function maximized during learning is: $$J(\pi) = \sum_{o \in \mathcal{O}} \int d(s) \;\pi(o | s) \int \pi(\tau | s, o) R(s, \tau) \;d\tau \;ds$$ Each option’s contribution is weighted by both the gating policy and its expected return.

2. Mode Identification via Return-Weighted Density Estimation

A distinguishing feature of the “Hierarchical Policy Search via Return-Weighted Density Estimation” (HPSDE) method (Osa et al., 2017) is its use of density estimation to partition the trajectory space according to return. The procedure is as follows:

1. The optimal policy $\pi^*$ is postulated as:

$\pi^*(\tau | s) = \frac{f(R(s, \tau))}{Z}$

where $f$ is monotonic and $Z$ is a normalization factor.

2. Importance weights are calculated for samples from the current policy:

$$W(s_i, \tau_i) = \frac{f(R(s_i, \tau_i))}{Z \;\pi_{old}(\tau_i|s_i)}$$

After normalization:

$$\tilde{W}(s_i, \tau_i) = \frac{f(R(s_i, \tau_i))/\pi_{old}(\tau_i|s_i)}{\sum_j f(R(s_j, \tau_j))/\pi_{old}(\tau_j|s_j)}$$

These return-weighted samples are used to fit a mixture model (such as a Gaussian mixture via variational Bayesian EM).

3. This clustering automatically assigns samples to options and determines the number and location of policy components, localizing option policies around modes of the return function.

3. Comparative Features and Algorithmic Trade-offs

HPSDE brings several advantages compared to prior hierarchical RL methods such as HiREPS:

Feature	HiREPS	HPSDE
Mode identification	Manual or heuristic	Automatic via density estimation
Hyperparameter requirements	High	Low; only $O_{max}$ needed
Structure adaptation	Limited	VB-EM prunes unnecessary options
Performance on multimodal reward	Averaging risk	Each option fits a distinct mode

HPSDE’s variational Bayesian EM clustering promotes sparsity, prunes superfluous options, and avoids explicit constraints to prevent mode averaging. Empirical results show improved learning speed and return performance on both benchmark and complex tasks (Osa et al., 2017).

4. Applications in High-Dimensional Motion Planning

A core demonstration involves redundant robotic manipulator control. For a KUKA LWR with seven degrees of freedom and obstacles present, multiple postural solutions exist for a single workspace target. Collision costs are penalized via $R(q) = -d(q) - C(q)$, with $d(q)$ for distance and $C(q)$ for collisions.

HPSDE learns distinct option policies for each feasible maneuver (e.g., circumventing an obstacle left or right), and the gating policy (potentially implemented via Gaussian Processes) selects the most appropriate for the current context. This unsupervised route identification goes beyond classic trajectory optimizers (CHOMP, TrajOpt), which cannot directly choose among redundant end configurations.

5. Hyperparameter Management and Scalability

The approach significantly reduces burdensome tuning requirements. Only the upper bound $O_{max}$, the maximal number of options, is set manually; the actual number is self-selected via variational Bayesian estimation. Importance weighting through $f(R)$ (typically $e^{R}$) and policy updates via off-the-shelf procedures (e.g., REPS, RWR) further contribute to robust learning and initialization insensitivity.

This design supports deployment from toy domains to challenging robotics settings with nontrivial reward topology and high state-action dimensionality.

6. Synthesis and Significance

The HPN, as instantiated in the HPSDE framework, demonstrates how return-weighted density estimation combined with hierarchical decomposition yields:

• Automatic and data-driven structure discovery in multimodal RL settings,
• Robust specialization of low-level controllers,
• Reduction in hyperparameter and structural engineering overhead,
• Empirical gains in return optimization and sample efficiency,
• Practical scalability to real-world problems such as redundant manipulation.

This body of work positions hierarchically decomposed, return-weighted density-driven policies as a viable architecture for scalable, interpretable, and efficient reinforcement learning in multimodal and complex domains.