---
title: Hierarchical Policy for Cluttered Manipulation
url: https://www.emergentmind.com/topics/hierarchical-policy-for-cluttered-scene-long-horizon-manipulation-hclm
type: topic
---

# Hierarchical Policy for Cluttered Manipulation

Hierarchical Policy for Cluttered-scene Long-horizon Manipulation (HCLM) refers to a vision-based framework for solving complex multi-step manipulation tasks in environments characterized by severe object occlusion and clutter. This approach leverages hierarchical reinforcement learning (HRL) and option-based policy decomposition to instantiate and sequence primitives such as push, pick, and place, enabling agents to generalize to diverse clutter densities and long task horizons [2312.02697].

## 1. Formal Policy Decomposition and Option Structure

HCLM implements a dual-level policy: a high-level discrete planner and a set of option modules corresponding to parameterized manipulation primitives. The central agent operates on an orthographic RGB-D observation \( o_t \in \mathbb{R}^{H \times W \times 4} \) at each step, producing the tuple \( a_t = (a_t^H, a_t^L) \) where:

- \( a_t^H \in \{\text{push}, \text{pick+place}\} \) indicates the primitive selection, output via Q-value maximization \( a^H_t = \arg\max_{a^H} Q_H(o_t, a^H) \).
- \( a_t^L = (x, y, \theta) \) specifies spatial parameters determined by the corresponding option module.

Push is parameterized by 2D pixel location and one of 12 discrete directions, pick utilizes a Transporter-based Q-map for location/angle selection, and place elaborates over 36 discrete placement orientations.

## 2. Training Paradigm: Behavior Cloning and Hierarchical RL

The pick and place sub-policies (options) are trained independently using supervised behavior cloning (BC) on data \( D = \{(o_j, a_j)\} \) sampled from oracle demonstrations. Training minimizes the cross-entropy loss
\[
L_{\text{BC}} = -\mathbb{E}_{(o,a) \sim D} [Y \cdot \log \text{softmax}\, Q(o)],
\]
where Y is a one-hot label expansion corresponding to ground-truth actions.

The high-level policy and push option are then trained jointly via hierarchical RL. Exploration is managed through ε-greedy scheduling with PER. Special algorithmic innovations are introduced:

- **Spatially Extended Q-Update (SEQ):** For push, a learned Q-map incorporates an anisotropic Gaussian spatial filter in the local action frame. The target is
\[
Y_t^u = R_t \cdot \text{Filter} + \gamma \eta Q_u(o_{t'}, \arg\max_{a_{t'}} Q_u(o_{t'}, a_{t'})),
\]
leveraging only successful transitions for Q-value bootstrapping.

- **Two-Stage Update Scheme (TSUS):** High-level Q-values are updated with adaptive weighting depending on the epoch and the origin (random/policy) of the push action, suppressing non-stationarity in HRL optimization.

## 3. Mathematical Agent Specification

The HCLM agent is formally defined as an MDP \( (\mathcal{S}, \mathcal{A}, P, R, \gamma) \) with:

- State: RGB-D top-down image,
- Action: tuple as above,
- Reward: stepwise task progression (STP) function,
\[
R_t = 
\begin{cases}
\Delta_{\text{task\_progress}}, & \text{if } \Delta_{\text{task\_progress}} < 0,\\
\mathcal{W}(a_t^H) \mathcal{I}(a_t, s_{t+1}), & \text{otherwise},
\end{cases}
\]
where \( \mathcal{W}(a_t^H) \) weights by primitive, and \( \mathcal{I}(a_t, s_{t+1}) \) is a success indicator.

- High-level and low-level Q-maps are trained with Huber losses on temporal-difference errors.

## 4. Network Architectures and Implementation

The "Dual-Level Action Network" (DLAN) serves as the backbone for joint high-level and push option inference:

- **Encoders:** Two streams (RGB via frozen CLIP ResNet50; Depth via custom CNN) fuse spatial features for downstream convolutional decoding.
- **Push option:** Up-convolution layers reconstruct dense Q-maps.
- **High-level branch:** Fused features traverse conv + FC layers to produce Q-values over \(\{\text{push}, \text{pick}+ \text{place}\}\).
- **Pick/place transporters:** Share encoder architecture but omit the high-level branch, utilizing key-query attention and dense Q-maps.

All input images are normalized and rotated per action angle before processing, preserving geometric invariance.

## 5. Experimental Protocol and Results

Evaluation occurs on the ClutteredRavens benchmark (PyBullet, UR5e + suction gripper), across six manipulation scenarios with increasing clutter:

- **Metrics:** Success rate (within maximum steps) and episode length.
- **Baselines:** Two-stream Transporter (pick+place-only), alternating push policies, RoManNet (RL for GR-ConvNets).
- **Quantitative Outcomes:** HCLM repeatedly outperforms all baselines, e.g., in "cluttered-block-insertion," HCLM achieves 97% success vs. strongest baseline at 56%. Efficiency (average episode length) is improved by 10–30% in the majority of tasks.

Robustness to increased clutter is demonstrated: in "cluttered-stack-block-pyramid," HCLM maintains ≥70% success even with 16 additional blocking objects, while best baseline performance does not exceed 34%.

Ablation studies confirm contributions:
- No hierarchical policy: complete failure (0%).
- No TSUS: 52% success.
- No SEQ: 79%.

## 6. Design Innovations: SEQ and TSUS

The SEQ mechanism spreads Q-learning targets for push over a local affine region centered on the selected pixel and direction, improving spatial credit assignment. Adjacent direction slices receive a degraded bootstrapped target via factor \(\kappa < 1\), enabling broader reward propagation and sample efficiency.

TSUS addresses instability inherent to HRL by dynamically selecting which training samples contribute to the high-level policy loss, suppressing the detrimental influence of non-policy (random) explorations early in training, then gradually integrating these as training progresses.

## 7. Generalization, Limitations, and Future Perspectives

HCLM demonstrates strong generalization to unseen clutter densities and longer manipulation horizons. Its architecture supports efficient option re-use and rapid inference. However, current formulations rely on frozen pick/place policies, limiting credit assignment across options. Furthermore, learning remains sensitive to exploration schedules and visual normalization heuristics. Prospective enhancements could include end-to-end hierarchical credit propagation, vision-based domain randomization, and adaptive curriculum strategies.

HCLM sets a precedent for hierarchical visual policy learning in cluttered, long-horizon manipulation, combining option-based policy structure with novel spatial and temporal update mechanisms, and achieves state-of-the-art empirical results across diverse task benchmarks [2312.02697].

Source: https://www.emergentmind.com/topics/hierarchical-policy-for-cluttered-scene-long-horizon-manipulation-hclm