Hierarchical Entity-Centric RL (HECRL)

• HECRL is a two-level framework that integrates a low-level value-based agent with a high-level conditional diffusion model to decompose complex tasks.
• It generates sparse, entity-factored subgoals that simplify long-horizon, multi-entity manipulation tasks and improve reachability.
• Empirical evaluations demonstrate that HECRL outperforms baseline methods in multi-entity settings and exhibits strong zero-shot generalization.

Hierarchical Entity-Centric Reinforcement Learning (HECRL) is a modular, two-level framework for offline goal-conditioned reinforcement learning (GCRL), designed to address the combinatorial and temporal complexity of long-horizon manipulation tasks in domains populated by multiple interacting entities. HECRL decomposes the global goal into sparse, entity-factorized subgoals and integrates conditional diffusion-based subgoal generation with a value-based GCRL agent, yielding marked performance gains in sparse-reward, high-dimensional domains and enabling robust generalization with respect to increasing numbers and arrangements of entities (Haramati et al., 2 Feb 2026).

1. Two-Level Hierarchical Framework

HECRL employs a compositional structure comprising (1) a low-level value-based GCRL agent and (2) a high-level entity-factored subgoal diffusion generator.

Low-Level GCRL Agent:

The agent operates on the goal-conditioned Markov Decision Process $(S, A, \mu, p, r)$, where the goal space $\mathcal{G} = \mathcal{S}$ and reward is sparse ($r(s, g) = 1_{s=g} - 1$). Its architecture includes:

• A Q-network $Q_\phi(s,a,g)$ trained via offline data.
• An implicit policy $\pi(a|s,g)$ extracted using Deep Deterministic Policy Gradient with Behavioral Cloning (DDPG+BC).
• A value network $V_\psi(s,g) = \max_a Q_\phi(s,a,g)$. The competence radius $R^{V_\pi}$ defines the maximal subgoal distance the agent can reliably traverse without significant temporal-difference (TD) error accumulation.

High-Level Subgoal Diffuser:

This component uses a conditional diffusion model $\mathcal{D}_\theta : (s,g) \rightarrow \tilde{g}$, which, given the current state and final goal, produces an intermediate subgoal $\tilde{g}$ expected to be reachable within $K$ steps. States and goals are represented in a factored form $\mathcal{G} = \mathcal{S}$0, $\mathcal{G} = \mathcal{S}$1, with a set-Transformer architecture employed to denoise individual entity subgoal factors, encouraging subgoals that typically modify only a few entities.

2. Mathematical Formulation and Model Architectures

Factored Spaces and Reward Structure

• State space: $\mathcal{G} = \mathcal{S}$2, component-wise structured.
• Action space: $\mathcal{G} = \mathcal{S}$3 (e.g., gripper pose changes for robotic manipulation).
• Goal space and reward: $\mathcal{G} = \mathcal{S}$4, $\mathcal{G} = \mathcal{S}$5 if $\mathcal{G} = \mathcal{S}$6, $\mathcal{G} = \mathcal{S}$7 otherwise (strictly sparse).

Value and Policy Objectives

• Value Loss: Expectile regression for $\mathcal{G} = \mathcal{S}$8,

$\mathcal{G} = \mathcal{S}$9

with $r(s, g) = 1_{s=g} - 1$0.

• Q-Function Loss: Regression to TD targets,

$r(s, g) = 1_{s=g} - 1$1

• Policy Extraction: DDPG+BC objective,

$r(s, g) = 1_{s=g} - 1$2

Subgoal Diffusion Generation

• Training Data: Triplets $r(s, g) = 1_{s=g} - 1$3 obtained by sampling $r(s, g) = 1_{s=g} - 1$4, final goal $r(s, g) = 1_{s=g} - 1$5, and intermediate subgoal $r(s, g) = 1_{s=g} - 1$6 from offline trajectories.
• Diffusion Process: Forward noise process

$r(s, g) = 1_{s=g} - 1$7

for $r(s, g) = 1_{s=g} - 1$8; denoiser $r(s, g) = 1_{s=g} - 1$9 is trained with

$Q_\phi(s,a,g)$0

• Generation: At test time, $Q_\phi(s,a,g)$1 reverse steps yield $Q_\phi(s,a,g)$2 candidate subgoals per query.

3. Training Procedures and Test-Time Composition

Independent Training of Components

• RL Agent: Dataset $Q_\phi(s,a,g)$3 with $Q_\phi(s,a,g)$4M transitions. Approximated using Adam optimizer ($Q_\phi(s,a,g)$5 learning rate), batch size $Q_\phi(s,a,g)$6, $Q_\phi(s,a,g)$7–$Q_\phi(s,a,g)$8M updates, $Q_\phi(s,a,g)$9, expectile $\pi(a|s,g)$0, DDPG+BC $\pi(a|s,g)$1.
• Diffusion Subgoal Generator: Uses same data $\pi(a|s,g)$2 for $\pi(a|s,g)$3 examples with $\pi(a|s,g)$4; 8-layer set-Transformer, hidden dim $\pi(a|s,g)$5, $\pi(a|s,g)$6 heads, $\pi(a|s,g)$7 diffusion steps, $\pi(a|s,g)$8 optimization steps.

Test-Time Subgoal Selection and Execution

• On every $\pi(a|s,g)$9 step interval, $V_\psi(s,g) = \max_a Q_\phi(s,a,g)$0 subgoal candidates are sampled from $V_\psi(s,g) = \max_a Q_\phi(s,a,g)$1. Only those with $V_\psi(s,g) = \max_a Q_\phi(s,a,g)$2 (reachability threshold) are retained. The candidate maximizing $V_\psi(s,g) = \max_a Q_\phi(s,a,g)$3 is selected; if no improvement over $V_\psi(s,g) = \max_a Q_\phi(s,a,g)$4, the agent targets the final goal directly. The low-level policy acts toward the selected subgoal for one step, repeating until the global goal is achieved.

4. Empirical Evaluation and Benchmarks

HECRL is evaluated on a suite of sparse-reward, multi-entity manipulation tasks:

• PPP-Cube: Pick/place/push with 3 cubes, using state or dual-view RGB observations.
• Stack-Cube: Block stacking.
• Scene: Drawer, window, button, and cube entities.
• Push-Tetris: 2D block pushing to target positions and orientations.

Performance Table (mean$V_\psi(s,g) = \max_a Q_\phi(s,a,g)$5std, 4 seeds):

Task	EC-SGIQL	EC-IQL	EC-Diffuser	HIQL	IQL
PPP³-State	82.5±3.1	51.5±4.4	44.8±6.7	48.3±7.3	34.3±4.9
Stack³-State	43.5±1.9	29.0±2.9	43.8±9.2	0.0±0.0	19.3±3.0
PPP³-Image	64.3±4.9	25.0±5.7	0.3±0.5	0.0±0.0	0.0±0.0
Scene-Image	61.5±5.9	53.0±5.5	3.3±2.5	8.3±1.3	17.5±2.7
Push-Tetris (cov)	61.4±3.3	31.6±1.3	7.9±0.5	5.2±0.8	3.4±0.8

Zero-Shot Generalization:

PPP-Cube trained on 3 cubes generalizes to 4/5/6 cubes with EC-SGIQL achieving $V_\psi(s,g) = \max_a Q_\phi(s,a,g)$6 success, compared to EC-IQL's $V_\psi(s,g) = \max_a Q_\phi(s,a,g)$7.

5. Ablation Studies and Analytical Findings

• Subgoal Selection: Ablations demonstrate both the necessity of value-threshold filtering and value-guided selection for high success rates; diffusion-based sampling outperforms AWR-based deterministic high-levels.
  • Max-Value (no reachability): $V_\psi(s,g) = \max_a Q_\phi(s,a,g)$8
  • Random-Sample (no $V_\psi(s,g) = \max_a Q_\phi(s,a,g)$9-guidance): $R^{V_\pi}$0
  • AWR-based: $R^{V_\pi}$1
• Factor Sparsity: Entity-centric diffusion results in subgoals modifying an average of $R^{V_\pi}$2 cubes per step (vs $R^{V_\pi}$3 in AWR), promoting local reachability and efficient decomposition.
• Hyperparameter Robustness:
  • $R^{V_\pi}$4: Robust if $R^{V_\pi}$5.
  • $R^{V_\pi}$6: Optimal around $R^{V_\pi}$7.
  • $R^{V_\pi}$8: Peak performance for $R^{V_\pi}$9.

6. Implementation Details and Practical Considerations

Model Architectures:

• Low-level: 4-layer MLPs (state-based) or 3-layer Entity Interaction Transformer (hidden dim=256, heads=8).
• Subgoal diffuser: 8-layer set-Transformer, hidden dim=256, heads=8.
• Vision encoders: DLPv2 (latent particles), VQ-VAE (codebook size 2048, embedding dim=16).

Training Regime:

• RL: $\mathcal{D}_\theta : (s,g) \rightarrow \tilde{g}$0–$\mathcal{D}_\theta : (s,g) \rightarrow \tilde{g}$1M gradient steps, Adam ($\mathcal{D}_\theta : (s,g) \rightarrow \tilde{g}$2), batch $\mathcal{D}_\theta : (s,g) \rightarrow \tilde{g}$3, on $\mathcal{D}_\theta : (s,g) \rightarrow \tilde{g}$4–$\mathcal{D}_\theta : (s,g) \rightarrow \tilde{g}$5 A100 GPUs ($\mathcal{D}_\theta : (s,g) \rightarrow \tilde{g}$6 hours).
• Diffusion: $\mathcal{D}_\theta : (s,g) \rightarrow \tilde{g}$7K steps, Adam ($\mathcal{D}_\theta : (s,g) \rightarrow \tilde{g}$8), batch $\mathcal{D}_\theta : (s,g) \rightarrow \tilde{g}$9 ($\tilde{g}$0 hours).

Compute:

All stages—including encoder pretraining, RL, and diffusion—fit within a single 8-GPU node.

7. Summary of Approach and Ongoing Directions

HECRL's entity-centric, hierarchical architecture enables scalable RL in environments where direct goal-reaching is difficult due to entity combinatorics and sparse rewards. Its factored diffusion subgoal generator, composed modularly with any value-based GCRL agent, empirically achieves over $\tilde{g}$1 higher success on benchmark image-based multi-entity manipulation tasks, demonstrates zero-shot scalability with increasing entity counts, and shows minimal sensitivity to key hyperparameters. Future research directions include automatic inference of optimal subgoal horizon $\tilde{g}$2 or competence threshold $\tilde{g}$3 from data, hybridization of diffusion and value-based guidance for subgoal optimality, and extensions to real-world robotics as structured object-centric video representations mature. All code, datasets, and pretrained models are publicly available (Haramati et al., 2 Feb 2026).