Pose-Based Object Exploration (POE)

• Pose-Based Object Exploration is a modular paradigm that segments complex robotic tasks into discrete, pose-aligned stages with tailored rewards.
• It improves learning efficiency and policy robustness by assigning stage-specific rewards and costs to simplify exploration in high-complexity tasks.
• Empirical results demonstrate significant gains in success rates, convergence speed, and data efficiency for tasks ranging from acrobatic locomotion to deformable object handling.

Pose-Based Object Exploration (POE), subsumed in recent literature under Stage-Aligned Reward (SAR), Stage-Wise Reward Shaping, or Stage-Aware Reward Modeling, is a paradigm for robotic learning that structures exploration, reward design, and policy optimization by decomposing complex long-horizon tasks into a sequence of discrete semantic stages. Rather than specifying a monolithic reward function over the entire task, POE/SAR assigns each stage of the task a tailored reward (and potentially cost) function, with gating based on current pose, progress, or other state features. This modular structure is especially potent in high-complexity, contact-rich, or long-horizon robotic manipulation and locomotion, enabling efficient learning, improved data efficiency, and policy robustness. It has been instantiated for deep reinforcement learning (DRL), imitation learning, and combined model-based/model-free frameworks for tasks ranging from acrobatic locomotion to deformable object handling (Peng et al., 2020, Kim et al., 24 Sep 2024, Chen et al., 29 Sep 2025, Escoriza et al., 3 Mar 2025).

1. Formalization of Stage-Aligned Reward and POE

The central idea of POE/SAR is to model robotic tasks as a sequence of $K$ ordered stages $S_1,\ldots,S_K$, each corresponding to a semantically coherent subtask—for example, “approach,” “grasp,” “lift,” “fold left sleeve,” etc. The agent’s state is mapped to the current stage using a scheduler or stage-detection policy $\sigma: \mathcal{S} \rightarrow \{1,\ldots,K\}$, which can be deterministic (via thresholds/checkpoints) or learned. Given this decomposition, per-stage rewards $r_i$ and potential costs $c_i$ are defined for each stage in the trajectory: $$r_i(s,a) = [r_{i,1}(s,a), \ldots, r_{i,N_i}(s,a)]^\top,\qquad c_i(s,a) = [c_{i,1}(s,a), \ldots, c_{i,M_i}(s,a)]^\top.$$ The agent receives only the rewards and costs for the active stage; all others are masked out. This gating can be “hard” (binary indicator) or “soft” (distance/gating function). This structure frames robotic exploration as a progression through pose-aligned substages, each incentivizing the specific transitions or behaviors optimal for the current phase (Kim et al., 24 Sep 2024, Escoriza et al., 3 Mar 2025).

2. Reward Modeling and Progress Estimation

Stage-aligned reward modeling is critical for POE, enabling stable learning across highly variable demonstrations and long, multi-phase episodes. In “SARM: Stage-Aware Reward Modeling for Long Horizon Robot Manipulation” (Chen et al., 29 Sep 2025), a two-head transformer-based reward model jointly predicts the current stage and regresses fine-grained (continuous) progress within each stage, conditioned on visual and proprioceptive inputs. Reward targets are derived using natural-language task annotations: each demonstration is segmented into $K$ subtasks, with normalized progress assigned according to the proportion of frames in each subtask and position within it: $$y_t = P_{k-1} + \bar\alpha_k \tau_t, \quad \tau_t = \frac{t-s_k}{e_k-s_k},$$ where $P_{k-1}$ is the cumulative prior proportion, $[s_k, e_k]$ are the local bounds for subtask $k$, and $\bar\alpha_k$ is the average dataset-level prior for subtask $k$. The model outputs a scalar reward $R(x_t) = \hat y_t \in [0,1]$, yielding a smooth, robust progress signal for both evaluation and policy learning.

This approach circumvents the inherent calibration and scaling problems of frame-index based or hand-crafted dense rewards, and is robust to non-uniform demonstration speeds, failures, and out-of-domain behaviors. Empirical results on deformable object manipulation indicate substantial improvements in both rollout evaluation and policy learning when leveraging stage-aware reward models (Chen et al., 29 Sep 2025).

3. Algorithmic Implementation: Actor–Critic and CMORL Variants

Practical implementation of POE/SAR follows standard DRL actor-critic paradigms, adapted to account for stage-specific rewards and costs. In trajectory planning (Peng et al., 2020), reward computation for each step involves three terms: posture reward (distance/direction), stride reward (joint movement), and stage incentive (hard or soft gating based on proximity to subgoal): $$R_{\text{total}}(s,a) = R_{\text{posture}}(s,a) + R_{\text{stride}}(s,a) + R_{\text{stage}}(s,a),$$ with $R_{\text{stage}}$ modulated according to current pose.

For stage-wise constrained multi-objective reinforcement learning (CMORL), as in “Stage-Wise Reward Shaping for Acrobatic Robots” (Kim et al., 24 Sep 2024), each stage has a reward and vector of constraints. Policy and critic networks maintain separate value estimates per stage, and optimization proceeds via a single surrogate advantage function aggregating per-stage advantages and penalties for constraint violations: $$A^\pi(s,a)=\frac{\sum_{i}\omega_i\hat A_{r,i}^\pi(s,a)}{\mathrm{Std}[\sum_i\omega_i\hat A_{r,i}^\pi]} - \eta\sum_{i}\frac{A_{c,i}^\pi(s,a)}{\mathrm{Std}[A_{c,i}^\pi]}1_{J_{c,i}>d_i}.$$ Policy improvement is carried out using PPO with value and cost critics normalized per stage.

Several implementations augment stage-aligned RL with demonstration data and learned world models. In DEMO3 (Escoriza et al., 3 Mar 2025), stage-aligned discriminators are trained to predict the probability of reaching subsequent stages from current world-model latent states, with their output incorporated as a dense shaping bonus for each stage. Training proceeds bi-phasically: demonstration-based pretraining (behavioral cloning), followed by reward learning and RL with joint gradient updates over policy, world model, and reward heads.

4. Integration in Imitation and Reinforcement Learning

POE/SAR serves as a unifying abstraction for reward design, policy learning, and demonstration-augmented learning in complex robotics:

• Imitation Learning: Stage-aware reward models (as in SARM) are used to filter and reweight demonstration data in behavior cloning. The Reward-Aligned Behavior Cloning (RA-BC) loss

$$\mathcal{L}_{\text{RA-BC}} = \frac{\sum_{i=1}^N w_i \ell(\pi_\theta(o_i), a_i)}{\sum_i w_i + \varepsilon},$$

applies data-dependent weights $w_i$ based on estimated progress deltas, promoting sample efficiency and higher final policy success rates, especially on long-horizon, contact-rich tasks.

• Reinforcement Learning: Stage alignment ensures agents optimize for the relevant skills/behaviors in each subtask, simplifying both exploration (as the search space is locally reduced) and reward specification (no need for globally balanced reward coefficients). In constrained multi-objective settings, per-stage costs directly facilitate hard safety/physicality constraints.
• Model-Based RL: By leveraging learned world models and per-stage discriminators (as in DEMO3), POE/SAR enables dense progress signals and substantially improves data efficiency, outperforming global reward shaping and pure sparse-reward RL.

5. Empirical Evaluation and Data Efficiency

Extensive experiments confirm the advantages of POE/SAR across multiple domains:

• Trajectory Planning: Soft stage-aligned rewards reduce convergence time by up to 46.9%, increase mean return by 4.4–15.5%, and decrease standard deviation by 21.9–63.2%, with planning success rates up to 99.6% (Peng et al., 2020).
• Acrobatic and Multi-Phase Locomotion: Stage-wise reward shaping achieves ~100% success on challenging acrobatic benchmarks (e.g., back-flip, side-roll) where monolithic baselines fail completely (Kim et al., 24 Sep 2024).
• Long-Horizon Manipulation: Stage-aware reward models with RA-BC yield up to 83% success on folding from a flattened shirt state and 67% from a crumpled state on real robots, far exceeding vanilla BC and alternative reward models (Chen et al., 29 Sep 2025).
• Data Efficiency: Incorporating semantic stage information and demonstrations, frameworks such as DEMO3 improve data efficiency by 40% on average, and by up to 70% on particularly hard manipulation tasks (Escoriza et al., 3 Mar 2025).

These gains result from effective mitigation of exploration challenges, improved reward continuity/calibration, and focused learning via per-stage reward and constraint shaping.

6. Extensions, Practical Considerations, and Future Prospects

The stage-aligned reward principle underlying POE generalizes to any task admitting a meaningful temporal or semantic segmentation, including mobile navigation, legged locomotion, assembly, and contact-rich manipulation.

Practical deployment of POE/SAR requires:

• Explicit or learnable stage scheduling (via threshold logic, finite-state machines, or learned classifiers);
• Stage-aligned reward and cost head definitions, often learned from demonstrations;
• Per-stage value, cost, and, if desired, world-model critics to maintain signal locality and calibration;
• Proper hyperparameter choices, e.g., reward bonus scale $\beta$ (usually $\leq 1/3$), mixing ratio of demonstration/RL data, and network capacities (see (Escoriza et al., 3 Mar 2025, Chen et al., 29 Sep 2025)).

Future work includes automatic stage discovery (joint learning of $\sigma(s)$ with the policy), hierarchical and compositional reward models, and deeper integration with model-based planning and task decomposition frameworks.

7. Significance and Impact

Pose-Based Object Exploration and its instantiations as SAR/SARM have materially advanced the efficiency, stability, and practical scalability of robotic learning systems. By transforming reward shaping from a global weight-tuning problem to a modular, stage-wise design, POE circumvents the “reward hacking” and signal imbalance common in monolithic shaping. Empirical validation across diverse tasks and robots confirms its broad applicability and underlines the potential for automated, scalable, and robust long-horizon skill acquisition in real-world settings (Peng et al., 2020, Kim et al., 24 Sep 2024, Chen et al., 29 Sep 2025, Escoriza et al., 3 Mar 2025).