- The paper introduces a contact-conditioned RL policy that integrates kinematic tracking with explicit contact labels for seamless humanoid whole-body control.
- It employs a novel hindsight scene reconstruction pipeline that generates 7.5 hours of contact-rich data to train robust free-space and contact interaction behaviors.
- Empirical results show significant improvements, with terrain interaction and object grasp success rates reaching 100% and 95%, respectively.
SceneBot: A Unified Framework for Contact-Conditioned General Humanoid Whole-Body Tracking
Introduction
"SceneBot: Contact-Prompted General Humanoid Whole Body Tracking with Scene-Interaction" (2606.27581) introduces a unified control framework for humanoid robots that integrates general motion tracking with explicit contact reasoning for complex scene interactions. Unlike prior reinforcement learning (RL)-based policies, which are typically limited to free-space motion and are ineffective in environments rich with contact events such as uneven terrain or object manipulation, SceneBot leverages a dual conditioning interface: reference kinematic motion and fine-grained, per-link contact labels. This architecture enables whole-body humanoid behaviors that seamlessly transition across free-space locomotion, terrain adaptation, and bimanual object manipulation within a single low-level control policy.
Methodological Contributions
SceneBot's core is a contact-aware RL policy trained to simultaneously track reference motion and realize specified physical contacts with the environment. The reference motion defines the generic kinematic sequence, while a binary contact label vector determines for each key body link which types of scene contact (terrain/object) should be established and exploited. SceneBot thus generalizes previous tracking paradigms by making contact intention an explicit policy input. The interface is strictly body-centric—avoiding reliance on vision or object-centric planning and maintaining the abstraction level necessary for robust low-level control.
Hindsight Scene Reconstruction
The absence of paired robot-scene interaction datasets is addressed using a novel data generation pipeline. Given retargeted human motion, SceneBot reconstructs interaction graphs by identifying contact events—based on low relative velocity/acceleration between the robot’s links and putative scene nodes (terrain or object)—over time. Subsequently, procedurally synthesized scene assets (terrains as 2.5D elevation maps; objects as plates aligned to contact patches) ensure semantic and kinematic consistency between motion and environment. This closed-loop scene generation produces 7.5 hours of contact-rich data without requiring environment-annotated mocap, significantly expanding data availability and diversity for RL.
Policy Training
Training employs PPO, with a reward function aggregating standard tracking error terms and two contact-oriented rewards: contact correctness encourages the establishment of desired contacts, and contact duration provides a dense signal for maintaining and releasing scene contact as appropriate. Heuristic stabilizing forces are applied during object manipulation to mitigate failures prior to force-closure grasps. The RL policy uses accurate kinematic state inputs, proprioceptive feedback, and high-frequency root odometry signals.
Infrastructure-Free State Estimation
Real-world deployment leverages a hybrid odometry stack (SuperOdometry), combining LiDAR-inertial sensing for global position/velocity and a dedicated pelvis-mounted IMU for robust root orientation estimation through Kalman filtering. This ensures drift-free, global whole-body tracking—even during aggressive motions—compared to prior work that relied solely on local, head-mounted IMU estimates.
Empirical Evaluation
SceneBot achieves strong quantitative and qualitative results across multiple task classes—free-space, terrain interaction, sit, and general object manipulation—on the Unitree G1 hardware and in MuJoCo simulation. It is the only evaluated method that unifies all these behaviors using a single policy. For example, SceneBot realizes complex, long-horizon behaviors such as carrying a box upstairs and simultaneously engaging terrain and object contacts without decomposing the task or requiring hand-coded transitions.
Empirically, on simulated tasks—using datasets such as AMASS, OMOMO, Bones, and Lafan—SceneBot matches state-of-the-art free-space tracking performance while vastly outperforming alternatives (e.g., SONIC [4], baselines without global state, or without contact labels) in contact-rich and environment-dependent scenarios. Notable figures include:
- Terrain interaction and object grasp success rates of 100% and 95%, respectively, relative to SONIC’s 15% and 5%.
- Root position/orientation errors: 0.1471 m/0.1017 rad (terrain); 0.1215 m/0.1225 rad (object).
- Policy generalizes successfully to unseen motions and scene configurations, demonstrating contact conditioning as an effective interface.
- Ablations confirm necessity of both contact labels and global root input: dropping either significantly impairs scene interaction performance (object manipulation success drops to near zero without hand contact labels).
Comparative Discussion
SceneBot is compared not only to pure motion tracking frameworks but also recent attempts at terrain-aware or object-interaction controllers. Unlike works such as HDMI [31] or OmniRetarget [21], which require precise environment annotation or cannot scale to diverse bimanual loco-manipulation, SceneBot’s interaction conditioning and synthetic scene generation pipeline remove a key bottleneck in both dataset construction and policy generalization. Policies trained with scene reconstruction data have measurably lower error rates and higher success rates compared to those using scene-aware retargeting, largely because automatic reconstruction guarantees feasibility—and relieves the RL pipeline from solving scene-motion mismatches.
Theoretical and Practical Implications
SceneBot demonstrates that explicit per-link contact conditioning is a minimal and generic yet highly effective mechanism for closing the gap between general whole-body motion controllers and the requirements of contact-rich behaviors in the physical world. The approach is compatible with future hierarchical or vision-language-based planners as a robust low-level primitive for perception-driven interaction. The pipeline for generating physically consistent scene-robot interaction datasets points toward scalable, environment-agnostic learning of complex skills, obviating the need for accurate, hand-annotated mocap-environment pairs and potentially supporting future unsupervised or generative interaction learning.
Limitations and Future Directions
SceneBot’s trajectory-scene pairing depends on high-fidelity motion retargeting; physically inconsistent or noisy kinematic inputs (e.g., synthetic motion, video reconstructions) degrade interaction graph quality and thus policy effectiveness. Furthermore, the current approach relies on externally specified contact sequences. Integrating this with high-level decision mechanisms for dynamic contact planning, perception-based scene parsing, or leveraging foundation models for end-to-end interactive skill synthesis constitute promising next steps.
Conclusion
SceneBot establishes a unified and data-efficient framework for general humanoid whole-body tracking with robust scene interaction via explicit contact conditioning. The method effectively bridges the longstanding gap between free-space and contact-rich behaviors within a single low-level policy. This offers both a practical path forward for general-purpose, robust locomotion-manipulation controllers and a theoretical template for scalable, scene-aware robot skill learning. Future developments could focus on autonomous perception-driven interaction and further abstraction of the contact interface for downstream policy composition and zero-shot generalization.
