SceneBot: Contact-Aware Whole-Body Tracking

• SceneBot is a contact-prompted humanoid tracking framework that integrates free-space locomotion, terrain traversal, and whole-body manipulation using explicit per-link contact labels.
• It employs a binary contact prompt interface conditioned on reference motion and proprioceptive states to resolve ambiguities in contact-rich scenarios.
• Trained with PPO on 7.5 hours of reconstructed contact-rich data, SceneBot achieves high success rates in free-space, object, terrain, and sitting tasks.

SceneBot is a contact-prompted humanoid whole-body tracking framework for scene-interactive control. It is designed to overcome a specific limitation of contemporary humanoid reinforcement-learning trackers: strong free-space motion imitation does not by itself resolve the physical ambiguities of contact-rich behavior such as stepping onto stairs, lifting or carrying objects, or sitting. SceneBot addresses this by conditioning a single policy on both reference motion and explicit per-link contact labels, thereby unifying free-space locomotion, terrain traversal, and whole-body manipulation within one controller. The framework is trained on 7.5 hours of reconstructed contact-rich data and is presented as generalizing to unseen motions and environments (Chen et al., 25 Jun 2026).

1. Problem setting and conceptual scope

SceneBot targets a regime in which humanoid control must remain faithful to a reference motion while also interacting intentionally with the environment. The motivating observation is that pose tracking alone is underdetermined in contact-rich settings. A hand near a box can represent reaching, touching, or grasping; a foot near a stair edge can represent hovering, stepping, or slipping. In these cases, kinematic similarity is insufficient because the controller must resolve external force interaction and support usage, not merely reproduce body geometry (Chen et al., 25 Jun 2026).

The framework is therefore positioned against the dominant free-space tracking paradigm. Standard trackers can reproduce locomotor or acrobatic motion when scene geometry is irrelevant, but failures become structural when the task depends on terrain support, object loading, or long-horizon coordinated contacts. The paper identifies several failure sources: kinematic ambiguity, dependence on physical support, catastrophic consequences of global drift, and the scarcity of motion datasets annotated with scene geometry and contact intent.

A central point is that SceneBot is not a high-level task planner. Its role is lower in the control stack: it is a contact-aware tracking controller that follows reference motion while respecting explicit expectations about which body parts should interact with terrain or objects. This design choice is what allows the same policy to cover free-space locomotion, uneven-terrain traversal, and whole-body object manipulation within a unified formulation.

2. Contact prompting as the control interface

SceneBot’s defining interface is a binary contact prompt over robot-link and scene-type pairs. The key robot links are

$$\mathcal{K} \in \{\text{left-wrist}, \text{right-wrist}, \text{left-foot}, \text{right-foot}, \text{pelvis}\},$$

and the scene types are terrain and object. The contact label vector is written as

$\bm{c}_t = [c^{i,j}_t], \quad c_t \in \{0,1\}^n,$

where an active entry indicates that link $i$ is expected to establish contact with scene type $j$ (Chen et al., 25 Jun 2026).

This interface is deliberately minimal. The labels are body-centric rather than semantic, and they specify interaction intent without requiring a full symbolic task description. The paper emphasizes three properties of this representation: it is low-level, because it is not task language or vision prompting; minimal, because it only indicates expected contact structure; and scene-aware, because it explicitly encodes how the robot should use terrain or objects. Free-space behavior is recovered as the degenerate case in which all contact labels are zero.

The tracking policy is formulated as

$$\bm{a}_t=\pi(\bm{c}_t,\bm{g}_t,\bm{s}_t,\bm{p}_{\text{root},t}),$$

where $\bm{g}_t \in \mathbb{R}^{93}$ is the reference-motion command, $\bm{s}_t$ is the proprioceptive state, and $\bm{p}_{\text{root},t}$ is the root state. The command includes desired lower-body joint angles and velocities, 6D head and wrist poses in the root frame, and global root position/orientation errors between current and target root states. The proprioceptive state includes current joint angles and velocities, projected gravity in the root frame, and root angular velocity. The root state includes root position, orientation, and linear velocity (Chen et al., 25 Jun 2026).

A common misconception is to treat SceneBot as a scene-understanding or vision-language policy. Its conditioning variable is not an open-ended scene description; it is a compact contact schedule tied to specific robot links. This suggests that SceneBot’s generality comes less from semantic richness than from using contact intent as a universal abstraction for physically grounded tracking.

3. Hindsight scene reconstruction and scene-interaction graphs

A major obstacle for contact-aware tracking is the absence of large-scale datasets pairing motion, scene geometry, and contact labels. SceneBot addresses this with hindsight scene reconstruction. The procedure begins with human motion data, retargets that motion to the humanoid, infers likely link-scene interactions over time, reconstructs plausible terrain and object geometry consistent with those interactions, and then uses the resulting motion-scene pairs for reinforcement learning (Chen et al., 25 Jun 2026).

The intermediate representation is a scene-interaction graph. Robot nodes are induced from the key links in $\mathcal{K}$, while scene nodes are of two types: terrain and object. A temporal edge records interaction time, the involved robot link node or nodes, the scene node, and a spatial location. Candidate edges are proposed when a robot link exhibits low relative velocity and acceleration with respect to the scene, which is treated as evidence of likely contact. Edges are then pruned if they induce collisions outside the interaction interval or violate force-closure constraints in object interactions.

The graph supports two distinct reconstruction processes. Terrain is represented as a 2.5D elevation map. For each terrain interaction edge, the method extracts the contact location $\mathbf{p}=(p_x,p_y,p_z)$, adds a square plateau centered at $\bm{c}_t = [c^{i,j}_t], \quad c_t \in \{0,1\}^n,$0, and sets its height to $\bm{c}_t = [c^{i,j}_t], \quad c_t \in \{0,1\}^n,$1. Similar isolated plateaus are merged, and collision-inducing regions are carved out. Objects are reconstructed as collections of plates parallel to their contact surfaces, with centers and trajectories estimated from the mean positions of the interaction edges involved in the grasp.

The paper argues that this reconstruction route is preferable to scene-aware retargeting. Retargeting to a preexisting scene can introduce object-hand misalignment, penetration, and infeasible support geometry, which degrades downstream policy learning. Reconstruction instead makes the scene consistent with the motion, reducing mismatch between reference behavior and trainable physical realization.

4. Reinforcement learning objective and contact-specific rewards

SceneBot is trained with PPO. Its reward design combines standard motion-tracking terms with two rewards specific to scene interaction. The first is a contact correctness reward,

$\bm{c}_t = [c^{i,j}_t], \quad c_t \in \{0,1\}^n,$2

which checks whether actual simulator contacts match desired contacts when the desired label is active. The second is a contact duration reward,

$\bm{c}_t = [c^{i,j}_t], \quad c_t \in \{0,1\}^n,$3

where $\bm{c}_t = [c^{i,j}_t], \quad c_t \in \{0,1\}^n,$4 is cumulative contact time for the relevant link-scene pair (Chen et al., 25 Jun 2026).

The asymmetry of the correctness reward is technically important. Unspecified or accidental contact is not penalized. This allows SceneBot to incorporate large amounts of flat-terrain motion data that lack explicit interaction annotations, rather than discarding them or forcing artificial negative labels. The duration reward is clipped at $\bm{c}_t = [c^{i,j}_t], \quad c_t \in \{0,1\}^n,$5 s so that the controller is encouraged to establish stable contact without becoming reluctant to release an object once the task requires disengagement.

Beyond these two terms, the paper lists standard tracking rewards for root orientation, root position, body-link position, body-link orientation, linear velocity, and angular velocity, together with penalties on action rate, joint-limit violations, and undesired contacts. Training for object interactions additionally uses a contact-mismatch termination condition: if the policy fails to establish the desired contact within a specified time window, the episode terminates. During object reconstruction, the framework also applies a heuristic stabilizing force, termed a “magic force,” so that a reconstructed object follows the intended trajectory until a stable force-closure grasp is achieved.

The overall reward structure indicates that SceneBot does not replace classical kinematic tracking objectives; rather, it augments them with enough contact supervision to disambiguate physically meaningful interaction.

5. Empirical behavior, deployment, and failure modes

The reported training corpus comprises 7.5 hours of reconstructed contact-rich data drawn from AMASS, OMOMO, Bones, and Lafan (Chen et al., 25 Jun 2026). The evaluation covers four categories: free-space, terrain interaction, object interaction, and sitting. SceneBot is compared against SONIC and several ablations, including No label, No foot label, and Local. The paper states that SceneBot is on par with SONIC for free-space motion while substantially outperforming SONIC and the ablations on terrain, object, and sitting tasks.

Category	Reported success rate
Free-space	100%
Object	95%
Terrain	100%
Sit	100%

Qualitatively, the framework is reported to handle stepping onto stairs of varying heights, picking up and carrying boxes, sitting down, agile kicking and running, and carrying a box while ascending stairs. The last capability is particularly significant because it requires simultaneous terrain interaction and object manipulation over a long horizon.

The ablations isolate the value of explicit contact prompting. Without hand contact labels, grasp success drops near zero. Without foot contact labels, terrain traversal becomes unstable and the robot falls. At the same time, disabling contact labels only at test time still performs surprisingly well for terrain traversal. This suggests that contact labels function not only as a runtime prompt but also as a stabilizing training interface that shapes the policy’s internal representation of scene interaction.

Deployment depends on global state estimation. SceneBot uses SuperOdometry plus IMU information to estimate root position, orientation, and linear velocity. The paper notes that a head-mounted IMU alone is inadequate because head motion can decouple from pelvis motion during aggressive behavior. Accordingly, pitch and roll are computed from a pelvis-mounted root IMU, yaw is fused by a Kalman filter combining root-IMU yaw angular velocity with SuperOdometry yaw, and state estimation runs at 200 Hz. This emphasis on global state is consistent with the paper’s finding that terrain and object interaction are sensitive to drift.

The reported limitations are also tied to data quality. Hindsight scene reconstruction requires high-fidelity robot motion and is sensitive to poorly retargeted trajectories such as foot sliding. Motion derived from raw videos or generative models may lack sufficient physical consistency, which can destabilize scene-interaction graph construction.

6. Position within scene-grounded humanoid systems

SceneBot occupies the low-level, contact-aware tracking layer in a broader class of scene-grounded humanoid systems. This role is explicit in subsequent work on vision-language-kinematics supervision, where a contact-aware whole-body tracker based on SceneBot converts predicted short-horizon kinematic trajectories into joint-level PD targets for a physical Unitree G1 (Wang et al., 29 Jun 2026). In that architecture, a high-level policy predicts future whole-body kinematics from egocentric observations and language instructions, while the SceneBot-style tracker executes those predictions under contact constraints.

This division of labor clarifies what SceneBot contributes. It is not a full perception-to-action system, nor a scene-language reasoning model. Its main contribution is an interface for making whole-body tracking compatible with environmental interaction. That interface is sufficiently general to support free-space locomotion when all labels are zero, terrain traversal when foot and pelvis contacts matter, and object manipulation when wrist-object contacts must be established and maintained.

A plausible implication is that SceneBot’s broader importance lies in standardizing contact conditioning as a modular control abstraction. Because the prompt is interpretable, sparse, and tied to specific links, it can be supplied by hand-authored motion, reconstructed interaction data, or an upstream perception-and-planning system. In that sense, SceneBot links classical motion imitation and scene-grounded embodied control without collapsing them into a single monolithic policy.