PhysLadder: A Hierarchical Benchmark for Robotic Manipulation

• PhysLadder is hierarchical sim–real benchmark designed to evaluate physical interaction in contact-rich robotic tasks with three success levels consisting of 16 household task.
• The benchmark facilitated a comprehensive evaluation of the capabilities of the GPT-6 Astra model in tasks ranging from simple contact selection (L1) to complex task strategy maintenance (L3.RE

PhysLadder is a hierarchical sim–real benchmark for evaluating physical interaction in contact-rich robotic manipulation. It comprises 16 household tasks organized into three cumulative levels: L1, contact selection and geometric alignment; L2, force regulation during sustained contact; and L3, coordinated contact reconfiguration with force regulation. Each task is instantiated in physically and visually matched simulated and real-world environments with shared task definitions and success criteria. The benchmark evaluates complete-task success rather than isolated motor actions, thereby emphasizing whether a system can select mechanically useful contacts, maintain support, regulate forces and torques, reconfigure contacts, and accumulate local physical progress into successful manipulation (Yu et al., 25 Oct 2026).

1. Benchmark rationale and hierarchical organization

PhysLadder addresses a limitation of aggregate manipulation benchmarks: successful grasping or placement does not necessarily demonstrate competence in sustained, contact-mediated interaction. A robot may identify an object and move an end effector near it while failing to establish a contact that permits the intended force direction, rotation, insertion, support, or deformation control.

The hierarchy is defined by two interaction dimensions:

• Contact configuration: where and how the robot contacts objects, tools, containers, or supports.
• Force regulation: how contact forces or torques produce, constrain, or regulate object motion or deformation.

A contact is functionally useful only if it enables the intended operation. A secure grasp can nevertheless be incorrect if it blocks an opening, prevents rotation, or makes the required force direction inaccessible. Contact reconfiguration denotes releasing and re-establishing contacts to continue the main operation; ordinary grasping, release, sliding, rolling, or recovery from a failed grasp does not automatically make a task L3.

Level	Interaction requirement	Typical failure
L1	Select suitable contacts and achieve geometric alignment	Misalignment or a grasp that obstructs placement
L2	Regulate task-level forces during sustained contact without required reconfiguration	Slip, stalled motion, crushing, spilling, or displacement
L3	Coordinate force regulation with contact changes and re-establish contacts to continue	Loss of torque or support after regrasping, or local progress that does not accumulate

L1 includes contact selection and alignment without force regulation beyond ordinary support, transport, and geometric placement. L2 adds sustained force or torque control while the main contact mode remains unchanged. L3 combines force regulation with reconfiguration, such as repeated regrasping during unscrewing or changing pushing contacts during rotation.

The levels describe particular task versions, initial configurations, and process constraints rather than intrinsic properties of task names. Sweeping two blocks near a dustpan may be L2-like if a stable contact mode is sufficient, whereas sweeping three dispersed blocks is L3 because the brush must establish successive contacts and adapt its direction. A nearly loosened cap may require only sustained torque regulation, while a cap requiring repeated regrasping belongs to L3.

2. Task suite

The benchmark contains five L1 tasks, seven L2 tasks, and four L3 tasks.

L1 tasks

1. Marker Placement: A capped marker is placed with either end inside a wide-mouth cup. The task requires stable grasping, alignment with the opening, release, and independent support by the cup.
2. Charger Alignment: A charger is lifted and its two prongs are aligned with corresponding socket holes. Insertion and electrical operation are not required; the principal requirement is precise relative alignment.
3. Hat Hanging: A visor is picked up and its head opening is passed over a specified coat-rack branch. The opening must remain accessible during grasping, and the released visor must be independently supported.
4. Cup Nesting: One cup is placed at a central location and a second cup is nested inside it. Both cups must remain nested and independently supported after release.
5. Flower Transfer: A flower is retrieved from one cup, inserted by its stem into another cup, and released. The grasp must leave sufficient stem exposed for insertion, and the flower must remain supported afterward.

L2 tasks

1. Marker Cap Removal: One arm stabilizes the marker body while the other pulls the cap along the marker axis. Both grasps must remain stable while cap retention is overcome.
2. Book Pulling: The right arm stabilizes the upper book while the left arm pulls the middle book outward by approximately 5 cm. The pulling force must overcome inter-book friction without moving the stack.
3. Marble Pouring: A source cup is picked up and tilted to pour twelve marbles into a fixed receiving cup, after which the source cup is returned and released. At least six marbles must remain in the receiver, the source cup must be empty, and the receiving cup must remain fixed.
4. Orange Handling: A deformable orange segment is lifted by 5–20 cm, returned, and released without visible damage. The grasp must provide support without excessive compression.
5. Drawer Pull–Return: A partly open drawer is pulled approximately 5 cm farther and then pushed back to its recorded starting position. The cabinet must remain fixed, and the same grasp must transmit both pulling and pushing forces along the drawer guide.
6. Jelly Handling: A deformable jelly is lifted by 5–20 cm, returned, and released without crushing or damage. Success depends on grasp position, grip pressure, support, and motion.
7. Whiteboard Wiping: An eraser is used to remove all writing from a whiteboard. The eraser must maintain adequate pressing force while sliding across the board; executing a nominal wiping trajectory is insufficient if writing remains visible.

L3 tasks

1. Bottle Cap Unscrewing: A fixed bottle cap is unscrewed and regrasped when the required rotation exceeds the range of one grasp. The fully disengaged cap must then be lifted 3–5 cm from the opening.
2. Nut Unscrewing: One arm stabilizes a threaded rod while the other unscrews the nut until complete separation. Regrasping is required, and the arms must coordinate torque transmission with reaction-force support.
3. Stacked-Book Pushing Rotation: Two stacked books are pushed, not grasped or lifted, from one region to another while completing a full $360^\circ$ in-plane rotation. Pushing contacts must change as the stack rotates while preserving stacking and support.
4. Block Sweeping: A brush is used to sweep three spatially separated blocks into a fixed dustpan. The gripper may not manipulate the blocks directly; the task requires successive brush–block contacts, changing sweep directions, and adapting contact locations.

L3 tasks therefore combine multiple potential sources of difficulty: contact reconfiguration, force and torque regulation, bimanual coordination, evolving geometry, support preservation, and longer-horizon execution. The hierarchy is not a controlled isolation of a single causal variable. Perception, reachability, spatial reasoning, embodiment, and setup constraints remain relevant at every level.

3. Sim–real environment design

The real-world platform uses two ARX R5 six-joint robotic arms, R5-series coupled two-finger grippers, one external Orbbec Gemini Pro camera, and two wrist-mounted Orbbec Gemini 305 cameras. The external camera provides a workspace-level view, while the wrist cameras provide local information about grasps and contacts. The model receives RGB images, proprioceptive states, execution feedback, and interaction history, but no depth images or direct contact-force measurements. Gripper commands specify opening position rather than contact force.

The simulated environment is implemented in NVIDIA Isaac Sim 5.1.0 with a dual-arm ARX R5 model and corresponding external and wrist-camera configuration. Scene reconstruction uses multiple real-scene video captures, segmentation, measured dimensions, reconstructed geometry, textures, collision surfaces, friction parameters, and calibrated object properties.

Matching is performed across four components:

1. Workspace and initialization: table geometry, object poses, support and fixation conditions, and mechanical engagement, including screw–nut and cap–thread engagement.
2. Assets and contact response: object shape, color, texture, scale, mass, collision geometry, friction, and relevant constraints. Drawer joints and fixed supports are represented explicitly. Orange and jelly are modeled with finite-element models. Wiping uses a force-weighted ink-removal proxy.
3. Robot embodiment: arm geometry, joint layout, gripper geometry, mounting positions, and the relationship between the robots and tabletop.
4. Camera configuration and appearance: camera poses, projection parameters, image geometry, lighting, and background appearance.

The matching objective is interaction-relevant rather than purely visual. Sliding, deformation, support, and contact behavior are considered alongside image similarity. Some matching evidence remains qualitative: for example, jelly frames are compared visually after a shared color adjustment, but quantitative deformation and contact-force agreement are not reported.

Each simulated and real task shares an initial-state specification, natural-language objective, permitted and prohibited actions, process constraints, and observable success criterion. A trial begins from reset object and arm states and receives three attempts per task in each environment. Simulation trials can terminate on independently detected success, task-defined failure, model-call or time-budget exhaustion, or certain simulator failures. The default simulated budget is 150 model calls and five hours. Real trials have no preset wall-clock or model-call limit but may terminate through model termination, operator intervention, infeasible execution, or a protective robot stop. Operator assistance does not count as autonomous success.

In simulation, GPT-6 Astra returns structured end-effector targets and gripper openings; cuRobo produces collision-checked trajectories and Isaac Sim executes them. In the real environment, tool calls drive the ARX control service through bounded motion segments and safety protections. In both settings, the system receives new observations and must verify whether the intended grasp, alignment, motion, support, or separation was actually achieved.

A trial is successful only if the complete task goal and all process constraints are satisfied. Partial progress does not constitute success: rotating a cap without fully removing it, moving a drawer without completing the outward-and-return sequence, partially rotating a book stack, moving blocks without placing them in the dustpan, or wiping while leaving writing visible are all failures.

4. Evaluation protocol and results

For task $i$ and environment $e\in\{\mathrm{sim},\mathrm{real}\}$, the task-level success rate is

$\mathrm{SR}_{i,e}=\frac{k_{i,e}}{n_{i,e}},$

where $k_{i,e}$ is the number of successful trials and $n_{i,e}=3$. For a task set $\mathcal{T}$, the reported rate is the equally task-weighted mean

$$\mathrm{SR}_e(\mathcal{T})= \frac{1}{|\mathcal{T}|} \sum_{i\in\mathcal{T}}\mathrm{SR}_{i,e}.$$

Thus, level means are not pooled trial rates; each task contributes equally regardless of its number of successful trials.

GPT-6 Astra is evaluated zero-shot with reasoning effort set to xhigh. No demonstrations, prior task executions, or task-specific strategies are supplied. The real-world basic skills provide only task-agnostic interface, embodiment, observation, verification, coordination, and recovery guidance.

Level	Tasks	Simulation	Real world
L1	5	66.7% (10/15)	80.0% (12/15)
L2	7	47.6% (10/21)	47.6% (10/21)
L3	4	0.0% (0/12)	0.0% (0/12)
Overall	16	41.7% (20/48)	45.8% (22/48)

At the task level, Marker Placement and Flower Transfer achieve $3/3$ in both environments. Hat Hanging fails in both environments. All four L3 tasks achieve $0/3$ in both simulation and reality. L2 performance is heterogeneous: Marble Pouring succeeds $i$0 in both settings, while Marker Cap Removal and Jelly Handling fail $i$1 in both.

Ten of the 16 tasks have identical success counts in simulation and reality. The remaining six differ by one successful trial:

• Charger Alignment: simulation $i$2, real $i$3.
• Cup Nesting: simulation $i$4, real $i$5.
• Book Pulling: simulation $i$6, real $i$7.
• Orange Handling: simulation $i$8, real $i$9.
• Drawer Pull–Return: simulation $e\in\{\mathrm{sim},\mathrm{real}\}$0, real $e\in\{\mathrm{sim},\mathrm{real}\}$1.
• Whiteboard Wiping: simulation $e\in\{\mathrm{sim},\mathrm{real}\}$2, real $e\in\{\mathrm{sim},\mathrm{real}\}$3.

The signed real-minus-simulation differences are $e\in\{\mathrm{sim},\mathrm{real}\}$4 percentage points for L1, $e\in\{\mathrm{sim},\mathrm{real}\}$5 for L2, $e\in\{\mathrm{sim},\mathrm{real}\}$6 for L3, and $e\in\{\mathrm{sim},\mathrm{real}\}$7 overall. The corresponding absolute differences are $e\in\{\mathrm{sim},\mathrm{real}\}$8, $e\in\{\mathrm{sim},\mathrm{real}\}$9, $\mathrm{SR}_{i,e}=\frac{k_{i,e}}{n_{i,e}},$0, and $\mathrm{SR}_{i,e}=\frac{k_{i,e}}{n_{i,e}},$1 percentage points. These are descriptive comparisons; the study does not report confidence intervals, hypothesis tests, or a formal agreement coefficient.

The paper also contains an intermediate real-world accounting block labelled “real-zs,” with different denominators and rates of 66.7% for L1, 16.7% for L2, 0% for L3, and 29.6% overall. The principal narrative and main performance table use the three-trial-per-task results shown above.

5. Capability profile and failure modes

Spatial reasoning and execution precision

Astra often localizes objects and moves the gripper near a target but fails to establish the required final geometry. In Hat Hanging, it may grasp and lift the visor without aligning its opening with the rack branch. In Charger Alignment, it can reach the socket vicinity but fail to align both prongs sufficiently precisely. In Block Sweeping, the relevant spatial relation changes after every sweep because block positions and useful brush directions evolve.

The benchmark therefore distinguishes semantic localization from complete spatial reasoning. Recognizing an object is insufficient when success depends on relative orientation, independent support, insertion geometry, or a changing contact configuration.

Contact selection

Astra frequently chooses contacts that are visually plausible but mechanically ineffective. Marker Cap Removal requires a pulling grasp on the cap rather than merely a grasp near it. Orange and Jelly Handling require a grasp that simultaneously supports the deformable object and avoids damage. Book Pulling requires access to the middle book while another arm stabilizes the upper book. Nut Unscrewing requires contacts that permit later torque transmission and regrasping. Block Sweeping requires a useful brush-handle grasp and an orientation that transmits force toward the dustpan.

Contact selection depends jointly on geometry, accessibility, material properties, and force direction. A contact can be stable yet unsuitable for the intended operation.

Force regulation

Established contact does not guarantee sufficient or appropriate physical effect. Observed failure patterns include insufficient pull to remove a marker cap, disturbance of a book stack during extraction, incomplete drawer pull–return, excessive or insufficient compression of deformable objects, cap slippage during torque application, and wiping without adequate pressure.

The platform provides no dedicated force sensors and commands gripper opening rather than contact force. Astra must therefore infer contact effectiveness from visual and proprioceptive consequences.

Coordination and contact reconfiguration

The most severe failures involve multiple contacts or sequential stages. Jelly Handling couples grasp location and force within one interaction. Nut Unscrewing requires one arm to stabilize the rod while the other repeatedly restores a torque-producing contact. Stacked-Book Pushing Rotation requires changing pushing contacts while preserving stacking and support. Block Sweeping requires successive brush–block contacts with changing directions and force application.

These tasks expose a distinction between selecting a locally reasonable action and maintaining a coherent manipulation strategy over time.

Failure to accumulate progress

A recurrent pattern is successful local interaction without complete task completion. Astra may rotate a cap without fully disengaging it, move a drawer in only one direction, translate or rotate a book stack without reaching the target, move blocks while leaving some outside the dustpan, traverse the board with an eraser while leaving visible writing, or lift an object without leaving it independently supported.

The benchmark uses effective contact as a diagnostic concept: contact that transmits the force or torque needed for the main operation. Effective contact is not equivalent to success. Partial cap rotation, for example, demonstrates useful contact but not complete unscrewing and controlled separation.

6. Interpretation, validity, and future development

For GPT-6 Astra, the results indicate relatively strong performance in semantic object identification, localization, grasping, orientation adjustment, and basic placement. Reliability decreases when sustained contact must regulate resistance, pressure, deformation, or constrained motion. No L3 task was completed in either environment.

The observed boundary should not be interpreted as proof that Astra cannot perform any component of L3 manipulation. Several unsuccessful L3 trials contain local rotation, translation, contact, or partial sweeping. The measured limitation is reliable completion under cumulative physical requirements.

The matched simulation results support the use of simulation as a repeatable testbed for this model and task suite. The same monotonic decline across levels, identical success counts on 10 of 16 tasks, one-trial differences on the other six, and similar intermediate failures indicate descriptive correspondence between environments.

Several qualifications constrain this interpretation:

• Small sample size: Three trials per task provide limited statistical precision.
• Limited real-world throughput: Manual resets and low parallelization restrict repetition.
• No formal sim–real agreement analysis: Confidence intervals, significance tests, rank correlation, and formal agreement coefficients are absent.
• Human judgment in reality: Real outcomes are judged from videos, whereas simulation uses executable criteria.
• Unsynchronized observations: The cameras are not hardware synchronized.
• Approximate physical models: Jelly elasticity and whiteboard erasure are simulated proxies rather than directly measured material laws.
• Non-identical initial configurations: Simulated variants and real resets reproduce task specifications without necessarily matching every object pose one-to-one.
• No controlled factor ablation: Spatial reasoning, force regulation, contact reconfiguration, sensing, embodiment, prompting, and action interfaces are not independently varied.
• Platform scope: Findings apply to the tested ARX R5 platform, cameras, grippers, objects, task versions, and initial conditions.

Future benchmark development suggested by these limitations includes larger trial counts, formal sim–real agreement statistics, controlled task-factor ablations, broader robot embodiments and grippers, quantitative validation of contact forces and deformation, synchronized sensing, depth and tactile-sensing studies, systematic variation of initial conditions and interaction length, and metrics for intermediate progress and effective contact in addition to binary success.

PhysLadder’s principal methodological contribution is the organization of physical interaction by cumulative contact demands rather than by nominal task labels alone. Its evaluation target is not simply whether a robot can produce a local motion, but whether it can select mechanically useful contacts, regulate the resulting forces, reconfigure contacts when necessary, maintain support, verify environmental consequences, and convert local progress into complete task success. The benchmark consequently provides a structured basis for studying the capability boundary between basic manipulation and sustained, coordinated physical interaction.