---
title: 'SIMPLE: Humanoid Loco-Manipulation Benchmark'
url: https://www.emergentmind.com/topics/simple
type: topic
---

# SIMPLE: Humanoid Loco-Manipulation Benchmark

Searching arXiv for the specified paper and closely related benchmark context.
SIMPLE, short for **Simulation-Based Policy Learning and Evaluation for Humanoid Loco-manipulation**, is a unified simulation framework for humanoid policy learning and evaluation that targets the open problem of scalable, reproducible benchmarking for whole-body humanoid loco-manipulation [2606.08278]. It is designed around a dual-simulator architecture that couples MuJoCo’s contact-rich rigid-body dynamics with Isaac Sim’s photorealistic rendering, and it integrates standardized task definitions, large-scale assets and scenes, data generation pipelines, policy interfaces, and evaluation protocols into a single testbed [2606.08278]. The framework is motivated by the mismatch between the rapid progress of humanoid foundation models and the cost, slowness, and limited reproducibility of real-world whole-body testing, particularly for tasks requiring simultaneous locomotion, balance, and manipulation [2606.08278].

## 1. Historical motivation and problem setting

SIMPLE was introduced to address a specific benchmark gap in humanoid robotics: existing simulation benchmarks had driven progress in tabletop and wheeled manipulation, but they either omitted locomotion and whole-body balance or focused primarily on locomotion while offering limited object diversity and photorealistic perception [2606.08278]. The paper situates RLBench, LIBERO, Behavior-1K, RoboCasa365, ManiSkill-HAB, RoboVerse, and Simpler-Env on the tabletop or wheeled side, and Isaac Gym/Lab and HumanoidBench on the locomotion-focused side [2606.08278].

The central claim of SIMPLE is that humanoid loco-manipulation requires a benchmark that simultaneously supports accurate contact dynamics, visually realistic perception, scalable data collection, and standardized evaluation across diverse tasks and environments [2606.08278]. In this formulation, the benchmark is not merely a simulator wrapper but a full-stack infrastructure spanning physics, rendering, data pipelines, policy integration, and metrics [2606.08278].

Tasks are modeled as a Markov decision process $(S, A, P, R, \gamma)$, where $S$ includes RGB observations and proprioception, $A$ includes upper-body targets and locomotion commands, $P$ is the MuJoCo transition, $R$ encodes task success criteria, and $\gamma \in (0,1]$ [2606.08278]. The policy objective is
$$
J(\pi) = \mathbb{E}_{\tau \sim \pi}\left[\sum_{t=0}^{T} \gamma^t r_t\right].
$$
For behavior cloning, SIMPLE reports
$$
L_{\mathrm{BC}}(\theta)=\mathbb{E}_{(s,a)\sim D}\left[\|a-\pi_\theta(s)\|^2\right]
$$
for continuous actions and
$$
L_{\mathrm{CE}}=-\mathbb{E}_{(s,a)\sim D}[\log \pi_\theta(a\mid s)]
$$
for discrete actions [2606.08278]. Domain randomization is formalized by sampling $\phi \sim p(\phi)$ and optimizing the expected return
$$
J(\pi; p)=\mathbb{E}_{\phi\sim p}\left[\mathbb{E}_{\tau\sim \pi,\phi}\left[\sum \gamma^t r_t\right]\right].
$$

## 2. Dual-simulator architecture and control design

The defining systems contribution of SIMPLE is its decoupled simulation stack. MuJoCo handles rigid-body dynamics, contact resolution, collision geometries, and control integration, while Isaac Sim provides photorealistic ray-traced USD rendering [2606.08278]. Physics is integrated at **500 Hz**, whereas Isaac Sim renders RGB at **50 Hz** through a producer–consumer loop in which MuJoCo writes synchronized robot and object states into a shared buffer consumed by Isaac Sim [2606.08278].

This separation is central to the framework’s operating logic. The high-level policy runs at **50 Hz**, but whole-body controllers produce low-level joint position commands to MuJoCo at **500 Hz** [2606.08278]. SIMPLE therefore treats control and perception as partially decoupled subsystems: the policy predicts upper-body kinematic targets and navigation commands, while lower-body tracking controllers such as **AMO** or **SONIC** maintain balance and execute locomotion [2606.08278].

The action space is a **36-D whole-body action** composed of a **32-D upper-body joint target** and a **4-D locomotion command** $(v_x, v_y, v_{\mathrm{yaw}}, q_{\mathrm{yaw}})$ [2606.08278]. The upper-body portion includes arms, hands, and waist, with the paper specifying **14 DoF hands + 14 DoF arms + waist RPY + base height** [2606.08278]. Observations consist of Isaac Sim RGB frames plus a **32-D proprioceptive state**, and each task also exposes a natural-language instruction string [2606.08278]. Depth is not required for the benchmark policies; the framework focuses on RGB and proprioception [2606.08278].

A practical consequence of the architecture is the need to decouple high-fidelity rendering from interactive data collection. Isaac Sim ray tracing is reported to render at approximately **4 FPS/GPU**, so large-scale datasets are rendered offline rather than during policy execution or teleoperation [2606.08278]. During VR teleoperation, egocentric stereo RGB is instead streamed from MuJoCo’s native renderer to minimize latency [2606.08278]. The paper also notes that specific integrator, contact solver, or friction parameters are not modified beyond MuJoCo defaults [2606.08278].

## 3. Scale, task taxonomy, and asset pipeline

SIMPLE is notable for the breadth of its environment content. The framework includes **60 diverse whole-body loco-manipulation tasks**, **50 complete indoor scenes**, and **over 1,000 object assets**; the paper further specifies **53 GraspNet objects** and **more than 1,500 Objaverse assets processed** [2606.08278]. Scenes come from **HSSD-200**, and camera viewpoints, lighting, and materials are randomized during offline replay [2606.08278].

The task suite spans multiple categories. These include rigid pick-and-place tasks such as **BendPick** and **BendPickAndPlace**, bimanual handover tasks such as **Handover** and **BendHandover**, non-prehensile pushing such as **PushOfficeChair**, articulated object manipulation such as **OpenTrashCan**, **OpenFaucet**, **OpenOven**, and **CloseDoor**, and mobile whole-body coordination tasks such as **XMovePick**, **XMoveBendPick**, **LocomotionPickBetweenTables**, and **PickAndPlaceAndHugContainer** [2606.08278].

Asset preprocessing is a substantial part of the framework. **CoACD convex decomposition** is used to generate collision geometries for MuJoCo; stable resting poses are enumerated by dropping each object in simulation; assets are converted to USD with high-resolution textures for Isaac Sim; and **BoDex** synthesizes dexterous grasp configurations for each stable pose, caching them for reuse [2606.08278]. This pipeline links physical stability, manipulation affordances, and visual fidelity into a shared asset representation.

A concise summary of the environment scale is given below.

| Component | Reported scale | Notes |
|---|---:|---|
| Tasks | 60 | Whole-body loco-manipulation |
| Scenes | 50 | Complete indoor scenes from HSSD-200 |
| Objects | >1,000 | Concretely 53 GraspNet and >1,500 Objaverse processed |

This scale matters because SIMPLE is intended not only for policy evaluation but also for data generation and robustness testing under standardized domain randomization [2606.08278]. A plausible implication is that the framework treats diversity in scenes, assets, and task logic as a first-class benchmark variable rather than as an implementation detail.

## 4. Data generation, rendering, and domain randomization

SIMPLE integrates two data generation pipelines: **automated motion planning** and **low-latency VR teleoperation** [2606.08278]. The motion-planning pipeline uses **CuRobo**, including its dynamics-aware v2 variant, to generate kinematic dual-arm trajectories from BoDex grasp targets [2606.08278]. A scripted policy decomposes tasks into atomic actions such as approach, grasp, lift, and place, while a scripted base policy coordinates lower-body movement [2606.08278]. Stable pose enumeration and cached grasps ensure feasible starts, and CuRobo provides minimum-jerk trajectory smoothing with collision checks embedded in planning [2606.08278].

The teleoperation pipeline uses a **PICO XR headset** and streams egocentric stereo video from MuJoCo directly to the headset [2606.08278]. Operator hand motions are retargeted to the humanoid upper body via inverse kinematics, and joystick inputs specify navigation commands, while AMO or SONIC handle lower-body and balance control [2606.08278]. Demonstrations are recorded at **50 Hz**, with low-level control still running at **500 Hz**, and trajectories are exported in **LeRobot trajectory format** [2606.08278].

The paper reports **6,000 rendered episodes** for policy learning and benchmarking [2606.08278]. It also reports collection throughput for three representative task groups:

| Collection mode | T1 pick-place | T2 handover | T3 mobile pick-place |
|---|---|---|---|
| Motion Planning (Sim) | 58.9 demos/hr, 61.1 s | 32.7 demos/hr, 109.8 s | 24.0 demos/hr, 150.0 s |
| Teleoperation (Real) | 206.8 demos/hr, 17.4 s | 130.9 demos/hr, 27.5 s | 87.2 demos/hr, 41.3 s |
| Teleoperation (Sim) | 310.3 demos/hr, 11.6 s | 197.8 demos/hr, 18.2 s | 156.5 demos/hr, 23.0 s |

The framework’s visual dataset is produced through **offline replay** in Isaac Sim, where trajectories collected through planning or teleoperation are rerendered with photorealistic RGB [2606.08278]. Domain randomization includes materials via **vMaterials**, lighting positions and intensities, camera viewpoints, target and distractor instances, initial poses, and language instructions [2606.08278]. The paper explicitly states that this randomization is intended to close the visual sim-to-real gap and support robust perception [2606.08278].

This separation between low-latency collection and high-fidelity offline replay is one of SIMPLE’s more distinctive design choices. It allows interactive control to remain feasible while preserving a photorealistic observation channel for large-scale training and evaluation [2606.08278].

## 5. Policy integration and training configurations

SIMPLE natively benchmarks three broad policy classes: lightweight imitation networks, large VLAs, and world action models [2606.08278]. The lightweight class includes **Diffusion Policy (DP)** and **ACT**. The VLA class includes **Ψ0**, **π0.5**, **GR00T-N1.6**, **InternVLA-M1**, and **EgoVLA**. The WAM class is represented by **DreamZero**, which jointly predicts future video and actions to plan in image space [2606.08278].

The paper reports concrete training configurations for each integrated policy. For example, **Ψ0** is fine-tuned for **40k steps on 8×A100**, with per-device batch **16**, cosine learning rate **1e−4**, **1,000 warmup**, **bf16**, a **36-D action head**, and chunk size **30** [2606.08278]. **DreamZero** uses **LoRA** fine-tuning with rank **4** and $\alpha=4$, also for **40k steps on 8×A100**, with per-device batch **1**, learning rate **1e−4**, **5% warmup**, action horizon **48**, RGB resolution **320×176**, and context frames **17/25/33** for chunk sizes **2/3/4** [2606.08278].

Other reported settings include **GR00T-N1.6** at **20k steps on 3×A100**, **InternVLA-M1** with a frozen VLM backbone and a fine-tuned action head for **30k steps**, **DP** with a **ResNet-18** visual encoder and **100 denoising steps at inference**, and **ACT** with a **36-D action head**, chunk size **100**, and a **4 encoder, 1 decoder layer** LeRobot configuration [2606.08278].

These details are significant because SIMPLE is intended as a cross-paradigm benchmark rather than a benchmark optimized for a single policy family. The paper’s integrated remote policy server abstraction for policies such as **Ψ0**, **π0.5**, and **GR00T** further supports this role by decoupling inference compute from the simulation host via HTTP or WebSocket interfaces [2606.08278].

## 6. Evaluation methodology, quantitative results, and transfer findings

SIMPLE standardizes evaluation through three domain-randomization levels. **Level 0** uses base scenes with distractor objects and background variation; **Level 1** adds material and lighting randomization; **Level 2** adds spatial randomization of object and robot initial poses [2606.08278]. Success is measured by programmatic task-specific checks such as lifting an object above **5 cm**, placing it into a container, opening an articulated part beyond a threshold, pushing beyond a target distance, or transferring between tables [2606.08278].

The primary metric is **success rate over $N$ trials**, often **10 per level**, with confidence intervals computed using the **Wilson score interval** [2606.08278]. The framework also reports data efficiency, training-step budgets, and robustness across Levels 0–2, including domain-randomization and data-scaling ablations [2606.08278].

Across six representative tasks, the benchmark highlights several distinct policy profiles. **Ψ0** is reported as strong overall, with results such as **XMovePick 10/10/6**, **BendPick 10/10/10**, **Handover 7/7/10**, **TabletopGrasp 10/10/8**, and **XMoveBendPick 10/9/9** across Levels 0/1/2 [2606.08278]. **DreamZero** shows strong generalization, including **XMovePick 10/10/10** and **Handover 7/8/9** [2606.08278]. **π0.5** performs well on some tasks, such as **BendPick 10/10/8**, but is reported as weaker on precise base-motion tasks, for example **XMoveBendPick 0/0/0** [2606.08278]. **ACT** is described as competitive and data-efficient, with results such as **BendPick 10/9/9**, **Handover 7/7/10**, and **XMoveBendPick 9/10/10** [2606.08278].

Extended-task evaluation with **Ψ0** shows a clear difficulty gradient. **CloseDoor** and **PushOfficeChair** achieve **10/10/10**, whereas **OpenTrashCan** yields **7/9/9**, **OpenOven** yields **7/5/4**, and **OpenFaucet** yields **3/3/4**, which the paper uses to confirm the greater difficulty of fine-grained contact tasks [2606.08278].

The ablation studies sharpen several methodological points. Training only on **Level 0** randomization yields success **0.80** on Eval Set 0 but drops to **0.50** on Set 2, whereas mixed **Level 0+1** training improves Set 2 to **0.80** while retaining in-distribution performance [2606.08278]. In teleoperation data scaling, increasing from **10 trajectories** to **100 trajectories** changes **XMoveBendPick** performance from **0.50/0.60/0.30** to **1.00/0.90/0.90** across Levels 0/1/2 [2606.08278]. Teleoperation-only data also outperforms motion-planning-only data on complex whole-body tasks, with average **7.56 vs 5.00** across **BendPick**, **Mobile pick-place**, and **XMoveBendPick** [2606.08278].

A central empirical claim is that simulation rankings echo real-world results, indicating that SIMPLE is a faithful proxy, although exact Pearson $r$, confidence intervals, or $p$-values are not reported [2606.08278]. The paper also demonstrates **zero-shot transfer** from SIMPLE-trained policies to physical humanoid robots under similar settings: for **pick-and-place**, simulation yields **9/10 = 0.90** and real world **8/10 = 0.80**; for **handover**, simulation yields **10/10 = 1.00** and real world **8/10 = 0.80** [2606.08278]. The reported evidence suggests that SIMPLE’s combination of contact fidelity and photorealistic visuals bridges part of the sim-to-real gap [2606.08278].

## 7. Reproducibility, limitations, and research significance

SIMPLE emphasizes reproducibility through a **gymnasium**-style environment API, a unified **BaseDualSimEnv** orchestration layer, standardized domain-randomization modules, remote policy servers, and **LeRobot** trajectory logging and replay [2606.08278]. Scenario authoring encapsulates success criteria, instruction strings, automated planning decomposition, and reset randomization in task classes [2606.08278]. The authors state that they will open-source the entire codebase, and note that Isaac Sim is **Apache-2.0** and that SIMPLE builds on open toolchains [2606.08278].

The principal systems limitation is rendering throughput. Photorealistic ray tracing in Isaac Sim is reported at roughly **4 FPS/GPU**, which makes large-scale RGB dataset generation time-intensive [2606.08278]. The paper therefore identifies faster rasterized pipelines and hybrid rendering strategies as natural future directions [2606.08278]. A second limitation is the **rigid-body assumption**: deformables such as cloth, rope, and food are out of scope [2606.08278]. The authors also identify broader sensor modalities, richer language grounding, more articulated assets, broader locomotion terrains, and standardized sim-to-real calibration recipes as future work [2606.08278].

In comparative terms, SIMPLE extends the principle demonstrated by Simpler-Env—that simulation can be a faithful proxy for real-world manipulation—into the harder setting of whole-body humanoid loco-manipulation [2606.08278]. Its contribution is therefore not just a benchmark dataset or a rendering pipeline, but a standardized experimental substrate that aligns physics, perception, policy APIs, data collection, and evaluation under one framework [2606.08278]. This suggests a shift in humanoid robotics benchmarking away from narrow task suites and toward integrated, reproducible infrastructures capable of supporting both foundation-model training and cross-policy evaluation.

SIMPLE can thus be understood as a full-stack benchmark for humanoid robotics: a system in which locomotion, manipulation, contact-rich control, photorealistic perception, scalable demonstration collection, and sim-to-real evaluation are co-designed rather than treated as separate engineering layers [2606.08278].

Source: https://www.emergentmind.com/topics/simple