SIMPLE: Humanoid Loco-Manipulation Benchmark

• SIMPLE is a unified simulation framework for humanoid loco-manipulation that integrates high-fidelity physics and photorealistic rendering through a dual-simulator design.
• The framework supports 60 diverse tasks across complete indoor scenes with over 1,000 objects, enabling standardized evaluation and scalable data generation.
• Its architecture decouples high-frequency physics control from lower-frequency visual processing, enhancing reproducibility and sim-to-real transfer in policy learning.

Searching arXiv for the specified paper and closely related benchmark context. SIMPLE, short for Simulation-Based Policy Learning and Evaluation for Humanoid Loco-manipulation, is a unified simulation framework for humanoid policy learning and evaluation that targets the open problem of scalable, reproducible benchmarking for whole-body humanoid loco-manipulation (Wei et al., 6 Jun 2026). It is designed around a dual-simulator architecture that couples MuJoCo’s contact-rich rigid-body dynamics with Isaac Sim’s photorealistic rendering, and it integrates standardized task definitions, large-scale assets and scenes, data generation pipelines, policy interfaces, and evaluation protocols into a single testbed (Wei et al., 6 Jun 2026). The framework is motivated by the mismatch between the rapid progress of humanoid foundation models and the cost, slowness, and limited reproducibility of real-world whole-body testing, particularly for tasks requiring simultaneous locomotion, balance, and manipulation (Wei et al., 6 Jun 2026).

1. Historical motivation and problem setting

SIMPLE was introduced to address a specific benchmark gap in humanoid robotics: existing simulation benchmarks had driven progress in tabletop and wheeled manipulation, but they either omitted locomotion and whole-body balance or focused primarily on locomotion while offering limited object diversity and photorealistic perception (Wei et al., 6 Jun 2026). The paper situates RLBench, LIBERO, Behavior-1K, RoboCasa365, ManiSkill-HAB, RoboVerse, and Simpler-Env on the tabletop or wheeled side, and Isaac Gym/Lab and HumanoidBench on the locomotion-focused side (Wei et al., 6 Jun 2026).

The central claim of SIMPLE is that humanoid loco-manipulation requires a benchmark that simultaneously supports accurate contact dynamics, visually realistic perception, scalable data collection, and standardized evaluation across diverse tasks and environments (Wei et al., 6 Jun 2026). In this formulation, the benchmark is not merely a simulator wrapper but a full-stack infrastructure spanning physics, rendering, data pipelines, policy integration, and metrics (Wei et al., 6 Jun 2026).

Tasks are modeled as a Markov decision process $(S, A, P, R, \gamma)$, where $S$ includes RGB observations and proprioception, $A$ includes upper-body targets and locomotion commands, $P$ is the MuJoCo transition, $R$ encodes task success criteria, and $\gamma \in (0,1]$ (Wei et al., 6 Jun 2026). The policy objective is

$$J(\pi) = \mathbb{E}_{\tau \sim \pi}\left[\sum_{t=0}^{T} \gamma^t r_t\right].$$

For behavior cloning, SIMPLE reports

$$L_{\mathrm{BC}}(\theta)=\mathbb{E}_{(s,a)\sim D}\left[\|a-\pi_\theta(s)\|^2\right]$$

for continuous actions and

$$L_{\mathrm{CE}}=-\mathbb{E}_{(s,a)\sim D}[\log \pi_\theta(a\mid s)]$$

for discrete actions (Wei et al., 6 Jun 2026). Domain randomization is formalized by sampling $\phi \sim p(\phi)$ and optimizing the expected return

$S$0

2. Dual-simulator architecture and control design

The defining systems contribution of SIMPLE is its decoupled simulation stack. MuJoCo handles rigid-body dynamics, contact resolution, collision geometries, and control integration, while Isaac Sim provides photorealistic ray-traced USD rendering (Wei et al., 6 Jun 2026). Physics is integrated at 500 Hz, whereas Isaac Sim renders RGB at 50 Hz through a producer–consumer loop in which MuJoCo writes synchronized robot and object states into a shared buffer consumed by Isaac Sim (Wei et al., 6 Jun 2026).

This separation is central to the framework’s operating logic. The high-level policy runs at 50 Hz, but whole-body controllers produce low-level joint position commands to MuJoCo at 500 Hz (Wei et al., 6 Jun 2026). SIMPLE therefore treats control and perception as partially decoupled subsystems: the policy predicts upper-body kinematic targets and navigation commands, while lower-body tracking controllers such as AMO or SONIC maintain balance and execute locomotion (Wei et al., 6 Jun 2026).

The action space is a 36-D whole-body action composed of a 32-D upper-body joint target and a 4-D locomotion command $S$1 (Wei et al., 6 Jun 2026). The upper-body portion includes arms, hands, and waist, with the paper specifying 14 DoF hands + 14 DoF arms + waist RPY + base height (Wei et al., 6 Jun 2026). Observations consist of Isaac Sim RGB frames plus a 32-D proprioceptive state, and each task also exposes a natural-language instruction string (Wei et al., 6 Jun 2026). Depth is not required for the benchmark policies; the framework focuses on RGB and proprioception (Wei et al., 6 Jun 2026).

A practical consequence of the architecture is the need to decouple high-fidelity rendering from interactive data collection. Isaac Sim ray tracing is reported to render at approximately 4 FPS/GPU, so large-scale datasets are rendered offline rather than during policy execution or teleoperation (Wei et al., 6 Jun 2026). During VR teleoperation, egocentric stereo RGB is instead streamed from MuJoCo’s native renderer to minimize latency (Wei et al., 6 Jun 2026). The paper also notes that specific integrator, contact solver, or friction parameters are not modified beyond MuJoCo defaults (Wei et al., 6 Jun 2026).

3. Scale, task taxonomy, and asset pipeline

SIMPLE is notable for the breadth of its environment content. The framework includes 60 diverse whole-body loco-manipulation tasks, 50 complete indoor scenes, and over 1,000 object assets; the paper further specifies 53 GraspNet objects and more than 1,500 Objaverse assets processed (Wei et al., 6 Jun 2026). Scenes come from HSSD-200, and camera viewpoints, lighting, and materials are randomized during offline replay (Wei et al., 6 Jun 2026).

The task suite spans multiple categories. These include rigid pick-and-place tasks such as BendPick and BendPickAndPlace, bimanual handover tasks such as Handover and BendHandover, non-prehensile pushing such as PushOfficeChair, articulated object manipulation such as OpenTrashCan, OpenFaucet, OpenOven, and CloseDoor, and mobile whole-body coordination tasks such as XMovePick, XMoveBendPick, LocomotionPickBetweenTables, and PickAndPlaceAndHugContainer (Wei et al., 6 Jun 2026).

Asset preprocessing is a substantial part of the framework. CoACD convex decomposition is used to generate collision geometries for MuJoCo; stable resting poses are enumerated by dropping each object in simulation; assets are converted to USD with high-resolution textures for Isaac Sim; and BoDex synthesizes dexterous grasp configurations for each stable pose, caching them for reuse (Wei et al., 6 Jun 2026). This pipeline links physical stability, manipulation affordances, and visual fidelity into a shared asset representation.

A concise summary of the environment scale is given below.

Component	Reported scale	Notes
Tasks	60	Whole-body loco-manipulation
Scenes	50	Complete indoor scenes from HSSD-200
Objects	>1,000	Concretely 53 GraspNet and >1,500 Objaverse processed

This scale matters because SIMPLE is intended not only for policy evaluation but also for data generation and robustness testing under standardized domain randomization (Wei et al., 6 Jun 2026). A plausible implication is that the framework treats diversity in scenes, assets, and task logic as a first-class benchmark variable rather than as an implementation detail.

4. Data generation, rendering, and domain randomization

SIMPLE integrates two data generation pipelines: automated motion planning and low-latency VR teleoperation (Wei et al., 6 Jun 2026). The motion-planning pipeline uses CuRobo, including its dynamics-aware v2 variant, to generate kinematic dual-arm trajectories from BoDex grasp targets (Wei et al., 6 Jun 2026). A scripted policy decomposes tasks into atomic actions such as approach, grasp, lift, and place, while a scripted base policy coordinates lower-body movement (Wei et al., 6 Jun 2026). Stable pose enumeration and cached grasps ensure feasible starts, and CuRobo provides minimum-jerk trajectory smoothing with collision checks embedded in planning (Wei et al., 6 Jun 2026).

The teleoperation pipeline uses a PICO XR headset and streams egocentric stereo video from MuJoCo directly to the headset (Wei et al., 6 Jun 2026). Operator hand motions are retargeted to the humanoid upper body via inverse kinematics, and joystick inputs specify navigation commands, while AMO or SONIC handle lower-body and balance control (Wei et al., 6 Jun 2026). Demonstrations are recorded at 50 Hz, with low-level control still running at 500 Hz, and trajectories are exported in LeRobot trajectory format (Wei et al., 6 Jun 2026).

The paper reports 6,000 rendered episodes for policy learning and benchmarking (Wei et al., 6 Jun 2026). It also reports collection throughput for three representative task groups:

Collection mode	T1 pick-place	T2 handover	T3 mobile pick-place
Motion Planning (Sim)	58.9 demos/hr, 61.1 s	32.7 demos/hr, 109.8 s	24.0 demos/hr, 150.0 s
Teleoperation (Real)	206.8 demos/hr, 17.4 s	130.9 demos/hr, 27.5 s	87.2 demos/hr, 41.3 s
Teleoperation (Sim)	310.3 demos/hr, 11.6 s	197.8 demos/hr, 18.2 s	156.5 demos/hr, 23.0 s

The framework’s visual dataset is produced through offline replay in Isaac Sim, where trajectories collected through planning or teleoperation are rerendered with photorealistic RGB (Wei et al., 6 Jun 2026). Domain randomization includes materials via vMaterials, lighting positions and intensities, camera viewpoints, target and distractor instances, initial poses, and language instructions (Wei et al., 6 Jun 2026). The paper explicitly states that this randomization is intended to close the visual sim-to-real gap and support robust perception (Wei et al., 6 Jun 2026).

This separation between low-latency collection and high-fidelity offline replay is one of SIMPLE’s more distinctive design choices. It allows interactive control to remain feasible while preserving a photorealistic observation channel for large-scale training and evaluation (Wei et al., 6 Jun 2026).

5. Policy integration and training configurations

SIMPLE natively benchmarks three broad policy classes: lightweight imitation networks, large VLAs, and world action models (Wei et al., 6 Jun 2026). The lightweight class includes Diffusion Policy (DP) and ACT. The VLA class includes Ψ0, π0.5, GR00T-N1.6, InternVLA-M1, and EgoVLA. The WAM class is represented by DreamZero, which jointly predicts future video and actions to plan in image space (Wei et al., 6 Jun 2026).

The paper reports concrete training configurations for each integrated policy. For example, Ψ0 is fine-tuned for 40k steps on 8×A100, with per-device batch 16, cosine learning rate 1e−4, 1,000 warmup, bf16, a 36-D action head, and chunk size 30 (Wei et al., 6 Jun 2026). DreamZero uses LoRA fine-tuning with rank 4 and $S$2, also for 40k steps on 8×A100, with per-device batch 1, learning rate 1e−4, 5% warmup, action horizon 48, RGB resolution 320×176, and context frames 17/25/33 for chunk sizes 2/3/4 (Wei et al., 6 Jun 2026).

Other reported settings include GR00T-N1.6 at 20k steps on 3×A100, InternVLA-M1 with a frozen VLM backbone and a fine-tuned action head for 30k steps, DP with a ResNet-18 visual encoder and 100 denoising steps at inference, and ACT with a 36-D action head, chunk size 100, and a 4 encoder, 1 decoder layer LeRobot configuration (Wei et al., 6 Jun 2026).

These details are significant because SIMPLE is intended as a cross-paradigm benchmark rather than a benchmark optimized for a single policy family. The paper’s integrated remote policy server abstraction for policies such as Ψ0, π0.5, and GR00T further supports this role by decoupling inference compute from the simulation host via HTTP or WebSocket interfaces (Wei et al., 6 Jun 2026).

6. Evaluation methodology, quantitative results, and transfer findings

SIMPLE standardizes evaluation through three domain-randomization levels. Level 0 uses base scenes with distractor objects and background variation; Level 1 adds material and lighting randomization; Level 2 adds spatial randomization of object and robot initial poses (Wei et al., 6 Jun 2026). Success is measured by programmatic task-specific checks such as lifting an object above 5 cm, placing it into a container, opening an articulated part beyond a threshold, pushing beyond a target distance, or transferring between tables (Wei et al., 6 Jun 2026).

The primary metric is success rate over $S$3 trials, often 10 per level, with confidence intervals computed using the Wilson score interval (Wei et al., 6 Jun 2026). The framework also reports data efficiency, training-step budgets, and robustness across Levels 0–2, including domain-randomization and data-scaling ablations (Wei et al., 6 Jun 2026).

Across six representative tasks, the benchmark highlights several distinct policy profiles. Ψ0 is reported as strong overall, with results such as XMovePick 10/10/6, BendPick 10/10/10, Handover 7/7/10, TabletopGrasp 10/10/8, and XMoveBendPick 10/9/9 across Levels 0/1/2 (Wei et al., 6 Jun 2026). DreamZero shows strong generalization, including XMovePick 10/10/10 and Handover 7/8/9 (Wei et al., 6 Jun 2026). π0.5 performs well on some tasks, such as BendPick 10/10/8, but is reported as weaker on precise base-motion tasks, for example XMoveBendPick 0/0/0 (Wei et al., 6 Jun 2026). ACT is described as competitive and data-efficient, with results such as BendPick 10/9/9, Handover 7/7/10, and XMoveBendPick 9/10/10 (Wei et al., 6 Jun 2026).

Extended-task evaluation with Ψ0 shows a clear difficulty gradient. CloseDoor and PushOfficeChair achieve 10/10/10, whereas OpenTrashCan yields 7/9/9, OpenOven yields 7/5/4, and OpenFaucet yields 3/3/4, which the paper uses to confirm the greater difficulty of fine-grained contact tasks (Wei et al., 6 Jun 2026).

The ablation studies sharpen several methodological points. Training only on Level 0 randomization yields success 0.80 on Eval Set 0 but drops to 0.50 on Set 2, whereas mixed Level 0+1 training improves Set 2 to 0.80 while retaining in-distribution performance (Wei et al., 6 Jun 2026). In teleoperation data scaling, increasing from 10 trajectories to 100 trajectories changes XMoveBendPick performance from 0.50/0.60/0.30 to 1.00/0.90/0.90 across Levels 0/1/2 (Wei et al., 6 Jun 2026). Teleoperation-only data also outperforms motion-planning-only data on complex whole-body tasks, with average 7.56 vs 5.00 across BendPick, Mobile pick-place, and XMoveBendPick (Wei et al., 6 Jun 2026).

A central empirical claim is that simulation rankings echo real-world results, indicating that SIMPLE is a faithful proxy, although exact Pearson $S$4, confidence intervals, or $S$5-values are not reported (Wei et al., 6 Jun 2026). The paper also demonstrates zero-shot transfer from SIMPLE-trained policies to physical humanoid robots under similar settings: for pick-and-place, simulation yields 9/10 = 0.90 and real world 8/10 = 0.80; for handover, simulation yields 10/10 = 1.00 and real world 8/10 = 0.80 (Wei et al., 6 Jun 2026). The reported evidence suggests that SIMPLE’s combination of contact fidelity and photorealistic visuals bridges part of the sim-to-real gap (Wei et al., 6 Jun 2026).

7. Reproducibility, limitations, and research significance

SIMPLE emphasizes reproducibility through a gymnasium-style environment API, a unified BaseDualSimEnv orchestration layer, standardized domain-randomization modules, remote policy servers, and LeRobot trajectory logging and replay (Wei et al., 6 Jun 2026). Scenario authoring encapsulates success criteria, instruction strings, automated planning decomposition, and reset randomization in task classes (Wei et al., 6 Jun 2026). The authors state that they will open-source the entire codebase, and note that Isaac Sim is Apache-2.0 and that SIMPLE builds on open toolchains (Wei et al., 6 Jun 2026).

The principal systems limitation is rendering throughput. Photorealistic ray tracing in Isaac Sim is reported at roughly 4 FPS/GPU, which makes large-scale RGB dataset generation time-intensive (Wei et al., 6 Jun 2026). The paper therefore identifies faster rasterized pipelines and hybrid rendering strategies as natural future directions (Wei et al., 6 Jun 2026). A second limitation is the rigid-body assumption: deformables such as cloth, rope, and food are out of scope (Wei et al., 6 Jun 2026). The authors also identify broader sensor modalities, richer language grounding, more articulated assets, broader locomotion terrains, and standardized sim-to-real calibration recipes as future work (Wei et al., 6 Jun 2026).

In comparative terms, SIMPLE extends the principle demonstrated by Simpler-Env—that simulation can be a faithful proxy for real-world manipulation—into the harder setting of whole-body humanoid loco-manipulation (Wei et al., 6 Jun 2026). Its contribution is therefore not just a benchmark dataset or a rendering pipeline, but a standardized experimental substrate that aligns physics, perception, policy APIs, data collection, and evaluation under one framework (Wei et al., 6 Jun 2026). This suggests a shift in humanoid robotics benchmarking away from narrow task suites and toward integrated, reproducible infrastructures capable of supporting both foundation-model training and cross-policy evaluation.

SIMPLE can thus be understood as a full-stack benchmark for humanoid robotics: a system in which locomotion, manipulation, contact-rich control, photorealistic perception, scalable demonstration collection, and sim-to-real evaluation are co-designed rather than treated as separate engineering layers (Wei et al., 6 Jun 2026).