LabUtopia Benchmark for Lab Manipulation

• LabUtopia Benchmark is a simulation framework for evaluating scientific lab manipulation, integrating high-fidelity physics, chemistry, and procedural scene generation.
• It combines LabSim’s physical and chemical engines, LabScene’s asset-driven procedural generation, and LabBench’s hierarchical task design for comprehensive performance evaluation.
• The benchmark specifically targets challenges such as fluid handling, long-horizon planning, and precise instrument control under both in-distribution and out-of-distribution conditions.

Searching arXiv for the primary LabUtopia papers and closely related benchmark papers mentioned in the provided data. arxiv_search({"query":"id:(Li et al., 28 May 2025) OR id:(Ren et al., 11 Jun 2026) OR id:(Jin et al., 9 Jun 2026)", "max_results": 10, "sort_by": "submittedDate", "sort_order": "descending"}) I couldn’t retrieve tool output in this interface, so I will rely on the provided arXiv metadata while keeping citations limited to the listed papers. LabUtopia Benchmark is a simulation-and-benchmark framework for scientific embodied agents operating in laboratory settings rather than household or generic manipulation environments. In its 2025 formulation, LabUtopia integrates LabSim, LabScene, and LabBench to support high-fidelity simulation, procedural lab-scene generation, and hierarchical evaluation of scientific embodied behavior; a later 2026 paper uses LabUtopia as the evaluation benchmark for vision-language-action policies on scientific laboratory manipulation under in-distribution and out-of-distribution conditions (Li et al., 28 May 2025, Ren et al., 11 Jun 2026). Across these formulations, the benchmark targets laboratory-specific requirements such as perception of physical and chemical transformations, precise instrument handling, long-horizon planning, transparent liquids, and variation across robot embodiments and scene layouts.

1. Scope, motivation, and benchmark identity

LabUtopia was introduced to address a gap in embodied-AI evaluation for scientific laboratories. The 2025 paper argues that laboratory work is fundamentally harder than standard embodied-AI settings because it requires perception of physical and chemical transformations, precise instrument handling, long-horizon planning across multi-step experimental protocols, and generalization across varying laboratory layouts, instruments, and materials (Li et al., 28 May 2025). The later 2026 paper frames the same benchmark family as a high-fidelity simulation benchmark for scientific embodied agents, designed specifically to evaluate vision-language-action (VLA) policies on scientific laboratory manipulation rather than household or tabletop tasks (Ren et al., 11 Jun 2026).

A central point in both papers is that prior benchmarks are misaligned with laboratory execution. The cited comparison set includes household or generic embodied benchmarks such as ALFRED, Behavior / Behavior-1K, RLBench, Ravens, VLMbench, VIMA-Bench, ManiSkill3, ClevrSkills, Arnold, and manipulation benchmarks such as Meta-World, RLBench, ManiSkill, CALVIN, LIBERO, RoboCasa, BridgeData v2, Open X-Embodiment, DROID. The reported limitation is not merely domain shift in object appearance, but the absence of laboratory-specific phenomena: instrument-specific interactions, liquid handling, protocol-level sequences, varying embodiments and camera setups, and state changes such as heating, opening, and pouring (Li et al., 28 May 2025, Ren et al., 11 Jun 2026).

The benchmark’s identity is somewhat layered. In the 2025 paper, LabBench is the explicit benchmark component within the larger LabUtopia suite. In the 2026 paper, LabUtopia itself is used as the benchmark name for VLA evaluation. This suggests that “LabUtopia Benchmark” can refer either to the full hierarchical LabBench inside the suite or to the later standardized LabUtopia evaluation harness used for laboratory VLA policies.

2. Infrastructure: LabSim, LabScene, and later benchmark generation

The 2025 formulation defines three main components. LabSim is a high-fidelity simulator for physical and chemical interactions; LabScene is a procedural lab-scene generator with diverse assets; and LabBench is the hierarchical benchmark for evaluation (Li et al., 28 May 2025). The benchmark depends on these lower layers because laboratory evaluation requires physically and chemically meaningful state evolution rather than only rigid-body interaction.

LabSim is built on Isaac Sim, supports rigid, deformable, and fluid objects, uses PhysX 5 physics, includes a GPU-accelerated fluid simulator, and adds a chemical engine. The chemical engine is built from a curated database of 200 common chemical substances from PubChem, stores properties like color, molar mass, and pH, and uses GPT-4o mini to infer chemical transformations from reactants (Li et al., 28 May 2025). The paper presents these mechanisms as necessary for scientifically meaningful tasks such as fluid transfer and reaction-driven state transitions.

LabScene is described as a procedural generation pipeline that uses a curated asset library and combines grid stochastic sampling with constraint-aware search. The 2025 paper reports over 100 diverse and physically plausible lab environments, over 100 laboratory scenes and 100 scientific instruments, and in the abstract more than 200 scene and instrument assets; the appendix further specifies about 60 categories of laboratory equipment assets and around 80 types of transparent glassware and plasticware assets (Li et al., 28 May 2025). The same paper also states that the authors collected over one thousand candidate scenes from designer websites and filtered and standardized them with expert review.

The 2026 paper extends the benchmark’s construction logic through RoboGenesis, a simulation-based workflow and data engine linked tightly to LabUtopia evaluation. In that account, scenes are built from a curated LabAssetLibrary and LabTextureLibrary, with 2,947 annotated assets, over 1,000 texture images, and 10,000 laboratory scenes. The scene-construction pipeline uses a placement solver with numeric constraints, performs ten validation checks, and assigns a 0–100 quality score, rejecting scenes below threshold (Ren et al., 11 Jun 2026). This suggests an evolution from a simulator-and-benchmark suite toward a larger procedural data-and-evaluation ecosystem for laboratory embodied learning.

3. Hierarchical task design and benchmark contents

The 2025 paper presents LabBench as a five-level hierarchy of increasing complexity, intended to evaluate primitive skills, compositional behavior, distribution-shift robustness, long-horizon procedures, and navigation-plus-manipulation (Li et al., 28 May 2025).

Level	Focus	Tasks listed
1	Atomic Manipulation Tasks	Pick, Pour, Place, Press, Shake, Stir, Open Door, Close Drawer / Close Door
2	Short-Horizon Manipulation Tasks	Pour Liquid, Shake Beaker, Transport Beaker, Heater Beaker, Stir GlassRod, Operate Drawer
3	Generalizable Short Manipulation Tasks	Pick, Press, Transport Beaker, Heater Beaker
4	Long-Horizon Manipulation Tasks	Clean Beaker, Drying Beaker
5	Mobile Manipulation Tasks	Navigation and Manipulation

At Level 1, the benchmark evaluates single-step laboratory actions such as Pick, Pour, Place, Press, Shake, Stir, Open Door, and Close Drawer / Close Door. These are framed as laboratory-specific primitives: grabbing beakers or flasks without tipping them, pouring liquids into target regions, pressing instrument buttons, stirring with a glass rod, and opening or closing cabinet elements (Li et al., 28 May 2025).

At Level 2, the benchmark composes 2–3 atomic actions into compound objectives. The listed tasks are Pour Liquid, Shake Beaker, Transport Beaker, Heater Beaker, Stir GlassRod, and Operate Drawer. The paper gives examples such as Pour Liquid = Pick + Pour, Transport Beaker = Pick + Place, Heater Beaker = Pick + Place + Press, Stir GlassRod = Pick + Stir, and Operate Drawer = Open + Close (Li et al., 28 May 2025).

At Level 3, the benchmark keeps short manipulation structure but evaluates under distribution shifts, including unseen object shapes, unseen sizes, different appearances or materials, different button colors or positions, different desktop materials, and different target locations. The appendix lists Pick, Press, Transport Beaker, and Heater Beaker at this level (Li et al., 28 May 2025).

At Level 4, LabBench evaluates full laboratory procedures. The listed tasks are Clean Beaker and Drying Beaker, with long action sequences such as Clean Beaker: Pick, Pour, Place, Pick, Shake, Pour, Place and Drying Beaker: Open, Pick, Place, Pick, Place, Close (Li et al., 28 May 2025). At Level 5, Navigation and Manipulation integrates path planning with a terminal pick action.

The 2026 paper presents a narrower but standardized benchmark slice with six laboratory operations: Pick Up, Press Button, Open Door, Pour Liquid, Heat Beaker, and Transport Beaker (Ren et al., 11 Jun 2026). It characterizes Press Button as near-saturated and relatively easy, Heat Beaker as also relatively easy for strong models, Pick Up and Open Door as harder due to grasping and articulated interaction, Transport Beaker as harder due to stable grasp and precise placement, and Pour Liquid as the hardest task overall. This suggests that the later VLA evaluation harness emphasizes Levels 1–2 of the original hierarchy while keeping the laboratory domain constraints central.

4. Evaluation protocol, splits, and metrics

The benchmark is fundamentally simulation-based. A common misconception is to treat LabUtopia as a real-world benchmark. The 2025 paper defines it as a simulation-and-benchmark suite, and the 2026 paper explicitly distinguishes LabUtopia = simulated benchmark from real Franka experiments = additional validation of transfer (Li et al., 28 May 2025, Ren et al., 11 Jun 2026).

The 2025 evaluation setup reports 150 episodes per task for training and 60 test episodes per task, with two cameras, each at 256 × 256 RGB resolution. A task instance is successful only if the success condition is satisfied and the state remains within tolerance for 2 seconds after the final action (Li et al., 28 May 2025). For navigation tasks, the paper additionally requires that a collision-free path exists, uses occupancy maps, and specifies a collision detection radius of 60 cm and map resolution of 0.5 meters per pixel/unit.

The 2026 VLA-oriented benchmark uses in-distribution (ID) and out-of-distribution (OOD) evaluation. It reports 120 episodes per task per setting, with OOD perturbing object placement, appearance, or scene configuration (Ren et al., 11 Jun 2026). The principal reported metric is success rate (%), including per-task success rates and average success rate across the six tasks. The later paper does not highlight a more complicated composite score in the main benchmark table.

Cross-embodiment evaluation is a defining design choice in the 2026 account. The robot profiles mentioned include single-arm robots, bimanual robots, and mobile manipulators, with examples such as Franka Panda, FR3, UR-series arms, Piper, Rizon4, Festo, ARX X5, ARX R5, Split ALOHA, Lift2, FR3 Duo, Ridgebase-mounted variants (Ren et al., 11 Jun 2026). For LabUtopia evaluation, methods are adapted to the LabUtopia robot schema. The paper states that LabVLA uses

$$A_t^r = [a_t^r,\ldots,a_{t+K-1}^r] \in \mathbb{R}^{K \times d_r}$$

as a continuous action chunk prediction, with embodiment-specific action dimension $d_r$ and padded maximum $d_{\max}$.

5. Baselines and empirical behavior

The 2025 paper benchmarks two imitation-learning baselines: ACT and Diffusion Policy (DP). At Level 1, both models perform relatively well on atomic actions. Reported examples include Stir: ACT 86.7, DP 95.0, Press: ACT 93.3, DP 96.7, and Close Drawer: ACT 96.7, DP 100.0 (Li et al., 28 May 2025). The interpretation given in the paper is that atomic manipulation is comparatively easy for current models.

Performance drops at higher levels. At Level 2, the paper reports Heater Beaker: ACT 86.7, DP 25.0, Operate Drawer: ACT 73.3, DP 10.0, and Stir w/ GlassRod: ACT 55.0, DP 10.0, and notes that DP often stalled or failed to execute actions successfully (Li et al., 28 May 2025). At Level 3, the reported ID/OOD success rates show a generalization gap, with examples such as Pick: ACT 81.7 / 71.7, DP 53.3 / 41.7, Heater Beaker: ACT 86.7 / 80.0, DP 21.6 / 8.3, and Transport Beaker: ACT 77.5 / 73.3, DP 67.5 / 15.0. A separate table on object-size generalization reports near-zero OOD performance, including Pick: ACT 1.7 OOD, DP 0.0 OOD and Pour Liquid: ACT 0.0 OOD, DP 0.0 OOD. At Level 4, long-horizon tasks show compounding-error collapse; for example, Clean Beaker drops from A1 99.3 to 1.6 by A7 for ACT, while DP falls to 0.0 in later stages (Li et al., 28 May 2025).

The 2026 paper compares a broader VLA set: SmolVLA, X-VLA, GR00T N1.5, $\pi_0$, $\pi_{0.5}$, $\pi_0$-FAST, InternVLA-A1, Wall-oss-flow, and LabVLA (Ren et al., 11 Jun 2026). In that benchmark, LabVLA achieves the highest average success rate, with ID: 71.1% and OOD: 70.0%, beating the next best policy, $\pi_0$, by +7.8 pp in ID and +6.8 pp in OOD. Task-wise, Press Button is near saturated for many methods, with LabVLA at 100% ID / 98.3% OOD; Pick Up is 49.2% ID / 48.3% OOD; Open Door is 65.0% ID / 65.8% OOD; and Pour Liquid remains the hardest task, with no method exceeding 50% (Ren et al., 11 Jun 2026). The paper also highlights a 1.1 percentage point drop from ID to OOD for LabVLA, from 71.1% to 70.0%.

Taken together, these results define the benchmark’s difficulty profile. Atomic manipulation is not the limiting case; the difficult regimes are compositional execution, OOD robustness, object-size and appearance variation, articulated interaction, liquid handling, and error accumulation in multi-stage protocols.

6. Scientific relevance, distinctive features, and limitations

LabUtopia’s main distinguishing property is that it treats laboratory work as a first-class embodied domain rather than a special case of household manipulation. The 2025 paper states that LabUtopia supports Fluid, Physics, and Chemistry, and presents this combination as absent from generic embodied benchmarks (Li et al., 28 May 2025). The 2026 paper adds that laboratory failure modes include transparent liquids, precise tilt angles, safety/contact constraints, and instrument-specific contact patterns (Ren et al., 11 Jun 2026). In both accounts, the benchmark is meant to evaluate the integration of perception, planning, and control under scientific-purpose constraints.

Another distinctive feature is the hierarchical organization of competence. The 2025 hierarchy runs from atomic actions to long-horizon and mobile manipulation. The 2026 paper makes this more explicit through a four-tier capability pyramid—Apprentice, Technician, Specialist, Scientist—and states that LabVLA is positioned at Level 2 (Technician), while LabUtopia mainly tests Levels 1–2: single-step manipulations and fixed multistep lab procedures (Ren et al., 11 Jun 2026). This suggests that the benchmark is intended not only to rank policies, but also to localize capability boundaries.

The benchmark is also explicit about its limitations. The 2025 paper notes that it is currently simulation-only, supports only two robot embodiments, and does not test VLA models (Li et al., 28 May 2025). The 2026 paper states that most validation is still in simulation, real-world testing is limited to a single Franka platform and a small set of benchtop tasks, and the system does not yet handle adaptive scientific decision-making, changing protocols based on observations, reagent substitution, safety-aware real wet-lab autonomy, or explanation or collaboration with human scientists (Ren et al., 11 Jun 2026).

A further point concerns task-count reporting. The 2025 abstract states that LabUtopia supports 30 distinct tasks, while the detailed benchmark description says LabBench comprises over 50 tasks (Li et al., 28 May 2025). This suggests that the suite-level task count and the benchmark-level enumeration are reported at different granularities, likely reflecting family-level tasks in one place and subtasks or benchmark instances in another. The papers do not resolve that discrepancy explicitly.

In the current literature, LabUtopia therefore occupies a specific niche: a high-fidelity, laboratory-centered, hierarchical benchmark for embodied agents, with later work showing how the same benchmark family can be adapted into a standardized cross-embodiment VLA evaluation harness. Its reported results indicate that liquid handling remains a major open challenge, that long-horizon execution still breaks under compounding error, and that laboratory-specific generalization is materially different from performance on household or tabletop benchmarks (Li et al., 28 May 2025, Ren et al., 11 Jun 2026).