---
title: LabUtopia Benchmark for Lab Manipulation
url: https://www.emergentmind.com/topics/labutopia-benchmark
type: topic
---

# LabUtopia Benchmark for Lab Manipulation

Searching arXiv for the primary LabUtopia papers and closely related benchmark papers mentioned in the provided data.
arxiv_search({"query":"id:2505.22634 OR id:2606.13578 OR id:2606.10382", "max_results": 10, "sort_by": "submittedDate", "sort_order": "descending"})
I couldn’t retrieve tool output in this interface, so I will rely on the provided arXiv metadata while keeping citations limited to the listed papers.
LabUtopia Benchmark is a simulation-and-benchmark framework for scientific embodied agents operating in laboratory settings rather than household or generic manipulation environments. In its 2025 formulation, LabUtopia integrates **LabSim**, **LabScene**, and **LabBench** to support high-fidelity simulation, procedural lab-scene generation, and hierarchical evaluation of scientific embodied behavior; a later 2026 paper uses **LabUtopia** as the evaluation benchmark for vision-language-action policies on scientific laboratory manipulation under in-distribution and out-of-distribution conditions [2505.22634] [2606.13578]. Across these formulations, the benchmark targets laboratory-specific requirements such as perception of **physical and chemical transformations**, precise **instrument handling**, **long-horizon planning**, transparent liquids, and variation across robot embodiments and scene layouts.

## 1. Scope, motivation, and benchmark identity

LabUtopia was introduced to address a gap in embodied-AI evaluation for scientific laboratories. The 2025 paper argues that laboratory work is fundamentally harder than standard embodied-AI settings because it requires perception of **physical and chemical transformations**, precise **instrument handling**, **long-horizon planning** across multi-step experimental protocols, and **generalization** across varying laboratory layouts, instruments, and materials [2505.22634]. The later 2026 paper frames the same benchmark family as a **high-fidelity simulation benchmark for scientific embodied agents**, designed specifically to evaluate **vision-language-action (VLA)** policies on **scientific laboratory manipulation** rather than household or tabletop tasks [2606.13578].

A central point in both papers is that prior benchmarks are misaligned with laboratory execution. The cited comparison set includes household or generic embodied benchmarks such as **ALFRED, Behavior / Behavior-1K, RLBench, Ravens, VLMbench, VIMA-Bench, ManiSkill3, ClevrSkills, Arnold**, and manipulation benchmarks such as **Meta-World, RLBench, ManiSkill, CALVIN, LIBERO, RoboCasa, BridgeData v2, Open X-Embodiment, DROID**. The reported limitation is not merely domain shift in object appearance, but the absence of laboratory-specific phenomena: instrument-specific interactions, liquid handling, protocol-level sequences, varying embodiments and camera setups, and state changes such as heating, opening, and pouring [2505.22634] [2606.13578].

The benchmark’s identity is somewhat layered. In the 2025 paper, **LabBench** is the explicit benchmark component within the larger LabUtopia suite. In the 2026 paper, **LabUtopia** itself is used as the benchmark name for VLA evaluation. This suggests that “LabUtopia Benchmark” can refer either to the full hierarchical **LabBench** inside the suite or to the later standardized LabUtopia evaluation harness used for laboratory VLA policies.

## 2. Infrastructure: LabSim, LabScene, and later benchmark generation

The 2025 formulation defines three main components. **LabSim** is a high-fidelity simulator for physical and chemical interactions; **LabScene** is a procedural lab-scene generator with diverse assets; and **LabBench** is the hierarchical benchmark for evaluation [2505.22634]. The benchmark depends on these lower layers because laboratory evaluation requires physically and chemically meaningful state evolution rather than only rigid-body interaction.

**LabSim** is built on **Isaac Sim**, supports **rigid, deformable, and fluid objects**, uses **PhysX 5** physics, includes a GPU-accelerated fluid simulator, and adds a **chemical engine**. The chemical engine is built from a curated database of **200 common chemical substances** from **PubChem**, stores properties like **color, molar mass, and pH**, and uses **GPT-4o mini** to infer chemical transformations from reactants [2505.22634]. The paper presents these mechanisms as necessary for scientifically meaningful tasks such as fluid transfer and reaction-driven state transitions.

**LabScene** is described as a **procedural generation pipeline** that uses a curated asset library and combines **grid stochastic sampling** with **constraint-aware search**. The 2025 paper reports **over 100 diverse and physically plausible lab environments**, **over 100 laboratory scenes** and **100 scientific instruments**, and in the abstract **more than 200 scene and instrument assets**; the appendix further specifies **about 60 categories of laboratory equipment assets** and **around 80 types of transparent glassware and plasticware assets** [2505.22634]. The same paper also states that the authors collected **over one thousand candidate scenes** from designer websites and filtered and standardized them with expert review.

The 2026 paper extends the benchmark’s construction logic through **RoboGenesis**, a simulation-based workflow and data engine linked tightly to LabUtopia evaluation. In that account, scenes are built from a curated **LabAssetLibrary** and **LabTextureLibrary**, with **2,947 annotated assets**, **over 1,000 texture images**, and **10,000 laboratory scenes**. The scene-construction pipeline uses a placement solver with numeric constraints, performs **ten validation checks**, and assigns a **0–100 quality score**, rejecting scenes below threshold [2606.13578]. This suggests an evolution from a simulator-and-benchmark suite toward a larger procedural data-and-evaluation ecosystem for laboratory embodied learning.

## 3. Hierarchical task design and benchmark contents

The 2025 paper presents **LabBench** as a five-level hierarchy of increasing complexity, intended to evaluate primitive skills, compositional behavior, distribution-shift robustness, long-horizon procedures, and navigation-plus-manipulation [2505.22634].

| Level | Focus | Tasks listed |
|---|---|---|
| 1 | Atomic Manipulation Tasks | Pick, Pour, Place, Press, Shake, Stir, Open Door, Close Drawer / Close Door |
| 2 | Short-Horizon Manipulation Tasks | Pour Liquid, Shake Beaker, Transport Beaker, Heater Beaker, Stir GlassRod, Operate Drawer |
| 3 | Generalizable Short Manipulation Tasks | Pick, Press, Transport Beaker, Heater Beaker |
| 4 | Long-Horizon Manipulation Tasks | Clean Beaker, Drying Beaker |
| 5 | Mobile Manipulation Tasks | Navigation and Manipulation |

At **Level 1**, the benchmark evaluates single-step laboratory actions such as **Pick**, **Pour**, **Place**, **Press**, **Shake**, **Stir**, **Open Door**, and **Close Drawer / Close Door**. These are framed as laboratory-specific primitives: grabbing beakers or flasks without tipping them, pouring liquids into target regions, pressing instrument buttons, stirring with a glass rod, and opening or closing cabinet elements [2505.22634].

At **Level 2**, the benchmark composes **2–3 atomic actions** into compound objectives. The listed tasks are **Pour Liquid**, **Shake Beaker**, **Transport Beaker**, **Heater Beaker**, **Stir GlassRod**, and **Operate Drawer**. The paper gives examples such as **Pour Liquid = Pick + Pour**, **Transport Beaker = Pick + Place**, **Heater Beaker = Pick + Place + Press**, **Stir GlassRod = Pick + Stir**, and **Operate Drawer = Open + Close** [2505.22634].

At **Level 3**, the benchmark keeps short manipulation structure but evaluates under **distribution shifts**, including unseen object shapes, unseen sizes, different appearances or materials, different button colors or positions, different desktop materials, and different target locations. The appendix lists **Pick**, **Press**, **Transport Beaker**, and **Heater Beaker** at this level [2505.22634].

At **Level 4**, LabBench evaluates full laboratory procedures. The listed tasks are **Clean Beaker** and **Drying Beaker**, with long action sequences such as **Clean Beaker: Pick, Pour, Place, Pick, Shake, Pour, Place** and **Drying Beaker: Open, Pick, Place, Pick, Place, Close** [2505.22634]. At **Level 5**, **Navigation and Manipulation** integrates path planning with a terminal pick action.

The 2026 paper presents a narrower but standardized benchmark slice with **six laboratory operations**: **Pick Up**, **Press Button**, **Open Door**, **Pour Liquid**, **Heat Beaker**, and **Transport Beaker** [2606.13578]. It characterizes **Press Button** as near-saturated and relatively easy, **Heat Beaker** as also relatively easy for strong models, **Pick Up** and **Open Door** as harder due to grasping and articulated interaction, **Transport Beaker** as harder due to stable grasp and precise placement, and **Pour Liquid** as the hardest task overall. This suggests that the later VLA evaluation harness emphasizes Levels 1–2 of the original hierarchy while keeping the laboratory domain constraints central.

## 4. Evaluation protocol, splits, and metrics

The benchmark is fundamentally **simulation-based**. A common misconception is to treat LabUtopia as a real-world benchmark. The 2025 paper defines it as a simulation-and-benchmark suite, and the 2026 paper explicitly distinguishes **LabUtopia = simulated benchmark** from **real Franka experiments = additional validation of transfer** [2505.22634] [2606.13578].

The 2025 evaluation setup reports **150 episodes** per task for training and **60 test episodes** per task, with **two cameras**, each at **256 × 256** RGB resolution. A task instance is successful only if the success condition is satisfied **and the state remains within tolerance for 2 seconds after the final action** [2505.22634]. For navigation tasks, the paper additionally requires that a collision-free path exists, uses occupancy maps, and specifies a collision detection radius of **60 cm** and map resolution of **0.5 meters per pixel/unit**.

The 2026 VLA-oriented benchmark uses **in-distribution (ID)** and **out-of-distribution (OOD)** evaluation. It reports **120 episodes per task per setting**, with OOD perturbing **object placement**, **appearance**, or **scene configuration** [2606.13578]. The principal reported metric is **success rate (%)**, including per-task success rates and average success rate across the six tasks. The later paper does not highlight a more complicated composite score in the main benchmark table.

Cross-embodiment evaluation is a defining design choice in the 2026 account. The robot profiles mentioned include **single-arm robots**, **bimanual robots**, and **mobile manipulators**, with examples such as **Franka Panda, FR3, UR-series arms, Piper, Rizon4, Festo, ARX X5, ARX R5, Split ALOHA, Lift2, FR3 Duo, Ridgebase-mounted variants** [2606.13578]. For LabUtopia evaluation, methods are adapted to the **LabUtopia robot schema**. The paper states that LabVLA uses
$$
A_t^r = [a_t^r,\ldots,a_{t+K-1}^r] \in \mathbb{R}^{K \times d_r}
$$
as a **continuous action chunk** prediction, with embodiment-specific action dimension \(d_r\) and padded maximum \(d_{\max}\).

## 5. Baselines and empirical behavior

The 2025 paper benchmarks two imitation-learning baselines: **ACT** and **Diffusion Policy (DP)**. At **Level 1**, both models perform relatively well on atomic actions. Reported examples include **Stir: ACT 86.7, DP 95.0**, **Press: ACT 93.3, DP 96.7**, and **Close Drawer: ACT 96.7, DP 100.0** [2505.22634]. The interpretation given in the paper is that atomic manipulation is comparatively easy for current models.

Performance drops at higher levels. At **Level 2**, the paper reports **Heater Beaker: ACT 86.7, DP 25.0**, **Operate Drawer: ACT 73.3, DP 10.0**, and **Stir w/ GlassRod: ACT 55.0, DP 10.0**, and notes that DP often **stalled** or failed to execute actions successfully [2505.22634]. At **Level 3**, the reported ID/OOD success rates show a generalization gap, with examples such as **Pick: ACT 81.7 / 71.7, DP 53.3 / 41.7**, **Heater Beaker: ACT 86.7 / 80.0, DP 21.6 / 8.3**, and **Transport Beaker: ACT 77.5 / 73.3, DP 67.5 / 15.0**. A separate table on object-size generalization reports near-zero OOD performance, including **Pick: ACT 1.7 OOD, DP 0.0 OOD** and **Pour Liquid: ACT 0.0 OOD, DP 0.0 OOD**. At **Level 4**, long-horizon tasks show compounding-error collapse; for example, **Clean Beaker** drops from **A1 99.3** to **1.6 by A7** for ACT, while DP falls to **0.0** in later stages [2505.22634].

The 2026 paper compares a broader VLA set: **SmolVLA**, **X-VLA**, **GR00T N1.5**, **\(\pi_0\)**, **\(\pi_{0.5}\)**, **\(\pi_0\)-FAST**, **InternVLA-A1**, **Wall-oss-flow**, and **LabVLA** [2606.13578]. In that benchmark, **LabVLA** achieves the highest average success rate, with **ID: 71.1%** and **OOD: 70.0%**, beating the next best policy, **\(\pi_0\)**, by **+7.8 pp** in ID and **+6.8 pp** in OOD. Task-wise, **Press Button** is near saturated for many methods, with LabVLA at **100% ID / 98.3% OOD**; **Pick Up** is **49.2% ID / 48.3% OOD**; **Open Door** is **65.0% ID / 65.8% OOD**; and **Pour Liquid** remains the hardest task, with no method exceeding **50%** [2606.13578]. The paper also highlights a **1.1 percentage point drop** from ID to OOD for LabVLA, from **71.1%** to **70.0%**.

Taken together, these results define the benchmark’s difficulty profile. Atomic manipulation is not the limiting case; the difficult regimes are compositional execution, OOD robustness, object-size and appearance variation, articulated interaction, liquid handling, and error accumulation in multi-stage protocols.

## 6. Scientific relevance, distinctive features, and limitations

LabUtopia’s main distinguishing property is that it treats laboratory work as a first-class embodied domain rather than a special case of household manipulation. The 2025 paper states that LabUtopia supports **Fluid**, **Physics**, and **Chemistry**, and presents this combination as absent from generic embodied benchmarks [2505.22634]. The 2026 paper adds that laboratory failure modes include **transparent liquids**, **precise tilt angles**, **safety/contact constraints**, and **instrument-specific contact patterns** [2606.13578]. In both accounts, the benchmark is meant to evaluate the integration of perception, planning, and control under scientific-purpose constraints.

Another distinctive feature is the hierarchical organization of competence. The 2025 hierarchy runs from atomic actions to long-horizon and mobile manipulation. The 2026 paper makes this more explicit through a four-tier capability pyramid—**Apprentice, Technician, Specialist, Scientist**—and states that LabVLA is positioned at **Level 2 (Technician)**, while LabUtopia mainly tests **Levels 1–2**: single-step manipulations and fixed multistep lab procedures [2606.13578]. This suggests that the benchmark is intended not only to rank policies, but also to localize capability boundaries.

The benchmark is also explicit about its limitations. The 2025 paper notes that it is **currently simulation-only**, supports only **two robot embodiments**, and does not test **VLA models** [2505.22634]. The 2026 paper states that most validation is still in **simulation**, real-world testing is limited to a **single Franka platform** and a small set of benchtop tasks, and the system does not yet handle **adaptive scientific decision-making**, **changing protocols based on observations**, **reagent substitution**, **safety-aware real wet-lab autonomy**, or **explanation or collaboration with human scientists** [2606.13578].

A further point concerns task-count reporting. The 2025 abstract states that LabUtopia supports **30 distinct tasks**, while the detailed benchmark description says **LabBench comprises over 50 tasks** [2505.22634]. This suggests that the suite-level task count and the benchmark-level enumeration are reported at different granularities, likely reflecting family-level tasks in one place and subtasks or benchmark instances in another. The papers do not resolve that discrepancy explicitly.

In the current literature, LabUtopia therefore occupies a specific niche: a high-fidelity, laboratory-centered, hierarchical benchmark for embodied agents, with later work showing how the same benchmark family can be adapted into a standardized cross-embodiment VLA evaluation harness. Its reported results indicate that **liquid handling remains a major open challenge**, that long-horizon execution still breaks under compounding error, and that laboratory-specific generalization is materially different from performance on household or tabletop benchmarks [2505.22634] [2606.13578].

Source: https://www.emergentmind.com/topics/labutopia-benchmark