Labimus: A Simulation and Benchmark for Humanoid Dexterous Manipulation in Chemical Laboratory

Abstract: Laboratory automation has made remarkable progress through robotic platforms and AI-driven scientific reasoning. However, many laboratory operations (e.g., solid--solid transfer) remain inherently dynamic and require real-time adaptation to different materials and experimental conditions. Such precision-critical manipulations are difficult to standardize, motivating the use of humanoid robots with dexterous hands. Despite this opportunity, no existing benchmark evaluates humanoid manipulation in precision-critical laboratory environments. We present Labimus, to our knowledge, the first benchmark for humanoid dexterous manipulation in organic chemistry laboratories. Labimus reconstructs over 30 functionally faithful assets from real organic chemistry workstations through real-to-sim modeling, collectively covering the core operations of routine organic chemistry experiments. The benchmark integrates articulated laboratory instruments, particle-based powder physics, and closed-loop instrument readouts, enabling a complete manipulation-to-measurement pipeline. It further defines six atomic operations and a seven-step solid-weighing workflow derived from real laboratory standard operating procedures. We introduce a precision-aware evaluation protocol designed to jointly measure task completion, experimental precision, and long-horizon execution. We benchmark three representative policies under procedural layouts and environmental perturbations. Results reveal a precision gap: policies that successfully complete laboratory tasks can still fail to satisfy the quantitative tolerances required by experimental protocols. Our benchmark exposes a fundamental disconnect between task completion and experimental validity, providing a new testbed for developing reliable humanoid robots for scientific laboratories.

• The paper presents Labimus, a benchmark that evaluates humanoid dexterous manipulation in chemical labs using over 30 real-to-simulation assets and protocol-driven tasks.
• It introduces a high-fidelity simulation framework with particle-based granular physics and closed-loop instrumentation to capture scientifically valid precision metrics.
• Evaluation protocols benchmark binary success, quantitative precision (e.g., 15mm placement tolerance and ±0.001g powder transfer), and multi-step procedural execution under varied conditions.

Labimus: A Benchmark for Humanoid Dexterous Manipulation in Chemical Laboratories

Motivation and Context

Automation in scientific laboratories has advanced considerably with robotic platforms and AI-driven experimental design, yet many precision-critical procedures, such as granular transfer in organic synthesis, remain challenging for conventional fixed-arm robots and non-dexterous grippers. These tasks require real-time adaptation, fine force control, and multi-finger dexterity—capabilities integral to human laboratory work but not yet adequately benchmarked in robotic systems. Existing simulation and benchmarking platforms for manipulation focus on household or industrial scenarios, proxy binary success criteria, and lack the task granularity or fidelity needed to evaluate scientific validity in laboratory contexts.

Labimus directly addresses this gap by introducing, for the first time, a benchmark for humanoid dexterous manipulation centered on precision-critical laboratory tasks, instantiated with more than 30 real-to-simulation reconstructed assets and a suite of evaluation protocols reflecting scientific standards.

Figure 1: Overview of Labimus, illustrating real-to-sim chemistry workstation assembly with 30+ assets and the Tianyi 2.0 humanoid demonstrating fine-grained manipulation involving instrument state readouts, particle-based powder physics, and dexterous tasks.

Simulation Environment, Asset Design, and Physics

High-fidelity simulation of laboratory procedures necessitates modeling beyond rigid-body and visual realism. Labimus’ simulation framework is distinguished by its integration of:

• Functional Asset Library: Over 30 containers, tools, and portable or instrumented devices recreated from actual organic chemistry setups, with accurate collision modeling, articulated components (e.g., sliding doors, tare buttons), and instrument state observables.
• Particle-based Granular Physics: A micro-element scheme for powder manipulation where individual grains are modeled as rigid bodies, resolving contacts via GPU-accelerated dynamics in PhysX. Critically, powder accumulation and transfer are quantified in real-time by simulated analytical balances, with direct digital output for measurement-anchored evaluation.
• Closed-Loop Instrumentation: Simulated balances and other scientific instruments provide synchronized, stateful readouts (e.g., digital mass display), ensuring that manipulation is tied to scientifically significant outcomes rather than visual cues alone.

Figure 2: Representative functional assets, including containers, spatulas, balances, and articulated doors—each supporting task-relevant physical interaction and measurement.

A dedicated SOP-to-simulation pipeline, leveraging LLM-assisted parsing, ensures that every task specification is grounded in authentic laboratory protocols, not arbitrary designer choices. Scene assembly and object configurations are procedurally generated for robustness evaluation, with both digital-twin and parametric layouts supported.

Task Suite and Evaluation Hierarchy

Labimus’ task suite comprises six atomic operations—door open, door close, grasp-and-place, tare press, tool pickup, scoop-and-weigh—and a seven-step solid-weighing workflow, all derived from documented standard operating procedures in organic chemistry.

Figure 3: Labimus benchmark overview, illustrating the simulation environment (top-left), representative atomic operations (top-right), the three-tiered evaluation hierarchy (bottom-left), and perturbation conditions (bottom-right).

Evaluation is structured into a three-tier hierarchy:

1. Tier 1—Binary Task Completion: Measures simple goal achievement (e.g., door opened), providing a baseline metric consistent with extant manipulation benchmarks.
2. Tier 2—Quantitative Precision: Applies continuous metrics (e.g., container placement within 15 mm, powder transfer within ±0.001 g), capturing scientific validity. Substantial completion–precision gaps are quantified.
3. Tier 3—Long-Horizon and Procedural Execution: Evaluates multi-step workflows with step-wise and cumulative error tracing, dissecting compounding error and phase-wise failure localization.

Each tier is exhaustively benchmarked across four standardized test conditions—standard, lighting perturbation, texture perturbation, and combined environmental changes—producing a $3\times4$ (tier, condition) evaluation matrix.

Figure 4: Atomic operations grouped by manipulation type (top) and visualization of the full solid-weighing workflow (bottom), indicating the procedural and instrument-rich nature of the core Labimus tasks.

Experimental Results and Precision Gap

Three representative baseline policies—ACT (transformer-based IL), Diffusion Policy (conditional denoising), and $\pi_0$ (VLA foundation model)—were benchmarked on core tasks using 100 human teleoperation demonstrations per operation, with dexterous input captured via Manus gloves and isomorphic arms on the Tianyi humanoid.

Key numerical findings:

• Tier 1 success rates: For "door open", ACT achieved $56.7 \pm 3.4\%$, DP $49.3 \pm 2.5\%$, and $\pi_0$ $47.3 \pm 7.4\%$. More fine-grained tasks like "tare press" saw near-zero success (<2.0%); "door close" revealed strong baseline differentiation ($\pi_0$ $40.7 \pm 3.8\%$ vs. ACT $24.7 \pm 0.9\%$).
• Tier 2 (precision): For "grasp-and-place" with ACT, the completion rate $S$ was $\pi_0$0, yet only $\pi_0$1 satisfied the $\pi_0$2 mm placement tolerance, with less than $\pi_0$3 of completed trials being scientifically valid. This demonstrates a persistent precision gap.
• Robustness analysis: $\pi_0$4 maintained comparable performance under isolated lighting ($\pi_0$5) or texture ($\pi_0$6) perturbations, but combined perturbations further reduced success ($\pi_0$7).

Collectively, these results reveal that binary completion metrics systematically overestimate the scientific reliability of robotic laboratory execution. Task success, as conventionally defined, fails to capture whether experimental tolerances are met—a critical insight in the context of laboratory automation.

Implications and Future Directions

Labimus redefines the evaluation standard for robotic manipulation in structured scientific domains by requiring policies to meet protocol-specified tolerances, not just discrete goal states. This is of immediate practical import for applications in automated chemistry, biology, and scientific instrumentation: scientific reliability—not mere physical achievement—must be the yardstick for embodied AI in science.

The framework's simulation fidelity, protocol grounding, and precision-centric evaluation expose the intrinsic difficulty of closing the gap between human and robotic scientific skill. Empirically, humanoid robots with dexterous hands remain far from reliably executing even single-step, high-precision laboratory manipulations. Key bottlenecks include single-finger actuation (as in tare press), cumulative error in multi-step workflows, and the need for robust, closed-loop control policies beyond current IL and VLA approaches.

Labimus also provides a comprehensive foundation for developing and systematically benchmarking next-generation policies that integrate fine-motor control, protocol reasoning, robust perception, and multi-modal feedback. The modular SOP-to-task pipeline is designed for extensibility—as further laboratory domains (e.g., liquid handling, apparatus assembly) and novel sensor modalities are integrated, Labimus can scale to broader scientific manipulation challenges.

The authors note forthcoming extensions: multi-modal (liquid/solid/gas) tasks, multi-humanoid comparisons, real-world (sim-to-real) transfer studies, and benchmarking on alternative dexterous platforms. Labimus is thus poised to become a pivotal resource for researchers at the intersection of robotics, AI, and laboratory automation.

Conclusion

Labimus initiates a paradigm shift in evaluating embodied AI for laboratory science, where precision and protocol fidelity supersede binary task success. The benchmark's realistic simulation suite, scientifically faithful assets, and multi-tiered, precision-aware metrics expose critical deficits in current policy performance and provide a roadmap for progress toward reliable, autonomous laboratory agents. Looking ahead, Labimus enables the field to pursue and quantify scientific validity as a first-class objective for humanoid manipulation.