VLBiMan: Vision-Language Bimanual Framework

• The paper introduces a one-shot demonstration-based framework for generalizable robotic bimanual manipulation using vision-language anchoring.
• It segments a human demonstration into invariant and variable motion modules and adapts scene-dependent steps through semantic grounding and IK refinement.
• Empirical results show success rates up to 85% in basic tasks, outperforming baselines in novel, cluttered, and interference scenarios.

VLBiMan—short for Vision-Language Anchored One-Shot Demonstration Enables Generalizable Robotic Bimanual Manipulation—is a modular framework for generalizable robotic bimanual manipulation from a single human demonstration, using vision-language anchoring rather than retraining a policy for every new object arrangement or object instance. It extracts from one demonstration the motion components that should remain stable across executions, adapts scene-dependent components through semantic grounding and geometric adjustment, and recomposes executable dual-arm trajectories with inverse kinematics and collision-aware refinement. The framework is explicitly positioned between end-to-end imitation or VLA methods, which generally require many demonstrations and retraining, and modular perception-planning systems, which can be sample efficient but brittle in dynamic or cluttered scenes (Zhou et al., 26 Sep 2025).

1. Conceptual definition and operating regime

VLBiMan addresses real-world bimanual manipulation on dual-arm robots after seeing only one seed demonstration plus a short natural-language task description. The target task family includes plugpen, inserting, unscrew, pouring, pressing, and reorient, as well as longer compositions such as reorient + unscrew, unscrew + pouring, tool-use spoon, and tool-use funnel. The framework is designed to handle object repositioning, orientation changes, background changes / clutter, novel but semantically similar object instances, external disturbances before grasp, and transfer to different dual-arm robot embodiments without retraining (Zhou et al., 26 Sep 2025).

The paper formalizes the overall mapping as

$$\mathcal{F}_{\text{VLBiMan}} : (\mathcal{T}, \mathcal{D}, \mathcal{S}_{new}) \mapsto \{ \widetilde{\mathcal{A}^{new}_t}\}_{t=1}^{T'},$$

where $\mathcal{T}$ is a textual task description, $\mathcal{D}$ is a one-shot demonstration, $\mathcal{S}_{new}$ is a new scene, and $\{ \widetilde{\mathcal{A}^{new}_t}\}_{t=1}^{T'}$ is the synthesized dual-arm trajectory. The demonstration sequence is

$$\mathcal{D} = \{(\mathcal{O}_t, \mathcal{A}_t)\}_{t=1}^T,$$

recorded at 10 FPS, with each action

$\mathcal{A}_t \in \mathbb{R}^{14}$

comprising 6-DoF pose for the left arm, 6-DoF pose for the right arm, and binary gripper states for both hands. Observations include a visual frame from a third-person stereo RGB camera, end-effector poses of both arms, and gripper states.

The “single human demonstration” is operationalized through kinesthetic teaching on the robot. An operator manually drags the dual-arm robot to sparse key waypoints, records end-effector poses and intended gripper states, and then the robot replays the trajectory while synchronized camera observations are recorded. This is therefore a one-shot waypoint-based demonstrated execution rather than a demonstration dataset for policy learning. The method assumes rigid objects only, a calibrated stereo camera, hand-eye calibration, object geometry similarity checks within a tolerance $\epsilon_g$, and no force or tactile sensing. It does not require CAD models, explicit 6D pose annotations, or policy retraining.

2. Demonstration structure and task-aware bimanual decomposition

The first stage of VLBiMan is Task-Aware Bimanual Decomposition, whose purpose is to convert a single demonstration into reusable motion modules. The demonstration is segmented into temporal intervals using keypose-driven heuristics, with candidate waypoints detected from velocity discontinuities, acceleration spikes, and gripper open/close transitions. Each waypoint $\mathbf{w}_i$ defines a time slot

$\tau_i = [t_i, t_{i+1}]$

and a segment

$\mathcal{T}$0

An inverse-kinematics solver is used to validate segment feasibility (Zhou et al., 26 Sep 2025).

This decomposition is explicitly semi-automatic. The segmentation is first scripted or heuristic, then refined by a human to ensure temporal correctness, spatial continuity, and robust execution. The segmentation policy is written as

$\mathcal{T}$1

Each segment is then classified according to whether it corresponds to a robot-object coupled phase. The key predicate is

$\mathcal{T}$2

which indicates whether object $\mathcal{T}$3 is physically attached to end-effector $\mathcal{T}$4 at time $\mathcal{T}$5.

A segment is treated as invariant when

$\mathcal{T}$6

leading to the decomposition

$\mathcal{T}$7

Invariant segments are motion components that can be reused once the relevant object relation has already been established; examples given in the paper include lifting after grasp, bilateral alignment, pouring motion once bottle and mug relation is set, and dual-arm unscrewing once both bottle and cap are engaged. Variable segments are those that depend on scene configuration, such as initial reach-to-grasp, grasp alignment, pre-contact approach, and orientation-sensitive pickup.

The internal representation is not a symbolic planning language and not a learned latent policy. It consists of temporally segmented motion primitives $\mathcal{T}$8 with end-effector pose trajectories, binary gripper states, temporal ordering, inter-arm dependencies, contact or grasp state, and semantic labels such as invariant or variable. The paper summarizes this design philosophy as prioritizing what to achieve over a fixed low-level realization of how to execute it.

3. Vision-language anchoring and scene adaptation

The second stage, Vision-Language Anchored Adaptation, modifies the variable modules $\mathcal{T}$9 for a new scene. The grounding problem is defined as

$\mathcal{D}$0

mapping task language to task-relevant scene objects. In practice, VLBiMan extracts task-relevant prompts from the natural-language description, passes them to pretrained vision-language and vision-foundation models, and obtains semantic masks for relevant objects (Zhou et al., 26 Sep 2025).

The paper names Florence-2 for vision-language understanding and SAM2 for high-quality segmentation. In ablations it also compares against SAM + DINOv2. The framework explicitly states that it does not rely on LLM-based instruction parsing and task decomposition, and it avoids 6D pose estimation that requires CAD models, generic grasp proposal systems as the main driver, and ambiguous affordance or keypoint clusters.

VLBiMan uses simple, task-dependent object-centric anchors derived from masks: either the geometric center of the mask or plane-contact points on the table-facing boundary. Given representative 3D points from demonstration and new scene, $\mathcal{D}$1 and $\mathcal{D}$2, the relative position shift is

$\mathcal{D}$3

For orientation-sensitive objects, a principal axis is computed from second-order image moments, and orientation change is measured as

$\mathcal{D}$4

For category-level shape variation, object height is estimated from the point cloud as

$\mathcal{D}$5

which is then used to adjust vertical placement motion, inter-arm distances, and tool or container handling parameters.

The orientation module is entirely geometric rather than learned. For a binary mask $\mathcal{D}$6, the centroid is

$\mathcal{D}$7

with second-order central moments

$\mathcal{D}$8

and covariance

$\mathcal{D}$9

Eigen-decomposition yields the principal axis, and the final orientation is

$\mathcal{S}_{new}$0

This permits orientation-sensitive adaptation without explicit 6D pose models. The paper attributes robustness to background changes, clutter, object relocation, and semantically similar unseen objects to the combination of semantic segmentation and lightweight mask-based geometric anchoring.

4. Trajectory recomposition, inverse kinematics, and hybrid dual-arm control

Once the scene-dependent quantities have been estimated, VLBiMan reconstructs a new trajectory by aligning invariant modules $\mathcal{S}_{new}$1 and adapted variable modules $\mathcal{S}_{new}$2 according to the temporal structure of the demonstration. The paper describes this as Autonomous Trajectory Composition. The resulting trajectory is not a direct replay: it is recomposed and then refined for reachability and collision avoidance (Zhou et al., 26 Sep 2025).

For grasping motions, the framework applies progressive IK refinement

$\mathcal{S}_{new}$3

with interpolation density $\mathcal{S}_{new}$4 in experiments. The purpose is to approach the target pose through spline-interpolated intermediate poses rather than attempting a single large motion. To reduce premature contact, VLBiMan adds dynamic collision compensation

$\mathcal{S}_{new}$5

where $\mathcal{S}_{new}$6 is a planar compensation term and $\mathcal{S}_{new}$7 is a vertical clearance.

Feasibility is checked through IK solvability, collision compensation during approach, and one-time physical replay for identifying unintended collisions and adjusting trajectories. The paper characterizes this as a practical engineering pipeline rather than a formal constrained optimization framework. For pre-grasp disturbances, the appendix describes a loop that continuously segments and tracks the object, estimates $\mathcal{S}_{new}$8, waits until motion becomes stable over a short window, and attempts grasp only when pose variance falls below threshold $\mathcal{S}_{new}$9, for example motion less than 10 mm. If grasp fails or the object moves again, the system returns to observation and retries.

A distinctive claim of VLBiMan is that it preserves mixed synchronous and asynchronous use of both arms. Some primitives are synchronous, such as simultaneous alignment in plugpen or bringing bottle and cup together in pouring; others are asymmetric, such as one arm stabilizing while the other unscrews. Because decomposition preserves temporal structure per arm and recomposition is performed globally, the framework can trigger synchronous segments concurrently when feasible. The appendix reports that maximizing synchronous execution where possible yields about 22% average execution efficiency improvement across ten tasks.

5. Platforms, task suite, and empirical performance

The main platform comprises two fixed-base Aubo-i5 collaborative arms, 6 DoF each, in an opposite-side configuration, equipped with DH-Robotics parallel grippers of maximum width 80 mm, and a third-person stereo camera (Kingfisher R-6000, 960×540 RGB). Cross-embodiment transfer is evaluated on two Rokae xMate CR7 arms in a humanoid-style configuration with parallel grippers of maximum opening 75 mm and a head-mounted stereo camera. VLBiMan is compared against Robot-ABC + AnyGrasp, ReKep, ReKep+ with oracle-level initial grasp labels, adapted Mechanisms, and adapted MAGIC (Zhou et al., 26 Sep 2025).

Across the six basic tasks, using 20 trials per setting, VLBiMan achieved the following average success rates:

Setting	Average success
New placements + same objects, no interference	85.0%
New placements + novel instances, no interference	78.3%
Same objects + interference	70.0%
Novel instances + interference	59.2%

For the no-interference, same-object setting, the per-task results were 19/20 for plugpen, 18/20 for inserting, 16/20 for unscrew, 17/20 for pouring, 17/20 for pressing, and 15/20 for reorient. Under novel instances + interference, the baseline comparison was especially pronounced: Mechanisms 3.3%, MAGIC 11.7%, Robot-ABC 6.7%, ReKep 14.2%, ReKep+ 25.0%, versus VLBiMan 59.2%.

For long-horizon tasks, VLBiMan reached 52.5% for same objects without interference, 41.3% for novel instances without interference, 38.8% for same objects with interference, and 25.0% for novel instances with interference. The paper reports that these scores remain substantially higher than all baselines; for example, with novel instances and interference, ReKep+ achieved 12.5% while VLBiMan achieved 25.0%.

The ablation study on the six basic tasks under novel instances + interference reports 35.8% when replacing Florence-2/SAM2 with SAM + DINOv2, 31.7% when using AnyGrasp instead of semantic grasp alignment, 29.2% with no IK refinement, 34.2% with no collision avoidance, and 59.2% for the full system. Under uneven lighting + dynamic interference, performance degraded moderately from 70.0% to 67.5% for same objects and from 59.2% to 55.8% for novel instances. With repeated interference on the six basic tasks, success declined gradually from 85.0% at zero disturbances to 70.0%, 61.7%, 56.7%, 53.3%, and 50.8% as interference count increased from 1 to 5.

Cross-embodiment transfer on inserting, unscrew, pouring, and reorient shows similar performance on the two robots. For no interference, same objects, the contralateral platform achieved 82.5% and the humanoid platform 83.8%. For no interference, novel instances, the rates were 73.8% and 76.3%. For with interference, same objects, they were 68.8% and 70.0%. For with interference, novel instances, they were 55.0% and 58.8%. The paper presents these results as evidence that skill representations based on object-relative motion structure can be instantiated on different robotic platforms without retraining.

6. Limitations, interpretation, and place within robotic manipulation

VLBiMan is presented as a modular, object-centric, one-shot bimanual manipulation framework, not as an end-to-end policy and not as a generic open-world manipulation system. The paper explicitly limits its scope to rigid objects only and notes the absence of runtime anomaly detection or recovery, sensitivity to slippage, execution errors, and occlusion, and hardware constraints stemming from a fixed-base dual-arm platform without force or tactile sensing (Zhou et al., 26 Sep 2025).

The reported failure analysis identifies initial grasp execution and dual-arm coordination as the dominant failure modes, with perception and trajectory optimization contributing smaller fractions. The system also remains dependent on sufficiently reliable segmentation masks, and the paper does not provide a general online replanning mechanism for unexpected mid-task failures. Tool-use and long-horizon tasks exhibit the expected accumulation of error over multiple stages. The method also assumes calibrated stereo reconstruction and semantically interpretable task prompts.

Within those constraints, the paper argues that VLBiMan derives its generalization from decomposition and recomposition rather than statistical coverage. A single demonstration is converted into reusable atomic modules; only the scene-dependent modules are re-anchored through semantic segmentation and geometry; and execution is reconstructed through IK and collision-aware refinements. This suggests a design space in which sample efficiency, embodiment transfer, and robustness to moderate scene variation are achieved through structured task abstraction instead of policy retraining.

In the terminology of the paper, VLBiMan’s distinctive contribution lies in combining one-shot reusable bimanual skill extraction, vision-language anchored adaptation, hybrid dual-arm control, and cross-embodiment transfer without retraining. Empirically, the framework reaches 85.0% success on basic tasks with new placements of the same objects and 78.3% on novel instances without interference, while maintaining a substantial advantage over the compared baselines under interference, long-horizon composition, and embodiment transfer.