---
title: 'VLBiMan: Vision-Language Bimanual Framework'
url: https://www.emergentmind.com/topics/vlbiman
type: topic
---

# VLBiMan: Vision-Language Bimanual Framework

VLBiMan—short for **Vision-Language Anchored One-Shot Demonstration Enables Generalizable Robotic Bimanual Manipulation**—is a modular framework for **generalizable robotic bimanual manipulation from a single human demonstration**, using **vision-language anchoring** rather than retraining a policy for every new object arrangement or object instance. It extracts from one demonstration the motion components that should remain stable across executions, adapts scene-dependent components through semantic grounding and geometric adjustment, and recomposes executable dual-arm trajectories with inverse kinematics and collision-aware refinement. The framework is explicitly positioned between end-to-end imitation or VLA methods, which generally require many demonstrations and retraining, and modular perception-planning systems, which can be sample efficient but brittle in dynamic or cluttered scenes [2509.21723].

## 1. Conceptual definition and operating regime

VLBiMan addresses **real-world bimanual manipulation** on dual-arm robots after seeing only **one seed demonstration** plus a short natural-language task description. The target task family includes **plugpen**, **inserting**, **unscrew**, **pouring**, **pressing**, and **reorient**, as well as longer compositions such as **reorient + unscrew**, **unscrew + pouring**, **tool-use spoon**, and **tool-use funnel**. The framework is designed to handle **object repositioning**, **orientation changes**, **background changes / clutter**, **novel but semantically similar object instances**, **external disturbances before grasp**, and transfer to **different dual-arm robot embodiments** without retraining [2509.21723].

The paper formalizes the overall mapping as
\[
\mathcal{F}_{\text{VLBiMan}} : (\mathcal{T}, \mathcal{D}, \mathcal{S}_{new}) \mapsto \{ \widetilde{\mathcal{A}^{new}_t}\}_{t=1}^{T'},
\]
where \(\mathcal{T}\) is a textual task description, \(\mathcal{D}\) is a one-shot demonstration, \(\mathcal{S}_{new}\) is a new scene, and \(\{ \widetilde{\mathcal{A}^{new}_t}\}_{t=1}^{T'}\) is the synthesized dual-arm trajectory. The demonstration sequence is
\[
\mathcal{D} = \{(\mathcal{O}_t, \mathcal{A}_t)\}_{t=1}^T,
\]
recorded at **10 FPS**, with each action
\[
\mathcal{A}_t \in \mathbb{R}^{14}
\]
comprising **6-DoF pose for the left arm**, **6-DoF pose for the right arm**, and **binary gripper states for both hands**. Observations include a visual frame from a **third-person stereo RGB camera**, end-effector poses of both arms, and gripper states.

The “single human demonstration” is operationalized through **kinesthetic teaching on the robot**. An operator manually drags the dual-arm robot to sparse key waypoints, records end-effector poses and intended gripper states, and then the robot replays the trajectory while synchronized camera observations are recorded. This is therefore a one-shot waypoint-based demonstrated execution rather than a demonstration dataset for policy learning. The method assumes **rigid objects only**, a **calibrated stereo camera**, hand-eye calibration, object geometry similarity checks within a tolerance \(\epsilon_g\), and no force or tactile sensing. It does **not** require CAD models, explicit 6D pose annotations, or policy retraining.

## 2. Demonstration structure and task-aware bimanual decomposition

The first stage of VLBiMan is **Task-Aware Bimanual Decomposition**, whose purpose is to convert a single demonstration into reusable motion modules. The demonstration is segmented into temporal intervals using **keypose-driven heuristics**, with candidate waypoints detected from **velocity discontinuities**, **acceleration spikes**, and **gripper open/close transitions**. Each waypoint \(\mathbf{w}_i\) defines a time slot
\[
\tau_i = [t_i, t_{i+1}]
\]
and a segment
\[
\mathcal{M}_i = \{\mathcal{A}_t\}_{t \in \tau_i}.
\]
An inverse-kinematics solver is used to validate segment feasibility [2509.21723].

This decomposition is explicitly **semi-automatic**. The segmentation is first scripted or heuristic, then refined by a human to ensure **temporal correctness**, **spatial continuity**, and **robust execution**. The segmentation policy is written as
\[
\pi_{seg}: \mathcal{D} \rightarrow \{\mathcal{M}_i\}_{i=1}^N.
\]
Each segment is then classified according to whether it corresponds to a robot-object coupled phase. The key predicate is
\[
bind(o, r, t),
\]
which indicates whether object \(o\) is physically attached to end-effector \(r\) at time \(t\).

A segment is treated as **invariant** when
\[
\forall t \in \tau_i,\; bind(o_k, r, t)=1,\ \text{and}\ geometry(o_k) \approx geometry(o_k^{demo}),
\]
leading to the decomposition
\[
\mathcal{D} \Rightarrow \{\mathcal{M}_i^{inv}\}^{N_{inv}}_{i=1} \cup \{\mathcal{M}_j^{var}\}^{N_{var}}_{j=1}.
\]
Invariant segments are motion components that can be reused once the relevant object relation has already been established; examples given in the paper include lifting after grasp, bilateral alignment, pouring motion once bottle and mug relation is set, and dual-arm unscrewing once both bottle and cap are engaged. Variable segments are those that depend on scene configuration, such as initial reach-to-grasp, grasp alignment, pre-contact approach, and orientation-sensitive pickup.

The internal representation is not a symbolic planning language and not a learned latent policy. It consists of temporally segmented motion primitives \(\mathcal{M}_i\) with end-effector pose trajectories, binary gripper states, temporal ordering, inter-arm dependencies, contact or grasp state, and semantic labels such as invariant or variable. The paper summarizes this design philosophy as prioritizing **what to achieve** over a fixed low-level realization of **how to execute it**.

## 3. Vision-language anchoring and scene adaptation

The second stage, **Vision-Language Anchored Adaptation**, modifies the variable modules \(\mathcal{M}_j^{var}\) for a new scene. The grounding problem is defined as
\[
\mathcal{G} : \mathcal{T} \mapsto \{o_k\}_{k=1}^K,
\]
mapping task language to task-relevant scene objects. In practice, VLBiMan extracts task-relevant prompts from the natural-language description, passes them to pretrained vision-language and vision-foundation models, and obtains semantic masks for relevant objects [2509.21723].

The paper names **Florence-2** for vision-language understanding and **SAM2** for high-quality segmentation. In ablations it also compares against **SAM + DINOv2**. The framework explicitly states that it **does not rely on LLM-based instruction parsing and task decomposition**, and it avoids 6D pose estimation that requires CAD models, generic grasp proposal systems as the main driver, and ambiguous affordance or keypoint clusters.

VLBiMan uses simple, task-dependent object-centric anchors derived from masks: either the **geometric center of the mask** or **plane-contact points** on the table-facing boundary. Given representative 3D points from demonstration and new scene, \(\mathbf{p}^{demo}\) and \(\mathbf{p}^{new}\), the relative position shift is
\[
\Delta \mathbf{x} = \mathbf{p}^{new} - \mathbf{p}^{demo}.
\]
For orientation-sensitive objects, a principal axis is computed from second-order image moments, and orientation change is measured as
\[
\Delta \theta = \angle(\mathbf{v}^{new}, \mathbf{v}^{demo}).
\]
For category-level shape variation, object height is estimated from the point cloud as
\[
\Delta h_k = \max_z(\mathcal{P}_k^{3D}) - \min_z(\mathcal{P}_k^{3D}),
\]
which is then used to adjust vertical placement motion, inter-arm distances, and tool or container handling parameters.

The orientation module is entirely geometric rather than learned. For a binary mask \(\mathbf{M}\), the centroid is
\[
\bar{x} = \frac{1}{N}\sum_{i=1}^N x_i,\quad \bar{y} = \frac{1}{N}\sum_{i=1}^N y_i,
\]
with second-order central moments
\[
\mu_{20} = \frac{1}{N}\sum (x_i-\bar{x})^2,\quad
\mu_{02} = \frac{1}{N}\sum (y_i-\bar{y})^2,\quad
\mu_{11} = \frac{1}{N}\sum (x_i-\bar{x})(y_i-\bar{y}),
\]
and covariance
\[
\Sigma = \begin{bmatrix} \mu_{20} & \mu_{11}\\ \mu_{11} & \mu_{02} \end{bmatrix}.
\]
Eigen-decomposition yields the principal axis, and the final orientation is
\[
\theta = \left(\arctan2(a_y, a_x)\times \frac{180}{\pi}\right)\bmod 360.
\]
This permits orientation-sensitive adaptation without explicit 6D pose models. The paper attributes robustness to background changes, clutter, object relocation, and semantically similar unseen objects to the combination of semantic segmentation and lightweight mask-based geometric anchoring.

## 4. Trajectory recomposition, inverse kinematics, and hybrid dual-arm control

Once the scene-dependent quantities have been estimated, VLBiMan reconstructs a new trajectory by aligning invariant modules \(\mathcal{M}_i^{inv}\) and adapted variable modules \(\widetilde{\mathcal{M}_j^{var}}\) according to the temporal structure of the demonstration. The paper describes this as **Autonomous Trajectory Composition**. The resulting trajectory is not a direct replay: it is recomposed and then refined for reachability and collision avoidance [2509.21723].

For grasping motions, the framework applies **progressive IK refinement**
\[
\mathbf{q}^{(n+1)} = IK(\mathbf{T}_g^{(n)}), \quad \mathbf{T}_g^{(n)} = SplineInterp(\mathbf{T}_{start}, \mathbf{T}_{goal}, n),
\]
with interpolation density \(n = 6\) in experiments. The purpose is to approach the target pose through spline-interpolated intermediate poses rather than attempting a single large motion. To reduce premature contact, VLBiMan adds **dynamic collision compensation**
\[
\tilde{\mathbf{x}}^{goal} = \mathbf{x}^{goal} + \delta_{base}\mathbf{u}_{\parallel} + \delta_z\mathbf{u}_z,
\]
where \(\delta_{base}\) is a planar compensation term and \(\delta_z\) is a vertical clearance.

Feasibility is checked through IK solvability, collision compensation during approach, and one-time physical replay for identifying unintended collisions and adjusting trajectories. The paper characterizes this as a practical engineering pipeline rather than a formal constrained optimization framework. For pre-grasp disturbances, the appendix describes a loop that continuously segments and tracks the object, estimates \((\mathbf{p}_t,\theta_t)\), waits until motion becomes stable over a short window, and attempts grasp only when pose variance falls below threshold \(\epsilon\), for example motion less than **10 mm**. If grasp fails or the object moves again, the system returns to observation and retries.

A distinctive claim of VLBiMan is that it preserves **mixed synchronous and asynchronous use of both arms**. Some primitives are synchronous, such as simultaneous alignment in plugpen or bringing bottle and cup together in pouring; others are asymmetric, such as one arm stabilizing while the other unscrews. Because decomposition preserves temporal structure per arm and recomposition is performed globally, the framework can trigger synchronous segments concurrently when feasible. The appendix reports that maximizing synchronous execution where possible yields about **22% average execution efficiency improvement** across ten tasks.

## 5. Platforms, task suite, and empirical performance

The main platform comprises two fixed-base **Aubo-i5** collaborative arms, **6 DoF** each, in an opposite-side configuration, equipped with **DH-Robotics parallel grippers** of maximum width **80 mm**, and a third-person stereo camera (**Kingfisher R-6000**, **960×540 RGB**). Cross-embodiment transfer is evaluated on two **Rokae xMate CR7** arms in a humanoid-style configuration with parallel grippers of maximum opening **75 mm** and a head-mounted stereo camera. VLBiMan is compared against **Robot-ABC + AnyGrasp**, **ReKep**, **ReKep+** with oracle-level initial grasp labels, adapted **Mechanisms**, and adapted **MAGIC** [2509.21723].

Across the **six basic tasks**, using **20 trials** per setting, VLBiMan achieved the following average success rates:

| Setting | Average success |
|---|---:|
| New placements + same objects, no interference | **85.0%** |
| New placements + novel instances, no interference | **78.3%** |
| Same objects + interference | **70.0%** |
| Novel instances + interference | **59.2%** |

For the no-interference, same-object setting, the per-task results were **19/20** for plugpen, **18/20** for inserting, **16/20** for unscrew, **17/20** for pouring, **17/20** for pressing, and **15/20** for reorient. Under **novel instances + interference**, the baseline comparison was especially pronounced: **Mechanisms 3.3%**, **MAGIC 11.7%**, **Robot-ABC 6.7%**, **ReKep 14.2%**, **ReKep+ 25.0%**, versus **VLBiMan 59.2%**.

For **long-horizon tasks**, VLBiMan reached **52.5%** for same objects without interference, **41.3%** for novel instances without interference, **38.8%** for same objects with interference, and **25.0%** for novel instances with interference. The paper reports that these scores remain substantially higher than all baselines; for example, with novel instances and interference, **ReKep+** achieved **12.5%** while VLBiMan achieved **25.0%**.

The ablation study on the six basic tasks under **novel instances + interference** reports **35.8%** when replacing Florence-2/SAM2 with **SAM + DINOv2**, **31.7%** when using **AnyGrasp** instead of semantic grasp alignment, **29.2%** with **no IK refinement**, **34.2%** with **no collision avoidance**, and **59.2%** for the full system. Under **uneven lighting + dynamic interference**, performance degraded moderately from **70.0%** to **67.5%** for same objects and from **59.2%** to **55.8%** for novel instances. With repeated interference on the six basic tasks, success declined gradually from **85.0%** at zero disturbances to **70.0%**, **61.7%**, **56.7%**, **53.3%**, and **50.8%** as interference count increased from 1 to 5.

Cross-embodiment transfer on **inserting**, **unscrew**, **pouring**, and **reorient** shows similar performance on the two robots. For **no interference, same objects**, the contralateral platform achieved **82.5%** and the humanoid platform **83.8%**. For **no interference, novel instances**, the rates were **73.8%** and **76.3%**. For **with interference, same objects**, they were **68.8%** and **70.0%**. For **with interference, novel instances**, they were **55.0%** and **58.8%**. The paper presents these results as evidence that skill representations based on object-relative motion structure can be instantiated on different robotic platforms without retraining.

## 6. Limitations, interpretation, and place within robotic manipulation

VLBiMan is presented as a **modular, object-centric, one-shot bimanual manipulation framework**, not as an end-to-end policy and not as a generic open-world manipulation system. The paper explicitly limits its scope to **rigid objects only** and notes the absence of **runtime anomaly detection or recovery**, sensitivity to **slippage**, **execution errors**, and **occlusion**, and hardware constraints stemming from a fixed-base dual-arm platform without force or tactile sensing [2509.21723].

The reported failure analysis identifies **initial grasp execution** and **dual-arm coordination** as the dominant failure modes, with perception and trajectory optimization contributing smaller fractions. The system also remains dependent on sufficiently reliable segmentation masks, and the paper does not provide a general online replanning mechanism for unexpected mid-task failures. Tool-use and long-horizon tasks exhibit the expected accumulation of error over multiple stages. The method also assumes calibrated stereo reconstruction and semantically interpretable task prompts.

Within those constraints, the paper argues that VLBiMan derives its generalization from decomposition and recomposition rather than statistical coverage. A single demonstration is converted into reusable atomic modules; only the scene-dependent modules are re-anchored through semantic segmentation and geometry; and execution is reconstructed through IK and collision-aware refinements. This suggests a design space in which sample efficiency, embodiment transfer, and robustness to moderate scene variation are achieved through structured task abstraction instead of policy retraining.

In the terminology of the paper, VLBiMan’s distinctive contribution lies in combining **one-shot reusable bimanual skill extraction**, **vision-language anchored adaptation**, **hybrid dual-arm control**, and **cross-embodiment transfer without retraining**. Empirically, the framework reaches **85.0%** success on basic tasks with new placements of the same objects and **78.3%** on novel instances without interference, while maintaining a substantial advantage over the compared baselines under interference, long-horizon composition, and embodiment transfer.

Source: https://www.emergentmind.com/topics/vlbiman