Bimanual Manipulation & Recipe Adaptation

• Bimanual manipulation is defined as dual-arm robotic coordination using modular perception, language-driven planning, and motion primitives for executing multi-step culinary tasks.
• The system integrates advanced vision–language models with template-based recipe adaptation and real-time diffusion-based skill modulation for adaptive dual-arm control.
• Experimental evaluations reveal high code correctness (100% syntactic accuracy), 83.4% end-to-end success, and highlight visual mis-detection as a key challenge for broader applicability.

Bimanual manipulation and recipe adaptation address the integration of dual-arm robotic control, machine perception, and language-based instruction to enable flexible, context-aware execution and adaptation of multi-step tasks in unstructured environments such as domestic kitchens. Recent research has demonstrated both programmatic template-driven recipe conversion and real-time human-in-the-loop skill adaptation frameworks grounded in vision-LLMs and hierarchical policy learning.

1. Architectures for Bimanual Manipulation in Cooking

Leading architectures for bimanual robot manipulation in recipe-centric contexts maintain a modular structure separating perception, high-level reasoning, and action execution. The Bi-VLA system exemplifies this approach, comprising three principal modules:

• Vision Module: Implements a QWEN-VL vision–LLM to detect ingredient presence and bounding boxes, supported by pixel-to-world conversions for spatial localization. The mapping incorporates both intrinsic and extrinsic calibration matrices and Brown–Conrady radial-tangential undistortion to robustly map image coordinates to 3D world points:

$z_c \; ^pP = ^p_cT\,^c_w\!\widehat T\,^wP$

• Language Module: Utilizes Retrieval-Augmented Generation (RAG) to query a vector database of recipe templates and a two-stage Starling-LM-7B-alpha LLM pipeline for (i) decomposing user intent into high-level sequential plans, and (ii) translating them into executable Python code mapping to motion primitives.
• Action Module: Executes generated code by invoking atomic dual-arm primitives (e.g., grasp, cut, pour) via a motion controller over two UR3 manipulators (one with a gripper, one with a knife). The control stack relies on the commercial hardware’s built-in inverse kinematics.

The Bi-VLA data-flow is sequential: user request → RAG recipe retrieval → semantic plan → code generation → motion API execution, with visual feedback closing the perception-action loop (Gbagbe et al., 2024).

2. Kinematic Planning, Control, and Dual-Arm Coordination

Robust bimanual execution in culinary tasks requires precise mapping from visual detection to manipulator reference frames. In Bi-VLA, the vision module’s output is transformed into spatial waypoints for the robot arms via calibrated homographies and undistortion. The UR3 controller manages joint-space and end-effector coordination, though the system does not implement explicit trajectory optimization or impedance/force control laws.

In action sequencing, bimanual coordination emerges from the LLM’s plan—e.g., tool allocation (“right arm with knife, left arm with gripper”), subtask orchestration (“left arm hands over to right arm for cutting”), and time-ordered execution. Task-level logic (handoffs, parallel sub-tasks) is imposed in the language prompt, not by a dedicated bimanual planner.

This modularity enables language-driven definition of complex dual-arm behaviors without recourse to specialized motion-planning algorithms or hand-coded heuristics (Gbagbe et al., 2024).

3. Language-Guided Recipe Adaptation

Recipe adaptation is handled through a mix of retrieval-based and language-based reasoning pipelines:

• Template-based Adaptation (Bi-VLA): Each recipe variation (e.g., Greek Salad, Fruit Salad, Russian Salad) is encoded as a vector in a small database. RAG locates the closest match to a user’s request and yields the recipe template. Ingredient list verification is performed via vision; if necessary elements are missing, execution is aborted. There is no continuous or fine-grained optimization over ingredient quantities or method steps; adaptation equates to template selection conditional on context.
• Reward-guided Online Adaptation (BiSAIL): Supports zero-shot online skill modulation via interactive user language. BiSAIL maintains a reason–then–modulate structure:
  • Adaptation Objective Reasoning: The Embodied Skill Adaptation Chain-of-Thought (ESA-CoT) module, built on a multimodal LLM, infers a black-box reward function $R(\tau; l, o)$ from the robot’s observation $o$ and verbal feedback $l$. This reward encodes high-level task constraints (e.g., gentler pouring, altered stirring direction).
  • Motion Modulation: Task-agnostic bimanual motion priors $p_\phi(\tau)$ (diffusion models) are combined with task policies $p_\theta(\tau|o)$, steering a trajectory population toward high-reward behaviors. Compositional MCMC and denoising leverage the inferred reward for guided sampling.
  • Online Refinement: Executed motions are critiqued by the LLM and iteratively improved in response to additional user feedback or outcome observation (Li et al., 27 Mar 2026).

Adaptation thus spans both retrieval-based template switching (Bi-VLA) and continuous motion re-targeting based on dynamically inferred objectives (BiSAIL).

4. Diffusion-Based Skill Modulation for Real-Time Adaptation

In BiSAIL, adaptation of bimanual skills to user feedback is achieved through diffusion modulation. Bimanual trajectory $\tau$ is scored against the reward $R$, with the diffusion process guided by the score gradient of both the pre-trained policy and the reward objective:

$$\nabla_{\tau_t} \log p_{\theta}(\tau_t, t) \to \nabla_{\tau_t} \log p_{\theta}(\tau_t, t) + \lambda \nabla_{\tau_t} O_{\text{adapt}}(\tau_t)$$

At each denoising step during sampling, the policy is nudged toward satisfying the high-level adaptation constraints inferred from language (e.g., velocity or smoothness in pouring, temporal sequencing in stirring). Coordination patterns $c$ (symmetric, unimanual) act as constraints within the compositional sampling routine.

This mechanism allows non-expert users to issue corrections in plain language (e.g., “pour more slowly,” “stir counterclockwise for 10 seconds”), which propagate as reward shaping terms directing on-the-fly motion synthesis (Li et al., 27 Mar 2026).

5. Evaluation Metrics and Experimental Findings

The following table summarizes key evaluation aspects for Bi-VLA and BiSAIL:

Module/Level	Metric / Result	Domain
Language Module	100% syntactic code correctness (100 GPT-3.5 queries)	Bi-VLA
Vision Module	96.06% per-object captioning accuracy	Bi-VLA
Vision Module	83.4% full-ingredient detection (complete)	Bi-VLA
System Task Success	83.4% end-to-end correct execution	Bi-VLA
Adaptation Efficacy	Significant outperformance in human-in-the-loop adaptability and cross-embodiment scalability	BiSAIL

In Bi-VLA, system-level success is constrainted by visual mis-detection; language and code generation are robust. Recipe adaptation is limited by the granularity of the template database and visual inventory.

BiSAIL demonstrates on two dual-arm platforms across six bimanual tasks—including real-time culinary skill adjustment in response to high-level natural language correction. The adaptation loop is realized by population-based sampling, diffusion steps, and live execution/refinement (Gbagbe et al., 2024, Li et al., 27 Mar 2026).

6. Design Principles, Scalability, and Future Directions

Key observations from Bi-VLA and BiSAIL development inform principles for future bimanual manipulation and recipe adaptation systems:

• Modularity: Strict decoupling of perception, reasoning, and control enables extensibility—improved vision models, advanced controllers, or enriched LLMs can be hot-swapped with minimal system changes.
• Language-Driven Coordination: Embedding explicit arm–tool assignments and sub-task plans in prompts enables sophisticated dual-arm orchestration without bespoke task planners.
• Failure Modes: Visual perception (ingredient detection) remains the single greatest limitation in system robustness. Improving visual-LLM accuracy is crucial for broader recipe generalization and transfer to highly variable domestic settings.
• Task Extension: To scale beyond cooking (e.g., laundry folding), new primitives and templates can be integrated into the respective APIs and vector DBs. Prompt engineering remains critical for role and safety specification.
• Continuous Adaptation: Online diffusion-based skill modulation, as in BiSAIL, provides a path for fine-grained, context-aware motion correction driven by intuitive verbal feedback, enabling customization by non-expert operators.

Broader adoption of such frameworks hinges on advances in visual robustness, real-time policy adaptation, ergonomic safety in human-robot environments, and data-driven expansion of recipe/task corpora (Gbagbe et al., 2024, Li et al., 27 Mar 2026).