---
title: Imitation-Guided Bimanual Planning
url: https://www.emergentmind.com/topics/imitation-guided-bimanual-planning-framework
type: topic
---

# Imitation-Guided Bimanual Planning

Imitation-guided bimanual planning denotes a family of robotic frameworks that use demonstrations to generate, sequence, or constrain coordinated two-arm behavior for tasks in which dual-arm synchronization, object-relative geometry, contact dynamics, and long-horizon stage structure are all consequential. Across recent work, the shared objective is not merely to imitate arm trajectories, but to convert demonstrations into planning-relevant structure: movement primitives, keyposes, object-centric transforms, affordance maps, scene-graph predicates, symmetry operators, or latent future representations. These structures are then coupled to generative policies, task-and-motion planners, consistency or diffusion models, graph neural networks, or compliant controllers to produce stable execution under coordination, reachability, collision, and contact constraints [2010.05134] [2405.03666] [2406.10093] [2506.18825] [2601.12116] [2603.07165].

## 1. Problem setting and design rationale

Bimanual manipulation is repeatedly formulated as a setting in which two arms must coordinate continuously while remaining compatible with the manipulated object, the environment, and the task phase. The difficulty is attributed to several recurring factors: high-dimensional action spaces, multi-modal dynamics, strict spatial and temporal synchronization, contact-rich transitions, and long-horizon error accumulation. In the graph-based hierarchical formulation of "Deep Imitation Learning for Bimanual Robotic Manipulation," these issues appear as multi-modal dynamics, relational variability, and generalization across changed object locations [2010.05134]. In "BiKC+: Bimanual Hierarchical Imitation with Keypose-Conditioned Coordination-Aware Consistency Policies," the same setting is framed as a combination of dual-arm coordination and long-horizon, multi-stage structure, with the additional observation that failures or delays at one stage propagate to later stages [2601.12116].

Several systems make the object explicit in the planning problem. "RoTri-Diff: A Spatial Robot-Object Triadic Interaction-Guided Diffusion Model for Bimanual Manipulation" states that generating two-arm trajectories requires maintaining spatial consistency with the manipulated object over time, avoiding inter-arm collisions, and maintaining stable contacts [2603.07165]. "BiPreManip: Learning Affordance-Based Bimanual Preparatory Manipulation through Anticipatory Collaboration" focuses on asymmetric sequential tasks in which one arm must first reconfigure an object so that the other arm can perform the goal-directed action, such as pushing an iPad to the table edge before grasping it or lifting a pen body so that the other hand can remove its cap [2603.21679].

Other formulations emphasize execution conditions beyond nominal kinematics. "A System for Imitation Learning of Contact-Rich Bimanual Manipulation Policies" treats a bracket insertion task as a closed-chain, contact-rich process in which the policy must exploit contact rather than avoid it, and reports that force/torque data is crucial for phase estimation and robustness [2208.00596]. "Imitation-Guided Bimanual Planning for Stable Manipulation under Changing External Forces" instead centers the problem around changing external wrenches, grasp transitions between uni-manual and bi-manual support, and motion-performance optimization under stability constraints [2509.19261]. In mobile settings, "MoMaGen: Generating Demonstrations under Soft and Hard Constraints for Multi-Step Bimanual Mobile Manipulation" extends the planning problem to include mobile base placement and active camera placement, with hard reachability, collision, visibility, and temporal constraints [2510.18316].

A plausible implication is that imitation-guided bimanual planning has evolved from direct behavioral cloning of joint trajectories toward structured formulations in which demonstrations are used to define intermediate planning objects. Those intermediate objects differ across papers, but they all function as coordination scaffolds.

## 2. Representational foundations

A central distinction among frameworks concerns what demonstrations are converted into before planning or action generation begins.

| Framework | Core representation | Representative role |
|---|---|---|
| HDR-IL | Elemental movement primitives with recurrent graph neural network dynamics | Decomposes multi-modal trajectories into primitive-specific relational dynamics |
| ScrewMimic | Screw action $\sigma = (g_l, g_r, S, \tau_l)$ | Projects human bimanual motion into a compact serial-linkage manifold |
| BiKC / BiKC+ | Joint-space bimanual keyposes, with $m_j \in \{0,1\}$ in BiKC+ | Marks sub-stage completion and coordination mode |
| RoTri-Diff | 21D RoTri vector $R_t = [p_{(A\to B)}, p_{(A\to O)}, p_{(B\to O)}]$ | Encodes robot-object triadic geometry densely over time |
| SViP | Semantic scene graphs and object-centric motion primitives | Segments demonstrations into symbolic-contact structure for TAMP |
| EquiBim | Bilateral reflection group action on observations and actions | Enforces left-right equivariance during imitation learning |

In relational formulations, the goal is to encode how arms and objects constrain one another. HDR-IL uses a fully connected graph over state features, with Graph Attention message passing inside a GRU encoder-decoder, and explicitly reports that absolute pose features plus learned interactions outperform hand-crafted relational coordinates [2010.05134]. RoTri-Diff makes this coupling geometric: with end-effectors $A,B$ and object $O$, world-frame poses $T_A,T_B,T_O \in SE(3)$ induce relative transforms such as $T_{AO}=T_A^{-1}T_O$, $T_{BO}=T_B^{-1}T_O$, and $T_{AB}=T_A^{-1}T_B$, together with the triangular consistency condition
$$
T_{AO}T_{OB}T_{BA}=I.
$$
The associated loop-consistency loss,
$$
L_{\text{loop}} = \| \log(T_{AO}T_{OB}T_{BA}) \|_2,
$$
penalizes drift among pairwise relations [2603.07165].

Keypose-centered frameworks represent demonstrations as milestone states rather than dense trajectories. In BiKC, a keypose is a joint-space waypoint that is common across demonstrations and marks a sub-stage boundary; the low-level policy then generates action chunks toward that waypoint [2406.10093]. BiKC+ extends this by attaching a binary coordination indicator $m_j$, where $m_j=1$ requires both arms to reach the keypose simultaneously and $m_j=0$ allows either arm to reach independently, with coordination ranges derived from VLM-inferred contact graphs [2601.12116].

Other papers encode coordination in more task-specific abstractions. ScrewMimic models the two hands as a serial kinematic linkage with a virtual 1-DoF screw joint, and represents a bimanual behavior by the screw action $\sigma = (g_l, g_r, S, \tau_l)$, where $S$ is the screw axis and $g_l,g_r$ are grasp contacts [2405.03666]. BiPreManip predicts dense goal and preparatory affordance maps over a point cloud, together with object transformation $T^{obj}=(t^{obj},r^{obj}) \in SE(3)$, so that one arm can reconfigure the object into a configuration better suited for the other arm’s final action [2603.21679]. SViP instead monitors demonstrations through semantic scene graphs $G=\{V,E,L\}$ and turns key graph transitions into symbolic preconditions and effects for object-centric primitives such as Pick, Place, Transfer, and Align [2506.18825].

Symmetry-based work treats coordination as an inductive bias over observation-action structure. EquiBim defines a bilateral reflection group $G=\{e,g\}\cong Z_2$ acting on images, point clouds, proprioception, and actions, and imposes the equivariance condition
$$
\pi(g\cdot o)=g\cdot \pi(o),
$$
with an additional regularizer $L_{eq}$ added to the behavior cloning objective [2603.08541].

## 3. Hierarchical policy architectures and generative models

Most imitation-guided bimanual planners are hierarchical, but the level at which hierarchy is introduced differs markedly. HDR-IL uses an explicit three-layer decomposition: a high-level planner predicts the next primitive $z_t\in\{1,\dots,K\}$, a primitive-specific recurrent GNN forecasts a short horizon of future states, and an IK controller turns those states into joint commands [2010.05134]. BiKC and BiKC+ also split the problem into high- and low-level components, but replace primitive labels with predicted keyposes. In BiKC+, the factorization is written as
$$
k_{j+1}, m_{j+1} \sim g_\omega(k_j,o_t), \qquad
a_{t:t+H_a-1} \sim \pi_\theta(o_{t-H_o+1:t}, k_{j+1}),
$$
so that stage planning and trajectory generation remain distinct but tightly coupled [2601.12116].

Generative modeling has become a dominant mechanism for the low-level component. BiKC formulates the trajectory generator as a consistency model trained from scratch, which produces an action sequence in one inference step rather than by iterative denoising [2406.10093]. BiKC+ retains this consistency-model formulation and uses it jointly with a CM-based keypose predictor, with reported real-world latencies such as 34.9 ms on Screwdriver Packing and 27.8 ms on Conveyor [2601.12116]. The emphasis on one-step generation is shared by Bi-HIL, although Bi-HIL uses a different hierarchy: a high-level transformer predicts a subtask command and a scalar subtask-level progress rate, while a low-level transformer-based CVAE generates force-aware action chunks conditioned on that progress estimate and keyframe memory [2603.13315].

Diffusion-based hierarchical policies are more explicit about multi-scale decomposition. RoTri-Diff separates prediction into three stages: object pointflow and RoTri segment prediction, keypose generation, and dense continuous trajectory generation conditioned on the full RoTri segment [2603.07165]. Its diffusion formulation uses a forward noising process
$$
q(x_t \mid x_{t-1})=\mathcal{N}(\sqrt{1-\beta_t}\,x_{t-1}, \beta_t I),
$$
and a reverse denoising process with conditioning on scene tokens, language, proprioception, object initialization, and RoTri state. Diffusion-Based Imaginative Coordination trains a shared diffusion decoder for both actions and future visual latents, but imposes a unidirectional attention mask so that video prediction is conditioned on actions while action prediction remains independent of video tokens; the resulting action-only inference path is then used at test time [2507.11296].

Transformer-based chunking architectures push hierarchy into the temporal interface between policy and control. "Learning Bimanual Manipulation via Action Chunking and Inter-Arm Coordination with Transformers" separates per-arm encoders and inserts an Inter-Arm Coordinated transformer Encoder (IACE) before decoding action chunks, with decoder choice depending on whether the task is synchronous or asynchronous [2503.13916]. InterACT similarly predicts chunked action sequences, but uses a Hierarchical Attention Encoder with segment-wise and cross-segment attention, followed by a Multi-arm Decoder with a Synchronization Block that lets each arm’s decoder consume the other arm’s intermediate outputs before producing final actions [2409.07914].

A related but distinct trend is to insert learned policies into explicit planning systems. SViP does not rely on a single end-to-end low-level policy for an entire task; instead, it partitions demonstrations into contact-rich bimanual segments and uses learned visuomotor policies only for those segments, while scripted object-centric primitives and PDDLStream-based TAMP handle the surrounding transitions [2506.18825]. This suggests a broader interpretation of imitation-guided planning: the demonstrations need not directly specify continuous control everywhere, so long as they define where learned control is necessary and how the planner should enter and exit those regions.

## 4. Planning integration, control realization, and constraints

Execution layers in imitation-guided bimanual planning are typically classical control systems augmented by learned targets, constraints, or safety monitors. IK remains a standard bridge from learned end-effector or keypose targets to joint-space execution. HDR-IL uses stacked Jacobians for both arms and damped least-squares inverse kinematics, with
$$
\dot{q}=J^\dagger \dot{x} + (I-J^\dagger J)\dot{q}_{ns},
$$
for coordinated velocity control [2010.05134]. RoTri-Diff describes both IK and differential IK deployment, including the damped least-squares update
$$
\Delta \theta = J^\top (J J^\top + \lambda^2 I)^{-1}\xi,
$$
together with joint-limit, velocity, acceleration, and SDF-based collision penalties [2603.07165].

Constraint handling ranges from soft penalties to full constrained optimization. MoMaGen casts data generation itself as a constrained optimization problem with hard constraints for dynamics, kinematics, collision avoidance, visibility, preserved end-effector–object transforms during contact-rich segments, and task success. Its formulation jointly reasons over mobile base pose, active camera pose, dual-arm trajectories, timing, and subtask synchronization [2510.18316]. SViP uses PDDLStream with optimistic adaptive planning, and wraps learned modules as black-box samplers and validators: the switching-condition generator provides candidate continuous parameters, and a feasibility validator predicts whether a bimanual learned skill can be initiated safely under the current object arrangement [2506.18825].

Contact-rich systems often replace pure trajectory tracking with compliant control. In the multipoint insertion system of [2208.00596], the low-level controller is Cartesian admittance:
$$
M_d \ddot{x} + D_d \dot{x} + K_d(x-x_d)=F_{ext},
$$
with the learned Bayesian Interaction Primitive estimating phase and future references from multimodal observations that include force/torque. SIMPLe instead deploys Cartesian impedance policies learned from kinesthetic demonstrations, saturates attractor displacement and stiffness to enforce explicit velocity and force limits, and modulates stiffness by epistemic uncertainty so that the robot can be safely reshaped during active kinesthetic correction [2210.16220]. Bi-HIL uses bilateral control during data collection and then conditions its low-level policy on torque, joint angle, angular velocity, language, and progress-rate information, with leader–follower servo constraints $\theta_l-\theta_f=0$ and $\tau_l+\tau_f=0$ shaping the demonstrations themselves [2603.13315].

Stability under task-specific physical requirements has generated additional planning layers. The grasp-transition framework of [2509.19261] models uni-manual and bi-manual grasps as grasp manifolds $M_u$ and $M_b$, seeks stable intersection states $I=M_u\cap M_b$, plans the minimal-regrasp sequence over a graph of stable configurations, and then executes motions with a hierarchical architecture comprising ProMP-based global path generation and a QP-driven local planner with manipulability optimization and obstacle avoidance. ScrewMimic also constrains execution geometrically, but through a different mechanism: once a screw action has been estimated from human video, the two hands are synchronized by construction because the relative motion satisfies $T_{21}(t)=\exp([S]\theta(t))$ and the waypoints are tracked by a stack-of-tasks IK-based whole-body controller [2405.03666].

A plausible synthesis is that imitation-guided bimanual planning has not displaced classical control; rather, it increasingly supplies the intermediate variables that make constrained control tractable.

## 5. Empirical findings across simulation and real robots

Reported performance gains are substantial but heterogeneous, reflecting differences in task families, observation modalities, and evaluation criteria.

| Framework | Reported outcome |
|---|---|
| RoTri-Diff | 80.9% average success on 11 RLBench2 tasks; outperforms baselines by 10.2%; real-world tasks: 5/5, 3/5, 4/5, 4/5 [2603.07165] |
| HDR-IL | Table lift: 100% success; peg-in-hole: 29% success [2010.05134] |
| InterACT | Transfer Cube: 82% vs ACT 50% on Transfer; Peg Insertion: 44% vs ACT 20% on Insert [2409.07914] |
| BiKC+ | Screwdriver Packing: 59.9%; Pants Hanging: 85.0%; Conveyor: 100.0% [2601.12116] |
| EquiBim | RoboTwin Image+Joint: 34.1 → 43.6; mirrored Banana Handover: 0/10 → 5/10 under shift [2603.08541] |
| Ag2x2 | 73.5% success across 13 diverse tasks [2507.19817] |

The evidence repeatedly supports the value of explicit coordination structure. HDR-IL reports that graph attention, residual target-object features, and primitive specialization are all consequential: on Table lift, HDR-IL achieved 100% success, while GRU-GRU reached 13%, and on Peg-in-hole, HDR-IL achieved 29% versus 1% for GRU-GRU [2010.05134]. InterACT reports systematic gains over ACT on both simulated and real tasks, such as Peg Insertion Insert 44% versus 20%, Insert Plug 42% versus 30%, and Sweep 52% versus 42%, with ablations showing that removing the Synchronization Block or cross-segment encoder degrades the coordination-heavy stages most strongly [2409.07914]. The related IACE-based transformer reports an 8–9% average success-rate improvement over ACT, with split decoders favored on asynchronous tasks and a single decoder favored on synchronous tasks [2503.13916].

Keypose-guided hierarchical models report benefits in both reliability and operational speed. BiKC shows that keypose conditioning improves sensitive stages such as contact and insertion, and that a one-step consistency policy can outperform diffusion baselines on dynamic tasks because diffusion latency causes missed pick opportunities [2406.10093]. BiKC+ extends this claim to more complex real-world tasks: 59.9% success on Screwdriver Packing, 85.0% on Pants Hanging, and 100.0% on Conveyor, while also reporting lower total duration than ACT on Screwdriver Packing despite ACT’s comparable per-step latency [2601.12116]. Bi-HIL reports 80% success on both 6-Cup Downstack and 4-Peg-in-Hole, versus 60% and 20% for Bi-ACT and lower numbers for ablations without keyframe memory or subtask-level progress rate [2603.13315].

Diffusion and latent imagination have produced notable gains when the visual future is used during training but not at inference. Diffusion-Based Imaginative Coordination reports a 24.9% increase on ALOHA, an 11.1% increase on RoboTwin, and a 32.5% increase in real-world experiments over ACT, while preserving higher control frequency than standard diffusion inference by omitting video prediction at test time [2507.11296]. RoTri-Diff reports especially large gains on precision-heavy RLBench2 tasks, including Pick plate 40.7% versus PPI 0.0% and AnyBimanual 8.0%, and Put item into drawer 87.0% versus 3DDA 52.7% [2603.07165].

Planning-integrated systems show a different kind of improvement: better recovery under OOD layouts or explicit task constraints. SViP reports 100% success on ID and XY-OOD peg-in-hole settings and 88% on XYH-OOD, while ACT falls to 2% in XYH-OOD; in real-world Object Handoff, Screwdriver Packing, and Cup-sleeve Insertion, SViP consistently exceeds diffusion-policy baselines under OOD and tilted setups using only 20 demonstrations [2506.18825]. MoMaGen reports that synthetic demonstration generation under hard reachability and visibility constraints can support policies trained from a single source demonstration and later fine-tuned with as few as 40 real-world demonstrations, with visibility constraints materially affecting both generated-data quality and downstream policy performance [2510.18316].

Contact- and force-aware systems likewise show clear empirical patterns. The contact-rich insertion system of [2208.00596] reports up to 90% success in the real bimanual task and 100% robustness under disturbances in simulation, with an approximately 20 percentage-point improvement when force/torque data is available under varied starts. The external-force planning framework of [2509.19261] reports shorter end-effector path lengths, higher minimum singular values, larger joint-limit margins, and lower execution times than minimal-regrasp, greedy, and random-sampling baselines in cutting and drilling scenarios.

## 6. Limitations, trade-offs, and future directions

The surveyed literature reports several recurring limitations. RoTri-Diff explicitly depends on rigid-body assumptions and accurate 6D object pose estimation, and notes degradation under calibration error or poor tracking, especially for tight-clearance or deformable-object tasks [2603.07165]. ScrewMimic similarly notes that pose-estimation noise and calibration errors can perturb the recovered screw axis and that tasks not well described by a single 1-DoF constraint require richer action sequences or mixtures of primitives [2405.03666]. BiPreManip identifies instability near small components, single-camera occlusion, and one-shot pushing failures, and argues that multi-step closed-loop pushing and multi-view perception would improve robustness [2603.21679].

Several frameworks depend on manual or heuristic task structure. HDR-IL uses manually labeled primitives and fixed-length call-and-return execution, and explicitly identifies automatic primitive discovery and termination learning as future directions [2010.05134]. BiKC and BiKC+ rely on heuristic keypose extraction and, in BiKC+, VLM-based coordination-range detection, which the paper notes may be viewpoint-sensitive and may benefit from wrist force/torque signals [2406.10093] [2601.12116]. Bi-HIL requires manual subtask annotation and visual keyframe supervision, and it reports no explicit collision-avoidance module, so safe operation remains heavily dependent on demonstration quality and bilateral compliance [2603.13315].

A second recurring trade-off concerns how explicitly coordination should be constrained. BiKC+ states that there is no separate coordination regularizer and that coordination is enforced structurally through synchronized keyposes, shared encoders, and runtime gating [2601.12116]. RoTri-Diff, by contrast, recommends loop-consistency, collision, alignment, antipodal, kinematic, and smoothness penalties for deployment [2603.07165]. This suggests a design spectrum rather than a single consensus: some systems encode coordination in representation and inference logic, while others regularize it directly in geometry or control space.

A third trade-off concerns inductive bias versus task asymmetry. EquiBim shows that bilateral equivariance improves robustness under mirrored distribution shifts, but also reports that tasks with functional asymmetry can show smaller gains or mild drops when symmetry is enforced too strongly [2603.08541]. More broadly, scene-agnostic planners such as the visual-affordance reasoning framework of [2512.09310] gain flexibility by deferring to rule-based constraints and foundation models, but then inherit possible LLM hallucination, role asymmetry errors, and dependence on accurate grounding.

Future work is correspondingly diverse. Reported directions include multi-object triadic sets, differentiable contact models, friction cones, tactile feedback, and embodiment-conditioned policies in RoTri-Diff [2603.07165]; whole-body loco-manipulation and improved planner efficiency in MoMaGen [2510.18316]; automatic success metrics and grasp optimization in ScrewMimic [2405.03666]; richer object-centric sub-goals and open-scene planning in scene-agnostic affordance planners [2512.09310]; and stronger integration of latent representations, demonstrations, and zero-shot skill transfer in Ag2x2, which shows that agent-agnostic visual representations can still be leveraged for imitation learning [2507.19817]. A plausible implication is that the next generation of imitation-guided bimanual planning frameworks will be less defined by any one policy class than by how effectively they couple demonstrations to geometry, contact, and structured task decomposition.

Source: https://www.emergentmind.com/topics/imitation-guided-bimanual-planning-framework