Collaborative Manipulation Task Graphs (CMTG)

• Collaborative Manipulation Task Graphs are directed, bipartite graphs that represent multi-robot manipulation tasks by encoding object dependencies, occlusions, and handover constraints.
• They are constructed through off-line motion planning, occlusion computations, and recursive graph assembly to create minimal, feasible action sequences.
• CMTGs integrate mixed-integer programming and Monte-Carlo Tree Search to generate optimal task skeletons and enable dynamic adaptation in robotic applications.

A Collaborative Manipulation Task Graph (CMTG) is a structured, graph-based abstraction designed to model, instantiate, and reason over multi-robot collaborative manipulation domains where object dependencies, occlusions, and robot assignments impose complex task and motion constraints. CMTGs encode the precedence, blocking, and feasibility relations among pick-and-place actions and movable objects, supporting the generation of minimal, feasible action sequences and skeletons that coordinate multiple robots, including explicit modeling of handovers and occlusion-driven dependencies (Zhang et al., 2023, Zhang et al., 2022).

1. Formal Definition and Structure

A CMTG is defined as a directed, labeled bipartite graph

$\mathcal{G} = (V, E, \mathcal{L})$

where:

• $V = V_O \cup V_A$, being the union of object nodes $V_O$ and action nodes $V_A$:
  • $V_O = \{ o_M : M \in \mathbf{M} \}$, a node for each movable object $M$.
  • $$V_A = \{ a_{\bar p} : \bar p \in \bar{\mathcal{A}} \}$$, an action node for each partially grounded pick-and-place action $\bar p$.
• $E$ comprises three edge sets:
  • $E_{OA} \subseteq V_O \times V_A$: "action edges," which specify which action moves which object.
  • $V = V_O \cup V_A$0: "block-pick" edges, capturing objects that block a pick action.
  • $V = V_O \cup V_A$1: "block-place" edges for objects blocking a place action.
• $V = V_O \cup V_A$2 assigns attributes and predicate-truth to nodes and edges, such as action details (objects, regions, robot assignments, grasps), object geometry/position, and blocking predicates (e.g., $V = V_O \cup V_A$3, $V = V_O \cup V_A$4).

This encoding allows the CMTG to represent dependencies necessary for synchronizing collaborative routines, handovers, and occlusion resolution in domains where monotonicity (no re-grasping/dropping) and synchrony are assumed (Zhang et al., 2023, Zhang et al., 2022).

2. Algorithmic Construction Procedure

Construction of a CMTG involves multiple stages:

1. Off-line motion-planner calls:

For each robot $V = V_O \cup V_A$5, object $V = V_O \cup V_A$6, and grasp $V = V_O \cup V_A$7:

• Test collision-free reachability for pick and place:
  • $V = V_O \cup V_A$8, $V = V_O \cup V_A$9.
  • If unsuccessful, invoke "minimum-constraint-removal" planning, ignoring movable obstacles.
• Check handover feasibility across robot pairs $V_O$0 at predefined points: $V_O$1.

2. Occlusion and blocking predicate computation:

For each feasible pick/place trajectory, compute swept volumes and test for occlusion predicates with all movable objects ($V_O$2, $V_O$3).

3. Graph assembly (recursive):

• For each goal object $V_O$4,
  • Add $V_O$5 to $V_O$6.
  • For each feasible $V_O$7 and $V_O$8: create $V_O$9 and add $V_A$0 to $V_A$1.
  • Recursively ensure blocking objects (from occlusion predicates) are present and create blocking edges.

All stages are parallelizable across robots and grasps for computational efficiency (Zhang et al., 2023, Zhang et al., 2022).

3. Mixed-Integer Programming over CMTGs

Once constructed, the CMTG admits a direct MIP formulation, which is core to synthesizing optimal task skeletons under constraint:

Variables:

• $V_A$2: action $V_A$3 on object $V_A$4 is scheduled at or after $V_A$5.
• $V_A$6: object $V_A$7 is scheduled to unblock $V_A$8 at $V_A$9.

Objective:

$V_O = \{ o_M : M \in \mathbf{M} \}$0

This minimizes the number of moved objects (favoring monotonic, uncluttered plans).

Key Constraints:

CMTG-MIP encodes at least the following (denoted as C1–C7):

1. Monotonicity: $V_O = \{ o_M : M \in \mathbf{M} \}$1
2. Blocking consistency: $V_O = \{ o_M : M \in \mathbf{M} \}$2 for all blocking edges.
3. Selective movement: Non-goal/non-blocking objects can only move to unblock others.
4. Robot action capacity: Each robot is assigned at most one action per timestep.
5. Goal coverage: Every goal object is moved.
6. Unique movement: Each object moved once.
7. Precedence: Blocking objects must be completed before dependent actions.

Constraints are linearized via the Big-M technique. For each feasible skeleton, the MIP yields a sequence order and robot/object assignments satisfying precedence and resource sharing requirements, including handover actions as needed (Zhang et al., 2023, Zhang et al., 2022).

4. Integration with Monte-Carlo Tree Search (MCTS)

The CMTG-MIP solution is wrapped in an MCTS-based meta-planner for exploration–exploitation tradeoff and dynamic adaptation:

• Task skeleton generation: MIP returns candidate plans (task skeletons).
• Plan grounding: A reverse-search attempts to ground a skeleton as a fully geometric, executable plan.
• MCTS tree structure: Nodes store partially-grounded prefixes; edges represent next skeletons.
• Selection: Chooses the skeleton maximizing

$V_O = \{ o_M : M \in \mathbf{M} \}$3

$V_O = \{ o_M : M \in \mathbf{M} \}$4 biases toward shorter skeletons.

• Expansion and evaluation: Attempts to ground skeletons fully; on failure, adds newly detected blockers to CMTG and replans.
• Backpropagation: Propagates composite rewards reflecting completeness and plan length.

This framework operates as an anytime planner, repeatedly solving the CMTG-MIP and updating the CMTG as the world evolves or as new blockers appear during execution (Zhang et al., 2023, Zhang et al., 2022).

5. Extensions: Semantic–Geometric CMTGs and Learning from Demonstration

The CMTG formalism generalizes to semantic–geometric task graphs constructed from human demonstration data. For bimanual or multi-agent human–robot collaboration:

• At each timestep $V_O = \{ o_M : M \in \mathbf{M} \}$5:

$V_O = \{ o_M : M \in \mathbf{M} \}$6

with $V_O = \{ o_M : M \in \mathbf{M} \}$7 objects + hands, $V_O = \{ o_M : M \in \mathbf{M} \}$8 fully connected, $V_O = \{ o_M : M \in \mathbf{M} \}$9=one-hot task encoding.

• Node features: one-hot identity + $M$0D position; Edge features: multi-hot semantic relations over time.
• A Message Passing Neural Network (MPNN) encoder, augmented with rotary positional encodings, propagates scene and temporal information.
• A Transformer-based decoder predicts next/future action-object pairs and regresses future motion trajectories.
• The model is optimized via a combined cross-entropy and MSE loss, evaluated for action/object classification and motion prediction on large-scale human and robotic demonstration datasets. Transfer to physical bimanual robotic platforms is achieved with minimal fine-tuning, maintaining high accuracy in online action selection (Herbert et al., 16 Jan 2026).

6. Empirical Results and Applications

Empirical evaluation demonstrates the utility of CMTG-driven planning for:

• Packaging and box-moving domains (multi-robot, multi-object, occlusion-rich).
• Underground mining (robotic arm and autonomous roof bolter collaboration).
• Human demonstration transfer to bimanual robotic systems with high online task success, low intervention, and near-perfect sequence accuracy (Zhang et al., 2023, Herbert et al., 16 Jan 2026, Zhang et al., 2022).

CMTG-based approaches consistently outperform baselines in metrics of planning time, plan optimality (plan length, number of objects moved), and generalization across task distributions, both in simulated and physical robot deployments.

7. Significance and Outlook

The CMTG provides a unifying abstraction for encoding, manipulating, and optimizing multi-robot collaborative manipulation tasks with explicit action-object dependencies, occlusion awareness, and support for handover synthesis. Its integration with MIP-backed skeleton generation and MCTS-based high-level control aligns with state-of-the-art requirements for scalable, anytime task-and-motion planning in dynamic, high-dimensional domains. The extension to learning semantic–geometric task graphs from demonstration suggests compatibility with emerging data-driven skill learning paradigms (Zhang et al., 2023, Herbert et al., 16 Jan 2026, Zhang et al., 2022).