---
title: Collaborative Manipulation Task Graphs (CMTG)
url: https://www.emergentmind.com/topics/collaborative-manipulation-task-graphs-cmtg
type: topic
---

# Collaborative Manipulation Task Graphs (CMTG)

A Collaborative Manipulation Task Graph (CMTG) is a structured, graph-based abstraction designed to model, instantiate, and reason over multi-robot collaborative manipulation domains where object dependencies, occlusions, and robot assignments impose complex task and motion constraints. CMTGs encode the precedence, blocking, and feasibility relations among pick-and-place actions and movable objects, supporting the generation of minimal, feasible action sequences and skeletons that coordinate multiple robots, including explicit modeling of handovers and occlusion-driven dependencies [2310.08802][2210.08005].

## 1. Formal Definition and Structure

A CMTG is defined as a directed, labeled bipartite graph
$$
\mathcal{G} = (V, E, \mathcal{L})
$$
where:
- $V = V_O \cup V_A$, being the union of object nodes $V_O$ and action nodes $V_A$:
  - $V_O = \{ o_M : M \in \mathbf{M} \}$, a node for each movable object $M$.
  - $V_A = \{ a_{\bar p} : \bar p \in \bar{\mathcal{A}} \}$, an action node for each partially grounded pick-and-place action $\bar p$.
- $E$ comprises three edge sets:
  - $E_{OA} \subseteq V_O \times V_A$: "action edges," which specify which action moves which object.
  - $E_{AP}^{\mathrm{pick}} \subseteq V_A \times V_O$: "block-pick" edges, capturing objects that block a pick action.
  - $E_{AP}^{\mathrm{place}} \subseteq V_A \times V_O$: "block-place" edges for objects blocking a place action.
- $\mathcal{L}$ assigns attributes and predicate-truth to nodes and edges, such as action details (objects, regions, robot assignments, grasps), object geometry/position, and blocking predicates (e.g., $\mathrm{OccludesPick}$, $\mathrm{OccludesGoalPlace}$).

This encoding allows the CMTG to represent dependencies necessary for synchronizing collaborative routines, handovers, and occlusion resolution in domains where monotonicity (no re-grasping/dropping) and synchrony are assumed [2310.08802][2210.08005].

## 2. Algorithmic Construction Procedure

Construction of a CMTG involves multiple stages:

**1. Off-line motion-planner calls:**  
For each robot $R$, object $M$, and grasp $g$:
- Test collision-free reachability for pick and place:
  - $\mathrm{ReachablePick}(M,g,R)$, $\mathrm{ReachablePlace}(M,Re,g,R)$.
  - If unsuccessful, invoke "minimum-constraint-removal" planning, ignoring movable obstacles.
- Check handover feasibility across robot pairs $(R_1, R_2)$ at predefined points: $\mathrm{EnableGoalHandover}(M, g_1, g_2, R_1, R_2)$.

**2. Occlusion and blocking predicate computation:**  
For each feasible pick/place trajectory, compute swept volumes and test for occlusion predicates with all movable objects ($\mathrm{OccludesPick}$, $\mathrm{OccludesGoalPlace}$).

**3. Graph assembly (recursive):**  
- For each goal object $M$,
  - Add $o_M$ to $V_O$.
  - For each feasible $(R^\mathrm{pick}, g^\mathrm{pick})$ and $(R^\mathrm{place}, g^\mathrm{place})$: create $a_{\bar p}$ and add $(o_M, a_{\bar p})$ to $E_{OA}$.
  - Recursively ensure blocking objects (from occlusion predicates) are present and create blocking edges.

All stages are parallelizable across robots and grasps for computational efficiency [2310.08802][2210.08005].

## 3. Mixed-Integer Programming over CMTGs

Once constructed, the CMTG admits a direct MIP formulation, which is core to synthesizing optimal task skeletons under constraint:

**Variables:**
- $X^t_{M,\bar a} \in \{0,1\}$: action $\bar a$ on object $M$ is scheduled at or after $t$.
- $X^t_{\bar a,M'} \in \{0,1\}$: object $M'$ is scheduled to unblock $\bar a$ at $t$.

**Objective:**
$$
\min \sum_{(M, \bar a) \in E_{OA}} X^1_{M,\bar a}
$$
This minimizes the number of moved objects (favoring monotonic, uncluttered plans).

**Key Constraints:**  
CMTG-MIP encodes at least the following (denoted as C1–C7):
1. **Monotonicity:** $X^t_{M,\bar a} \geq X^{t+1}_{M,\bar a}$
2. **Blocking consistency:** $X^t_{M,\bar a} = X^t_{\bar a,M'}$ for all blocking edges.
3. **Selective movement:** Non-goal/non-blocking objects can only move to unblock others.
4. **Robot action capacity:** Each robot is assigned at most one action per timestep.
5. **Goal coverage:** Every goal object is moved.
6. **Unique movement:** Each object moved once.
7. **Precedence:** Blocking objects must be completed before dependent actions.

Constraints are linearized via the Big-M technique. For each feasible skeleton, the MIP yields a sequence order and robot/object assignments satisfying precedence and resource sharing requirements, including handover actions as needed [2310.08802][2210.08005].

## 4. Integration with Monte-Carlo Tree Search (MCTS)

The CMTG-MIP solution is wrapped in an MCTS-based meta-planner for exploration–exploitation tradeoff and dynamic adaptation:
- **Task skeleton generation:** MIP returns candidate plans (task skeletons).
- **Plan grounding:** A reverse-search attempts to ground a skeleton as a fully geometric, executable plan.
- **MCTS tree structure:** Nodes store partially-grounded prefixes; edges represent next skeletons.
- **Selection:** Chooses the skeleton maximizing
  $$
  \mathrm{UCB}(D_j,E_i) = \frac{E_i.\mathrm{value}}{E_i.\mathrm{visits}+1} + c\,E_i.\mathrm{prior} \sqrt{\frac{D_j.\mathrm{visits}}{E_i.\mathrm{visits}+1}}
  $$
  $E_i.\mathrm{prior}=1/|\bar{\mathbf{S}}|$ biases toward shorter skeletons.
- **Expansion and evaluation:** Attempts to ground skeletons fully; on failure, adds newly detected blockers to CMTG and replans.
- **Backpropagation:** Propagates composite rewards reflecting completeness and plan length.

This framework operates as an anytime planner, repeatedly solving the CMTG-MIP and updating the CMTG as the world evolves or as new blockers appear during execution [2310.08802][2210.08005].

## 5. Extensions: Semantic–Geometric CMTGs and Learning from Demonstration

The CMTG formalism generalizes to semantic–geometric task graphs constructed from human demonstration data. For bimanual or multi-agent human–robot collaboration:
- At each timestep $t$:
  $$
  G^{(t)} = (V, E, u)
  $$
  with $V=$ objects + hands, $E=$ fully connected, $u$=one-hot task encoding.
- Node features: one-hot identity + $3$D position; Edge features: multi-hot semantic relations over time.
- A Message Passing Neural Network (MPNN) encoder, augmented with rotary positional encodings, propagates scene and temporal information.
- A Transformer-based decoder predicts next/future action-object pairs and regresses future motion trajectories.
- The model is optimized via a combined cross-entropy and MSE loss, evaluated for action/object classification and motion prediction on large-scale human and robotic demonstration datasets. Transfer to physical bimanual robotic platforms is achieved with minimal fine-tuning, maintaining high accuracy in online action selection [2601.11460].

## 6. Empirical Results and Applications

Empirical evaluation demonstrates the utility of CMTG-driven planning for:
- Packaging and box-moving domains (multi-robot, multi-object, occlusion-rich).
- Underground mining (robotic arm and autonomous roof bolter collaboration).
- Human demonstration transfer to bimanual robotic systems with high online task success, low intervention, and near-perfect sequence accuracy [2310.08802][2601.11460][2210.08005].

CMTG-based approaches consistently outperform baselines in metrics of planning time, plan optimality (plan length, number of objects moved), and generalization across task distributions, both in simulated and physical robot deployments.

## 7. Significance and Outlook

The CMTG provides a unifying abstraction for encoding, manipulating, and optimizing multi-robot collaborative manipulation tasks with explicit action-object dependencies, occlusion awareness, and support for handover synthesis. Its integration with MIP-backed skeleton generation and MCTS-based high-level control aligns with state-of-the-art requirements for scalable, anytime task-and-motion planning in dynamic, high-dimensional domains. The extension to learning semantic–geometric task graphs from demonstration suggests compatibility with emerging data-driven skill learning paradigms [2310.08802][2601.11460][2210.08005].

Source: https://www.emergentmind.com/topics/collaborative-manipulation-task-graphs-cmtg