---
title: Tool-Centric Inverse Dynamics Model
url: https://www.emergentmind.com/topics/tool-centric-inverse-dynamics-model-tc-idm
type: topic
---

# Tool-Centric Inverse Dynamics Model

The Tool-Centric Inverse Dynamics Model (TC-IDM) is an architectural and algorithmic approach for bridging the gap between high-level generative world models, typically realized as video-based vision-language planners, and the execution of low-level, physically-actionable robot control commands. By anchoring on the imagined trajectory of the robot's tool (i.e., end-effector) generated by a world model, TC-IDM provides a robust and interpretable intermediate representation—enabling the translation of visually planned actions into 6-DoF end-effector motions and corresponding gripper commands, even in scenarios involving novel tools, diverse end-effectors, and previously unseen object interactions, including with deformable materials. TC-IDM combines elements of geometry-grounded inverse dynamics, learned vision-driven grasp policy, and nonparametric modeling, supporting both analytic and data-driven components and facilitating viewpoint invariance and strong generalization properties [2601.18323][1901.03872].

## 1. Formal Definition and Interface

TC-IDM operates at the intersection of generative planning and low-level control. The primary interface consists of:

- **Inputs**: 
  - Generated RGB video $V_{\text{rgb-gen}} = \{I_{\text{rgb-gen}}^t\}_{t=0}^T$ produced by a video-based world model and accompanied by depth maps and camera poses.
  - An initial RGB-D observation and task instruction $L$.

- **Output**: 
  - A timed sequence of control commands $u(t) = \left( u_{\text{TCP}}(t), u_{\text{gripper}}(t)\right)$, where:
    - $u_{\text{TCP}}(t) \in SE(3)$ encodes the 6-DoF pose of the tool center point (TCP).
    - $u_{\text{gripper}}(t) \in \mathbb{R}$ is the continuous gripper aperture.

The planned trajectory is captured as a cloud of 3D points (waypoints) on the tool, $\tau_{\text{tool}}(t) = \{\mathbf{x}_i^t\}_{i=1}^K$, extracted from generated video and depth. This tool-centric representation is both more tolerant to scene complexity and directly mappable to robot motions [2601.18323].

## 2. Mathematical Formulation

TC-IDM comprises both analytic and learned components:

- **Analytic Geometry-Grounded Inverse Dynamics**: 
  - For each time pair $(t,\,t{+}1)$, given point correspondences $\mathcal{T}_{\text{gripper}}^t, \mathcal{T}_{\text{gripper}}^{t+1}$, solve for the rigid transformation:
    $$
    (\mathbf{R}_t, \mathbf{p}_t) = \arg\min_{\mathbf{R}\in SO(3),\,\mathbf{p}\in\mathbb{R}^3}\sum_{i=1}^K \left\| \mathbf{R} \mathbf{x}_i^t + \mathbf{p} - \mathbf{x}_i^{t+1} \right\|^2
    $$
  - The solution yields the 6-DoF TCP motion $u_{\text{TCP}}(t+1) = (\mathbf{R}_t, \mathbf{p}_t)$.

- **Learned Vision-Driven Gripper Control**:
  - Dense semantic embeddings $f_{\text{dino}}^t$ are extracted using a frozen DINOv3 encoder: $F_{\text{dino}} = \{f_{\text{dino}}^t\}_{t=0}^T$.
  - An MLP "GripperHead" regresses the gripper aperture:
    $$
    u_{\text{gripper}}(t) = \mathrm{GripperHead}(f_{\text{dino}}^t)
    $$
  - Trained by imitation, the loss is
    $$
    \mathcal{L}_{\text{gripper}} = \sum_{t=0}^{T} \left\|u_{\text{gripper}}(t) - u_{\text{gripper}}^\star(t)\right\|^2
    $$
    where $u_{\text{gripper}}^\star(t)$ is the ground-truth from demonstrations.

- **Combined loss (for learned branch)**:
    $$
    \mathcal{L} = \mathcal{L}_{\text{gripper}} + \lambda \mathcal{L}_{\text{TCP}}
    $$
    In the canonical TC-IDM, $\mathcal{L}_{\text{TCP}}$ is omitted due to analytic inversion.

## 3. Architectural Components

The architecture divides the inference and action mapping workflow into modular stages:

- **Segmentation and 3D Motion Estimation**:
  - Gripper/tool masks are generated via SAM 3 segmentation of the imagined video frames.
  - Conditioned on these masks, a 3D point tracker (SpatialTrackerv2) yields dense point trajectories.
  - Rigid-body filtering selects the $K$ point tracks on the tool that best fit a rigid transform, enforcing tool-centricity.

- **Decoupled Action Heads**:
  - A geometry-based head analytically computes TCP pose via rigid alignment; this branch is nonparametric and requires no learned weights.
  - A vision-driven head operates on high-dimensional DINOv3 embeddings to produce the gripper aperture; this branch is learned via MLP and is supervised by demonstration data.
  - The decoupling ensures semantic grasp cues (e.g., open/close, pinch) are distinguished from 3D spatial motion [2601.18323].

## 4. Operational Paradigm: Planning, Representation, and Translation

- **Plan-and-Translate**:
  - **Planning**: A video world model (e.g., WoW, Cosmos2, Kling) accepts initial state and instruction to generate the future RGB video encapsulating the tool’s full trajectory.
  - **Representation**: This video is transformed into 3D tool-point clouds using segmentation and depth alignment.
  - **Translation**: At each timestep, analytic pose extraction (for $u_{\text{TCP}}$) and learned gripper inference synergize to yield a final robot command $u(t)$, streamed at 100–500 Hz to the robot controller for real-time execution [2601.18323].

- **Inference Pipeline Summary**:

| Step | Operation | Method/Component              |
|------|-----------|------------------------------|
| 1    | Get initial state | Sensor RGB-D, text      |
| 2    | Generate video | World model (diff/transformer)   |
| 3    | Align depth/camera pose | Sensor fusion   |
| 4    | Segment gripper/tool | SAM 3              |
| 5    | Track points | SpatialTrackerv2           |
| 6    | Filter for tool | Rigid-body criterion    |
| 7    | Extract actions | Analytic + MLP heads    |
| 8    | Stream $u(t)$ | Robot control layer         |

## 5. Experimental Evaluation and Comparative Analysis

TC-IDM was evaluated on nine real-world manipulation tasks stratified by difficulty and on zero-shot deformable object manipulations (cloth removal, folding, hoodie folding). Primary metric was task success rate (binary), with results:

- Overall (across video models): 61.11%
- Simple (easy and medium): 77.7%
- Zero-shot deformable: 38.46%

Compared to end-to-end vision-language-action (VLA) baselines (e.g., $\pi_0$, OpenVLA, RT-2, Octo) and video-conditioned inverse dynamics models (AVDC, VidBot, AnyPos), TC-IDM yielded 30–50pp higher performance on hard tasks, near-perfect on easy cases, and unprecedented zero-shot capability for deformable objects. These results establish TC-IDM as a robust “last-mile” bridge, generalizing to new viewpoints (Apple Pro & D435i), cross-embodiment tasks (single- vs. dual-arm), and unseen cloth handling [2601.18323].

## 6. Foundations in Nonparametric Tool-Centric Inverse Dynamics

The TC-IDM concept generalizes earlier nonparametric, tool-aware inverse dynamics, as described by Haninger & Tomizuka [1901.03872]:
- In multimodal tool scenarios, explicit clustering via (soft) EM or collapsed Gibbs over Gaussian Process (GP) models associates each tool mode $k$ with a distinct inverse dynamics residual predictor $h_k(x) = \tau - g(q)$.
- The system leverages the tool's identity (mode) and online experience to learn or switch between GPs, with real-time assignment and disturbance (collision) rejection.
- Formal passivity is shown by constructing a composite storage function $S(q, \dot{q})$, guaranteeing the closed loop with impedance control remains passive.
- For a new tool, the framework discovers and adapts a new GP mode, while outlier detection mechanisms exclude transient disturbances, maintaining safe feedforward control.

## 7. Implementation and Practical Considerations

- **Data Acquisition**: Controlled explorations with low-gain impedance generate tool-specific data. For real-time operation, sparse GP mechanisms or buffer-based truncation are employed.
- **Online Inference**: At each control step, the system estimates expected torque (for the tool mode) and its confidence, reverting to fallback impedance control on uncertainty or exogenous perturbations [1901.03872].
- **Scalability**: Naive GP inference is cubic in sample count, necessitating fast approximate methods (FITC, SOD) or bounded buffers for deployment.
- **Safety and Generalization**: Passivity guarantees, disturbance rejection, and decoupling of tool-centric motion from semantic gripper cues contribute to robust and interpretable autonomous operation.

---

By leveraging the tool’s predicted trajectory, the Tool-Centric Inverse Dynamics Model framework enables a robust, modular, and generalizable approach for grounding high-level video-generated plans in executable robot actions, supporting complex manipulation across a range of tool, viewpoint, and object variability [2601.18323][1901.03872].

Source: https://www.emergentmind.com/topics/tool-centric-inverse-dynamics-model-tc-idm