---
title: 'Super-Mimic: Zero-Shot Robotic Imitation'
url: https://www.emergentmind.com/topics/super-mimic
type: topic
---

# Super-Mimic: Zero-Shot Robotic Imitation

Super-Mimic is a hierarchical framework for **zero-shot, long-horizon robotic imitation** that uses an **unscripted human demonstration video** as its primary instruction channel and optionally accepts an additional natural-language command. Its defining premise is that, for complex manipulation, a robot should first **watch** a human demonstration to infer the task’s procedural intent and then **imagine** how each inferred subtask should unfold in the robot’s current scene before acting. The framework therefore couples a **Human Intent Translator (HIT)** with a **Future Dynamics Predictor (FDP)** and an **Action Executor**, and is evaluated on real-world long-horizon manipulation tasks where task decomposition, contact structure, and intermediate motion all matter [2509.22205].

## 1. Problem setting and formal representation

Super-Mimic targets **open-world, long-horizon robotic manipulation without task-specific training**. The robot receives an unscripted human demonstration video \(V\) and optionally a language instruction \(L\), and must execute a multi-step task in a novel environment. The paper emphasizes two difficulties in this setting: **planning difficulty**, because a short instruction such as “tidy up the desk” does not specify the sequence of subtasks, object correspondences, or placements; and **execution difficulty**, because static scene understanding does not determine physically plausible object interactions, contact points, or intermediate motion [2509.22205].

The task is formalized as \(\mathcal{T}\). From \(V\) and optionally \(L\), the system must infer a sequence of subtasks
\[
\mathcal{P} = \{\tau_1, \dots, \tau_N\}.
\]
For each subtask \(\tau_n\), given the robot’s current RGB-D observation \(O_n\), the system generates a sub-plan \(k_n\), and the resulting sequence
\[
K = \{k_1, \dots, k_N\}
\]
provides spatio-temporal guidance to the executor. Each subtask is represented as
\[
\tau_n = (\text{desc}_n, \text{obj}_n, \text{loc}_n, \text{guide}_n, \text{precond}_n),
\]
where \(\text{desc}_n\) is a natural-language subtask description, \(\text{obj}_n\) is the semantic identifier of the target object, \(\text{loc}_n\) is the destination or relevant location, \(\text{guide}_n\) is the textual prompt used to guide future video generation in FDP, and \(\text{precond}_n\) is a set of preconditions required before execution.

This formulation makes the framework explicitly hierarchical. HIT addresses the question of **what should be done next**, while FDP addresses **what the world should look like as this step unfolds**. The executor then converts the imagined evolution into grasp and motion commands. A plausible implication is that the architecture is intended to separate symbolic procedural inference from dynamics-aware control, rather than forcing a single model to absorb both burdens.

## 2. System architecture and operational loop

The full pipeline is sequential and iterative. Starting from a human demonstration video, Super-Mimic first applies HIT to convert raw video into a robot-relevant symbolic plan. For the current subtask, FDP then synthesizes an imagined future video from the robot’s present view. A 3D waypoint trajectory is extracted from that generated video, and the Action Executor refines and executes the motion. Success is then verified; if the current subtask fails, the system replans from the updated observation [2509.22205].

| Module | Input | Output |
|---|---|---|
| HIT | Human demonstration video, current scene, optional language | Sequence of subtasks \(\mathcal{P}\) |
| FDP | Current RGB observation, subtask guide | Future rollout \(V_n^{future}\) |
| Action Executor | 3D waypoint trajectory, grasp candidates | Executable robot motion |

Operationally, the paper reconstructs the loop as:

1. Observe current scene \(O_n\).
2. Use HIT to produce or update subtask plan \(\mathcal{P}\).
3. Select current subtask \(\tau_n\).
4. Run FDP to generate imagined subtask video \(V_n^{future}\).
5. Track target object and estimate depth to recover \(k_n = \{p_m^{3D}\}\).
6. Plan grasp and optimize trajectory.
7. Execute.
8. Verify success with a VLM.
9. If success, increment \(n\); else replan from updated observation.

The system is largely a **composition of pretrained modules** rather than a jointly trained end-to-end model. HIT uses **Qwen2.5-VL 72B** and **MediaPipe**; FDP uses **Wan2.2-Lightning**, **Grounded SAM 2**, and **Video Depth Anything**; the executor uses **AnyGrasp** plus optimization-based path refinement. The paper states that these components are **trained separately by their original creators, not jointly within this paper**, and frames Super-Mimic itself as an inference-time orchestration framework.

## 3. Human Intent Translator

HIT is designed to infer procedural intent from unscripted human video. The input video is
\[
V = \{f_1, \dots, f_T\},
\]
and the first stage extracts a sparse keyframe sequence \(V_{key}\). The key idea is to use **MediaPipe hand landmark detection** to track the 2D wrist position \(p^{wrist}\) and to treat local wrist stationarity as a proxy for meaningful interaction events such as placing, grasping, or pausing during contact. A frame \(f_t\) is marked as a candidate keyframe if
\[
\frac{1}{|W_t|} \sum_{i \in W_t} ||p_i^{wrist} - p_{i-1}^{wrist}||_2 < \epsilon,
\]
where \(W_t = \{t-\Delta t, \dots, t+\Delta t\}\). Candidate frames are then temporally filtered to enforce a minimum interval between selected keyframes [2509.22205].

The paper treats this as an **intent-distillation step**, not merely temporal compression. After keyframe extraction, a VLM interprets the keyframe sequence and abstracts it into a baseline symbolic plan \(\mathcal{P}_{base}\). A second stage, **unified subtask planning**, takes \(\mathcal{P}_{base}\), the robot’s current RGB observation \(O^{rgb}\), and optionally a language command \(L\), and produces the final subtask sequence. The optional language condition can modify the demonstrated task; the paper gives the example “throw everything into the trash bin,” which changes how the demonstration is instantiated for the robot.

This design is central to the claim that Super-Mimic goes beyond direct replay. It supports both **task modification** and **skill transfer**, because the final plan is conditioned jointly on the demonstration abstraction and the robot’s current scene. The paper does not provide the exact prompts used with the VLM, nor an explicit reasoning template, so those details remain unspecified.

## 4. Future Dynamics Predictor and action execution

FDP takes the current RGB observation \(O_n^{rgb}\) and the textual guide \(\text{guide}_n\) for the current subtask and generates a short future rollout
\[
V_n^{future} = G(O_n^{rgb}, \text{guide}_n) = \{f'_1, f'_2, \dots, f'_Q\}.
\]
In experiments, \(G\) is instantiated with **Wan2.2-Lightning**. The rollout is not a single endpoint image but a temporally coherent imagined execution trace, intended to encode object interactions, contact structure, and intermediate motion [2509.22205].

To convert the generated video into actionable geometry, the system applies a three-stage extraction process. First, **Grounded SAM 2** tracks the target object and produces a dense 2D pixel trajectory. Second, the **Ramer-Douglas-Peucker (RDP)** algorithm sparsifies this trajectory into 2D waypoints. Third, **Video Depth Anything** estimates temporally consistent depth maps, and the 2D waypoints are lifted to 3D using camera intrinsics. The resulting subtask trajectory is
\[
k_n = \{p_1^{3D}, \dots, p_M^{3D}\}.
\]

The Action Executor then performs grasp planning and path refinement. The system uses **AnyGrasp** to produce grasp candidates and a multimodal language model to choose among them. For path optimization, it minimizes a weighted combination of smoothness and collision penalties. The paper gives
\[
C_{\text{smooth}} = \sum (1 - \cos \theta_m),
\]
where \(\theta_m\) is the angle between consecutive path segments, and
\[
C_{\text{coll}} = \sum \left(\min_j \|p_m^{3D} - \mathbf{o}_j\|_2 + \varphi\right)^{-1},
\]
where \(\mathbf{o}_j\) are obstacle points queried from a KD-tree and \(\varphi\) is a small offset term. The intended total objective is described as
\[
\min_{\{p_m^{3D}\}} \ \lambda_{\text{smooth}} C_{\text{smooth}} + \lambda_{\text{coll}} C_{\text{coll}},
\]
and is solved with a gradient-based local solver.

A key distinction from some earlier video-planning approaches is that Super-Mimic does **not** train a separate trajectory translator from generated video to robot actions. Instead, it directly lifts the generated visual plan into 3D using tracking and depth estimation. This is presented as one reason the framework remains “fully zero-shot.”

## 5. Experimental evaluation and empirical behavior

Experiments are conducted in the real world on a platform consisting of a **7-DoF xArm7** robot arm and an externally mounted **Orbbec Femto Bolt** RGB-D camera. The paper evaluates three long-horizon scenarios: **Meal Preparation** with 5 subtasks, **Tidy Up the Desk** with 5 subtasks, and **Irregular Traversal** with 8 subtasks. Performance is reported under four instruction conditions: **Short Simple language**, **Video**, **Video + Constraints**, and **Skill Generalization**. The main baselines are **ReKep** and **MOKA**, both text-driven zero-shot manipulation systems based on static scene understanding [2509.22205].

The evaluation metrics are **Task Success Rate (TSR)** and **Subtask Success Rate (SSR)** over \(N=20\) trials:
\[
\mathrm{TSR} = \frac{1}{N} \sum_{i=1}^{N} S_i, \qquad
\mathrm{SSR} = \frac{1}{N} \sum_{i=1}^{N} \left( \frac{n_i}{M} \right),
\]
where \(S_i = 1\) if all subtasks in trial \(i\) succeed, \(0\) otherwise, \(n_i\) is the number of successfully completed subtasks in trial \(i\), and \(M\) is the total number of subtasks.

The headline result is that Super-Mimic outperforms the strongest zero-shot baselines by **more than 20%** on complex long-horizon tasks. On **Meal Preparation**, the strongest baseline is MOKA with detailed language at **20% TSR**, while Super-Mimic with video reaches **50% TSR**. On **Tidy Up**, the strongest baseline reaches **20% TSR**, while Super-Mimic with video reaches **40% TSR**. On **Irregular Traversal**, both baselines obtain **0% TSR** even with detailed decomposition, while Super-Mimic reaches **20% TSR**.

The ablation study isolates the contributions of **watching** and **imagining**. The full model obtains:
- Meal Preparation: **50 / 82**
- Tidy Up: **40 / 78**
- Irregular Traversal: **20 / 58**

Removing FDP and replacing it with a static planner drops performance to:
- Meal Preparation: **20 / 58**
- Tidy Up: **20 / 62**
- Irregular Traversal: **0 / 24**

Keeping only the final destination point from the imagined video, rather than the full path, yields:
- Meal Preparation: **30 / 72**
- Tidy Up: **30 / 74**
- Irregular Traversal: **20 / 56**

Removing HIT and using only short language yields:
- Meal Preparation: **30 / 76**
- Tidy Up: **30 / 74**
- Irregular Traversal: **0 / 32**

Feeding the raw full video directly to the VLM planner instead of using key-action parsing yields:
- Meal Preparation: **40 / 70**
- Tidy Up: **30 / 70**
- Irregular Traversal: **10 / 44**

These results support the paper’s two-part explanation of why the system works: HIT contributes **semantic decomposition and transfer**, while FDP contributes **dynamic execution guidance**. The full imagined path matters more than a static destination, and the parsed demonstration matters more than raw video ingestion.

## 6. Failure modes, limitations, and relation to neighboring “Mimic” systems

The paper identifies three classes of failure. First, **HIT planning failure**, where the VLM misinterprets intent and produces the wrong symbolic plan, such as placing an apple into a cup. Second, **FDP prediction failure**, where the video model hallucinates physically implausible outcomes, such as deformed objects, which then yields unusable trajectories. Third, **execution failure**, where grasping or manipulation fails despite correct planning and prediction; the paper notes that this is the most common category [2509.22205].

The stated limitations are correspondingly modular. Super-Mimic depends on the quality of general-purpose video generation, on the reliability of tracking, depth estimation, and grasping, and on a relatively linear sequential replanning loop. The framework does not provide learned uncertainty estimates, formal guarantees of contact correctness, or a jointly optimized probabilistic treatment of planning and control. A plausible implication is that the method’s strength lies in cross-module orchestration rather than in a unified theory of uncertainty or dynamics.

The term **Super-Mimic** should also be distinguished from several contemporaneous systems with partially overlapping names. **MIMIC** denotes a framework for multimodal inversion and interpretation of autoregressive vision-language models rather than a robotics method [2508.07833]. **MimIC** denotes a method for approximating multimodal in-context demonstration effects through internal shift vectors in large multimodal models [2504.08851]. **VLMimic** uses vision-language models for fine-grained visual imitation learning from a small number of human videos [2410.20927]. **ZeroMimic** distills robot manipulation skills from human web videos without robot demonstrations for those skills [2503.23877]. This suggests a broader naming cluster around imitation, multimodality, and cross-domain transfer, but Super-Mimic itself refers specifically to the “From Watch to Imagine” framework for zero-shot long-horizon manipulation from unscripted human demonstration videos.

Source: https://www.emergentmind.com/topics/super-mimic