Super-Mimic: Zero-Shot Robotic Imitation

• Super-Mimic is a hierarchical framework that enables zero-shot, long-horizon robotic manipulation by watching unscripted human demonstration videos to infer procedural intent.
• It couples a Human Intent Translator with a Future Dynamics Predictor and Action Executor to decompose tasks and generate precise 3D motion plans.
• Evaluated on complex tasks like meal preparation and desk tidying, Super-Mimic outperforms baselines by dynamically coordinating planning and execution.

Super-Mimic is a hierarchical framework for zero-shot, long-horizon robotic imitation that uses an unscripted human demonstration video as its primary instruction channel and optionally accepts an additional natural-language command. Its defining premise is that, for complex manipulation, a robot should first watch a human demonstration to infer the task’s procedural intent and then imagine how each inferred subtask should unfold in the robot’s current scene before acting. The framework therefore couples a Human Intent Translator (HIT) with a Future Dynamics Predictor (FDP) and an Action Executor, and is evaluated on real-world long-horizon manipulation tasks where task decomposition, contact structure, and intermediate motion all matter (Ye et al., 26 Sep 2025).

1. Problem setting and formal representation

Super-Mimic targets open-world, long-horizon robotic manipulation without task-specific training. The robot receives an unscripted human demonstration video $V$ and optionally a language instruction $L$, and must execute a multi-step task in a novel environment. The paper emphasizes two difficulties in this setting: planning difficulty, because a short instruction such as “tidy up the desk” does not specify the sequence of subtasks, object correspondences, or placements; and execution difficulty, because static scene understanding does not determine physically plausible object interactions, contact points, or intermediate motion (Ye et al., 26 Sep 2025).

The task is formalized as $\mathcal{T}$. From $V$ and optionally $L$, the system must infer a sequence of subtasks

$\mathcal{P} = \{\tau_1, \dots, \tau_N\}.$

For each subtask $\tau_n$, given the robot’s current RGB-D observation $O_n$, the system generates a sub-plan $k_n$, and the resulting sequence

$K = \{k_1, \dots, k_N\}$

provides spatio-temporal guidance to the executor. Each subtask is represented as

$L$0

where $L$1 is a natural-language subtask description, $L$2 is the semantic identifier of the target object, $L$3 is the destination or relevant location, $L$4 is the textual prompt used to guide future video generation in FDP, and $L$5 is a set of preconditions required before execution.

This formulation makes the framework explicitly hierarchical. HIT addresses the question of what should be done next, while FDP addresses what the world should look like as this step unfolds. The executor then converts the imagined evolution into grasp and motion commands. A plausible implication is that the architecture is intended to separate symbolic procedural inference from dynamics-aware control, rather than forcing a single model to absorb both burdens.

2. System architecture and operational loop

The full pipeline is sequential and iterative. Starting from a human demonstration video, Super-Mimic first applies HIT to convert raw video into a robot-relevant symbolic plan. For the current subtask, FDP then synthesizes an imagined future video from the robot’s present view. A 3D waypoint trajectory is extracted from that generated video, and the Action Executor refines and executes the motion. Success is then verified; if the current subtask fails, the system replans from the updated observation (Ye et al., 26 Sep 2025).

Module	Input	Output
HIT	Human demonstration video, current scene, optional language	Sequence of subtasks $L$6
FDP	Current RGB observation, subtask guide	Future rollout $L$7
Action Executor	3D waypoint trajectory, grasp candidates	Executable robot motion

Operationally, the paper reconstructs the loop as:

1. Observe current scene $L$8.
2. Use HIT to produce or update subtask plan $L$9.
3. Select current subtask $\mathcal{T}$0.
4. Run FDP to generate imagined subtask video $\mathcal{T}$1.
5. Track target object and estimate depth to recover $\mathcal{T}$2.
6. Plan grasp and optimize trajectory.
7. Execute.
8. Verify success with a VLM.
9. If success, increment $\mathcal{T}$3; else replan from updated observation.

The system is largely a composition of pretrained modules rather than a jointly trained end-to-end model. HIT uses Qwen2.5-VL 72B and MediaPipe; FDP uses Wan2.2-Lightning, Grounded SAM 2, and Video Depth Anything; the executor uses AnyGrasp plus optimization-based path refinement. The paper states that these components are trained separately by their original creators, not jointly within this paper, and frames Super-Mimic itself as an inference-time orchestration framework.

3. Human Intent Translator

HIT is designed to infer procedural intent from unscripted human video. The input video is

$\mathcal{T}$4

and the first stage extracts a sparse keyframe sequence $\mathcal{T}$5. The key idea is to use MediaPipe hand landmark detection to track the 2D wrist position $\mathcal{T}$6 and to treat local wrist stationarity as a proxy for meaningful interaction events such as placing, grasping, or pausing during contact. A frame $\mathcal{T}$7 is marked as a candidate keyframe if

$\mathcal{T}$8

where $\mathcal{T}$9. Candidate frames are then temporally filtered to enforce a minimum interval between selected keyframes (Ye et al., 26 Sep 2025).

The paper treats this as an intent-distillation step, not merely temporal compression. After keyframe extraction, a VLM interprets the keyframe sequence and abstracts it into a baseline symbolic plan $V$0. A second stage, unified subtask planning, takes $V$1, the robot’s current RGB observation $V$2, and optionally a language command $V$3, and produces the final subtask sequence. The optional language condition can modify the demonstrated task; the paper gives the example “throw everything into the trash bin,” which changes how the demonstration is instantiated for the robot.

This design is central to the claim that Super-Mimic goes beyond direct replay. It supports both task modification and skill transfer, because the final plan is conditioned jointly on the demonstration abstraction and the robot’s current scene. The paper does not provide the exact prompts used with the VLM, nor an explicit reasoning template, so those details remain unspecified.

4. Future Dynamics Predictor and action execution

FDP takes the current RGB observation $V$4 and the textual guide $V$5 for the current subtask and generates a short future rollout

$V$6

In experiments, $V$7 is instantiated with Wan2.2-Lightning. The rollout is not a single endpoint image but a temporally coherent imagined execution trace, intended to encode object interactions, contact structure, and intermediate motion (Ye et al., 26 Sep 2025).

To convert the generated video into actionable geometry, the system applies a three-stage extraction process. First, Grounded SAM 2 tracks the target object and produces a dense 2D pixel trajectory. Second, the Ramer-Douglas-Peucker (RDP) algorithm sparsifies this trajectory into 2D waypoints. Third, Video Depth Anything estimates temporally consistent depth maps, and the 2D waypoints are lifted to 3D using camera intrinsics. The resulting subtask trajectory is

$V$8

The Action Executor then performs grasp planning and path refinement. The system uses AnyGrasp to produce grasp candidates and a multimodal LLM to choose among them. For path optimization, it minimizes a weighted combination of smoothness and collision penalties. The paper gives

$V$9

where $L$0 is the angle between consecutive path segments, and

$L$1

where $L$2 are obstacle points queried from a KD-tree and $L$3 is a small offset term. The intended total objective is described as

$L$4

and is solved with a gradient-based local solver.

A key distinction from some earlier video-planning approaches is that Super-Mimic does not train a separate trajectory translator from generated video to robot actions. Instead, it directly lifts the generated visual plan into 3D using tracking and depth estimation. This is presented as one reason the framework remains “fully zero-shot.”

5. Experimental evaluation and empirical behavior

Experiments are conducted in the real world on a platform consisting of a 7-DoF xArm7 robot arm and an externally mounted Orbbec Femto Bolt RGB-D camera. The paper evaluates three long-horizon scenarios: Meal Preparation with 5 subtasks, Tidy Up the Desk with 5 subtasks, and Irregular Traversal with 8 subtasks. Performance is reported under four instruction conditions: Short Simple language, Video, Video + Constraints, and Skill Generalization. The main baselines are ReKep and MOKA, both text-driven zero-shot manipulation systems based on static scene understanding (Ye et al., 26 Sep 2025).

The evaluation metrics are Task Success Rate (TSR) and Subtask Success Rate (SSR) over $L$5 trials: $L$6 where $L$7 if all subtasks in trial $L$8 succeed, $L$9 otherwise, $\mathcal{P} = \{\tau_1, \dots, \tau_N\}.$0 is the number of successfully completed subtasks in trial $\mathcal{P} = \{\tau_1, \dots, \tau_N\}.$1, and $\mathcal{P} = \{\tau_1, \dots, \tau_N\}.$2 is the total number of subtasks.

The headline result is that Super-Mimic outperforms the strongest zero-shot baselines by more than 20% on complex long-horizon tasks. On Meal Preparation, the strongest baseline is MOKA with detailed language at 20% TSR, while Super-Mimic with video reaches 50% TSR. On Tidy Up, the strongest baseline reaches 20% TSR, while Super-Mimic with video reaches 40% TSR. On Irregular Traversal, both baselines obtain 0% TSR even with detailed decomposition, while Super-Mimic reaches 20% TSR.

The ablation study isolates the contributions of watching and imagining. The full model obtains:

• Meal Preparation: 50 / 82
• Tidy Up: 40 / 78
• Irregular Traversal: 20 / 58

Removing FDP and replacing it with a static planner drops performance to:

• Meal Preparation: 20 / 58
• Tidy Up: 20 / 62
• Irregular Traversal: 0 / 24

Keeping only the final destination point from the imagined video, rather than the full path, yields:

• Meal Preparation: 30 / 72
• Tidy Up: 30 / 74
• Irregular Traversal: 20 / 56

Removing HIT and using only short language yields:

• Meal Preparation: 30 / 76
• Tidy Up: 30 / 74
• Irregular Traversal: 0 / 32

Feeding the raw full video directly to the VLM planner instead of using key-action parsing yields:

• Meal Preparation: 40 / 70
• Tidy Up: 30 / 70
• Irregular Traversal: 10 / 44

These results support the paper’s two-part explanation of why the system works: HIT contributes semantic decomposition and transfer, while FDP contributes dynamic execution guidance. The full imagined path matters more than a static destination, and the parsed demonstration matters more than raw video ingestion.

6. Failure modes, limitations, and relation to neighboring “Mimic” systems

The paper identifies three classes of failure. First, HIT planning failure, where the VLM misinterprets intent and produces the wrong symbolic plan, such as placing an apple into a cup. Second, FDP prediction failure, where the video model hallucinates physically implausible outcomes, such as deformed objects, which then yields unusable trajectories. Third, execution failure, where grasping or manipulation fails despite correct planning and prediction; the paper notes that this is the most common category (Ye et al., 26 Sep 2025).

The stated limitations are correspondingly modular. Super-Mimic depends on the quality of general-purpose video generation, on the reliability of tracking, depth estimation, and grasping, and on a relatively linear sequential replanning loop. The framework does not provide learned uncertainty estimates, formal guarantees of contact correctness, or a jointly optimized probabilistic treatment of planning and control. A plausible implication is that the method’s strength lies in cross-module orchestration rather than in a unified theory of uncertainty or dynamics.

The term Super-Mimic should also be distinguished from several contemporaneous systems with partially overlapping names. MIMIC denotes a framework for multimodal inversion and interpretation of autoregressive vision-LLMs rather than a robotics method (Jain et al., 11 Aug 2025). MimIC denotes a method for approximating multimodal in-context demonstration effects through internal shift vectors in large multimodal models (Jiang et al., 11 Apr 2025). VLMimic uses vision-LLMs for fine-grained visual imitation learning from a small number of human videos (Chen et al., 2024). ZeroMimic distills robot manipulation skills from human web videos without robot demonstrations for those skills (Shi et al., 31 Mar 2025). This suggests a broader naming cluster around imitation, multimodality, and cross-domain transfer, but Super-Mimic itself refers specifically to the “From Watch to Imagine” framework for zero-shot long-horizon manipulation from unscripted human demonstration videos.