MimicFunc: One-Shot Tool Manipulation

• MimicFunc is a framework for tool manipulation imitation that leverages a function frame—defined by three keypoints—to encode how a tool is used rather than how it appears.
• It employs a coarse-to-fine transfer combining VLM proposals and dense semantic correspondence to manage large intra-function variations in novel tools.
• Optimized with CasADi/IPOPT and data-efficient human demonstrations, it significantly enhances spatial, instance, and category generalization in robotic tasks.

to=arxiv_search.query 大发快三是国家json code {"query":"MimicFunc imitating tool manipulation from a single human video via functional correspondence arXiv", "max_results": 5, "sort_by": "relevance"} to=arxiv_search.query 天天好彩票json code {"query":"third-person imitation tool manipulation single human video ORION DINOBOT DITTO arXiv", "max_results": 10, "sort_by": "relevance"} MimicFunc is a framework for imitating tool manipulation from a single RGB-D human video and transferring that skill to novel tools that are functionally equivalent but may exhibit large intra-function variations in geometry, appearance, and grasp/use configuration. Its central representation is the function frame, a function-centric local coordinate frame defined from three functional keypoints—the function point, grasp point, and center point—so that manipulation is encoded in terms of how a tool is used relative to a target rather than how it looks. In the reported experiments, this representation supports one-shot generalization to novel tools for functionally equivalent tasks and also serves as a structured source of robot rollouts for downstream visuomotor policy learning (Tang et al., 19 Aug 2025).

1. Problem formulation and motivation

MimicFunc addresses one-shot imitation of tool manipulation from a single human demonstration video. The demonstration consists of a natural language description such as $$l_H = \text{"use the mug to pour contents into the bowl"}$$ and a video sequence $V_H = \{ I_t \}_{t=0}^{N-1}$, where each $I_t$ is an RGB-D frame. At test time, the robot observes a different tool and target arrangement and must generate a trajectory $T_R = \{ a_t \}_{t=0}^{N-1}$, with $a_t = (R_t, T_t) \in SE(3)$, where $R_t \in SO(3)$ and $T_t \in \mathbb{R}^3$ denote the end-effector pose (Tang et al., 19 Aug 2025).

The central difficulty is intra-function variation: tools that realize the same function may differ substantially in shape, topology, grasp affordances, and functional geometry. The reported examples include mugs, teapots, bottles, ladles, knives, cleavers, axes, spoons, shovels, and brushes. Standard imitation and policy-learning systems typically rely on visual similarity, geometric similarity, dense feature matching, or point-cloud registration, and therefore degrade when the demonstration tool and the test tool are not category-level neighbors. MimicFunc is explicitly designed for this regime by establishing function-level correspondences rather than raw geometric correspondences (Tang et al., 19 Aug 2025).

A second motivation is data efficiency for visuomotor learning. Typical behavioral cloning pipelines rely on repeated teleoperation for each new object configuration. MimicFunc instead uses a single human video per function, generalizes to novel tools and layouts, and produces robot rollouts that can be filtered by a Vision-LLM and reused as training data. The paper states that data collection is much faster in this regime, reporting approximately 5.1 s to record a human video versus 48 s per teleoperation demonstration (Tang et al., 19 Aug 2025).

2. Functional keypoints and the function frame

The core abstraction is a minimal functional skeleton defined by three 3D keypoints. For the human demonstration at time $t$, these are

$$K_t^H = [p_{\text{func}, t},\, p_{\text{grasp}, t},\, p_{\text{center}, t}] \in \mathbb{R}^{3 \times 3},$$

where $p_{\text{func}, t}$ is the function point, $V_H = \{ I_t \}_{t=0}^{N-1}$0 is the grasp point, and $V_H = \{ I_t \}_{t=0}^{N-1}$1 is the center point. For the robot test tool, the corresponding keypoints are

$V_H = \{ I_t \}_{t=0}^{N-1}$2

The function point denotes where the tool interacts with the target; the grasp point denotes where the hand contacts the tool; and the center point is the 3D bounding-box center, used as an object-agnostic anchor (Tang et al., 19 Aug 2025).

From these keypoints, MimicFunc constructs a function frame. For the demonstration, the function axis is

$V_H = \{ I_t \}_{t=0}^{N-1}$3

the grasp vector is

$V_H = \{ I_t \}_{t=0}^{N-1}$4

and the normal of the function plane is

$V_H = \{ I_t \}_{t=0}^{N-1}$5

The function plane is

$V_H = \{ I_t \}_{t=0}^{N-1}$6

Using $V_H = \{ I_t \}_{t=0}^{N-1}$7, the paper forms an orthonormal basis

$V_H = \{ I_t \}_{t=0}^{N-1}$8

with translation

$V_H = \{ I_t \}_{t=0}^{N-1}$9

so that each function frame is a rigid transform $I_t$0. The same construction is applied to the robot tool to obtain $I_t$1 (Tang et al., 19 Aug 2025).

This representation is intentionally function-centric rather than geometry-centric. The paper’s examples define the function point as the spout tip or lip of a container for pouring, the scoop bowl for scooping, the blade edge for cutting, the bristle region for brushing, and the hammer head bottom for pounding. A plausible implication is that the function frame is designed to preserve the operational invariants of tool use—interaction locus, actuation direction, and operational plane—while discarding much of the tool-specific geometry that destabilizes category-level transfer (Tang et al., 19 Aug 2025).

3. Perception pipeline and functional correspondence

MimicFunc first decomposes the human demonstration into a function plan with three keyframes: the initial keyframe $I_t$2, the grasp keyframe $I_t$3 at $I_t$4, and the function keyframe $I_t$5 at $I_t$6, with $I_t$7. These keyframes are discovered with VideoCLIP using phase-specific textual descriptions, such as before grasping, grasping, and function execution (Tang et al., 19 Aug 2025).

Functional keypoints are then extracted from the human video. The grasp point is computed at $I_t$8 by reconstructing a 3D hand mesh with HaMeR, identifying the intersection between the fingertip region and the tool mask, and taking the center of that contact region. The center point is obtained from the 3D bounding box of the segmented tool in $I_t$9. The function point is obtained at $T_R = \{ a_t \}_{t=0}^{N-1}$0 through mark-based visual prompting with a Vision-LLM, which is asked to mark the tool location responsible for the function; the marked 2D point is then lifted into 3D with depth. Afterward, CoTracker tracks the motion of these points through time, yielding a keypoint trajectory $T_R = \{ a_t \}_{t=0}^{N-1}$1 and a corresponding function-frame trajectory $T_R = \{ a_t \}_{t=0}^{N-1}$2 (Tang et al., 19 Aug 2025).

All 3D quantities are represented in the target frame rather than the camera frame. The target frame is estimated online from the segmented target point cloud using PCA-based principal axes and the target bounding-box center. This gives MimicFunc an object-centric reference system in which tool–target relations can be represented independently of the camera or robot base frame (Tang et al., 19 Aug 2025).

To establish correspondence on the test tool, MimicFunc uses a coarse-to-fine transfer procedure. For each semantic keypoint type $T_R = \{ a_t \}_{t=0}^{N-1}$3, the demonstration keypoint is projected into the demonstration image, a VLM proposes a coarse region $T_R = \{ a_t \}_{t=0}^{N-1}$4 on the test tool, and a dense semantic correspondence model $T_R = \{ a_t \}_{t=0}^{N-1}$5 refines the transfer to a specific 2D point: $T_R = \{ a_t \}_{t=0}^{N-1}$6 That point is back-projected into 3D, and the test-tool center point is computed from the test-tool bounding box, yielding

$T_R = \{ a_t \}_{t=0}^{N-1}$7

The paper characterizes this as a fusion of VLM commonsense for coarse functional region discovery and dense semantic correspondence for pixel-level transfer (Tang et al., 19 Aug 2025).

4. Alignment primitives and action generation

Once the demonstration and test-tool function frames are available, MimicFunc aligns them using three interaction primitives. The point primitive aligns the function point, the axis primitive aligns the function axis, and the plane primitive aligns the normal of the function plane. In effect, the desired test-time function keyframe is constrained so that the location of interaction, the operational direction, and the operational plane match the demonstration in the target frame (Tang et al., 19 Aug 2025).

The paper notes that geometric alignment can still be functionally invalid because of perception noise or structural differences between tools. MimicFunc therefore uses a VLM-based semantic evaluator. It renders the predicted arrangement of the test tool and target, asks whether the state is functionally valid for the specified task, and, if not, asks the VLM to attribute the failure to one or more primitives. New candidate points or axes are then resampled around the initial solution until a semantically valid alignment is found. This suggests a hybrid inference loop in which geometric constraints provide an initial alignment and language-conditioned semantic judgment filters or repairs it (Tang et al., 19 Aug 2025).

After alignment, MimicFunc generates a full function-frame trajectory for the robot tool by solving a constrained optimization problem: $T_R = \{ a_t \}_{t=0}^{N-1}$8 subject to

$T_R = \{ a_t \}_{t=0}^{N-1}$9

Here the translational term matches function-point trajectories and the Lie-log term matches frame orientations. The paper also introduces trajectory warping by symmetry rotation, alignment rotation, optional scaling, and translation: $a_t = (R_t, T_t) \in SE(3)$0

$a_t = (R_t, T_t) \in SE(3)$1

$a_t = (R_t, T_t) \in SE(3)$2

The optimization is implemented with CasADi and solved with IPOPT, with additional velocity and collision-avoidance constraints (Tang et al., 19 Aug 2025).

For execution, the optimized function-frame trajectory is transformed into the robot base frame. MimicFunc samples a grasp pose around $a_t = (R_t, T_t) \in SE(3)$3, executes the grasp, and then retargets the tool-centric function-frame motion to end-effector poses that keep the functional part at the desired position and orientation. This makes the overall procedure a specialized form of object-centric motion retargeting in which the preserved entity is the tool’s mode of use rather than the demonstrator’s limb kinematics (Tang et al., 19 Aug 2025).

5. Empirical evaluation and downstream policy learning

The reported experiments cover five functions—Pour, Cut, Scoop, Brush, and Pound—with five tasks per function spanning spatial, instance, and category generalization. Baselines are DINOBOT, DITTO, and ORION, with the same low-level execution pipeline but different correspondence modules. Evaluation uses average success rate over 25 tasks and 10 trials per task (Tang et al., 19 Aug 2025).

All methods achieve >70% success on spatial generalization, where the tool is unchanged and only the layout varies. The gap widens under instance and category generalization, where novel tools are introduced. The paper reports that DINOBOT is the strongest baseline under novel tools with approximately 57.5% average success, whereas MimicFunc reaches 79.5% average success across five functions for novel tool generalization. It also reports that geometry-only registration, as used in ORION, fails badly under large intra-function variations, while DINOBOT and DITTO still degrade by roughly 20–40% relative to the spatial-generalization setting (Tang et al., 19 Aug 2025).

Long-horizon tasks are modeled as chains of function plans. In the reported experiments, MimicFunc achieves 80.0% step completion and 76.0% task success. The paper attributes the dominant long-horizon failures to robot reach limitations in large layouts, and notes that function-frame trajectories are compatible with Task and Motion Planning systems such as PDDLStream (Tang et al., 19 Aug 2025).

Ablation results concentrate on keypoint transfer. The strongest variant uses Demo+VLM+DSC, meaning demonstration-anchored prompting, VLM region proposal, and dense semantic correspondence refinement. It achieves AKD 18.54 pixels and the best AP@15/30/45. Two weaker variants—Demo+DSC and VLM alone—perform worse, indicating that neither dense correspondence without coarse functional grounding nor VLM keypoint prediction without demonstration-anchored refinement is sufficient under large intra-function variation (Tang et al., 19 Aug 2025).

The framework is also used as a data generator for visuomotor policy learning. MimicFunc-generated rollouts are executed on novel objects, filtered by a VLM for success, and used to train or augment policies such as ACT and Diffusion Policy. On the Pour task, the paper reports that ACT+DA, which adds 30 MimicFunc-generated rollouts per novel object to a baseline of 50 teleoperation demonstrations, improves success by 41.6% at the instance level and 50.0% at the category level. The paper states that MimicFunc-generated data is more consistent and precise than teleoperation in this setting (Tang et al., 19 Aug 2025).

6. Context, limitations, and outlook

MimicFunc sits at the intersection of third-person imitation, affordance-centered manipulation, and motion retargeting. The paper relates it to correspondence-based imitation systems such as DINOBOT and DITTO, point-cloud registration systems such as ORION, keypoint-affordance methods including KPAM, K-VIL, Bi-KVIL, and mark-based prompting systems such as ReKep. Its distinguishing move is to define correspondence in a function-centric local frame rather than a geometry-centric or appearance-centric frame, and to couple that representation with VLM-guided semantic evaluation (Tang et al., 19 Aug 2025).

Several limitations are explicitly stated. The method currently requires RGB-D input, not pure RGB Internet video. It supports single-handed manipulation only, which excludes bimanual or multi-fingered tasks. Contact-rich tasks such as Brush and Pound remain challenging because the system lacks tactile feedback and explicit force control. Failures also arise from gripper limitations, depth accuracy, and segmentation. The reported failure breakdown attributes approximately 32.7% of failures to grasping, 26.5% to trajectory generation, 16.3% to functional keypoint transfer, 4.1% to function-frame alignment, and 20.4% to other sources such as segmentation and detection (Tang et al., 19 Aug 2025).

The paper proposes several extensions: leveraging monocular depth estimation to support RGB-only Internet videos, scaling data generation in simulation, and extending the framework to bimanual and multi-fingered manipulation. A plausible implication is that the function frame could serve as a general-purpose interface between human demonstration, robot execution, and downstream policy learning whenever the task is governed more by the structure of tool use than by category-level shape similarity (Tang et al., 19 Aug 2025).