---
title: 'FMimic: Fine-grained Action Learning Framework'
url: https://www.emergentmind.com/topics/fmimic
type: topic
---

# FMimic: Fine-grained Action Learning Framework

Searching arXiv for relevant papers on “FMimic” and closely related uses of the term.
arXiv search query: "FMimic OR ForceMimic"
FMimic is a visual imitation learning framework that uses foundation models to learn fine-grained manipulation skills directly from a small number of human demonstration videos and to execute those skills on a robot in new scenes, tasks, and object instances. In its principal formulation, FMimic is a full-stack pipeline rather than a single policy: it parses RGB-D human video into subtasks, grounds human-object motion, distills demonstrations into object-attached keypoints and object-relative waypoints, infers hierarchical semantic and geometric constraints with VLMs, adapts those constraints to novel objects through 6D keypoint transfer and iterative comparison, and, for high-precision tasks, refines execution with contact-aided optimization [2507.20622].

## 1. Scope, formulation, and conceptual position

FMimic addresses a specific limitation in recent foundation-model-driven robotics. Prior systems often use VLMs or LLMs to infer high-level plans from demonstrations and then rely on pre-defined motion primitives or separately trained low-level policies for execution. FMimic instead treats foundation models as fine-grained action learners: the low-level motor structure itself is induced from human videos, without requiring paired robot demonstrations, hand-written motion primitives, or gradient-based end-to-end policy training [2507.20622].

The framework is explicitly organized around four components: **human-object interaction grounding**, a **skill learner**, a **skill adapter**, and a **skill refiner**. A knowledge bank \(B\) stores both high-level plans and low-level skill constraints extracted from demonstrations. The pipeline begins from human RGB-D videos \(\mathcal{V}\), often as few as **1 video** per task, with extensive experiments using **5 videos**. It does not fine-tune the foundation models it uses; they remain frozen and are queried through prompting, feature extraction, and code generation [2507.20622].

A common misconception is to treat FMimic as another VLM-based symbolic planner. The reported method is more specific: it reframes fine-grained control as a problem of **constraint and pattern learning**. Human motion is first reduced to object-centric geometric structure, and VLMs reason over rendered keypoints and trajectories to synthesize executable skill code. This suggests that FMimic’s novelty lies less in direct control prediction than in converting observed manipulation into parameterized, object-relative procedural representations [2507.20622].

## 2. Human-object interaction grounding

FMimic’s first stage converts raw human video into structured interaction data. Task recognition begins by sampling keyframes \(\mathcal{K}\) from \(\mathcal{V}\). A VFM stack—described as using “Recognize Anything”, “Caption Anything”, and “Tokenize Anything”—produces detections \(T_d\), after which a VLM infers a task description \(T_t\) and structured object descriptions \(T_o\):
\[
T_d = \text{VFM}(\mathcal{K}), \quad T_t, T_o = \text{VLM}(T_d, \mathcal{K})
\]
The resulting text can encode object relations such as `"pie: (in plate)"` [2507.20622].

Video parsing is then performed by segmenting hands and task objects per frame with SAM-Track or Grounded-SAM and back-projecting the masks into point clouds \(\mathcal{P}\). FMimic defines an interaction distance
\[
d = \text{dist}(\mathcal{P})
\]
and uses a distance threshold \(\epsilon\) to detect contact start and end times:
\[
t_b = \{t \mid d^{t-1} > \epsilon \wedge d^t < \epsilon \}, \qquad
t_e = \{t \mid d^{t-1} < \epsilon \wedge d^t > \epsilon \}
\]
The video is partitioned into segments \(\{\tau_i\}\), and segments whose hand trajectory length is below \(\gamma\) are discarded. In the reported preprocessing pipeline, **keyframes are extracted every 3 s with a minimum of 5 frames**, \(\epsilon = 2~\text{cm}\), and \(\gamma = 5~\text{cm}\) [2507.20622].

For each segment, a VLM assigns a subtask text \(T_{\tau_i}\), identifies whether it is a grasp or a manipulation segment, and labels the master object \(O_m\) and slave object \(O_s\). Human motion is grounded through hand pose estimation with FrankMocap plus ICP, yielding a hand trajectory \(\xi_H\), which is mapped into a gripper trajectory \(\xi_G\). Object pose trajectories are estimated with FoundationPose,
\[
\xi_O = \{ x_C^{O^t} \},
\]
and handheld-camera motion is normalized with ORB-SLAM3. The grounded interaction for a segment is summarized as
\[
I = \{ \xi_{O_s}, \xi_{O_m} \}
\]
[2507.20622].

This representation is significant because it explicitly separates semantic parsing, contact segmentation, and geometry estimation. Rather than learning directly from pixels to robot actions, FMimic first constructs a structured interaction trace, which becomes the substrate for later abstraction and transfer [2507.20622].

## 3. Keypoints, waypoints, and hierarchical constraints

The skill learner distills grounded trajectories into object-centric keypoints and relative waypoints. For each manipulation segment, FMimic defines a keypoint frame rigidly attached to both master and slave objects. In the grasping phase, the master keypoint is the gripper pose at initial contact projected onto the object surface. In the manipulation phase, the slave keypoint \(c_s\) is the closest point on the slave object cloud to the master object cloud at contact onset:
\[
c_s = \arg\min_{p \in O_s^{t_b}} \|p - O_m^{t_b}\|
\]
The slave keypoint orientation is derived from the pre-contact motion direction and from an orthogonal surface direction; the master keypoint is chosen as the closest point on \(O_m^{t_b}\) to \(c_s\), with aligned orientation. Keypoint frames are stored in object coordinates for viewpoint invariance [2507.20622].

Waypoints are then computed by expressing the slave keypoint trajectory relative to the master keypoint trajectory and simplifying the resulting 6D trajectory with SQUISHE:
\[
\chi = \{ x_M^{\mathcal{F}_s^n} \}_{n=0}^N
\]
Thus each segment is represented as
\[
I = \{\mathcal{F}, \chi\}
\]
where \(\mathcal{F}\) denotes keypoint frames and \(\chi\) the object-relative waypoint sequence [2507.20622].

FMimic’s main abstraction layer is hierarchical constraints. It renders keypoints and waypoints on reconstructed object meshes and uses VLM prompting to infer both semantic constraints \(\Phi_s\) and geometric constraints \(\Phi_g\):
\[
I_V = \text{Render}([I, T_n], O), \qquad
\Phi_s = S_l(I_V), \qquad
\Phi_g = G_l(\Phi_s, I, P_o)
\]
For grasping, the geometric representation is a set of **Task Space Regions** \(\{\mathcal{R}_i\}_{i=1}^{N_C}\), obtained by semantically grouping keypoints and computing bounded regions over their frames. For manipulation, the geometric constraint is synthesized as executable code comprising a parameter estimation function
\[
f_p(P_o) \rightarrow \theta
\]
and a trajectory generation function
\[
f_s(\theta) \rightarrow \{x_M^{\mathcal{F}_s^n}\}
\]
where \(P_o\) includes object properties such as 3D bounding boxes [2507.20622].

The knowledge bank \(B\) stores both the high-level mapping \(T_t \rightarrow T_\tau\) and low-level subtask entries
\[
\{T_{\tau_i}, I=\{\mathcal{F}, \chi\}, O_m, O_s, \Phi_s, \Phi_g\}
\]
This makes FMimic interpretable in a way that end-to-end policies are not: grasp regions are represented as TSRs, while manipulation policies are represented as code conditioned on object geometry [2507.20622].

## 4. Adaptation to new objects and contact-aided refinement

FMimic adapts a demonstrated skill to new objects through a skill adapter that performs zero-shot 6D keypoint transfer and iterative correction. The transfer mechanism builds 3D implicit descriptor fields \(\phi(O)\) using Stable Diffusion and DINO features on multi-view observations and aligns reference and target objects through non-rigid registration in feature space. This produces a region-to-keypoint 6D mapping that warps the keypoint frame from the demonstrated object to a novel instance [2507.20622].

Constraint adaptation does not stop at geometric transfer. FMimic renders candidate motions in the new scene and uses a VLM comparison loop to refine the constraints by comparing the generated trajectory against the original demonstrations. Reported ablations show that in unseen environments, **0 iterations** of iterative comparison yield **0.62** success, **1** yields **0.67**, **2** yields **0.70**, and **3–5** increase performance to **0.75**. A separate fine-grained correction loop improves unseen-environment success from **0.69** with **0** correction iterations to **0.75** with **2**, with marginal gains beyond **2**, which is the reported default [2507.20622].

For high-precision tasks, FMimic adds a skill refiner. The paper describes three elements: **collision-minimal interaction optimization**, **reformulation of pose estimation as an in-hand relative pose estimation problem solved via contact-based particle filtering**, and **an information-gain-maximizing contact strategy selector**. The purpose is to refine both grounded interactions and transferred poses when visual grounding alone is insufficient, as in peg insertion or socket insertion [2507.20622].

This division between adaptation and refinement is methodologically important. Adaptation handles cross-instance geometry and scene changes using descriptor fields and VLM comparison, whereas refinement addresses the residual precision gap through contact reasoning. A plausible implication is that FMimic’s reported high-precision performance depends on this layered decomposition rather than on video imitation alone [2507.20622].

## 5. Experimental results and ablations

FMimic is evaluated in simulation, on a Franka robot in real-world manipulation, in unseen-task transfer, in long-horizon execution, and in high-precision insertion. On **12 RLBench tasks**, the reported overall single-task success rates are **0.65** for R3M-DP, **0.66** for DP, **0.12** for GraphIRL, **0.44** for CaP, **0.49** for Demo2Code, **0.75** for FMimic\(_{1v}\), and **0.87** for FMimic\(_{5v}\). In the RLBench multi-task setting, the reported overall success is **0.13** for R3M-DP, **0.15** for DP, **0.31** for CaP, **0.34** for Demo2Code, and **0.83** for FMimic\(_{5v}\) [2507.20622].

On a **Franka Emika** platform with **three ORBBEC RGB-D cameras** and **14 real-world tasks**, FMimic is evaluated in both seen and unseen environments, with **10 random initializations per task**. The reported overall success rates in the seen environment are **0.49** for R3M-DP, **0.55** for DP, **0.25** for GraphIRL, **0.39** for CaP, **0.43** for Demo2Code, **0.73** for FMimic\(_{1v}\), and **0.84** for FMimic\(_{5v}\). In the unseen environment they are **0.09**, **0.10**, **0.07**, **0.37**, **0.37**, **0.63**, and **0.75**, respectively. The paper summarizes these as **+29% absolute** over the best baseline in seen environments and **+38% absolute** in unseen environments [2507.20622].

For **6 unseen tasks** after training on the 14 seen-environment tasks, the reported success rates are **0.13** for R3M-DP, **0.07** for DP, **0.30** for CaP, **0.35** for Demo2Code, and **0.67** for FMimic\(_{5v}\). For **6 long-horizon tasks** with at least **7 subtasks**, the overall success rates are **0.13** for R3M-DP, **0.07** for DP, **0.02** for GraphIRL, **0.02** for CaP, **0.02** for Demo2Code, and **0.60** for FMimic\(_{5v}\), corresponding to an improvement of **more than 47% absolute** over the best baseline [2507.20622].

In **high-precision peg-insertion** tasks, FMimic is reported both with and without the skill refiner. In seen environments across **7 objects**, FMimic\(_{5v}\) without the refiner achieves **0.51**, while FMimic\(_{5v}\) with the skill refiner reaches **0.82**; the main baselines range from **0.16** to **0.48**. In unseen environments with **6 new shapes**, FMimic\(_{5v}\) without the refiner achieves **0.45**, and FMimic\(_{5v}\) with the refiner reaches **0.73**, while the baselines range from **0.12** to **0.25**. For two practical high-precision tasks, FMimic with the refiner achieves **0.70** on socket insertion and **0.80** on battery placement [2507.20622].

The ablation studies clarify which components matter. Removing the semantic layer yields **0.67** seen-environment success; using only semantic constraints without geometric formalization yields **0.74**; the full hierarchical representation yields **0.84**. For grasp learning, direct value prediction by VLM achieves **0.56**, DBSCAN clustering of grasp poses **0.66**, and **VLM grouping + TSRs** **0.84**. The number of demonstration videos increases success from **0.73** with **1 video** to **0.81** with **3**, **0.84** with **5**, and saturation around **0.85–0.86** with **7–11** videos. Robustness experiments report success varying from **0.69–0.75** across four camera configurations, degrading from **0.84** to **0.36** as object visibility drops from full visibility to **20%**, and decreasing only modestly from **0.86** to **0.80** as clutter and object count increase. With TRELLIS for 3D model generation, the reported overall result is **0.76** versus a **0.78** baseline [2507.20622].

## 6. Related systems, misconceptions, and terminological ambiguity

FMimic is explicitly distinguished from two neighboring classes of methods. Relative to VLM- or LLM-based planners such as CaP and Demo2Code, it does not stop at high-level pseudo-code or primitive invocation; it learns low-level action structure from video by synthesizing object-relative constraints and trajectories. Relative to cross-embodiment imitation and IRL methods such as GraphIRL, WHIRL, or XIRL, it does not rely on RL or large-scale robot interaction to realize skills, and the paper emphasizes strong performance with **1–5 demonstrations** as well as support for high-precision tasks through contact refinement [2507.20622].

The framework also has explicit limitations. The paper identifies difficulty in **fine-grained scene understanding**, especially part segmentation such as separating a pan surface from a handle; the burden of **object reconstruction**, particularly for transparent or thin objects; and the fact that VLMs do not reason about **joint limits, torque, IK failures, or robot dynamics**. Reported failure modes include IK infeasibility, collisions from incomplete environment perception, and mis-adapted trajectories when descriptor fields or segmentation are poor. These limitations are consistent with FMimic’s modular design: the system inherits failure modes from perception, reconstruction, planning, and foundation-model reasoning rather than from a single learned policy [2507.20622].

The name itself is not entirely unambiguous across arXiv-adjacent usage. In robotics, the most direct and formal use is the title **“FMimic: Foundation Models are Fine-grained Action Learners from Human Videos”** [2507.20622]. However, **“ForceMimic”** defines **FMimic** only as a shorthand for the full ForceMimic system, and states that there is **no separate component formally named FMimic** [2410.07554]. In a different domain, a waveform-design paper on integrated SAR and communications notes that **“FMimic” could be used informally** for an FM/LFM radar waveform that mimics conventional chirps while embedding communication data in frequency indices [2412.17035]. These uses are terminologically adjacent but conceptually unrelated.

In the strict sense established by the named robotics paper, FMimic denotes a program-synthesis-style visual imitation learning system in which foundation models are used as geometric and semantic reasoners over keypoints, waypoints, and object-relative constraints, with adaptation and contact refinement supplying the bridge from human video to executable robot behavior [2507.20622].

Source: https://www.emergentmind.com/topics/fmimic