---
title: Flow Before Imitation (FBI) for Dexterous Manipulation
url: https://www.emergentmind.com/topics/flow-before-imitation-fbi
type: topic
---

# Flow Before Imitation (FBI) for Dexterous Manipulation

Flow Before Imitation (FBI) is a visuotactile imitation learning framework for dexterous in-hand manipulation that dynamically fuses tactile interactions with visual observations through motion dynamics rather than through static multimodal feature fusion [2508.14441]. In the formulation reported for dexterous manipulation, FBI establishes a causal link between tactile signals and object motion via a dynamics-aware latent model, employs a transformer-based interaction module to fuse flow-derived tactile features with visual inputs, and trains a one-step diffusion policy for real-time execution [2508.14441]. The framework operates in both a **Vision-Only Mode**, which infers tactile cues from visual data using a learned model, and a **Visuotactile Mode**, which directly ingests tactile sensor readings [2508.14441].

## 1. Definition and conceptual basis

Within the dexterous manipulation literature, FBI denotes a dynamic visuotactile imitation learning architecture in which object motion flow is treated as the intermediary that connects contact dynamics to policy conditioning [2508.14441]. The central methodological departure from prior static fusion is that tactile forces are treated as the drivers of object motion, and temporal visual change is used to infer or refine contact information rather than merely concatenating vision and touch features frame by frame [2508.14441].

The framework is explicitly designed for dexterous in-hand manipulation, a regime characterized in the source paper by complex contact dynamics and partial observability [2508.14441]. FBI is therefore not only a multimodal policy, but a multimodal policy organized around motion dynamics. In the reported implementation, flow is used before imitation in the literal sense that motion flow is first estimated or represented, then converted into tactile contact features, and only then fused into the policy input [2508.14441].

A broader reading is also supported by related flow-based robot learning work. Several contemporaneous methods use flow as a precursor representation for action generation, retrieval, or hierarchical control rather than learning a direct state-to-action map. This suggests that “Flow Before Imitation” can also be read as a broader design orientation within robot learning, although the named FBI framework is the visuotactile dexterous-manipulation method introduced in [2508.14441].

## 2. System inputs, encoders, and operating modes

At each timestep, FBI takes robot states, visual observations, and optionally tactile readings as inputs [2508.14441]. The reported variables are
$s_{t-1}, s_t \in \mathbb{R}^{N_s}$ for robot states,
$P_{t-1}, P_t \in \mathbb{R}^{N_p \times 3}$ for two consecutive point clouds,
and $R_t \in \mathbb{R}^{N_r}$ for tactile readings [2508.14441].

The modality-specific encoders are specified as follows [2508.14441]:

| Component | Design | Output |
|---|---|---|
| Robot State Encoder | 3-layer MLP | $\mathcal{F}_s \in \mathbb{R}^{d_s}$ |
| Visual Encoder | 3-layer MLP + max-pooling | $\mathcal{F}_v \in \mathbb{R}^{d_v}$ |
| Tactile Encoder | 4-layer MLP | $\mathcal{F}_{tac} \in \mathbb{R}^{d_{tac}}$ |

The two operating modes differ in how tactile information is obtained [2508.14441]. In **Vision-Only Mode**, tactile cues are inferred from visual motion through the Flow2Tactile module, so the framework does not require physical tactile sensors. In **Visuotactile Mode**, measured tactile sensor readings are used directly, which refines contact state estimation [2508.14441]. This dual-mode construction is central to the framework’s claim that visuotactile reasoning can be retained even when only visual observations are available.

## 3. Flow2Tactile and dynamic visuotactile fusion

The Flow2Tactile module is the mechanism that converts motion information into tactile contact features [2508.14441]. Its motivation is that dense, physically accurate tactile sensing hardware is impractical in many real-world cases, so FBI predicts tactile contact states from motion flow derived from visual data [2508.14441].

The reported workflow has three stages [2508.14441]. First, object state flow is predicted using shortcut one-step flow matching, yielding flow vectors $f_{t-1 \to t}$ that map $P_{t-1}$ to $P_t$. Second, a pre-trained transformer searching module $T_s(\cdot)$ takes hand keypoints $K_t \in \mathbb{R}^{N_k \times 3}$ together with the flow and produces tactile readings:
$$
R_t = T_s(K_t, f_{t-1 \to t}).
$$
Third, contact keypoints are predefined on the hand surface; the example given is 456 keypoints for the Shadow Hand, providing a dense and transferable tactile representation [2508.14441].

The training losses reported for Flow2Tactile are [2508.14441]:
$$
\mathcal{L}_{FM} = CD(P_t, P_{t-1} + f_{t-1\to t}),
$$
$$
\mathcal{L}_{SC} = MSE\big(v_\theta(f^{t}, t, 2dt, P), v_{target}\big),
$$
$$
\mathcal{L} = \mathcal{L}_{FM} + \mathcal{L}_{SC},
$$
and
$$
\mathcal{L}_t = MSE(R_{gt}, T_s(K_t, f_{t-1\to t})).
$$

After encoding, FBI performs dynamic visuotactile fusion with a transformer module $T_f(\cdot)$ [2508.14441]. The visual feature is used as the template and the tactile feature as the searching feature. The resulting visuotactile feature is concatenated back with the visual feature to form the policy input $\mathcal{F}_{fuse}$ [2508.14441]. In the reported interpretation, this is the mechanism by which the model learns dependencies and contextual weighting across touch and vision, rather than relying on naive concatenation or averaging.

A frequent misunderstanding is to treat FBI as simply another multimodal concatenation policy. That characterization is inconsistent with the method description: the distinguishing claim is precisely that fusion is **dynamic** and **causality-minded**, with tactile interactions linked to motion flow and then fused through a transformer [2508.14441].

## 4. One-step shortcut policy and control generation

The policy head in FBI is a shortcut diffusion model trained for one-step inference [2508.14441]. The reported sampling equation is
$$
x^{t+dt} = x^t + v_\theta(x^t, t, dt, C)\,dt,
$$
with one-step mode setting $t=0$ and $dt=1$ [2508.14441]. The policy condition is
$$
C = \{\mathcal{F}_s, \mathcal{F}_{fuse}\},
$$
and the predicted action sequence over horizon $H$ is
$$
\mathbf{A}_t = \mathbf{A}^t + v_\theta(\mathbf{A}^t, t, dt, \mathcal{F}_s, \mathcal{F}_{fuse})\,dt,
$$
where $v_\theta$ is implemented by a U-Net backbone [2508.14441].

The policy training losses are reported as [2508.14441]:
$$
\mathcal{L}_{FM} = MSE\Big(\mathbf{A}^1 - \mathbf{A}^0, v_\theta(\mathbf{A}^t, t, dt, \mathcal{F})\Big),
$$
$$
\mathcal{L}_{SC} = MSE\Big(v_\theta(\mathbf{A}^t, t, 2dt, \mathcal{F}), v_{target}\Big),
$$
$$
v_{target} = \frac{1}{2}\Big[v_\theta(\mathbf{A}^t, t, dt, \mathcal{F}) + v_\theta(\mathbf{A}^{t+dt}, t+dt, dt, \mathcal{F})\Big],
$$
and
$$
\mathcal{L} = \mathcal{L}_{FM} + \mathcal{L}_{SC}.
$$

The practical significance of this design lies in real-time control. FBI is reported to run at 20Hz, with 51.2ms total step time, including 34.4ms model inference for Vision-Only and 17.4ms for Visuotactile [2508.14441]. The comparison given against standard diffusion is a DP3 baseline requiring 137.9ms per step [2508.14441]. In the source paper’s framing, the shortcut model preserves the generative-policy advantages of flow- or diffusion-style modeling while making them usable for online dexterous manipulation.

## 5. Empirical performance and ablation findings

The empirical evaluation covers two customized in-hand manipulation tasks and three standard dexterous manipulation tasks [2508.14441]. The simulation suite is described as including in-hand reorientation on cubes, apples, vases, rings, animal and daily objects, in-hand push, and three Adroit suite tasks: door, hammer, and pen [2508.14441].

In simulation, FBI achieves an average success rate of **64.7%** in Vision-Only mode and **66.5%** in Visuotactile mode across all tasks [2508.14441]. The summary reports that this is **16.6%-18.4% higher** than the second-best method, DP3, and lists other baselines as ManiCM at **47.4%**, Ada-Flow at **24.9%**, and Consistency Policy at **25.7%** [2508.14441]. The paper further states that FBI especially excels on hard tasks, with up to **21.4% gain on reorientation** [2508.14441].

In the real world, the reported average success rate is **33.5%** in Vision-Only mode and **35.0%** in Visuotactile mode, compared with a ManiCM baseline at **18.5%** [2508.14441]. The source describes these results as evidence of robustness and effective sim-to-real transfer with both modalities [2508.14441].

The ablation results identify several components as important [2508.14441]. Dense tactile keypoints provide the best performance relative to sparse or continuous encoding. Transformer fusion outperforms MLP fusion and addition fusion. FBI is also reported to generalize to unseen object shapes and sizes, outperforming baselines on extrapolative scenarios [2508.14441]. These observations indicate that the gains are not attributed only to one-step inference; they also depend on the particular tactile representation and the dynamic fusion mechanism.

## 6. Interpretation, misconceptions, and relation to adjacent flow-based research

FBI is best understood as a specialized visuotactile manipulation method rather than as a generic synonym for all flow-based imitation learning. In related work, flow serves several different roles. In multi-support humanoid manipulation, Flow Matching is used to generate multi-modal whole-body trajectories and is reported to be more appropriate for robotics than Diffusion and traditional behavior cloning, with real-robot demonstrations on Talos and a shared autonomy mode for automatic contact placement [2407.12381]. In “FlowPolicy,” consistency flow matching is used for single-step 3D flow-based policy generation, yielding a reported **7$\times$ increase in inference speed** while maintaining competitive average success rates [2412.04987]. In “Streaming Flow Policy,” action trajectories are treated as flow trajectories so that actions can be streamed to the robot on-the-fly during sampling [2505.21851]. In “HinFlow,” 2D point flows serve as high-level goals and hindsight-relabeled online interactions are used to improve a goal-conditioned imitation policy, with more than **$2\times$ performance improvement** over the base policy [2512.19269].

Against that background, FBI is distinctive because flow is not used only as an action generator or planner; it is also used to infer tactile interaction structure from visual motion before policy inference [2508.14441]. A second common misconception is therefore that Vision-Only FBI discards touch. The method description says the opposite: in Vision-Only Mode the framework infers tactile cues from visual data using a learned model, so tactile reasoning remains part of the representation even without physical tactile sensors [2508.14441].

The broader literature also suggests two useful contextual distinctions. First, some methods use flow as a cross-domain manipulation interface, where object flow bridges human videos, simulated robot play data, and real-world deployment [2407.15208]. Second, some methods use flow to retrieve motion-similar prior data for few-shot imitation learning, arguing that optical flow is a better proxy for transferable low-level behavior than purely visual or semantic similarity [2408.16944]. This suggests that FBI belongs to a larger family of approaches in which flow is treated as a compact, transferable, and causally meaningful intermediate representation; however, FBI’s specific contribution is the dynamic visuotactile fusion of flow-derived tactile features with visual inputs for dexterous in-hand manipulation [2508.14441].

Source: https://www.emergentmind.com/topics/flow-before-imitation-fbi