Flow Before Imitation (FBI) for Dexterous Manipulation

• Flow Before Imitation (FBI) is a dynamic visuotactile imitation learning framework that fuses motion-derived tactile features with visual inputs to enable dexterous in-hand manipulation.
• It employs a Flow2Tactile module to predict tactile signals from visual motion and integrates these through a transformer-based fusion, ensuring effective policy conditioning.
• Empirical evaluations show FBI enhances success rates by up to 21.4% on complex tasks and operates at 20Hz, indicating robust sim-to-real transfer.

Flow Before Imitation (FBI) is a visuotactile imitation learning framework for dexterous in-hand manipulation that dynamically fuses tactile interactions with visual observations through motion dynamics rather than through static multimodal feature fusion (Chen et al., 20 Aug 2025). In the formulation reported for dexterous manipulation, FBI establishes a causal link between tactile signals and object motion via a dynamics-aware latent model, employs a transformer-based interaction module to fuse flow-derived tactile features with visual inputs, and trains a one-step diffusion policy for real-time execution (Chen et al., 20 Aug 2025). The framework operates in both a Vision-Only Mode, which infers tactile cues from visual data using a learned model, and a Visuotactile Mode, which directly ingests tactile sensor readings (Chen et al., 20 Aug 2025).

1. Definition and conceptual basis

Within the dexterous manipulation literature, FBI denotes a dynamic visuotactile imitation learning architecture in which object motion flow is treated as the intermediary that connects contact dynamics to policy conditioning (Chen et al., 20 Aug 2025). The central methodological departure from prior static fusion is that tactile forces are treated as the drivers of object motion, and temporal visual change is used to infer or refine contact information rather than merely concatenating vision and touch features frame by frame (Chen et al., 20 Aug 2025).

The framework is explicitly designed for dexterous in-hand manipulation, a regime characterized in the source paper by complex contact dynamics and partial observability (Chen et al., 20 Aug 2025). FBI is therefore not only a multimodal policy, but a multimodal policy organized around motion dynamics. In the reported implementation, flow is used before imitation in the literal sense that motion flow is first estimated or represented, then converted into tactile contact features, and only then fused into the policy input (Chen et al., 20 Aug 2025).

A broader reading is also supported by related flow-based robot learning work. Several contemporaneous methods use flow as a precursor representation for action generation, retrieval, or hierarchical control rather than learning a direct state-to-action map. This suggests that “Flow Before Imitation” can also be read as a broader design orientation within robot learning, although the named FBI framework is the visuotactile dexterous-manipulation method introduced in (Chen et al., 20 Aug 2025).

2. System inputs, encoders, and operating modes

At each timestep, FBI takes robot states, visual observations, and optionally tactile readings as inputs (Chen et al., 20 Aug 2025). The reported variables are $s_{t-1}, s_t \in \mathbb{R}^{N_s}$ for robot states, $P_{t-1}, P_t \in \mathbb{R}^{N_p \times 3}$ for two consecutive point clouds, and $R_t \in \mathbb{R}^{N_r}$ for tactile readings (Chen et al., 20 Aug 2025).

The modality-specific encoders are specified as follows (Chen et al., 20 Aug 2025):

Component	Design	Output
Robot State Encoder	3-layer MLP	$\mathcal{F}_s \in \mathbb{R}^{d_s}$
Visual Encoder	3-layer MLP + max-pooling	$\mathcal{F}_v \in \mathbb{R}^{d_v}$
Tactile Encoder	4-layer MLP	$\mathcal{F}_{tac} \in \mathbb{R}^{d_{tac}}$

The two operating modes differ in how tactile information is obtained (Chen et al., 20 Aug 2025). In Vision-Only Mode, tactile cues are inferred from visual motion through the Flow2Tactile module, so the framework does not require physical tactile sensors. In Visuotactile Mode, measured tactile sensor readings are used directly, which refines contact state estimation (Chen et al., 20 Aug 2025). This dual-mode construction is central to the framework’s claim that visuotactile reasoning can be retained even when only visual observations are available.

3. Flow2Tactile and dynamic visuotactile fusion

The Flow2Tactile module is the mechanism that converts motion information into tactile contact features (Chen et al., 20 Aug 2025). Its motivation is that dense, physically accurate tactile sensing hardware is impractical in many real-world cases, so FBI predicts tactile contact states from motion flow derived from visual data (Chen et al., 20 Aug 2025).

The reported workflow has three stages (Chen et al., 20 Aug 2025). First, object state flow is predicted using shortcut one-step flow matching, yielding flow vectors $f_{t-1 \to t}$ that map $P_{t-1}$ to $P_t$. Second, a pre-trained transformer searching module $T_s(\cdot)$ takes hand keypoints $P_{t-1}, P_t \in \mathbb{R}^{N_p \times 3}$0 together with the flow and produces tactile readings:

$P_{t-1}, P_t \in \mathbb{R}^{N_p \times 3}$1

Third, contact keypoints are predefined on the hand surface; the example given is 456 keypoints for the Shadow Hand, providing a dense and transferable tactile representation (Chen et al., 20 Aug 2025).

The training losses reported for Flow2Tactile are (Chen et al., 20 Aug 2025):

$P_{t-1}, P_t \in \mathbb{R}^{N_p \times 3}$2

$P_{t-1}, P_t \in \mathbb{R}^{N_p \times 3}$3

$P_{t-1}, P_t \in \mathbb{R}^{N_p \times 3}$4

and

$P_{t-1}, P_t \in \mathbb{R}^{N_p \times 3}$5

After encoding, FBI performs dynamic visuotactile fusion with a transformer module $P_{t-1}, P_t \in \mathbb{R}^{N_p \times 3}$6 (Chen et al., 20 Aug 2025). The visual feature is used as the template and the tactile feature as the searching feature. The resulting visuotactile feature is concatenated back with the visual feature to form the policy input $P_{t-1}, P_t \in \mathbb{R}^{N_p \times 3}$7 (Chen et al., 20 Aug 2025). In the reported interpretation, this is the mechanism by which the model learns dependencies and contextual weighting across touch and vision, rather than relying on naive concatenation or averaging.

A frequent misunderstanding is to treat FBI as simply another multimodal concatenation policy. That characterization is inconsistent with the method description: the distinguishing claim is precisely that fusion is dynamic and causality-minded, with tactile interactions linked to motion flow and then fused through a transformer (Chen et al., 20 Aug 2025).

4. One-step shortcut policy and control generation

The policy head in FBI is a shortcut diffusion model trained for one-step inference (Chen et al., 20 Aug 2025). The reported sampling equation is

$P_{t-1}, P_t \in \mathbb{R}^{N_p \times 3}$8

with one-step mode setting $P_{t-1}, P_t \in \mathbb{R}^{N_p \times 3}$9 and $R_t \in \mathbb{R}^{N_r}$0 (Chen et al., 20 Aug 2025). The policy condition is

$R_t \in \mathbb{R}^{N_r}$1

and the predicted action sequence over horizon $R_t \in \mathbb{R}^{N_r}$2 is

$R_t \in \mathbb{R}^{N_r}$3

where $R_t \in \mathbb{R}^{N_r}$4 is implemented by a U-Net backbone (Chen et al., 20 Aug 2025).

The policy training losses are reported as (Chen et al., 20 Aug 2025):

$R_t \in \mathbb{R}^{N_r}$5

$R_t \in \mathbb{R}^{N_r}$6

$R_t \in \mathbb{R}^{N_r}$7

and

$R_t \in \mathbb{R}^{N_r}$8

The practical significance of this design lies in real-time control. FBI is reported to run at 20Hz, with 51.2ms total step time, including 34.4ms model inference for Vision-Only and 17.4ms for Visuotactile (Chen et al., 20 Aug 2025). The comparison given against standard diffusion is a DP3 baseline requiring 137.9ms per step (Chen et al., 20 Aug 2025). In the source paper’s framing, the shortcut model preserves the generative-policy advantages of flow- or diffusion-style modeling while making them usable for online dexterous manipulation.

5. Empirical performance and ablation findings

The empirical evaluation covers two customized in-hand manipulation tasks and three standard dexterous manipulation tasks (Chen et al., 20 Aug 2025). The simulation suite is described as including in-hand reorientation on cubes, apples, vases, rings, animal and daily objects, in-hand push, and three Adroit suite tasks: door, hammer, and pen (Chen et al., 20 Aug 2025).

In simulation, FBI achieves an average success rate of 64.7% in Vision-Only mode and 66.5% in Visuotactile mode across all tasks (Chen et al., 20 Aug 2025). The summary reports that this is 16.6%-18.4% higher than the second-best method, DP3, and lists other baselines as ManiCM at 47.4%, Ada-Flow at 24.9%, and Consistency Policy at 25.7% (Chen et al., 20 Aug 2025). The paper further states that FBI especially excels on hard tasks, with up to 21.4% gain on reorientation (Chen et al., 20 Aug 2025).

In the real world, the reported average success rate is 33.5% in Vision-Only mode and 35.0% in Visuotactile mode, compared with a ManiCM baseline at 18.5% (Chen et al., 20 Aug 2025). The source describes these results as evidence of robustness and effective sim-to-real transfer with both modalities (Chen et al., 20 Aug 2025).

The ablation results identify several components as important (Chen et al., 20 Aug 2025). Dense tactile keypoints provide the best performance relative to sparse or continuous encoding. Transformer fusion outperforms MLP fusion and addition fusion. FBI is also reported to generalize to unseen object shapes and sizes, outperforming baselines on extrapolative scenarios (Chen et al., 20 Aug 2025). These observations indicate that the gains are not attributed only to one-step inference; they also depend on the particular tactile representation and the dynamic fusion mechanism.

6. Interpretation, misconceptions, and relation to adjacent flow-based research

FBI is best understood as a specialized visuotactile manipulation method rather than as a generic synonym for all flow-based imitation learning. In related work, flow serves several different roles. In multi-support humanoid manipulation, Flow Matching is used to generate multi-modal whole-body trajectories and is reported to be more appropriate for robotics than Diffusion and traditional behavior cloning, with real-robot demonstrations on Talos and a shared autonomy mode for automatic contact placement (Rouxel et al., 2024). In “FlowPolicy,” consistency flow matching is used for single-step 3D flow-based policy generation, yielding a reported 7$R_t \in \mathbb{R}^{N_r}$9 increase in inference speed while maintaining competitive average success rates (Zhang et al., 2024). In “Streaming Flow Policy,” action trajectories are treated as flow trajectories so that actions can be streamed to the robot on-the-fly during sampling (Jiang et al., 28 May 2025). In “HinFlow,” 2D point flows serve as high-level goals and hindsight-relabeled online interactions are used to improve a goal-conditioned imitation policy, with more than $\mathcal{F}_s \in \mathbb{R}^{d_s}$0 performance improvement over the base policy (Zheng et al., 22 Dec 2025).

Against that background, FBI is distinctive because flow is not used only as an action generator or planner; it is also used to infer tactile interaction structure from visual motion before policy inference (Chen et al., 20 Aug 2025). A second common misconception is therefore that Vision-Only FBI discards touch. The method description says the opposite: in Vision-Only Mode the framework infers tactile cues from visual data using a learned model, so tactile reasoning remains part of the representation even without physical tactile sensors (Chen et al., 20 Aug 2025).

The broader literature also suggests two useful contextual distinctions. First, some methods use flow as a cross-domain manipulation interface, where object flow bridges human videos, simulated robot play data, and real-world deployment (Xu et al., 2024). Second, some methods use flow to retrieve motion-similar prior data for few-shot imitation learning, arguing that optical flow is a better proxy for transferable low-level behavior than purely visual or semantic similarity (Lin et al., 2024). This suggests that FBI belongs to a larger family of approaches in which flow is treated as a compact, transferable, and causally meaningful intermediate representation; however, FBI’s specific contribution is the dynamic visuotactile fusion of flow-derived tactile features with visual inputs for dexterous in-hand manipulation (Chen et al., 20 Aug 2025).