FlowHOI: Dexterous HOI Generation

• FlowHOI is a structured framework that decouples geometry-centric grasping from semantics-centric manipulation for generating explicit hand-object interaction scripts.
• It leverages egocentric observations, language cues, and 3D Gaussian splatting for scene reconstruction to ensure contact consistency and facilitate cross-robot transfer.
• Evaluations on GRAB, HOT3D, and real-robot setups show up to 1.7× higher success rates and 40× faster inference compared to diffusion-based methods.

FlowHOI is a hand-object interaction generation framework for dexterous robot manipulation that models manipulation as an explicitly structured interaction sequence rather than as robot-specific joint control. It generates semantically grounded, temporally coherent HOI sequences comprising hand poses, object poses, and hand-object contact states, conditioned on an egocentric observation, a language instruction, and a 3D Gaussian splatting scene reconstruction. Its central claim is that long-horizon, contact-rich manipulation fails when the underlying interaction structure is implicit; FlowHOI therefore uses a two-stage conditional flow-matching design that decouples geometry-centric grasping from semantics-centric manipulation, and evaluates the resulting representation on GRAB and HOT3D as well as on real-robot execution with Franka Emika Panda arms equipped with Allegro Hand v5 end-effectors (Zeng et al., 13 Feb 2026).

1. Problem setting and interaction representation

FlowHOI addresses dexterous manipulation in settings where success depends on establishing contact, preserving contact as the object changes pose or state, respecting scene geometry, and following a language instruction. The motivating observation is that recent vision-language-action models can generate plausible end-effector motions, yet often fail on long-horizon, contact-rich tasks because they do not explicitly represent HOI structure. FlowHOI treats this deficiency as a representational problem rather than only a planning problem (Zeng et al., 13 Feb 2026).

The framework therefore generates an embodiment-agnostic HOI script whose state variables are interaction-centric. The output includes hand poses over time, object poses over time, and contact-related features. In the grasping stage, each hand state is represented in MANO space using root translation relative to the object position at the grasp-transition frame, a 6D global orientation, MANO PCA pose coefficients, and per-joint signed-distance vectors to the object mesh. The signed-distance feature is defined by the difference between a hand joint and its closest point on the posed object mesh. The object state is represented by translation and 6D rotation. This makes the generated sequence more physically meaningful than a robot-specific action trace and also makes it easier to validate and retarget across embodiments (Zeng et al., 13 Feb 2026).

The paper explicitly argues that such a representation matters for four reasons: it separates geometry from semantics, improves physical plausibility because contact consistency can be enforced directly, supports cross-robot transfer, and provides a validation interface in which hand-object interactions can be checked for collision and contact before execution. A plausible implication is that FlowHOI is intended less as a standalone low-level controller than as an interaction-level intermediate layer between perception, language grounding, and downstream retargeting or control.

2. Two-stage conditional flow matching

FlowHOI uses a two-stage conditional flow-matching pipeline. The first stage is geometry-centric grasping, and the second is semantics-centric manipulation. The stated motivation is that humans typically establish a stable grasp first and then manipulate the object; the architecture mirrors that decomposition so that geometry and semantics are not solved simultaneously (Zeng et al., 13 Feb 2026).

In Stage 1, the object is treated as static and only hand states are generated. Conditioning includes a Basis Point Set encoding of the object geometry, a T5 encoding of a grasp-focused sub-instruction extracted from the full language prompt, and the initial hand-object state. The stage uses conditional flow matching but adopts $x$-prediction rather than direct velocity prediction. The interpolation path and general flow objective are:

$$\mathbf{x}_\tau = \bigl(1-(1-\sigma_{\min})\tau\bigr)\mathbf{x}_0 + \tau \mathbf{x}_1,$$

$$\mathcal{L}_{\mathrm{FM}} = \mathbb{E}\left[ \left\| \mathbf{v}_\theta(\mathbf{x}_\tau,\tau,\mathbf{c})-\mathbf{u}_\tau(\mathbf{x}_\tau\mid \mathbf{x}_1) \right\|_2^2 \right].$$

For grasping specifically, the model predicts the clean target sequence:

$$\mathcal{L}_{\text{flow}^g} = \mathbb{E}\left[ \left\| f^g_\theta(\mathbf{x}^g_\tau,\tau,\mathbf{c}^g)-\mathbf{x}^g_1 \right\|_2^2 \right].$$

The second stage generates the full HOI sequence, with only the post-grasp future treated as stochastic. Conditioning includes object geometry, the language instruction, local and global 3D scene tokens, and the transition grasp state. Continuity between stages is enforced by inserting the grasping output as a temporal prefix during manipulation and hard-clamping the transition state. The manipulation loss is a masked $x$-prediction objective over the unknown future segment:

$$\mathcal{L}_{\text{flow}^m} = \mathbb{E}\left[ \left\| \mathbf{M}\odot\left( f^m_\theta(\bar{\mathbf{x}^m_\tau},\tau,\mathbf{c}^m)-\mathbf{x}^m_1 \right) \right\|_2^2 \right].$$

Both stages are augmented with a motion-text alignment term, yielding

$$\mathcal{L}_{\text{grasp}}=\mathcal{L}_{\text{flow}^g}+\lambda_{\text{align}}\mathcal{L}_{\text{align}}, \qquad \mathcal{L}_{\text{manip}}=\mathcal{L}_{\text{flow}^m}+\lambda_{\text{align}}\mathcal{L}_{\text{align}},$$

with $\lambda_{\text{align}}=0.1$. The alignment loss is described as symmetric InfoNCE between text and motion embeddings. This design places language supervision at both the grasping and manipulation levels rather than only at final sequence generation (Zeng et al., 13 Feb 2026).

3. Scene grounding from egocentric observation and 3DGS

A defining feature of FlowHOI is that the manipulation stage is grounded not only in language and object geometry but also in a reconstructed 3D scene. From an initial egocentric observation, the method estimates the initial hand pose, object geometry, and initial object pose. It then reconstructs the scene with 3D Gaussian Splatting while masking out hands and objects, and encodes the resulting geometry into compact local and global scene tokens (Zeng et al., 13 Feb 2026).

The local token pathway begins by sampling scene points $\mathbf{X}\in\mathbb{R}^{N_s\times 3}$. Each point is assigned a geometric feature from Concerto and a semantic feature from SceneSplat. These are projected into a shared latent space and fused with a learnable channel gate,

$$\mathbf{F}=\tilde{\mathbf{E}}+\sigma(\boldsymbol{\alpha})\odot \tilde{\mathbf{U}},$$

after which Fourier positional encoding of coordinates is concatenated and the result is compressed by a Perceiver bottleneck into local scene tokens. In parallel, a global scene token is obtained by voxelizing the scene and encoding it with a ViT. The paper assigns different functions to the two token types: the global token captures coarse layout and helps avoid major collisions, while the local tokens capture fine-grained contact-relevant geometry and semantics (Zeng et al., 13 Feb 2026).

This scene-conditioning design is tied directly to the paper’s notion of semantics-grounded generation. Manipulation is not conditioned on scene layout in an implicit image-feature sense; instead, scene structure is represented explicitly as a tokenized geometric-semantic context. This suggests that FlowHOI aims to encode not only which interaction should occur, but also where in the scene it should remain feasible.

4. Reconstruction pipeline and the HOI prior

The paper identifies a supervision bottleneck: high-fidelity HOI data are scarce because hands and objects are frequently occluded and contact is difficult to annotate. To address that problem, FlowHOI introduces a reconstruction pipeline that recovers aligned hand-object trajectories and meshes from large-scale egocentric videos and uses them as a grasping prior (Zeng et al., 13 Feb 2026).

The reconstruction process has three steps. First, it detects the grasp-to-manipulation transition. Wrist trajectories are smoothed, and candidate transitions are identified by local speed minima and direction changes; supplementary details specify Gaussian smoothing, a 1-second local window, and orientation changes greater than $$\mathbf{x}_\tau = \bigl(1-(1-\sigma_{\min})\tau\bigr)\mathbf{x}_0 + \tau \mathbf{x}_1,$$0. Second, it reconstructs the target object by identifying it with SAM3, estimating metric depth with DepthAnything3, reconstructing the mesh with SAM3D, and transforming that mesh into the world frame using known camera extrinsics. Third, it performs hand-object alignment. MANO hand meshes are fitted by inverse kinematics, after which object translation offsets and hand pose corrections are optimized so that fingerpads contact the object without deep penetration (Zeng et al., 13 Feb 2026).

The hand-object alignment is formulated as an optimization over object translation offset and MANO pose correction:

$$\mathbf{x}_\tau = \bigl(1-(1-\sigma_{\min})\tau\bigr)\mathbf{x}_0 + \tau \mathbf{x}_1,$$1

The aligned grasp configuration is then propagated backward through the grasp phase, producing a full aligned HOI sequence used to pretrain the grasping model. This prior is derived from EgoDex for training, while GRAB and HOT3D are used for evaluation. The paper’s broader position is that reconstruction is a practical route to scalable HOI supervision when direct manual annotation is inadequate.

5. Evaluation, benchmarks, and real-robot execution

FlowHOI is evaluated on GRAB and HOT3D. GRAB provides high-fidelity mocap HOI data without scene context, whereas HOT3D adds egocentric recordings with hand and object annotations and reconstructed 3D scenes. EgoDex is used only for training the grasp prior, not for evaluation. Reported metrics cover physical interaction quality, motion quality, and realizable physical feasibility: interpenetration volume, interpenetration depth, contact ratio, interpenetration volume per contact unit, action recognition accuracy, sample diversity, overall diversity, heuristic physical plausibility score, simulation success rate, holding time, and inference time (Zeng et al., 13 Feb 2026).

On GRAB, the paper reports the following key numbers:

Measure	FlowHOI	Diffusion baselines
Simulation success rate	55.96%	DiffH2O: 33.03%; LatentHOI: 28.44%
Inference time	0.16 s	DiffH2O: 6.34 s; LatentHOI: 3.57 s

These numbers underlie two headline claims: FlowHOI achieves a $$\mathbf{x}_\tau = \bigl(1-(1-\sigma_{\min})\tau\bigr)\mathbf{x}_0 + \tau \mathbf{x}_1,$$2 higher physics simulation success rate than the strongest diffusion-based baseline, and up to a $$\mathbf{x}_\tau = \bigl(1-(1-\sigma_{\min})\tau\bigr)\mathbf{x}_0 + \tau \mathbf{x}_1,$$3 inference speedup. The paper also states that FlowHOI attains the highest action recognition accuracy and best or near-best physical interaction quality, and that on HOT3D it remains strong under real-world scene context while improving semantic correctness and contact consistency (Zeng et al., 13 Feb 2026).

The framework is additionally demonstrated on a real robot system composed of two Franka Emika Panda arms, each equipped with an Allegro Hand v5. The evaluated tasks are drinking from a cup, pouring liquid, tilting a container, and squeezing dressing. The execution pipeline reconstructs the 3D scene using Gaussian-LIC and 3DGS, retargets generated MANO trajectories to the Franka-plus-Allegro embodiment, refines them with DexTrack, and executes open-loop using joint impedance control. The reported significance of these demonstrations is not that FlowHOI is itself a dynamics controller, but that the generated HOI representation can be retargeted into real-robot execution pipelines while preserving contact-rich behavior (Zeng et al., 13 Feb 2026).

6. Relation to adjacent HOI research, naming ambiguity, and limitations

FlowHOI belongs to a broader shift in HOI research from label prediction toward structured generation, reconstruction, and affordance modeling. Within that landscape, it is important to distinguish several nearby threads. “HO-Flow: Generalizable Hand-Object Interaction Generation with Latent Flow Matching” is a separate framework for text-conditioned 3D hand-object motion synthesis from texts and canonical 3D objects, using an interaction-aware VAE and masked autoregressive flow matching in latent space (Chen et al., 12 Apr 2026). “H2OFlow” addresses 3D affordance learning from synthetic data via dense diffused point-cloud flows, focusing on contact, orientation, and spatial occupancy rather than robot-executable interaction scripts (Zhang et al., 17 Oct 2025). “VHOI” studies controllable HOI video generation from sparse trajectories via motion densification (Zhang et al., 10 Dec 2025), “ArtHOI” reframes articulated HOI synthesis as 4D reconstruction from monocular video priors (Huang et al., 4 Mar 2026), “HA-HOI” targets physically plausible monocular 4D HOI reconstruction and simulation readiness (Zhao et al., 14 May 2026), and “HOI-IDiff” recasts HOI detection outputs as images for diffusion-based detection rather than manipulation generation (Hui et al., 23 Mar 2025). FlowHOI is therefore specifically a dexterous robot manipulation method grounded in egocentric perception, language, and 3DGS scene context (Zeng et al., 13 Feb 2026).

A common misconception is to treat FlowHOI as merely a faster diffusion replacement. The paper supports a narrower interpretation: flow matching is important because it enables efficient generation, but the method’s distinguishing feature is the explicit HOI representation coupled with a stage decomposition that separates grasping geometry from manipulation semantics. Another possible misconception is to read the method as robot-specific. The paper argues the opposite: the generated output is embodiment-agnostic and intended for retargeting, validation, and cross-robot transfer.

The reported limitations are also explicit. FlowHOI depends on accurate initial hand and object state estimation, degrades under heavy occlusion or poor reconstruction, and produces outputs that are mainly kinematic and contact-consistent rather than fully dynamic; dynamics and compliance are delegated to downstream controllers. It also assumes a well-defined manipulation setup and does not yet handle more general cases such as mobile manipulation. Contact failures can still occur when object size is mismatched or when grasp establishment is incorrect (Zeng et al., 13 Feb 2026). These constraints place FlowHOI squarely in the class of interaction-structured generative front-ends whose practical value depends on the quality of perception, reconstruction, and retargeting components downstream.