Flowing from Reasoning to Motion: Learning 3D Hand Trajectory Prediction from Egocentric Human Interaction Videos (2512.16907v1)

Abstract: Prior works on 3D hand trajectory prediction are constrained by datasets that decouple motion from semantic supervision and by models that weakly link reasoning and action. To address these, we first present the EgoMAN dataset, a large-scale egocentric dataset for interaction stage-aware 3D hand trajectory prediction with 219K 6DoF trajectories and 3M structured QA pairs for semantic, spatial, and motion reasoning. We then introduce the EgoMAN model, a reasoning-to-motion framework that links vision-language reasoning and motion generation via a trajectory-token interface. Trained progressively to align reasoning with motion dynamics, our approach yields accurate and stage-aware trajectories with generalization across real-world scenes.

• The paper introduces EgoMAN, a modular framework that predicts 6DoF hand trajectories by integrating vision-language cues and explicit waypoint tokens.
• It employs a two-stage process where a reasoning module interprets visual and linguistic inputs, followed by a flow matching-based expert for smooth motion synthesis.
• Empirical results demonstrate significant improvements in trajectory accuracy and generalization, highlighting enhanced sample efficiency and intent-conditioned control.

Modular Vision-Language-to-Motion Reasoning for Egocentric 6DoF Hand Trajectory Prediction

Introduction and Problem Setting

"Flowing from Reasoning to Motion: Learning 3D Hand Trajectory Prediction from Egocentric Human Interaction Videos" (2512.16907) presents a modular architecture, EgoMAN, designed to predict 6DoF hand trajectories from a single egocentric RGB frame, wrist trajectory history, and natural language intent queries. The work explicitly addresses the constraints of prior approaches, which typically generalize poorly due to implicit semantic-action coupling, lack of interaction stage supervision, or reliance on object-centric affordances.

The authors propose the EgoMAN dataset, encompassing over 219K annotated hand trajectories with structured stage-aware annotations and 3M vision-language-motor QA pairs. The dataset spans diverse scenes and captures interaction structure necessary for mapping semantic intent to physical execution. The centerpiece is a two-stage model: (1) a Vision-Language Reasoning Module, built on QwenVL-7B, that interprets visual and linguistic cues and predicts semantic and waypoint tokens, and (2) a Flow Matching-based Motion Expert, which synthesizes temporally-aligned, physically plausible hand trajectories conditioned on the predicted waypoints and intentions.

Figure 1: The EgoMAN model’s architecture: reasoned intent and spatial context yield structured tokens, which drive a flow-matching motion generator for 6DoF trajectory synthesis.

EgoMAN Dataset: Semantically Structured, Stage-Aware Trajectories

The dataset contribution is central. EgoMAN aggregates 300+ hours and 1,500+ scenes from sources including EgoExo4D, Nymeria, and HOT3D-Aria. It features highly diverse hand–object interactions with precise wrist-centered 6DoF trajectory annotations aligned to distinct approach/manipulation segments. Structured stage annotations—critical for intent-mapped motion synthesis—are generated using automated (LLM-assisted) pipelines and human curation, resulting in high-quality ground truth that captures transitions between “approach” and “manipulation” at 10Hz.

EgoMAN's QA corpus covers semantic intent, geometric stage-specific queries, and “how” questions regarding expected motion, facilitating cross-modal supervision. The authors evaluate under two conditions: generalization to held-out (unseen) scenes and objects (“EgoMAN-Unseen”), and to out-of-distribution domains (HOT3D-OOD).

Modular Reasoning-to-Motion: Trajectory-Token Interface

The model implements a modular pipeline for future motion prediction, formalized as mapping $$[\text{RGB}, \text{past motion}, \text{intent}] \mapsto \text{future 6DoF trajectories}$$.

Reasoning Module

The module is a sequence model (Qwen2.5-VL), consuming visual/language streams and past pose, and emitting (i) natural language or (ii) a compact sequence of four trajectory tokens that parameterize intent and interaction stages: $<$ACT$>$ (action semantics), $<$START$>$, $<$CONTACT$>$, and $<$END$>$ (key approach/manipulation waypoints). Each waypoint token outputs a timestamp, 3D position, and 6D orientation. Pre-training jointly optimizes cross-modal QA, semantic (CLIP-projected) alignment, and trajectory regression losses.

Motion Expert via Flow Matching

The expert is a transformer-based conditional generative model utilizing Flow Matching (FM). Inputs are the predicted tokens (waypoints, semantics), past motion, and DINOv3 visual features, organized with temporally coherent positional embeddings. The model learns a velocity field minimizing endpoint and waypoint misalignment, yielding smooth, high-frequency, physically consistent trajectories for both manipulation and approach stages.

The interface between reasoning and motion is strictly structured: lowest-level semantic and spatial state is passed as discrete token embeddings, bridging interpretable reasoning and motion synthesis.

Empirical Results

Quantitative Evaluation

EgoMAN achieves consistent, significant improvements over leading baselines on EgoMAN-Bench and HOT3D-OOD:

• Outperforms HandsOnVLM by 27.5% (ADE, K=10) on both in-domain and OOD.
• Excels in waypoint prediction, with contact/trajectory errors reduced by 33-52% relative to affordance methods, at orders-of-magnitude higher inference FPS.
• SOTA in best-of-K generative accuracy for trajectory/rotational metrics; generalization holds for unseen environments, actions, and manipulation modes.

Ablations and Data Efficiency

Ablations reveal that progressive multi-stage pretraining (vision-language QA, tokenized stage-wise conditioning, low-level motion FM) is essential. Removing reasoning or FM pretraining causes severe performance collapse in ADE and FDE.

Data ablations highlight superior sample efficiency: with only 20% of the finetuning data, EgoMAN maintains strong generalization, while models lacking explicit waypoint supervision degrade rapidly.

Figure 2: Data efficiency: EgoMAN maintains low ADE/FDE down to 20% training data, driven by reasoning and waypoint-based modularization.

Qualitative Generalization & Intent Conditioning

EgoMAN robustly generates context-sensitive and intent-aligned trajectories for diverse real-world interactions, even under challenging visual ambiguity or with alternate intents issued over the same input stream.

Figure 3: EgoMAN outputs intent- and context-aligned 6DoF hand trajectories for varied activities.

Figure 4: Given different language intents and identical visual/motion context, EgoMAN produces distinct, controllable motion outputs.

Theoretical and Practical Implications

The modular action generation paradigm—explicit modularization of intent reasoning, semantic waypoints, and motion synthesis—yields both sample efficiency and OOD robustness. EgoMAN demonstrates the key advantage of a structured token interface: semantically interpretable, spatially explicit state intermediates that allow for controllable and traceable intent-to-motion mapping.

This work implies that future embodied AI and robot learning systems benefit from LLM-guided modularization and cross-modal pretraining for human-like semantic grounding. The dataset and architecture further bridge the gap between high-level task understanding and temporally precise low-level control, enabling language-driven, generalizable action policies directly from human videos.

Potential future directions include extending reasoning granularity to finer sub-stages, integrating articulated hand pose, and transferring policies to closed-loop robotic agents via sim-to-real adaptation and interactive planning with intent-tokenized intermediate states.

Conclusion

This paper establishes a new paradigm for egocentric 6DoF hand prediction via the careful integration of structured data, modular cross-modal reasoning, and FM-based generative motion. The EgoMAN pipeline unifies vision, language, and motion through a trajectory-token interface, yielding highly accurate, semantically aligned, and generalizable predictions. Results suggest this interface design and training pipeline can serve as a foundation for robust in-context robot action, intent-driven human–AI collaboration, and structured motion generation in embodied agents.