Traj2Action: Human-Robot Trajectory Transfer

• Traj2Action is a framework that represents human and robot demonstrations as 3D endpoint trajectories, facilitating high-level motion planning across different embodiments.
• It employs a dual-expert co-denoising architecture to refine coarse 3D trajectories into precise 7-D robot actions, separating planning from fine-grained control.
• Empirical results demonstrate notable performance improvements in robot manipulation tasks as human demonstration data scales, enhancing efficiency and scalability.

Traj2Action is a framework for manipulation skill transfer from human videos to robot actions that uses the 3D trajectory of the operational endpoint as a unified intermediate representation across human and robot embodiments. It addresses the morphology gap between human hands and robot grippers by first predicting a coarse trajectory that serves as a high-level motion plan and then conditioning robot-specific action synthesis on that plan within a co-denoising architecture. The formulation was introduced in "From Human Hands to Robot Arms: Manipulation Skills Transfer via Trajectory Alignment" (Zhou et al., 1 Oct 2025).

1. Problem setting and core abstraction

Traj2Action is motivated by a standard bottleneck in robot learning: real-world manipulation policies depend heavily on costly and hard-to-scale teleoperated demonstrations, whereas human videos are scalable but are separated from robot control by a substantial embodiment mismatch. In the formulation of Traj2Action, human hands and robot grippers differ in kinematics, control space, and appearance, so directly retargeting poses or rotations often fails. The central abstraction is therefore to represent every demonstration, human or robot, as a sequence of 3D points tracing the operational endpoint through space (Zhou et al., 1 Oct 2025).

This representation removes finger articulation and gripper jaw width from the shared transfer space and retains a high-level description of the intended manipulation. For a human demonstration, the operational endpoint is defined as the midpoint of the thumb and index fingertip after 3D hand recovery. For a robot demonstration, it is the robot end-effector position. In both cases, the shared representation is a trajectory $T\in\mathbb{R}^{H\times 3}$, which functions as a coarse plan for picking, placing, stacking, and related tasks.

A common misconception is that extra human data can simply be added to a robot action model by naive co-training. The reported ablations state that naïve co-training, implemented by padding human demonstrations to 7-D actions and mixing them with robot data, yields only $+2\%$ on PTT relative to the baseline’s $50\%\rightarrow 52\%$, whereas the unified trajectory representation is the mechanism that produces the substantive gains (Zhou et al., 1 Oct 2025).

2. Unified 3D end-effector trajectory representation

The representation pipeline is explicitly asymmetric across the two domains but identical at the level of the final trajectory. On the human side, Traj2Action uses MediaPipe 2D keypoints from three static cameras and one hand-mounted ego camera, fits the MANO hand model to obtain a temporally consistent 3D hand, and then takes the midpoint of the thumb and index fingertip as the end-effector position. On the robot side, it reads out the Franka arm’s end-effector position directly (Zhou et al., 1 Oct 2025).

The result is a shared 3D trajectory signal that abstracts away embodiment-specific details while preserving motion intent. This suggests that the transferred prior is not a pose prior but a motion-plan prior. In the terminology of the framework, the predicted trajectory is a coarse future trajectory $T_{t+1:t+H}\in\mathbb{R}^{H\times 3}$, and it is learned jointly from the union of human and robot data $D_h\cup D_r$ (Zhou et al., 1 Oct 2025).

The importance of the abstraction lies in the fact that the downstream robot policy still has to synthesize orientation and gripper behavior. Traj2Action therefore does not claim that the 3D trajectory is sufficient for the entire manipulation problem; rather, it is sufficient as a shared intermediate representation through which human and robot demonstrations can be aligned before robot-specific refinement (Zhou et al., 1 Oct 2025).

3. Dual-expert co-denoising architecture

Traj2Action extends a pretrained Vision-Language-Action backbone $\pi_0$ into two denoising experts trained jointly: a Trajectory Expert $g_T$ and an Action Expert $\pi_\theta$. The Trajectory Expert is initialized from $\pi_0$’s action expert weights and is trained on $D_h\cup D_r$ to predict the horizon-$+2\%$0 coarse future trajectory. The Action Expert is trained only on robot data $+2\%$1 and conditions on the coarse trajectory together with ego-view image $+2\%$2, language instruction $+2\%$3, and robot proprioception $+2\%$4 to synthesize a horizon-$+2\%$5 sequence of precise 7-D actions $+2\%$6, where each action is $+2\%$7 (Zhou et al., 1 Oct 2025).

Both experts are trained in a flow-matching / diffusion-style objective with time-indexed Gaussian noise. The trajectory denoising loss is

$+2\%$8

$+2\%$9

The action denoising loss, conditioned on $50\%\rightarrow 52\%$0, is

$50\%\rightarrow 52\%$1

$50\%\rightarrow 52\%$2

The total loss is

$50\%\rightarrow 52\%$3

Because $50\%\rightarrow 52\%$4 receives a noisy version of the trajectory and the denoising time $50\%\rightarrow 52\%$5, it learns to refine the coarse plan into robot-specific motion, orientation, and gripper commands. At inference, $50\%\rightarrow 52\%$6 and $50\%\rightarrow 52\%$7 are un-noised in parallel through a shared ODE or discretized denoising scheduler, producing a 3D trajectory and a set of 7-D actions with no extra latency (Zhou et al., 1 Oct 2025).

4. Data collection and preprocessing

The human data pipeline uses multi-view RGB at $50\%\rightarrow 52\%$8 FPS from three static cameras and one hand-wrist camera. The procedure is Google MediaPipe 2D keypoints followed by MANO 3D hand model fitting per frame through reprojection-error optimization, after which the thumb–index midpoint is extracted as $50\%\rightarrow 52\%$9. The robot data pipeline uses a Franka Research 3 with a UMI parallel gripper teleoperated by a SpaceMouse and records at $T_{t+1:t+H}\in\mathbb{R}^{H\times 3}$0 FPS wrist-cam RGB, static third-person RGB, end-effector position $T_{t+1:t+H}\in\mathbb{R}^{H\times 3}$1, quaternion orientation, and $T_{t+1:t+H}\in\mathbb{R}^{H\times 3}$2-D joint gripper width. The start pose is randomized for each demonstration to improve robustness (Zhou et al., 1 Oct 2025).

The preprocessing is not merely a conversion step; it determines the alignment quality between human and robot trajectories. This is reflected in the trajectory sampling-frequency ablation: the best reported setting samples human trajectories at $T_{t+1:t+H}\in\mathbb{R}^{H\times 3}$3 FPS and robot trajectories at $T_{t+1:t+H}\in\mathbb{R}^{H\times 3}$4 FPS, described as an approximately $T_{t+1:t+H}\in\mathbb{R}^{H\times 3}$5 speed ratio that aligns spatial displacement per time step across domains (Zhou et al., 1 Oct 2025).

A plausible implication is that trajectory alignment in Traj2Action depends not only on geometric consistency in $T_{t+1:t+H}\in\mathbb{R}^{H\times 3}$6 but also on temporal calibration between embodiments. That interpretation is consistent with the explicit emphasis placed on the sampling-frequency ablation.

5. Experimental results, scaling behavior, and ablations

The evaluation uses four real-world tasks on a Franka arm: Pick up water bottle (PWB), Pick up tomato + put in yellow/blue tray (PTT), Stack two rings on a pillar (SRP), and Stack three paper cups (SPC). The short-horizon tasks are measured by binary Success Rate over $T_{t+1:t+H}\in\mathbb{R}^{H\times 3}$7 trials, and the long-horizon tasks are measured by normalized Task Progress over $T_{t+1:t+H}\in\mathbb{R}^{H\times 3}$8 waypoints (Zhou et al., 1 Oct 2025).

Variant	Short-horizon	Long-horizon
Baseline $T_{t+1:t+H}\in\mathbb{R}^{H\times 3}$9 (robot only)	PWB $D_h\cup D_r$0, PTT $D_h\cup D_r$1	SRP $D_h\cup D_r$2, SPC $D_h\cup D_r$3
+Trajectory Expert†	PWB $D_h\cup D_r$4, PTT $D_h\cup D_r$5	SRP $D_h\cup D_r$6, SPC $D_h\cup D_r$7
+Traj. Expert + Human	PWB $D_h\cup D_r$8, PTT $D_h\cup D_r$9	SRP $\pi_0$0, SPC $\pi_0$1

† trained on robot data only.

In aggregate, the framework boosts performance by up to $\pi_0$2 and $\pi_0$3 over the $\pi_0$4 baseline on short- and long-horizon real-world tasks. The scaling study varies the number of human demonstrations while holding robot demonstrations fixed. On PTT, Success Rate rises from $\pi_0$5 with no human data to $\pi_0$6 with $\pi_0$7 human demonstrations and $\pi_0$8 with $\pi_0$9 human demonstrations. On SPC, Task Progress rises from $g_T$0 to $g_T$1 at $g_T$2 human demonstrations and $g_T$3 at $g_T$4 human demonstrations. The reported interpretation is monotonic improvement as human data scales (Zhou et al., 1 Oct 2025).

The ablations disentangle planning from fine-grained control. Adding the Trajectory Expert while training only on robot data improves short-horizon Success Rate by $g_T$5 and long-horizon Task Progress by $g_T$6. The decomposition result is summarized as: $g_T$7 supplies planning, $g_T$8 focuses on fine-grained control, and this separation drives most of the gain. A further ablation evaluates human data as a substitute for robot demonstrations: with only $g_T$9 human and $\pi_\theta$0 robot demonstrations, total collection time approximately $\pi_\theta$1 faster than $\pi_\theta$2 robot-only demonstrations, the model matches the $\pi_\theta$3 Success Rate of the robot-only model; at $\pi_\theta$4 human and $\pi_\theta$5 robot demonstrations, it exceeds that result with $\pi_\theta$6 (Zhou et al., 1 Oct 2025).

6. Relation to trajectory-centric literature

Traj2Action belongs to a broader family of trajectory-centric methods, but its task is manipulation transfer rather than video action recognition. "Revisiting hand-crafted feature for action recognition: a set of improved dense trajectories" introduces Trajectory-Set (TS), a motion-only descriptor built on improved Dense Trajectory, where $\pi_\theta$7 local trajectories are concatenated into a $\pi_\theta$8-dimensional vector and then Fisher-Vector encoded for video-level classification; it reports $\pi_\theta$9 on UCF50, $\pi_0$0 on UCF101, and $\pi_0$1 on HMDB51 (Matsui et al., 2017). "Localized Trajectories for 2D and 3D Action Recognition" clusters Dense Trajectories around human body joints provided by RGB-D cameras and encodes them with local Bag-of-Words, improving, for example, MSRDailyActivity3D from DenseTraj $\pi_0$2 to Localized (3D) $\pi_0$3 (Papadopoulos et al., 2019). "A Tube-and-Droplet-based Approach for Representing and Analyzing Motion Trajectories" constructs a thermal transfer field, a 3D tube, and a droplet vector for trajectory classification, anomaly detection, and 3D action recognition, reaching $\pi_0$4 on MSR-Action3D with Droplet+SVM and $\pi_0$5 when fused with Moving-Poselets (Lin et al., 2016). "Trokens: Semantic-Aware Relational Trajectory Tokens for Few-Shot Action Recognition" combines semantic-aware sampling, intra-trajectory HoD, inter-trajectory relational modeling, and a decoupled space-time transformer, reporting $\pi_0$6 on UCF101 and $\pi_0$7 on HMDB51 in $\pi_0$8-way, $\pi_0$9-shot evaluation (Kumar et al., 5 Aug 2025). "TrAction: Action Recognition with Sparse Trajectories" uses CoTracker3 trajectories augmented with monocular depth in a transformer with masked-trajectory pretraining, achieving $D_h\cup D_r$0 top-1 on Something-Something V2 and $D_h\cup D_r$1 on EPIC-Kitchens-100, while also showing complementarity with DINOv2 and V-JEPA 2 (Meier et al., 2 Jun 2026).

The comparison clarifies the scope of Traj2Action. The recognition-oriented literature uses trajectories as compact motion descriptors, relational tokens, or sparse transformer inputs for mapping videos to action labels. Traj2Action instead uses a 3D endpoint trajectory as a unified intermediate representation for transferring manipulation knowledge from human demonstrations to robot action sequences. Its closest conceptual link to the recognition literature is the shared assumption that trajectory structure can isolate task-relevant motion from appearance or embodiment details, but its output space is robot control rather than action classification.

The limitations reported for Traj2Action are also specific to this transfer setting. Its gains are tied to the quality of the extracted human and robot trajectories and to the alignment between the coarse plan and the fine action space. Conversely, its strongest empirical result is precisely that a simple 3D trajectory prior, when embedded in a dual-expert co-denoising architecture, can bridge the morphology gap sufficiently to improve real-world robot manipulation as human data scales (Zhou et al., 1 Oct 2025).