---
title: 'Traj2Action: Human-Robot Trajectory Transfer'
url: https://www.emergentmind.com/topics/traj2action
type: topic
---

# Traj2Action: Human-Robot Trajectory Transfer

Traj2Action is a framework for manipulation skill transfer from human videos to robot actions that uses the 3D trajectory of the operational endpoint as a unified intermediate representation across human and robot embodiments. It addresses the morphology gap between human hands and robot grippers by first predicting a coarse trajectory that serves as a high-level motion plan and then conditioning robot-specific action synthesis on that plan within a co-denoising architecture. The formulation was introduced in "From Human Hands to Robot Arms: Manipulation Skills Transfer via Trajectory Alignment" [2510.00491].

## 1. Problem setting and core abstraction

Traj2Action is motivated by a standard bottleneck in robot learning: real-world manipulation policies depend heavily on costly and hard-to-scale teleoperated demonstrations, whereas human videos are scalable but are separated from robot control by a substantial embodiment mismatch. In the formulation of Traj2Action, human hands and robot grippers differ in kinematics, control space, and appearance, so directly retargeting poses or rotations often fails. The central abstraction is therefore to represent every demonstration, human or robot, as a sequence of 3D points tracing the operational endpoint through space [2510.00491].

This representation removes finger articulation and gripper jaw width from the shared transfer space and retains a high-level description of the intended manipulation. For a human demonstration, the operational endpoint is defined as the midpoint of the thumb and index fingertip after 3D hand recovery. For a robot demonstration, it is the robot end-effector position. In both cases, the shared representation is a trajectory \(T\in\mathbb{R}^{H\times 3}\), which functions as a coarse plan for picking, placing, stacking, and related tasks.

A common misconception is that extra human data can simply be added to a robot action model by naive co-training. The reported ablations state that naïve co-training, implemented by padding human demonstrations to 7-D actions and mixing them with robot data, yields only \(+2\%\) on PTT relative to the baseline’s \(50\%\rightarrow 52\%\), whereas the unified trajectory representation is the mechanism that produces the substantive gains [2510.00491].

## 2. Unified 3D end-effector trajectory representation

The representation pipeline is explicitly asymmetric across the two domains but identical at the level of the final trajectory. On the human side, Traj2Action uses MediaPipe 2D keypoints from three static cameras and one hand-mounted ego camera, fits the MANO hand model to obtain a temporally consistent 3D hand, and then takes the midpoint of the thumb and index fingertip as the end-effector position. On the robot side, it reads out the Franka arm’s end-effector position directly [2510.00491].

The result is a shared 3D trajectory signal that abstracts away embodiment-specific details while preserving motion intent. This suggests that the transferred prior is not a pose prior but a motion-plan prior. In the terminology of the framework, the predicted trajectory is a coarse future trajectory \(T_{t+1:t+H}\in\mathbb{R}^{H\times 3}\), and it is learned jointly from the union of human and robot data \(D_h\cup D_r\) [2510.00491].

The importance of the abstraction lies in the fact that the downstream robot policy still has to synthesize orientation and gripper behavior. Traj2Action therefore does not claim that the 3D trajectory is sufficient for the entire manipulation problem; rather, it is sufficient as a shared intermediate representation through which human and robot demonstrations can be aligned before robot-specific refinement [2510.00491].

## 3. Dual-expert co-denoising architecture

Traj2Action extends a pretrained Vision-Language-Action backbone \(\pi_0\) into two denoising experts trained jointly: a Trajectory Expert \(g_T\) and an Action Expert \(\pi_\theta\). The Trajectory Expert is initialized from \(\pi_0\)’s action expert weights and is trained on \(D_h\cup D_r\) to predict the horizon-\(H\) coarse future trajectory. The Action Expert is trained only on robot data \(D_r\) and conditions on the coarse trajectory together with ego-view image \(I_t\), language instruction \(\mathcal{P}\), and robot proprioception \(q_t\) to synthesize a horizon-\(H\) sequence of precise 7-D actions \(a_{t+1:t+H}\in\mathbb{R}^{H\times 7}\), where each action is \(a_t=(\Delta x,\Delta y,\Delta z;\Delta\text{axis-angle }(3D);\Delta\text{gripper\_width})\) [2510.00491].

Both experts are trained in a flow-matching / diffusion-style objective with time-indexed Gaussian noise. The trajectory denoising loss is

$$
T_\tau = \tau\,T^*_{t+1:t+H} + (1-\tau)\,z
$$

$$
L_{\text{traj}}(g_T)=\mathbb{E}_{\tau,z,T^*}
\bigl[\|g_T(T_\tau,\tau,I_t,\mathcal{P},q'_t) - (z - T^*_{t+1:t+H})\|^2\bigr].
$$

The action denoising loss, conditioned on \(T_\tau\), is

$$
a_\tau = \tau\,a^*_{t+1:t+H} + (1-\tau)\,z
$$

$$
L_{\text{action}}(\pi_\theta)=\mathbb{E}_{\tau,z,a^*}
\bigl[\|\pi_\theta(a_\tau,T_\tau,\tau,I_t,\mathcal{P},q_t) - (z - a^*_{t+1:t+H})\|^2\bigr].
$$

The total loss is

$$
L = L_{\text{traj}} + L_{\text{action}}.
$$

Because \(\pi_\theta\) receives a noisy version of the trajectory and the denoising time \(\tau\), it learns to refine the coarse plan into robot-specific motion, orientation, and gripper commands. At inference, \(g_T\) and \(\pi_\theta\) are un-noised in parallel through a shared ODE or discretized denoising scheduler, producing a 3D trajectory and a set of 7-D actions with no extra latency [2510.00491].

## 4. Data collection and preprocessing

The human data pipeline uses multi-view RGB at \(30\) FPS from three static cameras and one hand-wrist camera. The procedure is Google MediaPipe 2D keypoints followed by MANO 3D hand model fitting per frame through reprojection-error optimization, after which the thumb–index midpoint is extracted as \(T^h(t)\in\mathbb{R}^3\). The robot data pipeline uses a Franka Research 3 with a UMI parallel gripper teleoperated by a SpaceMouse and records at \(30\) FPS wrist-cam RGB, static third-person RGB, end-effector position \(T^r(t)\in\mathbb{R}^3\), quaternion orientation, and \(1\)-D joint gripper width. The start pose is randomized for each demonstration to improve robustness [2510.00491].

The preprocessing is not merely a conversion step; it determines the alignment quality between human and robot trajectories. This is reflected in the trajectory sampling-frequency ablation: the best reported setting samples human trajectories at \(30\) FPS and robot trajectories at \(10\) FPS, described as an approximately \(3\times\) speed ratio that aligns spatial displacement per time step across domains [2510.00491].

A plausible implication is that trajectory alignment in Traj2Action depends not only on geometric consistency in \(\mathbb{R}^3\) but also on temporal calibration between embodiments. That interpretation is consistent with the explicit emphasis placed on the sampling-frequency ablation.

## 5. Experimental results, scaling behavior, and ablations

The evaluation uses four real-world tasks on a Franka arm: Pick up water bottle (PWB), Pick up tomato + put in yellow/blue tray (PTT), Stack two rings on a pillar (SRP), and Stack three paper cups (SPC). The short-horizon tasks are measured by binary Success Rate over \(50\) trials, and the long-horizon tasks are measured by normalized Task Progress over \(8\) waypoints [2510.00491].

| Variant | Short-horizon | Long-horizon |
|---|---|---|
| Baseline \(\pi_0\) (robot only) | PWB \(48\), PTT \(50\) | SRP \(23.75\), SPC \(37.75\) |
| +Trajectory Expert† | PWB \(58\), PTT \(60\) | SRP \(33.50\), SPC \(54.25\) |
| +Traj. Expert + Human | PWB \(76\), PTT \(76\) | SRP \(44.75\), SPC \(61.25\) |

† trained on robot data only.

In aggregate, the framework boosts performance by up to \(27\%\) and \(22.25\%\) over the \(\pi_0\) baseline on short- and long-horizon real-world tasks. The scaling study varies the number of human demonstrations while holding robot demonstrations fixed. On PTT, Success Rate rises from \(68\%\) with no human data to \(74\%\) with \(264\) human demonstrations and \(76\%\) with \(460\) human demonstrations. On SPC, Task Progress rises from \(37.75\%\) to \(57.50\%\) at \(264\) human demonstrations and \(61.25\%\) at \(460\) human demonstrations. The reported interpretation is monotonic improvement as human data scales [2510.00491].

The ablations disentangle planning from fine-grained control. Adding the Trajectory Expert while training only on robot data improves short-horizon Success Rate by \(10\%\) and long-horizon Task Progress by \(13.13\%\). The decomposition result is summarized as: \(g_T\) supplies planning, \(\pi_\theta\) focuses on fine-grained control, and this separation drives most of the gain. A further ablation evaluates human data as a substitute for robot demonstrations: with only \(240\) human and \(270\) robot demonstrations, total collection time approximately \(2.5\times\) faster than \(408\) robot-only demonstrations, the model matches the \(60\%\) Success Rate of the robot-only model; at \(635\) human and \(270\) robot demonstrations, it exceeds that result with \(62\%\) [2510.00491].

## 6. Relation to trajectory-centric literature

Traj2Action belongs to a broader family of trajectory-centric methods, but its task is manipulation transfer rather than video action recognition. "Revisiting hand-crafted feature for action recognition: a set of improved dense trajectories" introduces Trajectory-Set (TS), a motion-only descriptor built on improved Dense Trajectory, where \(K^2\) local trajectories are concatenated into a \(2\cdot L\cdot K^2\)-dimensional vector and then Fisher-Vector encoded for video-level classification; it reports \(95.0\%\) on UCF50, \(88.6\%\) on UCF101, and \(85.4\%\) on HMDB51 [1711.10143]. "Localized Trajectories for 2D and 3D Action Recognition" clusters Dense Trajectories around human body joints provided by RGB-D cameras and encodes them with local Bag-of-Words, improving, for example, MSRDailyActivity3D from DenseTraj \(64.4\) to Localized (3D) \(76.3\) [1904.05244]. "A Tube-and-Droplet-based Approach for Representing and Analyzing Motion Trajectories" constructs a thermal transfer field, a 3D tube, and a droplet vector for trajectory classification, anomaly detection, and 3D action recognition, reaching \(92.1\%\) on MSR-Action3D with Droplet+SVM and \(93.9\%\) when fused with Moving-Poselets [1609.03058]. "Trokens: Semantic-Aware Relational Trajectory Tokens for Few-Shot Action Recognition" combines semantic-aware sampling, intra-trajectory HoD, inter-trajectory relational modeling, and a decoupled space-time transformer, reporting \(94.0\%\) on UCF101 and \(69.8\%\) on HMDB51 in \(5\)-way, \(1\)-shot evaluation [2508.03695]. "TrAction: Action Recognition with Sparse Trajectories" uses CoTracker3 trajectories augmented with monocular depth in a transformer with masked-trajectory pretraining, achieving \(45.2\%\) top-1 on Something-Something V2 and \(54.1\%\) on EPIC-Kitchens-100, while also showing complementarity with DINOv2 and V-JEPA 2 [2606.03490].

The comparison clarifies the scope of Traj2Action. The recognition-oriented literature uses trajectories as compact motion descriptors, relational tokens, or sparse transformer inputs for mapping videos to action labels. Traj2Action instead uses a 3D endpoint trajectory as a unified intermediate representation for transferring manipulation knowledge from human demonstrations to robot action sequences. Its closest conceptual link to the recognition literature is the shared assumption that trajectory structure can isolate task-relevant motion from appearance or embodiment details, but its output space is robot control rather than action classification.

The limitations reported for Traj2Action are also specific to this transfer setting. Its gains are tied to the quality of the extracted human and robot trajectories and to the alignment between the coarse plan and the fine action space. Conversely, its strongest empirical result is precisely that a simple 3D trajectory prior, when embedded in a dual-expert co-denoising architecture, can bridge the morphology gap sufficiently to improve real-world robot manipulation as human data scales [2510.00491].

Source: https://www.emergentmind.com/topics/traj2action