3PoinTr: Embodiment-Agnostic Robot Manipulation

• 3PoinTr is a framework that leverages dense, embodiment-agnostic 3D point tracks to bridge the gap between human and robot manipulation.
• It employs a transformer-based predictor on initial point clouds to generate temporally consistent 3D trajectories with minimal robot demonstrations.
• The method integrates occlusion-aware training, Perceiver IO conditioning, and diffusion policy learning to achieve robust performance in simulation and real-world tasks.

3PoinTr is a framework for pretraining robot manipulation policies from casual, unconstrained human videos by leveraging dense, embodiment-agnostic 3D point tracks as an intermediate representation. The method enables sample-efficient behavior cloning with as few as 20 action-labeled robot demonstrations, and addresses the longstanding embodiment gap between human and robot demonstrations by modeling scene evolution independently of human or robot kinematics (Hung et al., 9 Mar 2026).

1. Problem Setting: Embodiment Gap in Robot Learning from Human Videos

3PoinTr addresses the challenge of cross-embodiment policy transfer, where the goal is to learn robot manipulation policies from demonstrations performed by humans in video. The embodiment gap is characterized by marked differences in kinematics, morphology, reachable poses, manipulation strategies, contact patterns, and action spaces between human bodies and robot manipulators. For example, in the "Right Glass" task, a human's hand can grasp a glass by the stem, but a robot gripper may be forced to manipulate the object differently. Retargeting human hand trajectories or predicting embodiment-specific flows has limited robustness, especially with unconstrained, naturally performed ("casual") human demonstrations. Prior approaches relied on geometric correspondences or choreographed demonstrations, which are difficult to scale and restrict the diversity of transferable skills.

3PoinTr formulates a two-stage learning framework. The first stage is to pretrain an embodiment-agnostic model that predicts the evolution of the 3D scene given an initial observation, with all embodiment points (human or robot) excluded from the input point cloud. The model learns a mapping from an initial point cloud $P\in\mathbb{R}^{N\times 3}$ to a set of dense 3D point tracks $X\in\mathbb{R}^{N\times T\times 3}$. The second stage learns a policy $\pi_\phi$ mapping these point tracks to a sequence of robot actions. The core insight is that 3D point tracks encode how the world evolves to accomplish a task, abstracting away the details of the agent performing the manipulation.

2. 3D Point Track Representation and Supervision

The central representation in 3PoinTr is a set of dense 3D point tracks, where each initial point $p^{(i)}$ is associated with a trajectory $$P^{(i)}=\{(x_t^{(i)},y_t^{(i)},z_t^{(i)})\}_{t=1}^T$$. The entire set $X\in\mathbb{R}^{N\times T\times 3}$ captures spatiotemporal scene evolution, goal specification, and object relationships in metric 3D space, but avoids encapsulating embodiment-specific parameters.

3D tracks are generated from stereo RGB video in the real-world setting via 2D tracking (CoTracker3), depth estimation (FoundationStereo), lifting to 3D, segmentation (SAM3), removal of embodiment points, and temporal resampling. For simulation, supervision uses ground-truth 3D points directly.

Training employs an L1 position loss on visible points and timesteps: $$\mathcal{L}_{\text{track}} = \sum_{i=1}^{N}\sum_{t=1}^{T} m_t^i\, \|\hat{X}_t^i-X_t^i\|_1$$ where $m_t^i$ is a visibility mask. Uniquely, 3PoinTr preserves partially observed tracks by masking only the occluded pairs, contrasting with baselines (e.g., General Flow), which drop any track with occluded frames. This occlusion-aware supervision allows the framework to utilize maximal information from videos even with significant occlusion—critical for realistic tabletop manipulation.

3. Transformer-Based Point Track Predictor

The architecture for 3D trajectory prediction consists of a lightweight transformer decoder that operates directly on the initial point cloud:

• Each input point is embedded with a 3-layer MLP (dimension 256, GELU activations).
• $N=2048$ points are used per sample, with unit-sphere normalization and Gaussian noise ($\sigma=0.01$) as augmentation.
• The model comprises two transformer decoder blocks (4 heads, head dimension 64, feed-forward expansion ratio 4).
• No separate positional encoding is introduced; the 3D coordinates are the only inputs to the tokenization MLP.

Multi-head self-attention is used to model geometric and contextual dependencies: $X\in\mathbb{R}^{N\times T\times 3}$0 with standard transformer updates via layer normalization and feed-forward modules. The output head is a linear layer predicting entire $X\in\mathbb{R}^{N\times T\times 3}$1 trajectories per point. The network jointly reasons about spatial structure through attention and predicts temporally consistent future point motion. This approach is more efficient than architectures using explicit video token stacks, VAEs, or heavy object-centric pipelines.

4. Policy Learning with Perceiver IO and Diffusion

After pretraining, the frozen point track predictor generates dense tracks given a new initial scene observation. The policy mapping from point tracks to robot actions leverages a Perceiver IO scheme followed by a Diffusion Policy:

• Each point trajectory $X\in\mathbb{R}^{N\times T\times 3}$2 is projected to a 128D token.
• A single learned query token cross-attends to the set of tokens (Perceiver-style cross-attention), producing a global compact scene/task representation.
• The resulting vector (dimension 256) conditions a Diffusion Policy (1D U-Net) that predicts the full open-loop sequence of 10-dimensional actions (end-effector position, 6D orientation, gripper command) for $X\in\mathbb{R}^{N\times T\times 3}$3 timesteps.
• The diffusion policy is trained via denoising loss as in Chi et al.

Behavior cloning is conducted open-loop, requiring only the initial scene and predicted point tracks—no visual feedback during execution. The two-stage design (point track prediction + compact policy conditioning) allows for strong sample efficiency: only 20 action-labeled robot demonstrations are needed for robust policy learning, as world dynamics and task semantics are distilled in pretraining.

5. Experimental Evaluation and Quantitative Results

Experiments evaluate on both simulation and real-world tasks using a UFactory xArm 7 robot with parallel-jaw gripper. Simulation tasks include block stacking, microwave opening, and glass uprighting; real-world tasks include drawer opening, glass righting, paper disposal, and sock folding, with randomized object positions and orientations to test for spatial generalization.

Evaluation uses the following metrics:

• 3D Average Displacement Error (ADE, mm): Mean per-point per-timestep Euclidean error.
• 5% ADE: ADE computed over the 5% of points with the largest motion—emphasizing task-critical regions.
• Policy success rate: Fraction of successful rollouts across diverse configurations.

Results show:

• 3PoinTr achieves lower ADE and 5% ADE compared to General Flow on all tasks, reducing error by 49.1% (ADE) and 61.8% (5% ADE) on average.
• With 20 demonstrations, 3PoinTr outperforms all baselines (AMPLIFY, ATM, Diffusion Policy, DP3), e.g., achieving 90.9% (Block Stack), 80.8% (Open Microwave), 95.2% (Glass) success rates in simulation. Policy success rates in real-world tasks are also consistently highest for 3PoinTr.
• Ablation indicates the full 3D point track representation is necessary; projecting tracks to 2D yields significantly lower performance.

6. Analysis: Qualitative Effects, Ablations, and Architectural Comparisons

Qualitative analysis shows that 3PoinTr produces denser, more accurate tracks, especially in cases of occlusion or non-rigid interactions. This improvement is attributed to:

• The architectural simplicity and expressiveness of the single transformer decoder,
• Occlusion-aware loss that leverages visible information even when objects become hidden, in contrast to prior methods discarding such tracks.

Architectural ablation reveals that the transformer-based point track prediction directly from initial point sets outperforms PointNeXt + CVAE-based baselines. Open-loop policy execution, enabled by robust pretraining, provides consistency and resilience to covariate shift and occlusion-induced perception failures seen in feedback-dependent baselines.

7. Insights, Limitations, and Prospective Directions

Key insights include:

• Scene evolution, rather than embodiment-specific motion, is a superior basis for transferring knowledge from human to robot manipulation.
• Dense, non-object-centric 3D point tracks provide a robust, interpretable, and flexible task specification.
• Conditioning downstream policies with compact summaries of dense pretraining yields strong few-shot generalization.

Limitations outlined by the authors:

• Policies are trained per-task; 3PoinTr is not yet a multi-task generalist.
• Real-world variations across backgrounds, viewpoints, and lighting are not extensively tested.
• Open-loop behavior cloning, while sample efficient, forgoes feedback-based adaptation during execution.
• Reliance on the quality of 2D/3D tracking and segmentation, which may be brittle for challenging visual conditions.
• Lack of multimodal (e.g., image or text) conditioning in the policy stage.
• Uniform temporal subsampling may not adequately capture high-frequency contact events.

Proposed avenues for future work include generalist policy training, use of internet-scale uncurated videos, closed-loop policy extensions, improved 3D tracking pipelines, adaptive temporal discretization, and multimodal action conditioning.

---

3PoinTr establishes 3D point tracks as a scalable, embodiment-agnostic interface enabling policy learning from unconstrained human demonstration videos, surpassing prior behavior cloning and video-pretraining frameworks when evaluated on both simulated and real-world manipulation tasks. The method demonstrates that focusing on scene evolution, coupled with transformer-based prediction and Perceiver IO-conditioned policy learning, can close the embodiment gap with minimal labeled robot data, providing a promising foundation for scalable robot acquisition of novel manipulation skills from diverse visual data sources (Hung et al., 9 Mar 2026).