Embodiment-Agnostic 3D Flow in Robotics

• Embodiment-agnostic 3D flow is a representation paradigm that defines manipulation and navigation goals as 3D geometric displacements independent of specific robot actuators.
• It leverages methodologies such as 3D point tracks, scene flow fields, and SE(3) transforms to predict object and part motions in dynamic environments.
• This approach enables efficient policy transfer and cross-embodiment adaptation, facilitating applications from robotic manipulation to mobile navigation.

Embodiment-agnostic 3D flow is an intermediate representation and computational paradigm in robotics and robot learning, in which the motion or control objectives for manipulation or navigation are described in terms of object- or scene-centric 3D geometric displacements rather than embodiment-dependent actuator variables. This approach enables learning, planning, and policy transfer across heterogeneous robots and even from human demonstration videos, by abstracting away the details of specific manipulators, sensors, or actuation spaces. Embodiment-agnostic 3D flow frameworks typically leverage world modeling via 3D point tracks, scene flow fields, or SE(3) transforms over object parts, and downstream policies or planners that “invert” these geometric flows into feasible robot actions based on each embodiment’s kinematics and dynamics.

1. Formal Definitions and Representations

Embodiment-agnostic 3D flow formalizes the robot’s plan or prediction as a metric 3D motion field that encodes what in the environment should move, where, and when, without specifying how any particular robot accomplishes this physically. The prevalent mathematical structures are as follows:

• 3D Point Tracks: A collection of N points $\{p^i \in \mathbb{R}^3\}_{i=1}^N$ tracked over T timesteps yields a tensor $X \in \mathbb{R}^{N \times T \times 3}$, with $X_t^i$ denoting the predicted 3D world position of point $i$ at time $t$ (Hung et al., 9 Mar 2026).
• 3D Scene Flow Fields: For any 3D point $x \in \mathbb{R}^3$ at time $t$, the scene flow $F_t(x) = (u(x), v(x), w(x)) \in \mathbb{R}^3$ represents the instantaneous velocity: $x_{t+1} = x_t + F_t(x_t)$ (Tang et al., 2024, Dharmarajan et al., 31 Dec 2025).
• Trace-Space: In compact form, trajectories are abstracted as time-indexed sequences of K points: $$\mathbf{T}^{1:L} = (p_1, ..., p_L) \in \mathbb{R}^{K \times L \times 3}$$, or with flow increments $X \in \mathbb{R}^{N \times T \times 3}$0 (Lee et al., 26 Nov 2025).
• Object-Part Transforms: For articulated or rigid object parts, per-frame SE(3) transforms $X \in \mathbb{R}^{N \times T \times 3}$1 are fit from per-point flows, representing frame-to-frame rigid motion to be realized by the end effector (Tang et al., 2024).

A key attribute of these representations is their lack of reference to the robot’s joint space or morphology: they encode the task as a desired geometric evolution of the scene, robust to the embodiment of the actor.

2. Core Methodological Components

Embodiment-agnostic 3D flow frameworks, as instantiated in research such as 3PoinTr, TraceGen, Dream2Flow, 3DFlowAction, and object-part flow planners, comprise several canonical processing stages:

1. Perceptual Encoding: Initial RGBD frames or point clouds are used to segment task-relevant objects or parts (via vision-language segmentation or foundation models such as Grounding DINO, SAM-2, LISA) and to lift selected pixels or grids to metric 3D via calibrated depth (Dharmarajan et al., 31 Dec 2025, Tang et al., 2024).
2. 3D Flow/Trace Prediction: A model, often a transformer or U-Net-based video diffusion model, predicts future object motion. For 3D point tracks, a two-block transformer predicts $X \in \mathbb{R}^{N \times T \times 3}$2 from initial points (Hung et al., 9 Mar 2026). For scene flow, a conditional diffusion model generates dense flow tensors or video (Dharmarajan et al., 31 Dec 2025, Zhi et al., 6 Jun 2025).
3. Supervision and Training: Supervised losses use ground-truth from simulated kinematics or 2D-3D lifted tracks, frequently masked by per-point visibility to accommodate occlusion (Hung et al., 9 Mar 2026). Large-scale auto-labeled datasets, such as ManiFlow-110k and corpora generated with TraceForge, facilitate cross-embodiment pretraining (Zhi et al., 6 Jun 2025, Lee et al., 26 Nov 2025).
4. Trajectory Extraction: Discrete 3D flows are aggregated into per-frame SE(3) transforms, tracing the pose trajectory for the object part; this is solved via SVD (Procrustes) (Tang et al., 2024, Zhi et al., 6 Jun 2025).
5. Policy/Planner Conditioning: The downstream planner or policy receives the 3D object flow as the goal to be achieved. Methods include optimization-based trajectory solvers, imitation learning via behavior cloning, or RL with flow tracking rewards (Dharmarajan et al., 31 Dec 2025, Yang et al., 27 Sep 2025).

This architecture separates geometric task specification (what, where) from embodiment-dependent execution (how).

3. Policy Inversion and Cross-Embodiment Application

Once an embodiment-agnostic 3D flow is predicted, it is inverted to robot actions through embodiment-specific planning modules:

• Inverse Kinematics and Trajectory Optimization: End-effector pose sequences extracted from SE(3) transforms (e.g., $X \in \mathbb{R}^{N \times T \times 3}$3) are mapped to joint-space via standard or QP-based inverse kinematics, enforcing collision, workspace, and smoothness constraints (Tang et al., 2024).
• Global Policy Conditioning: 3D flow tensors $X \in \mathbb{R}^{N \times T \times 3}$4 are compacted via cross-attention or Perceiver IO architectures to produce a global state vector conditioning diffusion or imitation learning policies, which then produce open-loop or closed-loop action sequences (Hung et al., 9 Mar 2026).
• Optimization and RL: Flow-following objectives are posed as constraints or reward terms in trajectory optimizers or RL, aiming to minimize the distance between predicted object points and the actual scene, plus robot-specific control costs (Dharmarajan et al., 31 Dec 2025, Zhi et al., 6 Jun 2025).
• Zero-Shot and Few-Shot Adaptation: By decoupling policy learning from embodiment, policies learned from human videos or one robot can be retargeted to new robots by replacing only the final planning stage, sometimes requiring minimal (≤5) new demonstrations for fine-tuning (Lee et al., 26 Nov 2025, Tang et al., 2024, Hung et al., 9 Mar 2026).

This paradigm enables transfer learning across morphologies, supporting applications such as training from unconstrained human demonstrations, adapting navigation policies between quadrupeds, bipeds, and aerial vehicles (Yang et al., 27 Sep 2025), and closed-loop manipulation from generative video predictions (Dharmarajan et al., 31 Dec 2025).

4. Empirical Performance and Comparisons

Published results across simulated and real domains consistently report substantial gains in transferability, generalization, and sample efficiency:

Method	Key Metric / Domain	Value(s)	Baseline Comparison
3PoinTr (Hung et al., 9 Mar 2026)	ADE (mm), Block Stack (sim)	0.20	General Flow: 0.47
	Real Tasks, 20 demos	9/10 all tasks	ATM <2/10, DP3 <8/10
TraceGen (Lee et al., 26 Nov 2025)	Real Franka Panda, 5 demos	80% success	<25% for from-scratch
	Human demo, 5 videos	67.5% success	–
3DFlowAction (Zhi et al., 6 Jun 2025)	Foundational tasks, zero-shot	70% success	AVDC, Rekep, etc.: 20–50%
Part-Flow (Tang et al., 2024)	Meta-World / Franka-Kitchen	+27.7%, +26.2%	vs. best prior RL/BC methods
Dream2Flow (Dharmarajan et al., 31 Dec 2025)	Real-world: Bread-in-Bowl	8/10	AVDC: 7/10, RIGVID: 6/10
CE-Nav (Yang et al., 27 Sep 2025)	Go2, Mean SR (success rate)	0.8575	PureRL: 0.64, 6h vs. 52h train

3PoinTr achieves ∼49% lower average displacement error and ∼43% higher real-world success rate than point-cloud–based and 2D-flow baselines, using only 20 labeled demonstrations. Ablations repeatedly show that replacing 3D flow/scene flow with 2D image-space flows or removing closed-loop planners yields 20–40 percentage point drops in task success (Hung et al., 9 Mar 2026, Zhi et al., 6 Jun 2025, Tang et al., 2024).

5. Embodiment-Agnostic 3D Flow in Navigation and Manipulation

While much of the literature focuses on manipulation, embodiment-agnostic 3D flow has also been applied to mobile navigation and cross-platform policy transfer:

• Navigation via Conditional Normalizing Flows: CE-Nav’s VelFlow learns the distribution over kinematically feasible body-frame velocity commands (e.g., $X \in \mathbb{R}^{N \times T \times 3}$5) independent of robot type, via maximum likelihood imitation learning from planner-generated datasets. The flow is then refined to account for dynamic and controller specifics of the target robot using RL on real or simulated hardware (Yang et al., 27 Sep 2025). This yields state-of-the-art navigation on quadrupeds (Go2 SR=0.86), bipeds, and quadrotors with up to an 8.7x reduction in adaptation time compared to pure-RL baselines.
• General Manipulation and Part/Entire Object Flows: Methods such as object-part flow and full-scene flow planners predict future transformations of both rigid and deformable objects, converting them into robot-agnostic reference trajectories to be realized by arbitrary arms (FANUC, Franka, Dobot) or even soft grippers, with demonstrated strong generalization to new object types, unseen backgrounds, and novel robot morphologies (Dharmarajan et al., 31 Dec 2025, Lee et al., 26 Nov 2025, Tang et al., 2024).

6. Data, Pretraining, and Real-World Considerations

Progress in embodiment-agnostic 3D flow has been driven by the construction of large, heterogeneous, cross-embodiment datasets, and by pretraining world-models and trace predictors at scale:

• Dataset Synthesis: ManiFlow-110k (110k clips), TraceForge (1.8M triplets), include data from both robot and human demonstrations across environments, object types, and camera setups (Zhi et al., 6 Jun 2025, Lee et al., 26 Nov 2025). Automatic moving-object detection, backprojected optical flow, and robust point tracking (CoTracker3) are integral for dataset construction.
• Generalization from Human Videos: Policies trained with human demonstration alone, without joint or torque labels, can directly guide robot manipulation by retargeting the inferred 3D flow to the robot’s kinematic chain. For example, part-flow planners trained solely on human videos without any robot-specific fine-tuning achieve 8/10 drawer opens on a FANUC arm (Tang et al., 2024).
• Efficiency and Speed: TraceGen demonstrates 50–600x faster inference compared to pixel-based video world models, critical for real-world deployment; similarly, dynamics-aware refiners and compact transformer architectures enable adaptation with minimal data and training time (Lee et al., 26 Nov 2025, Yang et al., 27 Sep 2025, Hung et al., 9 Mar 2026).
• Failure Modes: The main limitations are robustness to occlusion, multi-object and granular flows, video-generation artifacts, and slow pipeline components (e.g., video synthesis bottlenecks in Dream2Flow) (Dharmarajan et al., 31 Dec 2025). Masked loss functions on occluded points and open-loop/frozen-policy strategies have addressed some, but not all of these challenges.

7. Related Paradigms, Open Questions, and Trajectories

Embodiment-agnostic 3D flow sits at the intersection of geometric robot learning, world modeling, and cross-domain policy transfer. Related approaches include:

• Video Diffusion Models and Keypoint Trajectories: Some prior methods use pixel or keypoint-based video world models, but these struggle with generalization and inference speed (Lee et al., 26 Nov 2025, Zhi et al., 6 Jun 2025).
• Normalizing Flows and Distributional Planning: In navigation (CE-Nav), conditional flow models resolve multi-modality in action distributions, addressing the "disastrous averaging" problem by sampling diverse feasible commands (Yang et al., 27 Sep 2025).
• Behavioral Cloning from Flow: Conditioning BC policies on 3D flow or trace embeddings yields greater spatial generalization and sample efficiency than policies relying on object category, pose, or point clouds (Hung et al., 9 Mar 2026).

Open problems include improving robustness to occlusion and scene clutter, scaling to highly deformable or multi-object scenarios, integrating multi-view or 4D flows, and closing the gap to real-time performance for full closed-loop manipulation in the wild (Dharmarajan et al., 31 Dec 2025). A plausible implication is increasing convergence between geometric computer vision, foundation models for depth and segmentation, and continuous control via RL or optimization, with 3D flow serving as the shared interface.

---

Collectively, these developments establish embodiment-agnostic 3D flow as a central paradigm for disentangling geometric objectives from embodiment-specific actuation, with strong empirical evidence for its utility in scalable, generalizable robot learning and control (Tang et al., 2024, Zhi et al., 6 Jun 2025, Lee et al., 26 Nov 2025, Dharmarajan et al., 31 Dec 2025, Hung et al., 9 Mar 2026, Yang et al., 27 Sep 2025).