3D FlowMatch Actor: Real-Time 3D Trajectory Control

• 3D FlowMatch Actor is a neural policy architecture that uses a continuous-time optimal transport formulation to directly sample 3D trajectories from multi-modal inputs.
• It integrates visual, proprioceptive, and language cues via a Transformer with 3D relative attention, ensuring spatial alignment and efficient sequence modeling.
• The framework achieves state-of-the-art performance with significant reductions in training and inference times, setting new benchmarks in robotic and human motion tasks.

A 3D FlowMatch Actor is a neural policy architecture that leverages flow matching for efficient, high-fidelity 3D trajectory prediction and control in complex robotic manipulation and motion generation scenarios. Unlike traditional diffusion-based networks for time-series or trajectory prediction, 3D FlowMatch Actors employ a continuous-time optimal transport formulation to enable direct end-to-end sampling of 3D trajectories conditioned on high-dimensional perceptual inputs, such as multi-view visual features, proprioceptive states, and text or language instructions. Notable instantiations include the 3D FlowMatch Actor for robot manipulation (Gkanatsios et al., 14 Aug 2025) and FlowMotion for 3D human motion generation (Cuba et al., 2 Apr 2025).

1. Flow Matching Paradigm in 3D Trajectory Modeling

3D FlowMatch Actors supersede conventional denoising diffusion models (such as DDPM/DDIM) by treating generation as the integration of an ordinary differential equation (ODE) that continuously transports a noisy initial sample to a structured data point or trajectory over a normalized time interval. The generative process is parameterized as a velocity field $v_\theta(x, t, c)$, where $x$ is the (possibly noisy) current state, $t \in [0, 1]$ is an abstract time parameter, $c$ may denote conditional context such as language or scene encoding, and $\theta$ are trained parameters.

The training objective is a flow-matching loss

$$\mathcal{L}_{FM} = \mathbb{E}_{t \sim U[0,1],\,X_0, X_1} \left[\| v_\theta(X_t, t) - (X_1 - X_0) \|^2\right]$$

where $X_0 \sim \mathcal{N}(0, I)$, $X_1 \sim p_\text{data}$, and $X_t = (1-t)X_0 + t X_1$. This drives $v_\theta$ to point in the direction of the data along straight-line couplings, reducing the number of integration steps needed for accurate sampling compared to diffusion-based methods (Gkanatsios et al., 14 Aug 2025, Cuba et al., 2 Apr 2025).

2. Architecture and 3D Perceptual Scene Representation

The 3D FlowMatch Actor architecture fuses diverse input streams into a unified sequence modeling framework. Visual features are extracted from multi-view RGB-D images using pretrained 2D encoders (e.g., CLIP), then lifted into the world coordinate frame using depth and camera calibrations to form sparse sets of 3D "visual tokens." Proprioceptive features (e.g., end-effector pose for robotic arms) are embedded and augmented with 3D positional encodings, ensuring spatial alignment with the visual context (Gkanatsios et al., 14 Aug 2025).

A sequence is formed: $x$0 and processed by a Transformer encoder featuring a novel 3D relative attention mechanism. Each attention score incorporates learned 3D relative positional biases, capturing geometric dependencies between action and scene elements.

For text-conditioned motion generation (e.g., FlowMotion), each text prompt is encoded (CLIP, 512D), time $x$1 is embedded, and both are injected into the sequence as additional conditioning vectors at every timestep (Cuba et al., 2 Apr 2025).

3. Flow-Matching Objective, Conditioning, and Temporal Modeling

Rather than denoising stepwise as in DDPM, the FlowMatch Actor integrates the ODE: $x$2 backwards from noise $x$3 to the generated trajectory $x$4. For conditional contexts $x$5 (e.g., text or visual scene), the velocity predictor is trained to minimize

$x$6

where $x$7 is a target regression loss directly predicting $x$8 to further reduce high-frequency jitter and improve generation fidelity (Cuba et al., 2 Apr 2025). In the manipulation setting, both single-arm ($x$9) and dual-arm ($t \in [0, 1]$0 in $t \in [0, 1]$1) controls are unified under this ODE-based trajectory prediction (Gkanatsios et al., 14 Aug 2025).

4. System and Training Optimizations

Several architectural and system-level optimizations are critical to the computational efficiency and effectiveness of 3D FlowMatch Actors:

• Subsampling and Efficient Attention: Density-biased sampling (DBS) reduces the token count significantly. Fused attention kernels (e.g., Triton/C++) accelerate transformer computation.
• Mixed Precision and CUDA Graphs: Automatic low-precision casting and static CUDA graphs increase both training and inference throughput.
• Data Handling: Storing depth maps in fp16 and RGB frames in uint8, with GPU-side unprojection and augmentation, increases data pipeline efficiency.
• Keypose Sampling: For manipulation, sampling keyposes across all episodes enables higher data diversity.
• Reduced Camera Count: Using only front + wrist RGB-D views is sufficient for state-of-the-art results, further reducing compute cost (Gkanatsios et al., 14 Aug 2025).

Collectively, these yield a 30× reduction in training time and a 36× gain in inference speed compared to baseline diffusion models.

5. Unified Trajectory Generation for Manipulation and Motion

The 3D FlowMatch Actor unifies policy learning for both single- and dual-arm robotic manipulation:

Mode	Action Representation	Policy Output
Unimanual	$t \in [0, 1]$2	Dense T-step trajectory
Bimanual	$t \in [0, 1]$3, $t \in [0, 1]$4 each	Joint prediction for both arms

All action tokens are processed together with scene and proprioceptive tokens in a 6-layer 3D Relative Denoising Transformer, and the output velocity fields are integrated over $t \in [0, 1]$5 steps (typically $t \in [0, 1]$6 for 85.1% avg. success in bimanual PerAct2) (Gkanatsios et al., 14 Aug 2025). For motion generation, FlowMotion exploits a transformer-based conditional velocity predictor with a direct regression-to-target arm for jitter-free ODE-sampled motion (Cuba et al., 2 Apr 2025).

6. Empirical Performance and Comparative Evaluation

3D FlowMatch Actors establish new state-of-the-art results across multiple domains:

• Robotic Manipulation: On PerAct2 (bimanual), 3DFA yields 85.1% avg. success (next-best: 43.7%, model size: 3238M). In unimanual RLBench (74 tasks), 90.3% avg. (compared to Act3D's 83.0%). In real-world dual-arm tasks, 53.5% vs. 32.5% (π₀), with <54 ms inference time per sample (Gkanatsios et al., 14 Aug 2025).
• 3D Human Motion Generation: FlowMotion achieves FID $t \in [0, 1]$7 0.278 and jitter $t \in [0, 1]$8 39.5 (HumanML3D) and FID $t \in [0, 1]$9 0.396, jitter $c$0 52.4 (KIT-ML), outstripping diffusion and noise-predictor baselines while offering an order-of-magnitude improvement in sampling speed (Cuba et al., 2 Apr 2025).

Comprehensive ablations underscore the efficacy of flow-matching (e.g., 5 steps sufficient for $c$185% success, vs. DDPM/DDIM requiring $c$2100 steps), and of system-level pipeline advances.

7. Limitations and Prospective Directions

Limitations of the 3D FlowMatch Actor paradigm include:

• Dependence on Accurate Depth and Calibration: In robotic manipulation, noisy or poorly calibrated depth leads to degraded policy performance, due to reliance on accurate visual tokenization (Gkanatsios et al., 14 Aug 2025).
• Model Capacity and Generalization: While compact (e.g., 3.8M params in 3DFA), capacity can be insufficient for rare event handling or complex multimodal distributions (“put lid on wrong jar”).
• Force and Precision Control: Tasks demanding sub-millimeter or fine force feedback are not optimally handled by current instantiations.
• Planning-Motion Coupling: On benchmarks such as PerAct2, final execution is limited by the accuracy of external planners (e.g., RRT producing unrealistic rope arcs).

Future directions suggested by existing results include the exploration of joint 2D/3D architectures to ease depth/camera requirements, integration of higher-capacity vision-LLMs for richer conditional understanding, and flow models with adaptive integration schedules for increased precision and robustness (Gkanatsios et al., 14 Aug 2025).

---

The 3D FlowMatch Actor framework represents a significant methodological advance in real-time, data-, compute-, and performance-efficient 3D trajectory learning, generalizing across domains from robotic manipulation to text-driven human motion generation, with clear empirical superiority over prior diffusion-based generative approaches (Gkanatsios et al., 14 Aug 2025, Cuba et al., 2 Apr 2025).