3D FlowMatch Actor (3DFA)

• 3D FlowMatch Actor (3DFA) is a unified 3D policy that combines flow matching with 3D pretrained visual scene representations to enable both single- and dual-arm robot manipulation.
• It employs a 3D relative denoising transformer and an innovative tokenization pipeline to directly link geometric scene context with dense trajectory prediction.
• Empirical evaluations show 3DFA achieves over 30× faster training and 36× faster inference, setting a new state of the art on multiple benchmarks.

Searching arXiv for the specified paper and closely related works to ground the article with citations. arxiv_search(query="(Gkanatsios et al., 14 Aug 2025)", max_results=5, sort_by="submittedDate") 3D FlowMatch Actor (3DFA) is a unified 3D policy for single- and dual-arm robot manipulation that combines flow matching for trajectory prediction with 3D pretrained visual scene representations for learning from demonstration. It uses 3D relative attention between action and visual tokens during action denoising, and it is positioned as a fast alternative to prior 3D diffusion-based policies while retaining high task performance. In the reported evaluations, 3DFA is described as achieving over $30\times$ faster training and inference than previous 3D diffusion-based policies, establishing a new state of the art on the bimanual PerAct2 benchmark, and setting a new state of the art on 74 RLBench tasks in unimanual settings by directly predicting dense end-effector trajectories (Gkanatsios et al., 14 Aug 2025).

1. Scope and defining characteristics

3DFA is presented as a single policy architecture that spans both single-arm and dual-arm manipulation. Its unifying premise is the combination of two elements: flow matching as a replacement for DDPM-style diffusion in trajectory generation, and 3D-aware scene encoding with 3D relative attention for grounding action prediction in geometric scene structure. The system is explicitly framed as building on prior work in 3D diffusion-based single-arm policy learning, while extending the design to bimanual control and substantially faster execution (Gkanatsios et al., 14 Aug 2025).

At each control step $t$, the model consumes four input types: a language instruction embedding $l$ from a frozen CLIP-text encoder, a set of posed RGB-D images $o$, proprioception $c_L,c_R$ for the left and right arms, and a noisy trajectory estimate $x^i_L,x^i_R$, defined as a linear interpolation between Gaussian noise and the demonstration trajectory. These are converted into four token streams: visual tokens, proprioceptive tokens, action tokens, and language tokens.

The visual stream is constructed by encoding each RGB patch with a frozen CLIP-image encoder into a feature vector $f \in \mathbb{R}^d$, then lifting that feature into 3D by unprojecting the pixel center using depth and intrinsics and expressing it in the robot world frame. The resulting cloud is sparsified via density-biased sampling down to $N_v$ tokens. Proprioceptive tokens are described as a small set of learnable embeddings for each arm, indexed by the current end-effector 3D position. Action tokens are formed by encoding each future-horizon pose in the noisy trajectory with an MLP and lifting it to 3D using its noised 3D location. Language tokens are static CLIP-text embeddings.

The term “unified” therefore refers to architectural scope rather than to a task-agnostic abstraction divorced from embodiment. A plausible implication is that the same representational machinery is intended to handle both bilateral coordination and single-arm execution without changing the basic policy class.

2. 3D tokenization and denoising transformer

All 3D tokens—visual, proprioceptive, and action—are concatenated along the sequence dimension and processed by 6 layers of a “3D Relative Denoising Transformer.” Each layer uses self-attention with 3D-rotary positional embeddings so that pairwise attention weights depend on relative 3D offsets, rather than solely on absolute token positions. The network is further conditioned by FiLM on two variables: the current denoising step $i \in [0,1]$ and the last end-effector pose. Language tokens are fused through standard cross-attention at each layer (Gkanatsios et al., 14 Aug 2025).

The output heads attached to action tokens predict three quantities: a velocity vector $v$ corresponding to $t$0, a rotational velocity $t$1, and a binary gripper-open probability. This design makes the policy an explicit trajectory predictor rather than a keypose-only policy.

A central architectural claim concerns how action and visual tokens interact. For two tokens $t$2 with 3D positions $t$3, 3D relative attention uses an embedding of $t$4 in the attention computation. The stated effect is that visual context is tied directly to trajectory refinement: noisy action tokens can be steered toward geometric affordances such as “holes” or “handles” in the visual point cloud. Language influences this process by modulating attention keys and queries via FiLM at each layer.

This coupling between trajectory tokens and 3D scene tokens is one of the main distinctions between 3DFA and policies that treat perception and action generation as more weakly connected stages. The paper’s formulation suggests that geometric grounding is not merely an input preprocessing step, but a recurring bias throughout denoising.

3. Flow-matching formulation

3DFA uses flow matching, specifically Rectified Flow, in place of the forward and reverse stochastic processes used in DDPM-style diffusion. The forward process is described not as stochastic diffusion, but as straight-line interpolation between Gaussian noise and clean data. For a clean demonstration trajectory $t$5 and Gaussian noise $t$6, the interpolated sample at time $t$7 is

$t$8

The reverse process is an ODE defined by a learned velocity field $t$9:

$l$0

with the objective of transporting $l$1 along straight paths so that $l$2 (Gkanatsios et al., 14 Aug 2025).

The training objective parameterizes $l$3 and minimizes

$l$4

Because the interpolation path is linear, the ground-truth instantaneous velocity is the constant vector $l$5. In the bimanual setting, the paper defines the total loss as the sum of left-arm and right-arm flow losses together with binary cross-entropy terms for gripper openness:

$l$6

$l$7

$l$8

Noise scheduling is implemented by sampling $l$9, or discrete steps $o$0, from a logit-normal distribution $o$1. The reported purpose is to concentrate learning on mid-range noise levels while still covering $o$2. Inference uses $o$3 uniform steps with $o$4.

A common misconception in this area is that replacing DDPM-style diffusion with a faster ODE-based generator necessarily sacrifices policy quality. The reported results are presented precisely to argue against that interpretation: the speedup is not accompanied by reduced benchmark performance in the tested settings.

4. System-level and architectural optimizations

The paper attributes 3DFA’s runtime gains not to flow matching alone, but to a combination of modeling and systems decisions. Relative to 3DDA’s reported 21-day training time and $o$5 Hz inference, 3DFA is described as reducing training to 16 h and increasing inference to $o$6 Hz through several cumulative changes (Gkanatsios et al., 14 Aug 2025).

The listed optimizations are specific. Flow matching reduces denoising from 100 DDPM steps to 5 denoising steps. Random keypose sampling with zarr avoids loading whole episodes while ensuring fixed batch size and high diversity. Half-precision I/O loads uint8 RGB and fp16 depth on CPU and converts to fp32 on GPU. Depth-to-point-cloud unprojection and augmentation are moved to the GPU. Density-biased sampling is implemented in PyTorch with batches of $o$7 in place of FPS. Attention uses native FlashAttention or Triton-based kernels in half precision. Training uses mixed precision with autocast. Inference uses CUDA-graph compilation after refactoring control flow into a static graph.

The paper states that each optimization alone yields $o$8–$o$9 speedups, and that together they sum to approximately $c_L,c_R$0 faster training and approximately $c_L,c_R$1 faster inference. In this presentation, efficiency is not treated as a side effect of a lighter model; it is an explicit co-design outcome of the generative formulation, the tokenization pipeline, and the deployment path.

A plausible implication is that 3DFA should be understood as a systems architecture as much as a policy architecture. The empirical argument depends on the interaction between algorithmic simplification and implementation-level throughput.

5. Training and inference procedure

The training data are demonstrations represented as sequences $c_L,c_R$2, with $c_L,c_R$3. During training, the procedure randomly samples both a denoising time $c_L,c_R$4 and a keypose index within each episode. For PerAct2, the reported configuration is a batch of 256 keyposes across 4 GPUs; for HiveFormer, it is 16 trajectories on 1 GPU. Gaussian noise $c_L,c_R$5 is sampled together with $c_L,c_R$6 logit-normal to form $c_L,c_R$7, after which the token streams are encoded, processed by the 6-layer 3D relative Transformer, and used to predict $c_L,c_R$8 and gripper-open logits. Optimization minimizes $c_L,c_R$9. Total training steps are reported as approximately 300 k for PerAct and PerAct2, compared with 600 k in 3DDA, and approximately 100 k for HiveFormer (Gkanatsios et al., 14 Aug 2025).

Inference is described as a closed-loop procedure. Starting from $x^i_L,x^i_R$0, the policy iteratively updates

$x^i_L,x^i_R$1

for $x^i_L,x^i_R$2. The final trajectory $x^i_L,x^i_R$3 is then extracted as the predicted sequence of end-effector poses and sent through inverse kinematics, or through an RRT planner if keyposes are used.

This inference recipe is significant because it links the flow-matching formulation directly to robot control latency. In the unimanual RLBench setting, the paper explicitly states that the policy directly predicts dense end-effector trajectories, eliminating the need for motion planning. That claim is specific to the reported setting; it should not be generalized beyond the conditions described.

6. Empirical performance and ablation findings

The reported benchmark results span bimanual simulation, single-arm simulation, large-scale RLBench evaluation, and real-world mobile bimanual manipulation. The results are summarized below (Gkanatsios et al., 14 Aug 2025).

Setting	Reported result	Comparison stated in source
PerAct2, 13 multitask bimanual tasks, 100 demos each	$x^i_L,x^i_R$4 success $x^i_L,x^i_R$5	next best multitask $x^i_L,x^i_R$6: $x^i_L,x^i_R$7; training: 16 h on $x^i_L,x^i_R$8L40S vs 504 h / 21 days for 3DDA; inference: $x^i_L,x^i_R$9 Hz vs $f \in \mathbb{R}^d$0 Hz; params: $f \in \mathbb{R}^d$1 M vs $f \in \mathbb{R}^d$2’s $f \in \mathbb{R}^d$3 B
Single-arm PerAct, 18 tasks	$f \in \mathbb{R}^d$4 average success with 4 cameras	matches 3DDA with 6× less training and 28× faster inference; with 2 cameras: $f \in \mathbb{R}^d$5 while running 30× faster
HiveFormer, 74 RLBench tasks	$f \in \mathbb{R}^d$6 average success	$f \in \mathbb{R}^d$7 pp over Act3D; $f \in \mathbb{R}^d$8 on 8 continuous-interaction tasks vs ChainedDiffuser $f \in \mathbb{R}^d$9, PointFlowMatch $N_v$0
Real world, 10 bimanual tasks on Mobile Aloha	$N_v$1 average success @ $N_v$2 ms/control	$N_v$3: $N_v$4 @ $N_v$5 ms; iDP3: $N_v$6 @ $N_v$7 ms

On PerAct2, the source characterizes the result as a new state of the art, outperforming the next-best method by an absolute margin of $N_v$8. In real-world evaluation, the comparison is notable because the paper states that 3DFA surpasses strong baselines with up to $N_v$9 more parameters and significantly more pretraining.

The ablation studies are used to isolate the contributions of denoising formulation and systems optimization. For denoising steps, the paper reports that DDPM needs 100 steps to hit $i \in [0,1]$0, DDIM collapses below 25 steps, and flow matching sustains $i \in [0,1]$1 at 5 steps, $i \in [0,1]$2 at 3 steps, and $i \in [0,1]$3 at 1 step. For training time, it reports that moving from 3DDA to random keypose sampling halves data-load time, and that adding DB and the remaining optimizations yields a $i \in [0,1]$4 overall effect, turning 504 h into 16 h, summarized as $i \in [0,1]$5. For inference, flow matching with $i \in [0,1]$6 fewer steps produces $i \in [0,1]$7 Hz, adding DBS raises throughput to $i \in [0,1]$8 Hz, and adding CUDA-graph and optimized attention reaches $i \in [0,1]$9 Hz, summarized as $v$0.

These findings are used to support two claims. First, speed improvements are not attributed solely to replacing diffusion with flow matching. Second, the policy’s reported state-of-the-art performance depends on the combination of 3D-aware attention conditioning and flow matching, rather than on either element in isolation.