Action-Geometry Prediction with 3D Geometric Prior for Bimanual Manipulation

Abstract: Bimanual manipulation requires policies that can reason about 3D geometry, anticipate how it evolves under action, and generate smooth, coordinated motions. However, existing methods typically rely on 2D features with limited spatial awareness, or require explicit point clouds that are difficult to obtain reliably in real-world settings. At the same time, recent 3D geometric foundation models show that accurate and diverse 3D structure can be reconstructed directly from RGB images in a fast and robust manner. We leverage this opportunity and propose a framework that builds bimanual manipulation directly on a pre-trained 3D geometric foundation model. Our policy fuses geometry-aware latents, 2D semantic features, and proprioception into a unified state representation, and uses diffusion model to jointly predict a future action chunk and a future 3D latent that decodes into a dense pointmap. By explicitly predicting how the 3D scene will evolve together with the action sequence, the policy gains strong spatial understanding and predictive capability using only RGB observations. We evaluate our method both in simulation on the RoboTwin benchmark and in real-world robot executions. Our approach consistently outperforms 2D-based and point-cloud-based baselines, achieving state-of-the-art performance in manipulation success, inter-arm coordination, and 3D spatial prediction accuracy. Code is available at https://github.com/Chongyang-99/GAP.git.

• The paper introduces an RGB-only diffusion policy that jointly predicts bimanual action chunks and future dense 3D pointmaps using the pretrained π³ geometric foundation model.
• The method achieves the best average success rates across all three RoboTwin 2.0 categories—63.2% for Dominant-select, 51.3% for Sync-bimanual, and 50.4% for Seq-coordinate—and reaches 40% average success on four real-world tasks.
• Ablations show that geometric imagination and 3D perception improve performance, while the remaining limitations include single-step 3D prediction, no persistent spatial memory, and low absolute success on difficult real-world tasks.

Overview

This paper addresses a persistent representational gap in bimanual imitation learning: policies built on 2D features lack explicit spatial reasoning, while point-cloud-based methods require calibration, workspace cropping, and depth sensing that are fragile outside simulation. The authors propose an RGB-only framework that uses a pre-trained 3D geometric foundation model, $\pi^3$, as the perception backbone, and couples action generation with explicit prediction of future 3D scene structure. The policy jointly denoises a future action chunk and a 3D latent that decodes into a dense pointmap, so the model must anticipate how geometry evolves under its own actions. The approach is evaluated on the RoboTwin 2.0 benchmark and on a real AgileX Cobot Magic platform, where it reports the highest average success rates among compared baselines across all three simulated task categories.

Motivation and positioning

The paper situates itself against three lines of work. First, action-chunking and diffusion policies such as ACT and Diffusion Policy improve temporal smoothness but remain "geometry-flat," operating on 2D imagery without explicit structure. Second, 3D-centric methods such as DP3 and G3Flow condition on point clouds or lifted semantic flows; these achieve better geometric reasoning but depend on calibrated cameras, fixed workspaces, digital-twin assets, or sparse back-projection pipelines that the authors argue hinder real-world deployment. Third, predictive world models that forecast future observations operate largely in 2D image space, producing predictions without guaranteed geometric consistency.

The central question posed is whether a bimanual policy can exploit a feed-forward 3D foundation model to obtain RGB-only, 3D-aware predictive control without any explicit point cloud pipeline. This distinguishes the work from concurrent efforts that use VGGT-style features for action denoising: here the geometric prior is not merely a static feature source but supports predictive 3D reasoning tied to the action sequence.

Method

The architecture follows three stages. Three parallel encoders process the inputs: a geometry encoder feeds a 6-frame temporal window (five uniformly sampled past frames plus the current frame) into the $\pi^3$ backbone, extracting concatenated features from its last two layers into a 1024-dimensional vector $\mathbf{f}_{3d}$; a semantics encoder applies DINOv3 to the current frame for $\mathbf{f}_{2d}$; and an MLP embeds the 14-dimensional proprioceptive state (two arms × [6 joints + gripper]) into $\mathbf{f}_p$. These tokens are fused by a 4-layer DETR encoder into a Semantic-Geometric Fused Context $\mathbf{f}_c$.

A conditional diffusion decoder — a standard DETR decoder following ACT's chunking strategy — then jointly denoises two heterogeneous targets: an action chunk $a_{t:t+N} \in \mathbb{R}^{N \times 14}$ and a 3D latent $\mathbf{f}_{t+N}$ that decodes into a dense pointmap $P_{t+N} \in \mathbb{R}^{H \times W \times 4}$ (XYZ coordinates plus per-point confidence). Training minimizes an L1 loss over all three components of $x_0 = \{a_{t:t+N}, \mathbf{f}_{t+N}, P_{t+N}\}$ with balancing weights $\pi^3$0 and $\pi^3$1.

Two design choices deserve emphasis. The supervision target is the latent at horizon $\pi^3$2 only, forcing the model to infer the complete scene state resulting from the entire action sequence — a "look-ahead" mechanism the authors identify as crucial for long-horizon planning. Additionally, pseudo-ground-truth latents are pre-extracted from demonstrations using a temporal observation window rather than single frames, since naive single-frame $\pi^3$3 features proved noisy and unstable. At inference, decoding of the pointmap can be skipped entirely, so the 3D prediction branch adds no deployment-time cost beyond training.

Simulation results

Experiments use RoboTwin 2.0 with 100 expert demonstrations per task, three random seeds, and 100 evaluation rollouts per seed. Tasks are grouped into 16 Dominant-select, 8 Sync-bimanual, and 8 Seq-coordinate categories.

Task category	Best baseline	Ours
Dominant-select (16 tasks)	DP3, 61.2%	63.2%
Sync-bimanual (8 tasks)	Xu et al., 47.6%	51.3%
Seq-coordinate (8 tasks)	G3Flow, 46.3%	50.4%

On Dominant-select tasks, the method surpasses the 3D-native DP3 despite using only RGB input, with the largest margins on spatially demanding tasks such as Open Laptop (88.7%) and Open Microwave (86.0%). On Sync-bimanual tasks, the most notable result is Place Dual Shoes at 43.3%, versus 17.7% for DP3, which the authors attribute to DP3's reliance on a fixed sparse point sample being inadequate for modeling simultaneous two-arm interaction. On Seq-coordinate tasks, Hang Mug yields 40.0% against 26.7% for G3Flow and 25.3% for RDT, supporting the claim that predicting future states captures multi-step dependencies better than purely reactive policies.

A data-efficiency study on four tasks with 10–100 demonstrations shows that pre-trained features provide a clear advantage in low-data regimes where 2D baselines fail almost completely — DP scores 0.0% on Scan Object with 10 demos and 0.6% with 20, while the proposed method already exhibits a learning signal at 20 demos (24.7% on Place Empty Cup).

Ablation study

Module-level ablations on four bimanual tasks isolate each component's contribution:

Configuration	Avg. success (%)
Full model	25.1
w/o 2D semantic module	24.4
w/o geometric imagination	23.6
w/o 3D module + imagination	21.0

Removing the future-pointmap ("geometric imagination") branch costs 1.5 points, while removing both 3D perception and prediction costs 4.1 points. The authors conclude that predictive pointmap reasoning is the main driver, with semantic and geometric perception providing complementary gains. It should be noted that the absolute ablation deltas are modest relative to seed variance typical of this benchmark, and the ablation covers only four tasks, so the relative weighting of components may not generalize across the full suite.

Real-world evaluation

Deployment uses an AgileX Cobot Magic system with three RealSense D435i cameras, 50 teleoperated demonstrations per task, and 20 trials per task. The method achieves a 40% average success rate across four tasks, versus 32.5% for Xu et al., 25% for DP, and 23.8% for ACT. On Hanging Mug, ACT and DP fail entirely (0%) while the proposed method reaches 20%; on Place Dual Shoes it reaches 20% against 15% for the next best. These results support RQ3 — that performance transfers beyond simulation — though the absolute success rates remain low on the hardest tasks, indicating substantial headroom even for the best method.

Limitations and open questions

The paper concedes two principal limitations. The predictive horizon is single-step: the model forecasts one future 3D state at horizon $\pi^3$4 rather than a multi-step trajectory, constraining long-horizon reasoning. There is also no persistent 3D memory, so the policy cannot accumulate state across extended interactions — a notable gap given that continuous 3D perception models with persistent state exist in the reconstruction literature. Additional caveats bear on the reported results: G3Flow was excluded from some tasks due to digital-twin asset requirements, and Xu et al. encountered NaN failures on Open Microwave, so comparisons are incomplete on those entries. Whether the pseudo-ground-truth latent extraction remains stable under domain shift between simulation and real imagery, and whether joint 3D prediction helps on tasks requiring deformable-object reasoning, are left unexamined.

Conclusion

This paper demonstrates that a pre-trained 3D geometric foundation model can serve as the sole perception prior for RGB-only bimanual manipulation, and that jointly denoising actions with a future 3D latent measurably improves success rates, inter-arm coordination, and long-horizon task performance relative to both 2D-based and point-cloud-based baselines. The strongest evidence comes from consistent top rankings across all three RoboTwin task categories and a real-world average advantage of 7.5 points over the nearest baseline. The open questions — multi-step 3D trajectory prediction and persistent geometric memory — define the natural next step for extending predictive 3D reasoning to longer-horizon bimanual autonomy.