CL3R: 3D Learning for Robotic Manipulation

• The paper’s main contribution is a 3D pre-training framework that couples point-cloud MAE with contrastive alignment to improve spatial reasoning and semantic integration.
• CL3R combines unified coordinate transformation and random multi-view fusion to enhance robustness against viewpoint variations and occlusions.
• Empirical results show that CL3R outperforms 2D and 3D baselines, yielding higher success rates in simulation and real-world robotic manipulation tasks.

CL3R, short for 3D Reconstruction and Contrastive Learning for Enhanced Robotic Manipulation Representations, is a self-supervised 3D pre-training framework for robotic visuomotor policy learning that is designed to produce point-cloud representations that are both spatially aware and semantically meaningful. It addresses two limitations attributed to recent perception modules built around pre-trained 2D foundation models: difficulty in capturing 3D spatial information and limited generalization across diverse camera viewpoints. The framework combines a point-cloud Masked Autoencoder (MAE) with contrastive alignment to pre-trained 2D vision–LLMs, while also introducing a 3D visual representation pre-training framework based on unified coordinates and random fusion of multi-view point clouds. The resulting 3D encoder is frozen and used as the vision module in imitation-learning or reinforcement-learning pipelines for robotic manipulation (Cui et al., 11 Jul 2025).

1. Conceptual objective and problem setting

CL3R is formulated around the premise that a robust perception module is crucial for visuomotor policy learning. In the reported setting, the central problem is that 2D foundation-model features provide strong semantic understanding but are insufficiently grounded in 3D geometry and are sensitive to camera-view changes, which is especially consequential in fine-grained robotic manipulation. CL3R therefore targets a joint objective: preserving precise spatial structure from point clouds while transferring high-level semantics from large 2D vision–LLMs such as CLIP (Cui et al., 11 Jul 2025).

The framework’s role in policy learning is explicitly modular. A 3D encoder is pre-trained first, then frozen, and finally integrated as the vision component of a downstream policy. Features from this encoder are combined with robot state and passed to a policy network. The stated effect is to improve spatial reasoning and semantic grounding simultaneously, which in turn accelerates downstream policy training and improves generalization to novel camera viewpoints.

The design also establishes a specific position within robotic representation learning. CL3R is neither a purely geometric depth pre-training method nor a purely semantic transfer method. Its defining claim is that 3D reconstruction and cross-modal alignment are complementary rather than interchangeable. The ablation results reported later support that interpretation.

2. Point-cloud Masked Autoencoder

The geometric core of CL3R is a point-cloud MAE operating on raw point clouds $X\in\mathbb{R}^{N\times 3}$. Patch extraction uses farthest-point sampling to choose $n$ center points, and for each center the method gathers its $k$ nearest neighbors, producing $P\in\mathbb{R}^{n\times k\times 3}$. A high mask ratio is then applied: a subset of patches is withheld as ground truth, $P_{gt}\in\mathbb{R}^{mn\times k\times 3}$, and the visible subset is $P_v\in\mathbb{R}^{((1-m)n)\times k\times 3}$, with $m$ sampled uniformly in $[60\%,80\%]$ (Cui et al., 11 Jul 2025).

Each patch is embedded into a token by a small MLP, producing $T\in\mathbb{R}^{n\times d}$ for the full set and $T_v\in\mathbb{R}^{((1-m)n)\times d}$ for visible patches, with patch-centroid positional embeddings added. The encoder $n$0 is a standard Transformer; the reported configuration is 12 layers, hidden dimension $n$1, and 8-head attention. It processes the visible tokens into latent visible features $n$2. The decoder $n$3 receives these latent features together with masked placeholder tokens, adds full positional embeddings, and reconstructs the masked patches through a small Transformer decoder followed by an MLP head, yielding $n$4.

The reconstruction objective is the $n$5 Chamfer distance between predicted and withheld patches:

$n$6

This objective is described as forcing the model to “inpaint” missing geometry from context. In the logic of the framework, the MAE component is responsible for sharpening geometric priors and improving spatial reasoning.

3. Contrastive semantic transfer

The semantic component of CL3R aligns 3D scene representations with paired RGB images and text instructions. For each 3D scene $n$7 or fused multi-view point cloud, one associated RGB image $n$8 and one text instruction $n$9 are used. Positive pairs are defined as $k$0 and $k$1, where $k$2, $k$3, and $k$4. Negative pairs are mismatched samples within the batch, such as $k$5 for $k$6 (Cui et al., 11 Jul 2025).

Similarity is defined as

$k$7

where $k$8 is a learnable temperature. For a batch of size $k$9, the image–3D loss for sample $P\in\mathbb{R}^{n\times k\times 3}$0 is

$P\in\mathbb{R}^{n\times k\times 3}$1

A symmetric term $P\in\mathbb{R}^{n\times k\times 3}$2 is also used, and the combined 3D–image contrastive loss is

$P\in\mathbb{R}^{n\times k\times 3}$3

An analogous loss is defined for text:

$P\in\mathbb{R}^{n\times k\times 3}$4

The total semantic transfer loss is

$P\in\mathbb{R}^{n\times k\times 3}$5

Within the framework, this contrastive branch is not described as replacing geometric learning. Rather, it injects “rich concepts from 2D foundation models into 3D space.” The empirical interpretation supplied by the ablations is that the contrastive term adds semantic benefit without harming geometry.

4. Unified coordinates and multi-view fusion

A distinctive element of CL3R is its 3D visual representation pre-training framework for robotic tasks, which is built around coordinate unification across datasets and random fusion of multi-view point clouds. Each camera view $P\in\mathbb{R}^{n\times k\times 3}$6 provides an extrinsic matrix $P\in\mathbb{R}^{n\times k\times 3}$7 mapping camera-frame points to a shared robot/world frame. For a point $P\in\mathbb{R}^{n\times k\times 3}$8 in view $P\in\mathbb{R}^{n\times k\times 3}$9, the transformation is

$P_{gt}\in\mathbb{R}^{mn\times k\times 3}$0

All multi-view point clouds are transformed into this unified coordinate system before MAE and contrastive training. The stated purpose is to remove 2D “left/right/up/down” ambiguities across datasets (Cui et al., 11 Jul 2025).

Random fusion is then applied during pre-training. If $P_{gt}\in\mathbb{R}^{mn\times k\times 3}$1 is the set of available views, each training iteration randomly samples $P_{gt}\in\mathbb{R}^{mn\times k\times 3}$2 views, transforms each point cloud into the world frame, and fuses them:

$P_{gt}\in\mathbb{R}^{mn\times k\times 3}$3

The reported pseudocode is:

$T\in\mathbb{R}^{n\times d}$1

This procedure is described as improving robustness to viewpoint overlap, occlusions, and missing data, while also augmenting the data and resolving multi-view alignment ambiguity. The framework attributes test-time robustness to novel viewpoints to the combination of unified world coordinates and random multi-view fusion.

5. Training configuration and downstream policy integration

The reported pre-training datasets are RH20T and RLBench. RH20T is described as a real-world dataset containing 110K trajectories over 140 tasks with multi-view RGB and depth data. RLBench is used in simulation with 9 representative pick-and-place and utensil tasks, including examples such as close-jar, move-hanger, and stack-blocks (Cui et al., 11 Jul 2025).

Optimization uses AdamW with weight decay $P_{gt}\in\mathbb{R}^{mn\times k\times 3}$4, batch size $P_{gt}\in\mathbb{R}^{mn\times k\times 3}$5 fused point clouds per GPU, and total training for 100 epochs. The learning-rate schedule uses an initial learning rate of $P_{gt}\in\mathbb{R}^{mn\times k\times 3}$6, linear warmup for 10 epochs, and cosine decay thereafter. The MAE hyperparameters are specified as $P_{gt}\in\mathbb{R}^{mn\times k\times 3}$7 patches, $P_{gt}\in\mathbb{R}^{mn\times k\times 3}$8 points per patch, mask $P_{gt}\in\mathbb{R}^{mn\times k\times 3}$9, encoder depth $P_v\in\mathbb{R}^{((1-m)n)\times k\times 3}$0, decoder depth $P_v\in\mathbb{R}^{((1-m)n)\times k\times 3}$1, and embedding dimension $P_v\in\mathbb{R}^{((1-m)n)\times k\times 3}$2. The loss weights are $P_v\in\mathbb{R}^{((1-m)n)\times k\times 3}$3 for reconstruction, $P_v\in\mathbb{R}^{((1-m)n)\times k\times 3}$4 for 3D-to-image contrast, and $P_v\in\mathbb{R}^{((1-m)n)\times k\times 3}$5 for 3D-to-text contrast, with temperature $P_v\in\mathbb{R}^{((1-m)n)\times k\times 3}$6 initialized at $P_v\in\mathbb{R}^{((1-m)n)\times k\times 3}$7 and learned.

In downstream policy learning, the feature vector $P_v\in\mathbb{R}^{((1-m)n)\times k\times 3}$8 is concatenated with robot state and fed to a 3-layer MLP with 512 units in each layer. Training is performed by behavior cloning for 150 epochs on MetaWorld, or with imitation-learning rollouts every 10 epochs on RLBench. This downstream protocol is central to the paper’s evaluation because CL3R is presented not as a complete control architecture, but as a pre-trained perception module whose quality is measured by its effect on policy performance.

6. Empirical evaluation, interpretation, and scope

The experimental evidence is reported across simulation, real-world manipulation, instruction-guided multi-task learning, viewpoint shift, and ablations. On MetaWorld, evaluated on 15 tasks partitioned into 5 easy, 5 medium, and 5 hard tasks, Lift3D achieves mean success rate (SR) $P_v\in\mathbb{R}^{((1-m)n)\times k\times 3}$9 with $m$0 across the three difficulty groups, while CL3R achieves mean SR $m$1 with $m$2. On RLBench with 4 tasks, Lift3D reaches mean SR $m$3 and CL3R reaches mean SR $m$4. The paper states that CL3R outperforms all 2D- and 3D-based baselines listed there: R3M, VC-1, PointNet++, PointNext, DP3, and RVT-2 (Cui et al., 11 Jul 2025).

In real-world evaluation on 5 RLBench tasks executed on a Franka Emika platform with RealSense sensing, the baselines are VC-1, PointNet without pre-training, and Lift3D. The reported mean SR values are CL3R $m$5, Lift3D $m$6, PointNet $m$7, and VC-1 $m$8. In multi-task instruction-guided evaluation on MetaWorld using button-press, reach, and lever-pull with 18 text prompts, the reported scores are VC-1 $m$9, R3M $[60\%,80\%]$0, CLIP $[60\%,80\%]$1, and CL3R $[60\%,80\%]$2. The paper interprets this as demonstrating effective semantic grounding via contrastive alignment.

The viewpoint-generalization experiment directly targets the camera-robustness claim. When training from the “corner” camera and testing from the unseen “corner2” viewpoint, the paper reports that 2D methods drop from approximately $[60\%,80\%]$3 to approximately $[60\%,80\%]$4 SR, whereas CL3R retains $[60\%,80\%]$5 compared with $[60\%,80\%]$6 on the original view. This is presented as evidence that the unified coordinate system and fusion scheme yield strong generalization to novel viewpoints.

The ablation study on 4 MetaWorld tasks reports mean SR $[60\%,80\%]$7 for full CL3R, $[60\%,80\%]$8 without contrastive loss, $[60\%,80\%]$9 without MAE loss, and $T\in\mathbb{R}^{n\times d}$0 without any pre-training. These results are used to support three internal claims: MAE is critical for spatial reasoning, contrastive loss adds semantic benefit without harming geometry, and pre-training improves data efficiency. A common misunderstanding would be to reduce CL3R to only one of its two pre-training signals; the ablation pattern suggests that its reported gains depend on the joint use of geometry reconstruction and semantic alignment. A plausible implication is that the framework’s strongest contribution lies in the interaction between 3D structure learning, semantic transfer, and viewpoint-invariant pre-training rather than in any single component taken in isolation.