ManiVID-3D: 3D RL for Robotic Manipulation

• The paper introduces ManiVID-3D, combining explicit 3D alignment, self-supervised disentangled feature learning, and GPU-accelerated batch rendering to address camera viewpoint variations.
• Its architecture employs ViewNet for canonical alignment and a dual-head encoder to separate view-invariant from view-dependent features, enhancing robustness under sensor shifts.
• Experimental results demonstrate a 44.7% improvement over baselines with 80% fewer parameters, validating its efficiency and sim-to-real transfer capabilities across diverse tasks.

ManiVID-3D is a 3D visual reinforcement learning architecture for robotic manipulation that is designed to learn view-invariant representations under camera viewpoint changes. It was introduced in "ManiVID-3D: Generalizable View-Invariant Reinforcement Learning for Robotic Manipulation via Disentangled 3D Representations" and published on 2025-09-14 (Li et al., 14 Sep 2025). The method targets a recurrent failure mode in real-world deployment: a policy trained from a fixed front-facing camera may fail when the camera is shifted, even though such shifts are often unavoidable in real-world settings where sensor placement is hard to manage appropriately. ManiVID-3D addresses this by combining explicit 3D coordinate alignment, self-supervised disentangled feature learning, and GPU-accelerated batch rendering. Across 10 simulated and 5 real-world tasks, it is reported to achieve a 44.7% higher success rate than state-of-the-art methods under viewpoint variations while using 80% fewer parameters (Li et al., 14 Sep 2025).

1. Problem formulation and research context

The central problem addressed by ManiVID-3D is viewpoint variation in visual RL for robotic manipulation. In the reported setting, the camera’s pose often shifts slightly, with examples including $\pm 60^\circ$ yaw, $\pm 7.5^\circ$ pitch, and $\pm 10\%$ distance. Even small viewpoint changes induce large 2D geometric distortions that standard RGB or depth-based policies cannot easily undo (Li et al., 14 Sep 2025).

Within the paper’s framing, existing remedies fall into two categories. One category requires precise extrinsic calibration, with RoboUniView given as an example (Liu et al., 2024). The other relies on heavy data-centric augmentation or MAE-style reconstruction, with ManiWhere cited as a representative instance (Yuan et al., 2024). The reported limitation is that such approaches still degrade under large viewpoint offsets greater than $45^\circ$ (Li et al., 14 Sep 2025).

ManiVID-3D’s response is explicitly geometric. The method learns an explicit 3D coordinate alignment that is intended to “undo” arbitrary extrinsic shifts without calibration, and it separates viewpoint-invariant from viewpoint-dependent features through contrastive objectives. A plausible implication is that the framework treats viewpoint variation primarily as a representation and alignment problem rather than as a purely augmentation-driven robustness problem.

2. End-to-end architecture and disentangled representation learning

The end-to-end pipeline consists of parallel simulation and rendering, ViewNet, a disentanglement encoder with dual heads, and an actor-critic policy. In the reported block diagram, parallel simulation and rendering feed point-cloud observations into ViewNet; ViewNet aligns the observation into a unified coordinate system; the aligned point cloud is processed by a disentanglement encoder; and the actor-critic policy $\pi_\theta$ outputs robot torque commands (Li et al., 14 Sep 2025).

The observation model is 3D rather than image-centric. Observations are point-cloud sequences

$O_t \in \mathbb{R}^{N \times K \times C},$

where $N$ is the number of frames, $K$ is the number of points, and $C = 3 + \mathrm{RGB}$. After ViewNet alignment, the encoder $f_\theta$ branches into two components: a view-invariant head $\pm 7.5^\circ$0 that produces $\pm 7.5^\circ$1, and a view-dependent head $\pm 7.5^\circ$2 that produces $\pm 7.5^\circ$3. The policy consumes only $\pm 7.5^\circ$4, optionally concatenated with proprioception, to output actions (Li et al., 14 Sep 2025).

The disentanglement mechanism is enforced by three losses: InfoNCE for invariant features, InfoNCE for dependent features, and an orthogonality loss. The paper’s formulation therefore separates the representation used for control from the representation used to preserve viewpoint-dependent information. This suggests that viewpoint robustness is obtained not by erasing all camera information from the input pipeline, but by isolating the task-relevant latent that is passed to the policy.

3. ViewNet and canonical alignment in 3D

ViewNet is the architectural component that performs single-view point-cloud alignment into a unified “canonical” frame with zero extrinsic calibration. Its backbone is PointNet++, and it ingests

$\pm 7.5^\circ$5

A regression head predicts SIM(3) parameters: scale $\pm 7.5^\circ$6, rotation $\pm 7.5^\circ$7, and translation $\pm 7.5^\circ$8 (Li et al., 14 Sep 2025).

The alignment transform is defined as

$\pm 7.5^\circ$9

Training uses synthetic triplets $\pm 10\%$0. The reported loss combines an MSE term and a Chamfer term:

$\pm 10\%$1

and

$\pm 10\%$2

The full objective is

$\pm 10\%$3

This design couples supervised alignment to a geometric set-matching criterion, with $\pm 10\%$4 and $\pm 10\%$5 serving distinct supervisory roles (Li et al., 14 Sep 2025).

In the paper’s larger argument, ViewNet is the mechanism that removes the need for extrinsic calibration at deployment. That claim is important for sim-to-real transfer because it shifts the burden of camera handling from manual calibration to learned canonicalization.

4. GPU-accelerated rendering and reinforcement learning objective

The rendering subsystem is built on ManiSkill3’s parallel simulation (Tao et al., 2024). The paper batches $\pm 10\%$6 environments, each with $\pm 10\%$7 cameras, and renders depth images that are converted into point clouds. The preprocessing sequence in the pseudocode is crop, plane-remove, filter, and FPS, after which the point cloud is aligned by ViewNet and encoded by the disentangler (Li et al., 14 Sep 2025).

On an NVIDIA RTX 4090, with $\pm 10\%$8 environments, $\pm 10\%$9 cameras, and $45^\circ$0 K points per environment, the reported performance is

• $45^\circ$1 Hz
• total throughput $45^\circ$2 fps

The paper characterizes this as a GPU-accelerated batch point-cloud renderer achieving greater than 5000 fps throughput across hundreds of parallel environments (Li et al., 14 Sep 2025). Within the architecture, this throughput is not an auxiliary systems result; it is presented as a precondition for large-scale, high-throughput 3D RL training.

The RL problem is formulated as an MDP with unobserved simulator state space $45^\circ$3, observation function $45^\circ$4 frames of point clouds in $45^\circ$5, continuous action space $45^\circ$6 consisting of joint torques or end-effector commands, task-specific sparse or dense reward $45^\circ$7, MuJoCo transition dynamics $45^\circ$8, and discount $45^\circ$9. The goal is to maximize

$\pi_\theta$0

The policy network takes $\pi_\theta$1 as input, uses a 2- or 3-layer MLP head to map $\pi_\theta$2 to $\pi_\theta$3 and $\pi_\theta$4, and uses a separate critic $\pi_\theta$5 implemented as an MLP over $\pi_\theta$6. The RL objective is given in Soft Actor-Critic style:

$\pi_\theta$7

(Li et al., 14 Sep 2025)

5. Experimental results and benchmark comparisons

The evaluation covers 10 simulation tasks and 5 real-world tasks. In simulation, the robots are AIRBOT Play for 5 tasks, UR5 for 3 tasks, and Franka for 2 dex tasks. Testing views use yaw $\pi_\theta$8, pitch $\pi_\theta$9, and distance $O_t \in \mathbb{R}^{N \times K \times C},$0--$O_t \in \mathbb{R}^{N \times K \times C},$1. The baselines listed are Maniwhere, Maniwhere+3D-STN, ReViWo, MV-MWM, SRM, and MoVie (Li et al., 14 Sep 2025).

Method	Average success rate across 10 tasks
Maniwhere	88.9%
Maniwhere+3D	82.2%
ReViWo	57.6%
MV-MWM	55.9%
SRM	12.1%
MoVie	8.3%
ManiVID-3D	94.5%

From these results, the paper reports a 44.7% average improvement over the set of baselines, and $O_t \in \mathbb{R}^{N \times K \times C},$2 over Maniwhere (Li et al., 14 Sep 2025). It also reports that ManiVID-3D’s success varies by less than $O_t \in \mathbb{R}^{N \times K \times C},$3 when yaw offsets grow from $O_t \in \mathbb{R}^{N \times K \times C},$4 to $O_t \in \mathbb{R}^{N \times K \times C},$5, whereas Maniwhere drops approximately $O_t \in \mathbb{R}^{N \times K \times C},$6 under the same shift. The stated interpretation is robustness to severe perspective changes.

Efficiency metrics are likewise emphasized. The encoder is reported to have 84.4% fewer parameters than Maniwhere, with inference latency reduced by 52.6%, training time to convergence reduced by 39.1%, and FLOPs reduced by 69.8% (Li et al., 14 Sep 2025). These values place the method’s contribution simultaneously in generalization and computational efficiency.

The paper also includes a comparison with 3D imitation learning on the cube-lift task. DP3, described there as a 3D diffusion policy IL method, achieves $O_t \in \mathbb{R}^{N \times K \times C},$7, whereas ManiVID-3D achieves $O_t \in \mathbb{R}^{N \times K \times C},$8 (Ze et al., 2024). In real-world transfer on 5 AIRBOT tasks, evaluated under 4 viewpoint variations and 4 scene-variation scenarios, Maniwhere records 79 successes out of 140 trials (56.4%), while ManiVID-3D records 103 successes out of 140 trials (73.6%), corresponding to $O_t \in \mathbb{R}^{N \times K \times C},$9 (Li et al., 14 Sep 2025).

6. Sim-to-real transfer, ablations, and nomenclatural distinction

For zero-shot real-world deployment, the paper states that no extrinsic calibration is required because ViewNet aligns real-camera point clouds to the same canonical frame as simulation. It further states that multi-stage preprocessing—crop, plane removal, outlier filter, and normalization—closes the sim-real gap purely in 3D (Li et al., 14 Sep 2025). This places the transfer mechanism in the geometry-processing stack rather than in camera calibration or image-domain adaptation.

The ablation studies are reported on three tasks: Cube Lift, Pick & Place, and Laptop Close. Removing ViewNet reduces average success to 57.1%, which the paper characterizes as a catastrophic drop. Removing the invariant contrastive loss yields 87.9%, removing the dependent and orthogonal losses yields 89.5%, and replacing ViewNet with 3D-STN yields 78.1% (Li et al., 14 Sep 2025). The stated conclusion is that both ViewNet and the disentanglement objectives are essential for robust cross-view performance.

A recurrent naming confusion is possible because ManiVID-3D and ManiVideo are distinct systems. ManiVID-3D is a 3D visual RL architecture for robotic manipulation via disentangled 3D representations (Li et al., 14 Sep 2025). By contrast, ManiVideo is a diffusion-based framework designed to synthesize temporally coherent, 3D-consistent bimanual hand-object manipulation videos conditioned on driving signals such as hand pose parameters and object trajectories (Pang et al., 2024). This suggests that the two works occupy different research categories: policy learning and control in the former, conditional video generation in the latter.