PointWorld: Scaling 3D World Models for In-The-Wild Robotic Manipulation

Abstract: Humans anticipate, from a glance and a contemplated action of their bodies, how the 3D world will respond, a capability that is equally vital for robotic manipulation. We introduce PointWorld, a large pre-trained 3D world model that unifies state and action in a shared 3D space as 3D point flows: given one or few RGB-D images and a sequence of low-level robot action commands, PointWorld forecasts per-pixel displacements in 3D that respond to the given actions. By representing actions as 3D point flows instead of embodiment-specific action spaces (e.g., joint positions), this formulation directly conditions on physical geometries of robots while seamlessly integrating learning across embodiments. To train our 3D world model, we curate a large-scale dataset spanning real and simulated robotic manipulation in open-world environments, enabled by recent advances in 3D vision and simulated environments, totaling about 2M trajectories and 500 hours across a single-arm Franka and a bimanual humanoid. Through rigorous, large-scale empirical studies of backbones, action representations, learning objectives, partial observability, data mixtures, domain transfers, and scaling, we distill design principles for large-scale 3D world modeling. With a real-time (0.1s) inference speed, PointWorld can be efficiently integrated in the model-predictive control (MPC) framework for manipulation. We demonstrate that a single pre-trained checkpoint enables a real-world Franka robot to perform rigid-body pushing, deformable and articulated object manipulation, and tool use, without requiring any demonstrations or post-training and all from a single image captured in-the-wild. Project website at https://point-world.github.io/.

• The paper introduces a pre-trained, embodiment-agnostic 3D world model that unifies scene state and robot actions through dense point clouds.
• It employs a neural mapping and PointTransformerV3 architecture to predict per-point scene evolution from temporal robot actions using a massive dataset of over 2M trajectories.
• The model achieves robust zero-shot and finetune transfer across diverse robots, supporting real-time manipulation tasks in both simulated and real-world environments.

PointWorld: Scaling 3D World Models for In-The-Wild Robotic Manipulation

Introduction and Motivation

Robust world modeling underpins effective general-purpose robotic manipulation, particularly in unstructured, in-the-wild environments where perception and action are tightly coupled in 3D. "PointWorld: Scaling 3D World Models for In-The-Wild Robotic Manipulation" (2601.03782) introduces a large-scale, pre-trained 3D world model, $\text{\algo}$, that unifies environment state and robot action in the modality of 3D point flows. Unlike prior techniques that entangle embodiment specification with action representation, PointWorld leverages an embodiment-agnostic framework: both scene and agent are encoded as dense or structured 3D point clouds, enabling seamless conditioning and generalization across robots and tasks. The model predicts per-point scene evolution directly in response to a temporal sequence of robot actions, making no assumptions about objectness, material priors, or full observability.

This approach is fundamentally driven by the need for a generalizable, physically-grounded dynamics model operable from partial, raw sensory data and scalable across a variety of manipulation tasks. By leveraging advances in 3D vision, large model architectures, and massive data curation, the work makes it feasible to deploy model-predictive planning and real-time action inference directly from in-the-wild RGB-D captures.

Model Architecture and Action-State Parameterizations

The central formulation is multi-step action-conditioned point flow prediction over a finite horizon, using a neural mapping

$$\mathcal{F}^{H}_\theta : (\mathbf{s}_t, \mathbf{a}_{t:t+H-1}) \rightarrow \mathbf{s}_{t+1:t+H}$$

where $\mathbf{s}_t$ represents the environment’s state as a dense 3D point cloud, and actions $\mathbf{a}_{t:t+H-1}$ are robot point flows computed via forward kinematics from a robot URDF and a temporal sequence of joint-space commands.

PointWorld’s architectural pipeline transforms each input as follows. The scene is reconstructed from calibrated RGB-D and robot masks, producing a partial world-centric point set. Actions are sampled as prospective robot contact points (typically targeting grippers) and temporally stacked. The concatenated scene-action point cloud is then featurized using a frozen DINOv3 encoder for scene points, and temporal embeddings for robot points. The backbone, primarily PointTransformerV3 (PTv3), operates hierarchically with multi-head attention and local aggregation, yielding highly expressive representations for downstream per-point 3D flow prediction.

Figure 1: Overview of \algo’s action-conditioned 3D perception-processing pipeline—robot actions and scene are unified in a single point cloud, enabling geometry-aware world modeling.

Action representations as dense 3D gripper flows are shown to outperform global pose, joint-space, and sparse whole-body alternatives in both real and simulated multi-embodiment settings. This design directly encodes interaction geometry, supports partial observability, and mediates positive transfer across robots with heterogeneous morphologies.

Figure 2: Action representations as gripper point flows enable efficient, robust contact modeling and transfer across different robot bodies.

Dataset Curation and Annotation Pipeline

A core innovation is the construction of a massive action-conditioned 3D dynamics dataset, aggregating ∼2M trajectories and 500+ hours from open-world, teleoperated manipulation (DROID, ∼200 hours, and BEHAVIOR-1K, ∼1100 hours, after filtering) in both real and simulated domains. To enable high-fidelity labels, the work pioneers a fully markerless annotation pipeline that extracts robust depth and camera pose using FoundationStereo, refines extrinsics with mesh-depth alignment, and obtains dense correspondence fields from CoTracker3. This yields high-precision 3D point flow annotations even with complex, articulated, and deformable object interactions.

Figure 3: The annotation pipeline achieves high geometric alignment and annotation accuracy, outperforming previous depth/extrinsics releases.

This curated dataset supports rigorous empirical evaluation of backbone architectures, action representations, and scaling trends.

Training Objectives and Regularization

Training such a world model introduces two primary challenges: (1) highly imbalanced point motion—most points are static, and (2) substantial noise in real-world supervision. Addressing these, PointWorld employs a weighted regression objective where mover likelihoods focus the loss on points with significant displacement, thus amplifying signal-to-noise on informative interactions. Additionally, the model predicts an aleatoric log-variance per point to regularize supervision based on estimated reliability, further stabilized by a Huber loss over 3D residuals.

Figure 4: Movement weighting and uncertainty regularization balance training signal and prevent overfitting to unreliable data, with uncertainty further capturing action-conditioned physical variability.

Scaling Laws, Backbone Analysis, and Ablation

Systematic architectural and scaling ablations demonstrate that PTv3 delivers superior capacity and accuracy when extrapolated to order-of-magnitude larger parameter regimes. Model performance scales log-linearly with both the amount of data and the number of parameters (to ∼1B), mirroring known scaling laws in vision and language domains.

Figure 5: Roadmap showing consistent improvements from backbone modernization, stabilized objectives, pretrained features, and parameter scaling.

(Figure 6, right)

Figure 6 (right): Empirical scaling law—both dataset size and model capacity induce predictable, log-linear accuracy gains.

Further, chunked prediction (predicting $H$ steps per pass) leads to lower rollout drift and higher compute efficiency compared to both teacher-forced and autoregressive baselines. The model is robust to partial observability; augmenting with multiple views and randomized camera sampling in training confers generalization to variable input vantage.

Figure 7: Chunked prediction yields both lower rollout drift and substantial computational savings over stepwise autoregression.

Figure 8: Performance is robust across varying camera counts: multiple views and randomization at train/inference time further drive generalization.

Generalization, Zero-Shot & Finetune Transfer

PointWorld consistently generalizes both within and across domains. In zero-shot evaluation, models pretrained on DROID or jointly with BEHAVIOR-1K transfer to unseen real-world environments (e.g., out-of-distribution DROID labs) with only a minor performance drop, closing the majority of the gap with domain-specific specialists. Cross-domain finetuning (20× fewer updates) quickly matches or exceeds from-scratch specialist baselines. Real-to-sim transfer is more effective than sim-to-real, but joint (real+sim) pretrained models exhibit the best overall generalization.

Figure 9: Zero-shot and finetuned generalization to new lab environments—pretrained models close the specialist gap and benefit from additional finetuning.

Model-Based Manipulation and Real-World Deployment

PointWorld enables real-time (0.1s per forward pass) deployment as a drop-in model-predictive dynamics prior in MPC-based manipulation frameworks. With no additional demonstrations, reward engineering, or adaptation, a single model supports diverse manipulation tasks (rigid pushing, deformation, articulation, tool use) in previously unseen scenes, driven by high-dimensional 3D cost specification defined over the world model’s state.

(Figure 6, left)

Figure 6 (left): Zero-shot real-world deployment: \algo supports rigid, deformable, articulated, and tool-use tasks, showing high task success rates with no additional retraining.

Figure 10: Example rollouts from a single pretrained model on diverse domains with accurate dynamics predictions over rigid, deformable, and articulated objects.

Limitations and Future Directions

Current limitations include the quasi-static initial state assumption (no histories or exogenous dynamics), lack of explicit physical priors, and focus exclusively on geometric, rather than photometric, dynamics. The framework does not yet model robot embodiment deformations or actuation variability, instead assuming accurate forward-kinematic realizations. Annotating fine-scale or small objects remains challenging due to depth and calibration errors.

Future lines of work include integrating explicit physics constraints, coupling with appearance dynamics models (e.g., NeRF/Gaussians), supporting dynamic initializations, and extending to soft or compliant robots. Automatically specifying complex multi-stage cost/reward via VLMs or demonstration is a promising research avenue.

Conclusion

PointWorld presents a scalable, high-capacity, and physically-grounded approach to 3D environment modeling for robotic manipulation. By fusing massive high-fidelity data, expressive point-based architectures, robust training objectives, and embodiment-agnostic action encoding, the method achieves strong quantitative and qualitative results, including practical zero-shot deployment in the real world. This work demonstrates that representation- and data-centric scaling, combined with robust annotation, are essential levers for generalizable world models in robotics, laying groundwork for future advances in unified, action-conditioned 3D simulation and planning.