Deform360: A Massive Multi-view Visuotactile Dataset for Deformable World Models

Abstract: Predicting object dynamics (i.e., world modeling) is a fundamental challenge for robotic manipulation, and modeling deformable objects presents a particularly difficult case due to their high-dimensional state spaces and complex material properties. While current world models approach this through two distinct paradigms: learning the dynamics over the 2D pixel space or more explicit 3D geometric space. A systematic understanding of their relative strengths and limitations remains elusive due to the lack of diverse, large-scale real-world data. To address this, we present Deform360, a large-scale visuotactile dataset featuring 198 daily-life objects, 1,980 interaction sequences, and over 215 hours of observations from 41 surround-view cameras and bimanual tactile grippers to capture both global motion and contact-induced local deformations. Leveraging a novel markerless visuotactile 3D tracking pipeline to extract dense geometry and motion, we systematically evaluate current state-of-the-art world models, comparing 2D video models against 3D particle models. Finally, we provide a preliminary demonstration indicating the real-world applicability of our dataset by performing robot planning tasks on deformable objects. Our analysis reveals key insights into the trade-offs between structural priors and scalability, providing a solid benchmark for future research in generalizable deformable object-centric world modeling. Project website: https://deform360.lhy.xyz

• The paper presents Deform360, a large-scale dataset capturing 41-camera views and tactile signals across 198 deformable objects over 215 hours of interaction.
• The study introduces a markerless tracking pipeline that integrates 3D Gaussian Splatting, 2D feature tracking, and multi-view alignment to ensure precise deformation tracking even under occlusions.
• The paper benchmarks 2D versus 3D world models, revealing trade-offs between geometric supervision and large-scale visual pretraining, with practical implications for real-world robot planning.

Deform360: A Comprehensive Multi-View Visuotactile Dataset for Deformable World Models

Motivation and Dataset Overview

Robotic manipulation of deformable objects poses significant challenges due to high-dimensional state spaces, non-linear material properties, and frequent occlusions during interaction. The "Deform360" dataset (2607.05390) directly addresses these issues by providing a large-scale, multi-modal dataset featuring synchronized multi-view (41 cameras) and tactile recordings from 198 diverse real-world deformable objects across 1,980 interaction sequences and over 215 hours of data. Deform360 captures both global object motion and contact-induced local deformations, supporting research on both 2D and 3D object-centric world models, contact detection, and real-world robot manipulation tasks.

Figure 1: Deform360 dataset overview, demonstrating the scale and diversity of objects and the multi-view visuotactile capture setup.

Acquisition System and Object Taxonomy

The Deform360 capture system deploys 41 spatially calibrated RGB cameras for dense, surround-view observation at 720p and 30 FPS, paired with bimanual UMI grippers instrumented with tactile sensors. The protocol encompasses a spectrum of unimanual and bimanual tasks—poking, squeezing, stretching, folding, twisting—designed to exercise both global motion and fine-grained local deformations. Object diversity is a central strength, spanning:

• 1D deformables: ropes, cables, threads
• 2D deformables: cloths, garments, bags, papers
• 3D volumetric deformables: plush toys, foam, squeezable objects

  Figure 2: Visualization of 1D deformables such as various ropes, cables, and wire-like objects.

  

  Figure 3: Representative 2D deformables including fabrics, bags, and paper-like thin shells.

  

  Figure 4: Selected 3D volumetric deformables encompassing plush toys, foams, and squeezables.

This taxonomy facilitates research in both intra- and cross-class dynamic modeling, and the dataset’s scale surpasses prior real-world benchmarks in both sensory richness and annotated interactions.

Markerless Visuotactile Tracking Pipeline

To generate ground-truth dense geometry and motion annotations in the presence of heavy occlusions, the authors introduce a markerless tracking pipeline combining per-frame 3D Gaussian Splatting (GS), robust 2D feature tracking, and multi-view geometry alignment.

Figure 5: Annotation pipeline — per-frame 3D Gaussian Splatting, markerless 2D tracking, 2D-to-3D lifting, and cross-view/tactile consistency optimization.

This sequence decouples high-fidelity rendering from physical tracking, enabling the enforcement of:

• Temporal consistency and local rigidity (ARAP constraints)
• Spatial smoothness and tactile plausibility (via physically-informed losses)
• Geometric accuracy (bidirectional Chamfer losses against reconstructed surfaces)

Critically, tactile signals are leveraged as a supervisory signal to regularize particle trajectories during occluded contact events, resulting in persistent object-centric particle identities even under severe deformations.

Figure 6: Qualitative visualization of the particle tracking system across object categories, showing robust tracking under interaction.

Figure 7: Benefit of visuotactile integration—tracking with tactile input yields lower error and preserves fidelity under severe occlusion compared to vision-only baselines.

Empirically, visuotactile fusion reduces point cloud Chamfer error by 5$\times$ under occlusions, validating the importance of synchronized force sensing.

Benchmarking World Models: 2D vs. 3D

To systematically probe modeling capabilities, the authors formulate three core tasks: contact prediction, multi-modal world model benchmarking, and real robot planning. Two major paradigms are compared:

• 2D action-conditioned video models (e.g., Cosmos-Predict 2.5B): Direct latent-space video prediction, leveraging large-scale internet pre-training for generalization.
• 3D particle-based models (e.g., PhysTwin, PGND, ParticleFormer): Explicit geometry prediction via learning (GNNs, Transformers) or differentiable physical simulation.

Evaluation is staged under three generalization settings:

1. Per-episode (frame): Intra-sequence forecasting.
2. Multi-episode: Cross-sequence, same-object generalization.
3. Multi-object: Zero-shot to novel objects.

Key results and numerical claims:

• In the low-data per-episode regime, PhysTwin (differentiable simulation with strong priors) yields the lowest forecasting errors (CD, Track Error), outperforming learned particle models.
• Multi-episode generalization: Cosmos achieves superior image reconstruction (PSNR, SSIM), but ParticleFormer dominates in future prediction, demonstrating 3D structure is crucial for temporal extrapolation.
• Multi-object (zero-shot): Cosmos significantly surpasses all 3D methods in image quality metrics for unseen categories, a consequence of pre-training scale and diverse visual inductive biases.

  Figure 8: Multi-object generalization — predicted future frames for cable and bubble-wrap, highlighting comparative strengths across world models.

  

Figure 9: Dynamic reconstruction examples—novel view synthesis at different time steps for rope and cloth sequences.

Figure 10: Multi-episode generalization on glove-cloth and sack-cloth, illustrating cross-episode predictive robustness.

These benchmarks reveal a fundamental trade-off between inductive structure and scaling: 3D models provide strong priors and exploit geometric supervision in low-data settings, while 2D models exploit large-scale visual pretraining to generalize better on novel categories but have limited controllability and policy integration in open-loop settings.

Practical Robotics and Contact Prediction

Beyond world model evaluation, Deform360 enables downstream robotic planning. Models trained on Deform360 (notably PhysTwin) were deployed in a zero-shot manner on a different robot platform for model-predictive control (MPC) of real deformables, demonstrating the data’s cross-robot transfer potential.

Figure 11: Real robot planning via MPC — learned 3D world models guide deformable object manipulation in physical setups different from dataset capture.

Additionally, the dataset’s synchronized visuotactile streams permit high-accuracy contact prediction ($88.67\%$ mean accuracy, $0.8909$ F1), quantifying the learnability of tactile events from exteroceptive streams and confirming the tight visual-physical coupling in the dataset.

Limitations and Future Research Directions

Despite its scope, Deform360 faces inherent challenges:

• Severe self-occlusion and large plastic deformations remain difficult for vision-based tracking.
• Tactile sensors only capture normal-axis pressure, limiting slip detection and rich surface force recovery.
• Current large-scale video models exhibit action-command drift in long rollouts, likely due to action encoding deficiencies and lack of sufficiently action-conditioned pretraining.

Implications: The dataset sets the stage for several future avenues:

• Design of foundation 3D world models leveraging large-scale pretraining and hybrid physics learning
• Scaling multi-modal policy learning across robot platforms and interaction types
• Advanced tactile/force sensing enabling richer contact inference and model fidelity

Conclusion

Deform360 sets a new standard for deformable object world modeling by coupling large-scale, high-resolution multi-view video with dense, markerless 3D tracking and tactile data across an unprecedentedly diverse set of daily-life objects. Through systematic benchmarking, it reveals the scalability-structure trade-off in current world models, highlights the necessity of tactile supervision, and enables deployment of learned models in real-world robotic planning tasks. Its design and empirical insights provide a robust foundation for next-generation research aiming at physically grounded, generalizable, and robot-ready world models in deformable manipulation (2607.05390).