Deform360: Visuotactile Dataset & Benchmark

• Deform360 is a large-scale, multimodal dataset designed for evaluating deformable world models with synchronized 360° video, tactile sensing, and dense 3D tracking.
• It benchmarks two paradigms by comparing scalable 2D video diffusion models with explicit 3D particle-based representations for action-conditioned prediction and planning.
• The dataset addresses key challenges in deformable object modeling by providing high-fidelity 3D supervision, diverse object interactions, and integrated tactile feedback for improved physical reasoning.

Searching arXiv for the named work and closely related deformable-object world-modeling papers. arxiv_search(query="ti:\"Deform360\" OR abs:\"Deform360\" deformable world models visuotactile dataset", max_results=10, sort_by="submittedDate") arxiv_search(query="deformable object world model tactile dataset particle model video model robotics", max_results=10, sort_by="relevance") arxiv_search(query="Deform360: A Massive Multi-view Visuotactile Dataset for Deformable World Models", max_results=5, sort_by="relevance") Deform360 is a large-scale real-world visuotactile dataset and benchmark for deformable object-centric world modeling, introduced in “Deform360: A Massive Multi-view Visuotactile Dataset for Deformable World Models” (Li et al., 6 Jul 2026). It was designed for action-conditioned prediction of deformable object dynamics under robot interaction, with synchronized surround-view video, tactile sensing, dense 3D tracking, and downstream robot-planning evaluation. Its central scientific purpose is comparative: Deform360 enables a direct study of two major paradigms—learning dynamics in 2D pixel space and learning dynamics in explicit 3D geometric or particle space—under shared real-world data, supervision, and evaluation protocols.

1. Motivation and conceptual scope

Deform360 addresses a specific bottleneck in robotic world modeling: deformable objects have effectively high- or infinite-dimensional state, strong occlusions, and contact-induced local deformations. The paper situates this difficulty against a data problem as much as a modeling problem. Prior datasets are described as too small, too narrow in object diversity, synthetic, or missing crucial annotations such as multi-view 3D geometry and tactile contact signals.

The dataset was therefore constructed to overcome three limitations identified explicitly in the literature summary associated with the work: lack of scale, lack of multimodal observations, and lack of high-fidelity 3D supervision. In that formulation, Deform360 is not only a corpus of recordings but also an experimental instrument for benchmarking competing inductive biases. The benchmark compares 2D pixel-space or video world models with 3D explicit geometric or particle-based world models, and it does so on real deformable interactions rather than on synthetic rollouts or narrowly scoped object families (Li et al., 6 Jul 2026).

This framing gives Deform360 a dual identity. At the data level, it is a large multimodal capture effort over daily-life deformable objects. At the methodological level, it is a benchmark for studying the trade-off between structural priors and scalability. The paper’s own conclusion is nuanced rather than unilateral: explicit 3D priors and large-scale 2D video models are both valuable, but they dominate in different regimes.

2. Dataset composition and sensing configuration

The dataset scale is unusually large for real-world deformable manipulation. It contains 198 unique deformable objects, 1,980 interaction sequences, 23.3 million total frames, and 215.7 hours of cumulative multi-view footage. Capture is performed with 41 calibrated camera views at 30 FPS and $720 \times 1280$ RGB resolution per camera. After filtering out streams with sync or frame-loss issues, the paper reports 74,850 raw videos, with an average episode duration of 10.34 seconds (Li et al., 6 Jul 2026).

Component	Value
Objects and sequences	198 unique deformable objects; 1,980 interaction sequences
Visual capture	41 calibrated camera views; 30 FPS; $720 \times 1280$ RGB per camera
Aggregate scale	23.3 million total frames; 215.7 hours; 74,850 raw videos after filtering

The objects span 17 semantic categories and are grouped into three deformability classes. The 1D class contains 28 objects, including ropes, cables, wires, straps, and chains. The 2D class contains 98 objects, including cloth, fabric, garments, paper-like items, bags, and sheets. The 3D volumetric class contains 72 objects, including plush toys, stuffed animals, foam objects, squeezables, and slime-like objects. The paper emphasizes that these classes expose different modeling challenges: 1D objects involve knotting, self-contact, and long-range shape dependencies; 2D objects involve folding, wrinkling, and plasticity; 3D volumetric objects involve internal stress, volume change, and nonlocal deformation.

Each interaction sequence includes synchronized tactile streams from bimanual tactile-equipped UMI grippers, aligned with the visual stream at 30 Hz. Episodes also record robot proprioception, specifically 6D pose and gripper openness. The interaction repertoire includes unimanual and bimanual manipulation. The main summary mentions approximately 5 unimanual and 5 bimanual episodes per object, while the benchmark table lists 13 primitives: bend, close, drag, fold, lift, open, poke, press, roll, squeeze, stretch, twist, and wave.

The capture rig is explicitly described as being designed for 360° observability. That design choice is substantive rather than cosmetic: global object motion and local contact-induced deformation are both required for meaningful deformable-world supervision, and neither is reliably recoverable from sparse or monocular sensing alone.

3. Markerless visuotactile 3D tracking pipeline

A major technical contribution of Deform360 is its annotation pipeline, which converts raw synchronized multi-view video and tactile streams into dense 3D geometry and motion. The pipeline is markerless and visuotactile. Its architecture decouples per-frame geometry reconstruction from temporal tracking and identity preservation.

Preprocessing begins with calibration using ArUco grids. RGB streams are undistorted, and gripper pose and gripper opening are tracked using ArUco markers on the wrist and fingers. Multi-view marker estimates are fused with RANSAC. For geometry reconstruction, the paper uses 3D Gaussian Splatting on each frame rather than as the final tracking representation. Each Gaussian $k$ is parameterized by a mean $\boldsymbol{\mu}_k$, covariance $$\boldsymbol{\Sigma}_k = \mathbf{R}_k \mathbf{S}_k \mathbf{S}_k^T \mathbf{R}_k^T$$, opacity $\alpha_k$, and spherical harmonic color coefficients. The per-frame optimization uses

$$\mathcal{L}_{\text{gs}} = (1 - \lambda_{\text{gs}})\mathcal{L}_1 + \lambda_{\text{gs}}\mathcal{L}_{\text{SSIM}},$$

with $\lambda_{\text{gs}} = 0.2$ in the implementation.

Temporal correspondence is then established with CoTracker3, which tracks up to 1,600 grid points on the segmented object mask per view over 15-frame clips with stride 5. These 2D tracks are lifted to 3D using rendered depth from 3DGS:

$$\mathbf{P}_{n,t} = \mathbf{E}_n^{-1} \mathbf{D}_{n,t}(\mathbf{u}_{n,t}) \mathbf{K}_n^{-1} \tilde{\mathbf{u}}_{n,t}.$$

The lifted 3D estimates from multiple views are fused with RANSAC into a consistent global velocity field.

The final stage is physics-informed refinement with tactile supervision. The overall tracking objective is

$$\mathcal{L}_{\text{track}} = \mathcal{L}_{\text{shape}} + \lambda_{\text{local}} \mathcal{L}_{\text{local}} + \lambda_{\text{lap}} \mathcal{L}_{\text{lap}} + \lambda_{\text{tactile}} \mathcal{L}_{\text{tactile}},$$

with $720 \times 1280$0, $720 \times 1280$1, and $720 \times 1280$2. The shape term is a bidirectional Chamfer distance to the next-frame 3DGS point cloud. The local term is an ARAP-like local rigidity constraint. The Laplacian term enforces smoothness of the velocity field. The tactile term,

$720 \times 1280$3

encourages particles near active taxels to move consistently with measured contact motion and functions as a soft no-slip regularizer.

The paper emphasizes that tactile feedback is especially valuable under gripper-induced or self-occlusion. That is the reason Deform360 is formulated as visuotactile rather than purely visual.

4. Benchmark design and evaluated world-model paradigms

Deform360 benchmarks two classes of action-conditioned world models. On the 2D side, the paper evaluates Cosmos Predict 2.5 2B, a large pretrained video diffusion model. Because Cosmos does not natively accept robot actions, it is post-trained on Deform360 with action conditioning injected through the robot’s 7D action sequence—6D wrist pose plus gripper openness—via cross-attention in the DiT blocks (Li et al., 6 Jul 2026).

On the 3D side, the benchmark includes PGND, ParticleFormer, and PhysTwin. These methods operate on explicit particle states or differentiable physical simulation. The comparison is organized around three generalization settings: per-episode or frame generalization, multi-episode or episode generalization, and multi-object or object generalization. The first setting trains on early frames in one episode and predicts later frames from that same episode. The second trains on some episodes of the same object and tests on unseen episodes. The third trains on some objects and tests zero-shot on unseen object instances.

Evaluation uses both 3D and 2D metrics. The 3D metrics are Chamfer distance and track error. The 2D metrics are PSNR, SSIM, and LPIPS. The paper explicitly notes that 3D models are also evaluated by rendering predicted trajectories back into images, which makes the comparison cross-representational rather than representation-isolated.

This design makes Deform360 a controlled benchmark for asking whether better deformable prediction arises from stronger structure or from larger-scale generative pretraining. A plausible implication is that the benchmark was constructed to avoid rewarding only one representational family.

5. Empirical findings and revealed trade-offs

The empirical results support a clear trade-off between structural priors and scalability. In low-data regimes, 3D models dominate. On per-episode generalization, PhysTwin performs best in both reconstruction and prediction, with reported values of $720 \times 1280$4 for Chamfer distance and $720 \times 1280$5 for track error. The paper interprets this as evidence that strong 3D and physics-based priors are especially effective when only a short trajectory is available for adaptation.

As the amount of data broadens across episodes or objects, the picture changes. In the multi-episode setting, Cosmos achieves the strongest image-space reconstruction quality, with PSNR $720 \times 1280$6 and LPIPS $720 \times 1280$7, whereas ParticleFormer often gives stronger future dynamics prediction than the other 3D baselines. In the multi-object setting, Cosmos attains the best visual generalization to unseen object categories, with PSNR $720 \times 1280$8 and LPIPS $720 \times 1280$9, while ParticleFormer remains substantially stronger on 3D geometry metrics, with Chamfer distance $k$0 and error $k$1 (Li et al., 6 Jul 2026).

The benchmark also reports a notable failure mode for the large video model: action misalignment. The model can generalize visually yet fail to obey robot commands over long horizons with sufficient fidelity. By contrast, 3D particle models remain more grounded in physical state, even when they lag in zero-shot visual realism.

The quality of the Deform360 annotations is quantified independently of the world-model results. The paper reports global-average reconstruction quality of PSNR $k$2, SSIM $k$3, and LPIPS $k$4. A tactile ablation shows that visuotactile tracking reduces Chamfer error from $k$5 in the visual-only variant to $k$6, described as about a $k$7 improvement. The paper also trains a transformer for contact prediction from visual observations and robot action, obtaining mean accuracy $k$8 and F1-score $k$9, compared with a random baseline of $\boldsymbol{\mu}_k$0. These results indicate that the visual-tactile coupling captured by the dataset is not incidental; it is learnable and quantitatively useful.

6. Robot planning relevance, limitations, and scientific significance

Beyond passive prediction, Deform360 includes a preliminary real-world robot-planning demonstration. Learned models are deployed in a Model Predictive Control framework on an xArm in a different lab, under a zero-shot transfer setting relative to the new robot environment. The model used for planning is PhysTwin-based 3D representation, and the task is to move deformable objects toward a goal state.

The paper explicitly does not deploy Cosmos for planning. Three reasons are given: video models are sensitive to appearance shifts across environments, reward design in video space is difficult, and geometric metrics such as Chamfer distance are much easier to use with 3D models. This practical asymmetry is one of the benchmark’s most consequential findings. Even when 2D video models are visually stronger, explicit 3D representations remain more convenient for closed-loop robotic control.

The scientific significance of Deform360 lies in this separation of regimes. The benchmark concludes that 3D particle models offer better inductive bias, better low-data physical reasoning, and greater utility for planning, whereas 2D video models offer better scalability, better texture and appearance fidelity, and stronger zero-shot visual generalization when large-scale pretraining is available (Li et al., 6 Jul 2026). Deform360 therefore functions as a benchmark for a central question in deformable-object robotics: whether robust world modeling should be grounded primarily in geometric and physical structure or in scalable video modeling. The answer supplied by the benchmark is conditional rather than absolute. Both strategies are effective, but their strengths diverge across data regime, evaluation target, and downstream use.

In that sense, Deform360 is not merely a large dataset of deformable-object interactions. It is a real-world multimodal benchmark that exposes the operational boundary between physically grounded prediction and visually scalable prediction, while providing the synchronized sensing, dense 3D supervision, and planning interface needed to study that boundary rigorously.