SoftVTBench: A Safety-Aware Visuo-Tactile Benchmark for Physically Constrained Robotic Manipulation of Deformable Objects

Abstract: Deformable object manipulation poses challenges beyond task completion: successful execution must also maintain safe physical interaction, holding the object stably without slip or drop while avoiding excessive deformation. However, existing manipulation benchmarks are predominantly success-oriented and rarely evaluate whether a policy remains physically safe throughout execution. We present SoftVTBench, a safety-aware visuo-tactile benchmark for physically constrained deformable object manipulation. Built in Isaac Sim with finite-element-simulated deformable objects, SoftVTBench provides multi-view RGB observations, RGB tactile sensing with marker motion, proprioception, and language instructions, and defines four matched task suites over object type (deformable vs. rigid) and variation axis (object vs. spatial). It separately reports Goal Success and Safety Success; the latter additionally requires no drop and peak deformation below a calibrated object-specific threshold, measured from policy-hidden privileged Finite Element Method (FEM) states. We implement pi0.5-based baselines under this protocol. Experiments show that success-only evaluation substantially overstates policy performance, as a large fraction of goal-completing rollouts still violate physical safety. Furthermore, incorporating tactile sensing improves Safety Success (e.g., from 21.4% to 35.6% on object-centric deformable tasks) and reduces object deformation during execution, while maintaining comparable Goal Success. SoftVTBench provides a reproducible benchmark for studying visuo-tactile deformable manipulation under physical interaction constraints.

• The paper introduces a safety-aware benchmark that distinguishes between goal success and safe physical interactions when manipulating deformable objects.
• It employs synchronized multi-modal inputs including RGB camera views, tactile imaging with marker flow, and proprioceptive data to assess fine-grained deformation and slip using FEM simulations.
• Empirical results show that incorporating tactile sensing significantly boosts safety success in deformable-object tasks, highlighting the need for process-level safety evaluation in robotics.

SoftVTBench: A Safety-Aware Visuo-Tactile Benchmark for Physically Constrained Robotic Manipulation of Deformable Objects

Motivation and Benchmark Overview

SoftVTBench addresses a critical gap in robotic manipulation benchmarks: evaluating physical safety during deformable object manipulation, beyond mere task completion. Traditional benchmarks emphasize goal achievement over process-level interaction quality, leading to policies that may routinely over-deform or drop soft objects while still being reported as successful.

SoftVTBench is a visuo-tactile benchmark built in Isaac Sim with finite-element method (FEM)-simulated deformable assets, providing multi-view RGB, synchronized tactile (RGB+marker flow), full proprioception, and language-conditioned instructions. It defines four task suites along two axes—object type (deformable vs. rigid) and variation (object identity vs. spatial)—with nearly 2000 episodes spanning 33 assets in complex tabletop scenes.

Unlike standard benchmarks, SoftVTBench separately reports Goal Success (task completion) and Safety Success (completion plus stable grasp, no drop, and deformation below calibrated, object-specific FEM thresholds). This protocol directly exposes unsafe policies whose interaction strategies remain undisclosed under success-only metrics.

Figure 1: Overview of SoftVTBench with multi-modal input streams and safety-centric evaluation for physically constrained manipulation with deformable objects.

Protocol, Modalities, and Task Design

SoftVTBench advances visuo-tactile benchmarking by providing:

• Highly realistic sensory input: Multi-view RGB, dual-finger RGB tactile imaging with marker motion (GelSight Mini emulation), proprioceptive state, and natural language task instructions, all synchronized at 20 Hz.
• Action specification: Each policy step outputs a 3D end-effector pose and gripper closure command (binary or continuous), supporting ablation and multimodal control.
• Task formulation: Tasks include object-centric and spatial-variation suites for both deformable and rigid objects. Deformable tasks focus on safe grasp-and-place manipulation, where excessive force or imprecise contact risks deformation beyond task-specified safe envelopes.

Evaluation hinges on hidden, privileged FEM state access for post hoc safety discrimination—peak normalized RMS deformation, slip/drop detection—while observation is strictly limited to exteroceptive and tactile inputs.

Figure 2: Organization of SoftVTBench into four matched suites and illustration of the "Goal-Safety Gap": goal-reaching actions can violate hidden safety constraints.

Empirical Analysis

Baseline Architectures

SoftVTBench evaluates and compares two baseline configurations:

• $\pi_{0.5}$-Vision: Policy utilizes only third-person and wrist RGB, proprioceptive state, and language; gripper command is binary.
• $\pi_{0.5}$-Visuo-Tactile: Augments the above with tactile RGB (multi-frame, both fingers), marker motion, and continuous gripper-width command.

Both baselines are LoRA-fine-tuned policies derived from an open-world vision-language-action backbone, trained and evaluated on identical task distributions.

Main Numerical Results

Empirical results from SoftVTBench highlight several critical findings:

• On rigid-object suites, tactile augmentation does not consistently improve Goal Success (e.g., Object-Rigid: VT 32.4% vs. VO 38.8%), suggesting that visual and proprioceptive information is largely sufficient, and additional tactile signals contribute marginally or introduce noise.
• On deformable-object suites, visuo-tactile policies produce significant gains in Safety Success without compromising Goal Success. For instance, in Object-Soft, Safety Success improves from 21.4% (VO) to 35.6% (VT), while Goal Success remains comparable (71.8% vs. 70.4%). The effect is even pronounced in Spatial-Soft with Safety Success rising from 32.6% to 44.6%.

Figure 3: Goal Success vs. Safety Success across all task suites, highlighting the "Goal-Safety Gap" on deformable-object suites.

The explicit reporting of both metrics unveils the frequency of unsafe but goal-reaching rollouts—a phenomenon masked by standard evaluation. Visuo-tactile policies exhibit a capability to adapt grasping synergies, reducing excessive force and unstable contacts that commonly lead to deformation-induced failures.

Qualitative Tactile Observations

Inspection of tactile sensor activations (RGB plus marker motion) provides further evidence: each deformable asset yields object- and configuration-specific contact patterns, directly encoding compliance and slip cues unavailable to cameras.

Figure 4: Tactile RGB and marker motion for ten Object-Soft assets, capturing heterogeneity in geometry and compliance observable through the gripper interface.

Deformation Distribution Analysis

A granular comparison of continuous deformation statistics confirms the role of tactile sensing:

• Visuo-tactile policies shift the median and upper tail of max-deformation distributions downward relative to vision-only policies (e.g., Object-Soft P95: VT 38.81% vs. VO 44.70%), demonstrating not merely threshold effects but reduced propensity for severe unsafe contacts across the entire evaluation set.

Implications and Future Directions

From a theoretical standpoint, SoftVTBench formalizes a process-level safety paradigm for deformable-object manipulation: optimality is not solely a function of task completion, but a composite of goal attainment and constraint adherence over physically latent variables (i.e., deformation, slip). It operationalizes the tension between observability and physical safety, showing that tactile sensing is essential for controlling interaction-level safety in high-DOF, contact-rich environments.

Practically, SoftVTBench will catalyze:

• Benchmark-driven development of visuo-tactile policies with explicit safety considerations in manipulation, encouraging the design of architectures and training protocols that effectively exploit fine-grained contact feedback.
• Systematic study of the "Goal-Safety Gap" and the development of robust perceptual models and control algorithms that generalize safety, not just success, across complex asset geometries and materials.
• Extensions to richer physical domains: future work must address the limitations of synthetic simulation, expand deformable asset diversity, and integrate world-action or foundation-model based approaches to achieve process-level physical reasoning and policy transfer.

Conclusion

SoftVTBench establishes a new standard for manipulation evaluation by decoupling goal-reaching from interaction safety using FEM-based privileged evaluation in simulation. Its systematic use of visuo-tactile sensing demonstrates that tactile-instrumented policies achieve substantially higher rates of safe manipulation in deformable-object suites, while vision-based approaches systematically overestimate success due to unobserved unsafe contacts. This benchmark supplies a vital platform for advancing physically grounded, safety-aware manipulation research in robotic learning.