VibeAct: Vibration to Actions for Contact-Rich Reactive Robot Dexterity

Abstract: Dexterous manipulation depends on contact events that are fast, local, and often visually occluded. Piezoelectric microphones offer a compact and high-bandwidth way to sense these interactions, but the resulting vibro-acoustic signals are difficult to simulate faithfully enough for end-to-end sim-to-real policy learning on dexterous robot hands. We propose VibeAct, a framework that bridges real vibrotactile sensing and simulation-based reinforcement learning through a shared physical representation of contact and slip. In the real world, we embed piezoelectric microphones into a dexterous robot hand and collect vibro-acoustic data through teleoperation, then replay the recordings in a calibrated digital clone to automatically label per-finger contact and slip. A tactile estimator learns to predict contact and slip from real microphone waveforms, while manipulation policies are trained in simulation on the same representation computed directly from simulated contacts. This decoupling lets policies exploit rapid tactile feedback without simulating raw audio. Across five contact-rich tasks spanning regrasping, in-hand reorientation, and insertion, VibeAct consistently outperforms a proprioception-and-point-cloud baseline in simulation, with the largest gains on tasks requiring sustained reactive control, where the continuous slip-magnitude channel proves the most informative observation. The learned policies transfer to a physical dexterous hand-arm platform, improving success rates on deployed tasks. Project videos and additional details are at https://vibeact.github.io/.

• The paper presents a novel sim-to-real framework that decouples vibrotactile sensing from control using a physically grounded contact-and-slip representation.
• It details a tactile estimator with per-finger subnetworks that map high-bandwidth vibration signals to actionable contact metrics, boosting policy learning.
• Experimental results show significant task success improvements and robust sim-to-real transfer, validating enhanced performance in contact-rich manipulation tasks.

VibeAct: Bridging Vibrotactile Sensing and Simulation-Based Dexterous Manipulation

Framework Overview

VibeAct introduces a novel sim-to-real framework for dexterous manipulation by decoupling vibrotactile sensing and control through a shared physical representation of contact and slip. The system leverages piezoelectric microphones embedded in each robot fingertip, enabling high-bandwidth tactile sensing without the overhead of vision-based devices. A central challenge addressed is the impracticality of simulating raw vibration signals during policy learning—direct simulation is infeasible due to complex dependencies on finger materials, mounting arrangements, and background motor/acoustic noise. VibeAct circumvents this by training a tactile estimator to map microphone waveforms to a low-dimensional, physically grounded contact-and-slip vector, which is also directly computable from simulated contact dynamics. RL policies are trained in simulation using this intermediary representation as an additional observation channel beside proprioception and point clouds.

Figure 1: An overview of VibeAct, showing the tactile estimator architecture and its integration with RL policy learning via contact-and-slip observations.

Data Collection and Label Generation

The real-world data acquisition setup consists of an xArm7 with an anthropomorphic LEAP hand, each fingertip instrumented with two piezoelectric microphones (8 channels total). Teleoperated object interactions are recorded synchronously with high-rate microphone signals, robot/hand joint states, and object poses. To enable label generation without manual annotation, VibeAct replays these trajectories in a calibrated digital clone using MuJoCo's contact solver to automatically generate per-finger contact onset, slip presence, and slip magnitude labels. This process aligns vibration signals with physically defined tactile events based on real and simulated dynamics.

Figure 2: Vibrotactile data labeling setup, illustrating automatic contact and slip supervision via digital-clone replay of teleoperation recordings.

Tactile Estimator Architecture

VibeAct’s tactile estimator comprises four independent per-finger subnetworks, each processing multi-channel log-mel spectrograms derived from synchronized microphone pairs. A learnable gating layer suppresses spurious channels, followed by frequency-only convolutional pooling for temporal resolution retention, temporal convolutions, and attention pooling to fuse per-microphone data. Each subnetwork produces predictions for contact onset (sparse event), slip presence (binary state), and slip magnitude (continuous severity). Training employs a class-weighted binary cross-entropy for event detection and Huber loss for magnitude regression, with a pretrain-then-fine-tune strategy—general vibro-acoustic patterns are learned from fixed-object data, then adapted using in-hand manipulation data.

Policy Learning and Integration

Manipulation policies are trained entirely in simulation using PPO under domain randomization of object friction, mass, pose, and camera configurations. Each policy receives proprioceptive and point cloud inputs, augmented with the tactile estimator’s 12-D contact-and-slip vector (3 features per fingertip: onset, slip presence, slip magnitude). The observation pipeline integrates a PointNet-style encoder for geometric features and MLPs for proprioceptive and tactile streams. Rewards are designed per task for progress, drops, and success signals, ensuring robustness across diverse contact-rich scenarios.

Experimental Evaluation

Tactile Estimator Ablation and Performance

Extensive evaluation demonstrates that the sequential training strategy (fixed-object pretrain followed by moving-object fine-tune) maximizes F1 scores for contact onset (0.597) and slip presence (0.913), with minimal MAE in slip magnitude ($4.736\,\mathrm{mm/s}$). Shared encoder architectures degrade performance, underlining significant per-finger variation in contact dynamics and vibration propagation. Performance is consistent across the four instrumented fingers, emphasizing the estimator's reliability.

Task Success and Policy Ablation

VibeAct was evaluated on five contact-rich manipulation tasks, including finger gaiting, peg insertion, cube rotation, and nut rotation:

Figure 3: Training curves for VibeAct policies and baselines across five manipulation tasks, highlighting accelerated convergence and superior final success rates with tactile input.

Figure 4: Depiction of task suite: Box Climb, Can Climb, Peg in Hole, Cube Rotation, and Nut Rotation, designed for benchmarking contact-rich dexterity.

VibeAct exhibits pronounced gains in success rates, especially on tasks demanding sustained closed-loop reactive control. For instance:

• Cube Rotation success rises from 6.0% (baseline) to 57.0% (full VibeAct).
• Peg in Hole improves from 6.5% to 30.0%.
• Can Climb and Nut Rotation also display substantial improvements.

Ablation studies reveal that slip magnitude is the most informative channel; adding it alone markedly elevates task performance. Contact onset alone yields inconsistent benefits, while slip presence provides moderate improvements. The continuous slip signal is critical for modeling graded fingertip-object interactions.

Sim-to-Real Transfer

Policies trained in simulation transfer directly to hardware, with the tactile estimator replacing simulated contact inputs. Across three real-world testbeds (Box Climb, Can Climb, Nut Rotation), VibeAct outperforms the proprioception and point cloud baseline, demonstrating robustness under sensor and actuation noise.

Figure 5: Sim-to-real perspective alignment, facilitating consistent point cloud and tactile input mapping during policy deployment.

Implications and Future Directions

VibeAct establishes a practical paradigm for sim-to-real dexterous manipulation by leveraging physically grounded intermediate tactile representations. The explicit decoupling enables robust RL policy training without requiring infeasible acoustic simulation, while still exploiting high-bandwidth tactile channels at deployment. From a theoretical perspective, the approach validates that sparse and scalar contact/slip features suffice for many closed-loop manipulation behaviors, and that RL policies can generalize from such low-dimensional but task-critical tactile signals. Architecturally, the finding that per-finger subnetworks outperform shared encoders underscores the importance of modeling spatially heterogeneous contact dynamics.

Practically, VibeAct demonstrates significant gains for manipulation tasks beyond the reach of vision or proprioception alone, especially those involving visually occluded or dynamically shifting contacts. The outlined sim-to-real pipeline has relevance to multi-fingered hands, in-hand reorientation, and assembly tasks. Interfacing high-bandwidth, low-cost tactile sensing with RL policies in this manner opens the door to scalable dexterous automation in unstructured settings.

Limitations and Prospects

The present approach discards raw vibration information not encompassed by contact and slip (e.g., surface texture, precise contact location), limiting policy access to richer dynamic cues. The estimator is hardware-specific and may require recalibration for other hands or sensor placements. Automatic label generation depends on accurate object pose tracking, constraining applicability in environments where such calibration is difficult.

Future developments could explore richer intermediate tactile representations, hierarchical policies, temporal modeling of contact events, and adaptive recalibration methods. Improving sim-to-real acoustic fidelity could further bridge the gap between sensing and control for tactile tasks in open-world robot manipulation.

Conclusion

VibeAct presents a robust and scalable methodology for integrating vibrotactile sensing into sim-to-real dexterous robot control, leveraging a shared contact-and-slip representation as a bridge between real-world perception and simulation-based policy learning. Empirical results demonstrate significant improvements in manipulation task success rates, most notably where tactile feedback is essential for sustained reactive control. The framework delineates a path forward for incorporating high-bandwidth tactile sensing into RL-driven robot dexterity, supporting both the theoretical assumptions and practical demands of contact-rich manipulation.