Learning Bimanual Cloth Manipulation with Vision-based Tactile Sensing via Single Robotic Arm

Abstract: Robotic cloth manipulation remains challenging due to the high-dimensional state space of fabrics, their deformable nature, and frequent occlusions that limit vision-based sensing. Although dual-arm systems can mitigate some of these issues, they increase hardware and control complexity. This paper presents Touch G.O.G., a compact vision-based tactile gripper and perception/control framework for single-arm bimanual cloth manipulation. The proposed framework combines three key components: (1) a novel gripper design and control strategy for in-gripper cloth sliding with a single robot arm, (2) a Vision Foundation Model-backboned Vision Transformer pipeline for cloth part classification (PC-Net) and edge pose estimation (PE-Net) using real and synthetic tactile images, and (3) an encoder-decoder synthetic data generator (SD-Net) that reduces manual annotation by producing high-fidelity tactile images. Experiments show 96% accuracy in distinguishing edges, corners, interior regions, and grasp failures, together with sub-millimeter edge localization and 4.5° orientation error. Real-world results demonstrate reliable cloth unfolding, even for crumpled fabrics, using only a single robotic arm. These results highlight Touch G.O.G. as a compact and cost-effective solution for deformable object manipulation.

• The paper introduces Touch G.O.G., a single-arm end-effector combining decoupled width control, finger abduction, and DIGIT tactile sensors to emulate bimanual cloth manipulation without external cameras.
• Its SAM-based perception pipeline achieves 96% grasp-state classification accuracy and estimates cloth-edge position and orientation with 0.59 mm and 4.52° errors, respectively, aided by synthetic tactile data.
• Closed-loop corner-to-corner edge tracing succeeds in 24 of 35 flattened-cloth trials and 20 of 35 crumpled-cloth trials, demonstrating practical capability while exposing gaps between perception accuracy and task reliability.

Overview and motivation

Cloth manipulation remains difficult for robots because fabric state is high-dimensional, deformable, and frequently self-occluding, which undermines global vision-based perception. Dual-arm systems mitigate some of these issues but at the cost of hardware expense and control complexity. The paper introduces Touch G.O.G., a compact end-effector and perception/control framework that emulates bimanual dexterity on a single robotic arm by combining mechanical design, foundation-model-based tactile perception, and synthetic data generation (2603.10609). The system performs closed-loop edge tracing along cloth edges using only on-finger tactile feedback, with no external cameras during operation — a deliberate design choice motivated by the severe self-occlusion that occurs during edge sliding.

The work builds directly on the authors' earlier G.O.G. gripper-on-gripper design (2603.10609), which relied on passive compliance and open-loop kinematics; Touch G.O.G.'s stated contribution is to close the loop by integrating high-resolution vision-based tactile sensing into the control pipeline.

Mechanical design: D-WCG and T-VFG

The end-effector comprises two functionally distinct modules. The Decoupled Width Control Gripper (D-WCG) is a dual-rail linear prismatic base in which each finger carriage is independently actuated by a stepper motor through a timing-belt transmission. Decoupled actuation permits asymmetric finger positioning to adapt to irregular fabric geometries, while the rigid belt transmission ensures finger separation tracks motor commands accurately under fabric tension loads — essential for stable closed-loop expansion maneuvers.

Each D-WCG finger terminates in a Tactile Variable Friction Gripper (T-VFG) equipped with a DIGIT vision-based tactile sensor and driven by a DC motor with an incremental encoder. A DC motor housed within the D-WCG finger provides an additional abduction degree of freedom (DoF), giving each finger continuous position and orientation control within a planar workspace. Compared with the prior G.O.G., whose fingers were constrained to an approximately 1D locus, the abduction DoF substantially enlarges the reachable region, allowing grasps offset from the central line and approach angles adapted to edge orientation.

Both grasping and abduction motions are governed by PID position controllers with exponentially filtered derivative terms to suppress encoder-noise amplification and chattering. The loop uses dynamically measured sampling intervals $\Delta T$ to remain robust to interrupt-driven timing jitter. Grasping force is implicitly bounded by limiting motor effort, preventing excessive normal force on the fabric.

Perception pipeline: PC-Net, SD-Net, and PE-Net

All perception is built on the Segment Anything Model (SAM) ViT backbone [Kirillov_2023_ICCV], operating exclusively on DIGIT tactile images.

PC-Net classifies each grasp into four classes — Edge, Corner, In-Fabric, or Grasp Failure — from stacked sequences of five tactile frames, with temporal feature averaging after a convolutional head. It is trained on roughly 10,000 augmented samples per class from 100 real grasp trials per class across multiple fabrics. In evaluation against ResNet50, DenseNet121, and ViT baselines, PC-Net achieves 96% overall accuracy, with precision of 0.97 (Edge), 0.99 (Corner), 0.94 (In-Fabric), and 0.97 (Grasp Failure). Notably, standard augmentation suffices here; synthetic data is not required for classification. Accurate corner recognition matters because it triggers both the start and termination of sliding trajectories.

SD-Net addresses the annotation bottleneck for pose estimation. It is a SAM-backboned encoder–decoder that reconstructs high-fidelity tactile images from lightweight edge-annotation masks, trained with a combined MSE + SSIM loss. Starting from only 100 manually annotated images, SD-Net generates 30,000 synthetic samples spanning diverse edge poses and textures. A t-SNE analysis shows synthetic samples covering a broader latent region than the real data, indicating interpolation beyond the collected distribution rather than mere replication.

PE-Net regresses edge center $(x, y)$ and orientation $\theta$ from tactile images, trained with MSE plus a cosine-similarity angular loss to handle angle periodicity. Its quantitative results are the strongest claims in the paper:

Method	Distance error (mm)	Angle error (°)
Classical edge detection	3.38	50.22
ViT baseline	0.83	5.35
PE-Net without SD-Net	0.78	5.52
PE-Net (full)	0.59	4.52

Two implications follow directly. First, sub-millimeter localization is what makes closed-loop sliding feasible: the controller must keep the detected edge centered on the DIGIT sensing area throughout the trajectory. Second, the ablation isolates SD-Net's contribution — synthetic data reduces distance error from 0.78 mm to 0.59 mm and angular error from 5.52° to 4.52°, confirming that broader pose coverage translates into estimation accuracy rather than merely improving training-set statistics.

Closed-loop sliding control and real-world results

During sliding, two discrete PD controllers regulate the vertical offset $e_y$ and angular misalignment $e_\theta$ of the detected edge in the tactile image. Yaw commands drive the UR5 end-effector, while abduction commands re-orient the T-VFG, with a feed-forward term compensating the coupling between yaw rotation and perceived edge tilt; commands are clipped to ±5° (yaw) and ±30° (abduction). PC-Net additionally triggers discrete insertion-depth corrections: an In-Fabric classification causes withdrawal to a shallower grasp, while Grasp Failure triggers deeper insertion. Sliding terminates when both fingers classify Corner.

Evaluation covers seven fabrics (four template fabrics, a patterned towel, a linen blanket, a pink towel) under flattened and crumpled initial configurations, five trials each, for 70 total experiments. Success requires reaching the opposite corner with both fingers classified as Corner and sufficient unfolding. Results:

Configuration	Success rate
Flattened	24/35 (~69%)
Crumpled	20/35 (~57%)

These rates are notably lower than the perception accuracies, and the paper does not provide a failure-mode breakdown attributing errors to perception versus mechanics. The gap between near-perfect classification and ~60–70% task success indicates that perception accuracy alone does not determine system reliability; fabric-specific sliding dynamics and controller limits appear to be contributing factors, though this is not analyzed quantitatively. Still, achieving any reliable corner-to-corner tracing on crumpled, patterned fabrics using only local visuotactile feedback — no global vision — is a meaningful demonstration, since prior tactile edge-tracing work was largely restricted to simple square cloths and often required dual arms.

Limitations and open questions

Several limitations are acknowledged or evident. The sliding success rates leave substantial room for improvement, particularly for crumpled fabrics, and the paper does not decompose failures between misclassification, pose-estimation error, and mechanical slip limits. Some fabric orientations exceed the inherent sliding capability of the mechanism and require arm-level adjustments, implying the current policy is not universally applicable. Evaluation is confined to a single UR5 platform and relatively small fabric samples; generalization to larger garments such as shirts or dresses remains untested. The framework also assumes one finger starts from a correctly grasped corner, sidestepping autonomous initial corner acquisition. Finally, the authors note that extending to multi-step tasks such as folding or dressing would likely require additional sensing modalities (force–torque or global vision), leaving open whether purely tactile control scales beyond single-edge tracing.

Conclusion

Touch G.O.G. demonstrates that bimanual-style cloth manipulation can be achieved with a single arm by pairing a mechanically capable end-effector — decoupled width control plus abduction DoFs with integrated DIGIT sensors — with SAM-backboned visuotactile perception and synthetic data generation. The headline results are 96% cloth-part classification accuracy, 0.59 mm / 4.52° edge pose estimation, and successful corner-to-corner sliding on patterned and crumpled fabrics without external cameras. The principal open question raised by the results is closing the gap between component-level perception performance and end-to-end task success rates, particularly under crumpled initial conditions.