A Model-based Visual Contact Localization and Force Sensing System for Compliant Robotic Grippers

Abstract: Grasp force estimation can help prevent robots from damaging delicate objects during manipulation and improve learning-based robotic control. Integrating force sensing into deformable grippers negotiates trade-offs in cost, complexity, mechanical robustness, and performance. With the growing integration of RGB-D wrist cameras into robotic systems for control purposes, camera-based techniques are a promising solution for indirect visual force estimation. Current approaches mostly utilize end-to-end deep learning, which can be brittle when generalizing to new scenarios, while existing model-based approaches are unsuited to grasping and modern grasper geometries. To address these challenges, we developed a model-based visual force sensing approach integrating an iterative contact localization with generalization to unseen objects. The system extracts structural key points from wrist camera RGB-D images of deforming fin-ray-shaped soft grippers, and uses these key points to define parameters of an inverse finite element analysis simulation in Simulation Open Framework Architecture. The iterative contact localization sub-system utilizes a deep learning-based online 3D reconstruction and pose estimation pipeline to dynamically update contact location, and is robust to visual occlusion and unseen objects. Our system demonstrated an average root mean square error of 0.23 N and normalized root mean square deviation of 2.11% during the load phase, and 0.48 N and 4.34% over the entire grasping process when interacting with different objects under various conditions, showcasing its potential for real-time model-based indirect force sensing of soft grippers.

• The paper presents a model-based, vision-driven system that accurately localizes contact and estimates force in compliant robotic grippers using RGB-D data, deep keypoint extraction, and iFEA simulation.
• The method leverages iterative contact localization and mesh registration to overcome occlusion and sensor integration challenges, achieving sub-0.8 N RMSE in force estimation.
• The system generalizes robustly to unseen objects and supports real-time control, demonstrating practical improvements over purely data-driven models in both static and dynamic tests.

Model-Based Visual Force Sensing for Compliant Robotic Grippers: An Authoritative Analysis

Introduction

The presented work introduces a model-based, vision-driven contact localization and force estimation system for compliant robotic grippers, specifically fin-ray geometries. The system leverages real-time RGB-D data, deep keypoint extraction, mesh-based object pose estimation, and inverse FEA (iFEA) simulation in SOFA for the indirect measurement of contact forces. This approach addresses persistent limitations in sensorized and camera-based force estimation—namely, trade-offs among mechanical robustness, sensing accuracy, cost, the complexity of sensor integration, occlusion-induced errors, and the brittleness of end-to-end deep learning models.

Figure 1: System-level flowchart detailing the pipeline and frame rates for visual keypoint extraction, mesh reconstruction, pose estimation, contact localization, and iFEA-based force estimation.

System Architecture

The system comprises physical fin-ray soft gripper jaws, a wrist-mounted RGB-D camera, and a digital processing pipeline.

• Keypoint Extraction: DeepLabCut (DLC) identifies a sparse set of 3D structural keypoints from the deforming gripper in RGB-D images. Rigorous likelihood thresholding and spatial filtering mitigate errors from occlusion and misidentification.
• Mesh Reconstruction and Pose Estimation: A novel mesh scaling and alignment workflow leverages the SAM-3D Objects model, ICP refinement, and FoundationPose for robust mesh-object registration even without direct camera intrinsics. This resolves canonical mesh scaling ambiguities relative to the observed object.
• Iterative Contact Localization: The system introduces an iterative, model-based algorithm that computes gripper-object intersections within the SOFA simulation—dynamically updating the candidate contact position even under strong occlusions.
• iFEA Simulation and Force Computation: The system utilizes a per-jaw iFEA simulation at up to 40 Hz. It applies position effectors at tracked keypoints, a dynamically assigned actuator constraint at the inferred contact, and a QP-based solver with carefully tuned deformation energy weights ($\varepsilon$) per contact region.

  Figure 2: Physical dual-jaw gripper and its digital twin with overlaid iFEA mesh, actuator constraints, and keypoint locations.

Evaluation: Static and On-Robot Force Sensing

Static Benchtop Assessment

Comprehensive static evaluations were conducted with controlled loading of single gripper jaws across various object diameters (15/25/35 mm) and contact positions (upper, middle, lower), with ground-truth forces measured via a Nano17 F/T sensor.

Figure 4: Benchtop setup for static evaluation and contact positions: upper, middle, and lower (panels B–D).

Results demonstrate:

• Load-phase RMSE: Mean 0.23 N (NRMSD 2.11%)
• Full-cycle RMSE: Mean 0.48 N (NRMSD 4.34%)
• Error drivers: Internal strain nonlinearity (stiff contact regions yield underestimated force at increasing object diameter) and visibility loss of outer keypoints (overestimation at high-occlusion, softer regions).

  Figure 3: Error contributions from contact position and object size, and violin plots revealing estimation biases induced by occlusion and strain localization across all test conditions.

Necessity of Iterative Contact Estimation

Direct comparison between fixed actuator constraints and the proposed contact estimator shows that only the iterative contact localization algorithm enables robust force estimation under unknown and potentially shifting contact conditions. Fixed (incorrect) actuator positions cause substantial RMSE increases (>5 N).

On-Robot Generalization and Dynamic Evaluations

Three object geometries (cylinder, cube, asymmetric) with embedded load cells were grasped by the dual-jaw system. The mesh normalization workflow yields high-fidelity digital twins, preserving sub-4 mm Chamfer L2 error with minimal pose bias against CAD.

Figure 5: On-robot experimental setup, physical and digital objects with load cells, and force estimation results across conditions, including for previously unseen objects.

Key findings:

• Generalization: The model-based system preserves sub-0.8 N RMSE (1.75–7.04% NRMSD) in load and full phases across all tested objects, outperforming a ResNet50+Transformer deep learning baseline, especially on unseen objects where the latter fails.
• Robustness to Contact Shift: Reliable force estimation is maintained even during shifts in the contact region mid-grasp.
• Dynamic Range and Delicate Manipulation: Responsive control is demonstrated in the closed-loop grasp of fragile objects (e.g., a potato chip), achieving successful manipulation up to the fracture threshold.

  Figure 6: Manipulation force estimation during object rotation (left: polar force trajectory); potato chip grasp performance (right: force control and fracture threshold demonstration).

Practical and Theoretical Implications

This model-based, visually driven approach establishes a new standard in indirect force sensing for soft, compliant robotic grippers. It offers distinct advantages over data-driven models:

• Transferability: The system robustly transfers across geometric and material variations, requiring only minimal data for keypoint retraining (50–200 frames for DLC), with no object-specific force ground-truth supervision.
• Occlusion Robustness: Iterative contact localization significantly mitigates estimation failure modes due to occlusion or ambiguous deformations.
• Real-Time Control: The pipeline supports real-time feedback (up to 30 Hz limited by the camera rate) for dynamic manipulation tasks, including force-controlled grasping of delicate objects.
• Generalization Beyond Training: The approach generalizes to previously unseen objects, as evidenced empirically, in contradiction to the sharp generalization failures observed for purely data-driven visual models.

Theoretically, the work demonstrates the feasibility and utility of combining geometric simulation (iFEA), model-based reasoning (contact localization), and deep learning (keypoint extraction) for indirect force estimation. The use of mesh-based object registration and contact inference in the context of non-rigid, highly occluded, and unknown-contact scenarios extends the reach of visual-tactile estimation paradigms.

Limitations and Future Directions

• Material Modeling: Current iFEA implementations assume linear elastic behavior. Nonlinearities under large deformation, viscous effects, and material aging remain sources of error, particularly in highly strained or dynamic scenarios.
• Frame Rate Bottlenecks: FoundationPose and the overall contact estimator limit the pre-contact update rate to 10 Hz; future optimizations should parallelize or further optimize these components.
• Lighting Sensitivity: While robust under moderate lighting, severe occlusions or poor illumination degrade keypoint and mesh quality, requiring either high-quality point clouds or improved markerless vision networks.

Anticipated directions include integrating nonlinear viscoelastic simulation, augmenting the perception stack with multimodal cues (e.g., vibration sensing), and expanding materials and gripper morphologies validated by this paradigm.

Conclusion

This paper provides an authoritative demonstration that model-based, vision-centric force estimation for compliant grippers can achieve high accuracy, robustness to occlusion, and generalization to unseen objects—exceeding the limitations of traditional sensorized or end-to-end learned approaches. The system's practical utility is validated in dynamic robotic manipulation, and its methodological contributions supply a strong foundation for future research in indirect force and tactile inference in soft robotics.