FingerEye: Continuous and Unified Vision-Tactile Sensing for Dexterous Manipulation

Abstract: Dexterous robotic manipulation requires comprehensive perception across all phases of interaction: pre-contact, contact initiation, and post-contact. Such continuous feedback allows a robot to adapt its actions throughout interaction. However, many existing tactile sensors, such as GelSight and its variants, only provide feedback after contact is established, limiting a robot's ability to precisely initiate contact. We introduce FingerEye, a compact and cost-effective sensor that provides continuous vision-tactile feedback throughout the interaction process. FingerEye integrates binocular RGB cameras to provide close-range visual perception with implicit stereo depth. Upon contact, external forces and torques deform a compliant ring structure; these deformations are captured via marker-based pose estimation and serve as a proxy for contact wrench sensing. This design enables a perception stream that smoothly transitions from pre-contact visual cues to post-contact tactile feedback. Building on this sensing capability, we develop a vision-tactile imitation learning policy that fuses signals from multiple FingerEye sensors to learn dexterous manipulation behaviors from limited real-world data. We further develop a digital twin of our sensor and robot platform to improve policy generalization. By combining real demonstrations with visually augmented simulated observations for representation learning, the learned policies become more robust to object appearance variations. Together, these design aspects enable dexterous manipulation across diverse object properties and interaction regimes, including coin standing, chip picking, letter retrieving, and syringe manipulation. The hardware design, code, appendix, and videos are available on our project website: https://nus-lins-lab.github.io/FingerEyeWeb/

• The paper presents a novel sensor combining binocular RGB cameras and a compliant ring to provide continuous, fine-grained feedback for dexterous manipulation.
• It employs multi-tag AprilTag pose estimation and transformer-based policy learning, achieving over 30% improvement in execution success rates.
• Practical evaluations demonstrate high sensitivity and robust performance in delicate grasping tasks with a low-cost, reproducible design.

FingerEye: Continuous and Unified Vision-Tactile Sensing for Dexterous Manipulation

Motivation and Technical Context

Robotic dexterity in contact-rich manipulation necessitates fine-grained, continuous perception spanning pre-contact, contact initiation, and post-contact phases. State-of-the-art vision-based tactile sensors such as GelSight, TacTip, and their derivatives provide rich feedback post-contact but are limited in pre-contact perception, hindering closed-loop adaptation and stable contact regulation. Approaches like See-Through-Skin (STS) sensors partially bridge vision and touch but remain constrained by central sensing regions, specialized illumination, or complex fabrication, thus limiting practical reproducibility and peripheral contact sensitivity.

FingerEye introduces a novel vision-tactile sensing modality, combining binocular RGB cameras and a compliant ring structure with an AprilTag layout, yielding seamless, continuous feedback across all interaction phases. This enables unified perception streams suitable for high-fidelity, millimeter-level sensing and policy learning in dexterous manipulation. The sensor is mechanically optimized for fingertip integration, peripheral-contact awareness, and low-cost fabrication.

Figure 1: (a) The wedge-shaped FingerEye sensor's compact form factor, (b) cross-sectional view showing dual camera fields and extended sensing regions, (c) exploded hardware view.

Mechanical and Perceptual Design

FingerEye's mechanical architecture adopts a wedge-shaped, fingertip-scale module with dual cameras: one orthogonal to the acrylic cover (tip camera) enabling close-range pose and deformation estimation, and one tilted at the finger root (root camera) preserving broad scene context and stereo depth cues. The compliant ring—molded from silicone—surrounds the transparent acrylic cover, providing directional compliance for 6D wrench sensitivity and expanding the peripheral sensing region.

The integrated AprilTag layout, composed of densely distributed markers ($35$ tags, $2$mm each), supports multi-tag PnP pose estimation robust to partial occlusion, non-uniform deformation, and varying illumination (with passive lighting and CLAHE). The unified visual stream avoids modality switching, colored illumination, and heuristic sensing, ensuring reproducible, stable RGB observations critical for learning-based control.

Robustness and Sensing Validation

Central hardware experiments validate qualitative robustness of AprilTag-based pose estimation, sensitivity to small wrenches, deterministic wrench–deformation mapping, and responsiveness in delicate grasping.

Figure 2: AprilTag-based pose estimation remains stable under force perturbations (top) and lighting variation (bottom).

The multi-tag pose estimation maintains stability under significant deformation and low-light conditions, contrasting with keyline marker tracking, which fails under simultaneous perturbations. Sensitivity analysis yields minimum detectable forces of $[4.30,\,4.22,\,9.93]\text{ mN}$ and torques of $[0.32,\,0.13,\,8.55]\text{ mN}\cdot\text{m}$, confirming high precision suitable for delicate manipulation. Wrench–deformation correlation experiments demonstrate strong linearity ($R^2_{\text{test}}$ near unity; low RMSE), validating pose changes as reliable proxies for contact force and torque.

Figure 3: Mapping between predicted wrench values from ring deformation and ground-truth, affirming deterministic force-torque estimation.

Delicate grasping trials across fragile and deformable objects (e.g., chip, eggshell, balloon) show consistent, immediate contact detection via estimated normal deformation, enabling damage-free manipulation—a practical indicator of the sensor’s efficacy.

Figure 4: Visualization of fingertip normal deformation curves during delicate grasping, showing responsive contact onset detection.

Integrated Learning Platform and Policy Design

A platform comprising a fixed-base robotic arm (xArm7), LEAP Hand, and multiple FingerEye modules enables teleoperation data collection with synchronized visual and proprioceptive feedback. The learning framework includes:

• FingerEye Policy: A transformer-based imitation policy utilizing multi-view vision-tactile fusion and action chunking. Each camera's RGB feed is processed through a RADIO backbone, per-camera learnable embedding, and transformer encoder; cross-attention in the decoder conditions action queries on robot joint state and recent tag pose history.
• Simulation Digital Twin: Isaac Lab-based simulation supports large-scale, appearance-randomized synthetic data generation, facilitating representation learning for visual robustness.

Figure 5: Left: data collection interface for teleoperated demonstration streaming; Right: transformer-based policy architecture for multi-camera fusion and action chunking.

Simulation-augmented representation learning leverages domain-randomized synthetic observations supervised with object-level signals, enabling policies to generalize across significant appearance variations without deterioration from sim-real action bias.

Figure 6: Framework for simulation-augmented representation learning, promoting robust encoder supervision with auxiliary object decoders.

Figure 7: Digital twin and visual augmentation examples with randomized appearance and lighting.

Evaluation: Manipulation Tasks and Modalities

Policy efficacy is benchmarked across four manipulation tasks: chip picking (fragile rigid), coin standing (unstable rigid), letter retrieving (thin deformable), and syringe manipulation (articulated functional). Task setups span scenarios requiring precise pre-contact alignment, at-contact sensitivity, and post-contact force modulation.

Figure 8: Real-world rollouts for chip picking, coin standing, letter retrieval, and syringe manipulation tasks, demonstrating generality.

Figure 9: Training and testing configurations for task evaluation.

Comparative experiments address the benefits of local contact sensing (FingerEye vs. vision-only), binocular vs. monocular sensing, policy architecture efficiency, and simulation-augmented representation learning.

Figure 10: Representative failure modes of baseline policies lacking local tactile sensing.

Quantitative Results and Architectural Insights

Policies incorporating FingerEye sensors—especially binocular vision-tactile streams—exhibit substantial improvements ($>30\%$ average increase in execution success rates) over wrist-camera-only baselines. Binocular configurations further outperform monocular setups, significantly reducing pre-contact alignment errors and missed grasps. Policy architecture comparisons reveal that the FingerEye Policy achieves $2.8\times$ faster training speed and $+14.8\%$ higher success rate than strongest diffusion and ACT-style baselines, attributable to efficient multi-view tokenization and action chunking.

Figure 11: Task-wise quantitative results across modalities; FingerEye consistently outperforms vision-only baselines.

Figure 12: Simulation evaluation: success rates by modality (left), relative training speed and final success under identical visual inputs (right).

In sim-augmented representation learning, policies trained with auxiliary object supervision from domain-randomized simulation generalize robustly to unseen object appearances, outperforming both real-only and naive sim-real co-training, especially under significant color shifts.

Figure 13: Success rates across coin color variants for different training strategies; simulation-augmented representation learning enables strong generalization.

Comparative Analysis with Marker Designs and Sensor Variants

AprilTag-based pose estimation shows superior robustness to keyline markers, particularly under challenging deformation and illumination, attributed to multi-corner geometric constraints in PnP (see supplemental comparisons). In practical tasks, the peripheral deformation enabled by FingerEye’s compliant ring allows sensing at the fingertip edge—an area unaddressed by rigid ring designs or square-shaped sensors (e.g., GelSight)—facilitating wedging, sliding, and precise edge engagement.

Figure 14: Keyline-marker tracking fails under combined deformation and lighting variation; AprilTag pipeline retains performance.

Figure 15: GelSight fails to capture the coin in the coin standing task, illustrating limitations of traditional tactile sensors.

FingerEye’s fabrication is straightforward and reproducible, using commercially available components and simple molding procedures (total material cost $<\$61$).

Figure 16: Materials and components used for FingerEye module fabrication.

Figure 17: Experimental setup for wrench–deformation correlation assessment.

Figure 18: Visual task sequences for chip picking, coin standing, syringe manipulation, and letter retrieving across training and testing configurations.

Implications and Future Directions

FingerEye's unified and continuous vision-tactile sensing paradigm advances closed-loop dexterous manipulation by enabling consistent pre-contact, at-contact, and post-contact feedback. The validated deterministic wrench–deformation mapping and high sensitivity pave the way for precise force regulation, impedance and hybrid control strategies, and robust contact-driven policy learning.

Practically, the reproducibility, low domain-gap RGB observations, and peripheral contact awareness support scalable deployment across diverse robot platforms. The simulation-augmented representation learning framework enhances generalization, reducing the required volume of real-world demonstrations and facilitating sim-to-real transfer, which is critical for future foundation-model-based manipulation policies.

Theoretically, continuous unified sensing at the fingertip opens research into optimal sensor geometry, active deformation control, whole-body tactile fusion, and adaptable policy architectures. Extension to bimanual, mobile, or humanoid platforms, integration with multi-modal teleoperation, and application to hybrid imitation-reinforcement learning can be pursued.

Conclusion

FingerEye establishes a reproducible, cost-efficient, and robust vision-tactile sensing system that delivers continuous feedback across all phases of contact-rich dexterous manipulation. Its hardware and policy design demonstrate substantial improvements in manipulation reliability and learning efficiency. Simulation-augmented representation learning provides robust generalization to unseen visual contexts. The work motivates integrated sensing and scalable policy learning approaches for real-world dexterous robots, and suggests promising future research directions in sensor design, scalable data generation, and generalist manipulation policies.