2000 character limit reached
NoContactNoWorries: Estimating Contact through Vision and Proprioception for In-Hand Dexterous Manipulation
Published 23 Jun 2026 in cs.RO and cs.AI | (2606.24450v1)
Abstract: Perceiving physical contact is fundamental to dexterous manipulation. While robots often rely on dedicated hardware tactile sensors, humans exhibit a remarkable ability to infer contact by integrating visual information with an innate sense of their body's pose and movement. Inspired by this embodied perceptual skill, we investigate whether a robot can learn to infer contact from vision, an approach that also offers a scalable alternative to tactile hardware specifically for binary contact estimation, which faces practical challenges in cost, fragility, and integration. We present NoContactNoWorries, a transformer-based multimodal framework that fuses RGB-D vision with the robot's proprioception to infer binary contact states as a pseudo-tactile signal for hand-object interactions. We validate by training a single contact prediction model on multiple objects and show that the inferred contact signal supports downstream reinforcement learning agents for in-hand object reorientation, generalizing to novel objects. Experiments in both simulation and on a real-world robot validate our approach, highlighting the feasibility of inferring contact from vision and proprioception. Project Page: https://soham2560.github.io/no-contact-no-worries/
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Explain it Like I'm 14
Plain-English Summary of “NoContactNoWorries”
What is this paper about?
This paper is about teaching a robot hand to tell when its fingertips are touching an object without using special touch sensors. Instead, the robot “feels” contact by combining:
- what it sees with a small wrist camera (color + depth),
- and what it knows about its own finger positions and movements (its built-in body sense, called proprioception).
The goal is to create a “virtual touch” signal that says yes/no for touch at each fingertip, so the robot can do delicate tasks like spinning or reorienting objects in its hand.
What questions were the researchers trying to answer?
In simple terms, they asked:
- Can a robot figure out “am I touching?” just from vision and its own joint positions, without real tactile sensors?
- Will this “virtual touch” work well enough to control tricky in-hand tasks, like rotating objects, including new shapes the robot hasn’t seen before?
- Which pieces of information are most important: the camera view, the hand’s pose, the intended motion, or changes over time?
How did they do it? (Methods in everyday language)
Think of the robot like a person wearing thin gloves: you can’t feel pressure well, but you can still guess you’re touching something by watching your hand and noticing whether your fingers move as expected.
They built a learning system (a kind of AI model called a transformer) that does three big things:
- See the scene: It takes in a short video clip of RGB-D frames (color + depth) from a wrist camera and turns them into compact visual “tokens” (like a summary of important visual spots).
- Feel its own body: It reads the hand’s current finger positions (where the joints actually are) and the commanded positions (where it’s trying to move next). These are like “now” and “intent.”
- Focus and think over time:
- Cross-attention: The “now” and “intent” signals act like pointers that tell the model where to look in the camera features (e.g., “look where the index finger should be”).
- Temporal reasoning: It looks across a short time window (a few recent frames) to spot patterns that reveal contact, like slight motion slowdowns, small misalignments, or persistent occlusions. A single frame can be confusing, but a tiny clip makes contact clearer.
Then it outputs a simple result: for each fingertip, a 0/1 contact prediction (no/yes). This is the virtual touch.
How they trained and tested it:
- In simulation: They used a robot hand (LEAP Hand) to rotate objects. The simulator provides ground-truth touch labels (so the model knows the right answers during training).
- In the real world: They stuck very thin force sensors on the hand only to measure accuracy (not to control the robot). During actual use, the robot relied only on camera + proprioception.
- They tried both “seen” objects (used in training) and “novel” objects (not seen before), like a hexagonal prism and a letter “R.”
- They also compared many versions of their approach, like using only vision, only pose, or removing the time component, to see what matters most.
Key terms explained quickly:
- RGB-D: A camera image that has both color (RGB) and per-pixel distance (Depth).
- Proprioception: The robot’s internal sense of its joint positions and motor commands (like knowing where your fingers are without looking).
- Transformer with attention: A model that learns to “focus” on the most useful parts of the input (for example, the bit of the image near a fingertip).
- Binary contact: A simple yes/no touch signal at each fingertip.
What did they find, and why is it important?
Main results (in clear points):
- Accurate virtual touch: The full model (vision + proprioception + time) predicted fingertip contact very well in simulation and performed strongly on a real robot. In simulation, accuracy (F1 score) was around 0.9 on many shapes; in the real world, it was typically around 0.8. This is quite good for a yes/no contact guess without real tactile sensors.
- Works on new objects: The model generalized to object shapes it never saw during training (like the hex prism and letter “R”), meaning it learned useful, transferable cues.
- Handles occlusion: When the camera view is blocked by the hand or object, vision alone struggles. Adding proprioception and time helps fill in the gaps, keeping predictions reliable.
- Better than simpler tricks: A “just geometry” baseline (using depth + fingertip positions) and “vision-only” were clearly weaker. Using pose-only did okay (because tracking errors can hint at contact) but still worse than the full mix. Temporal modeling (using a short clip) boosted performance compared to single-frame guesses.
- Good enough for real control: A robot policy trained in simulation with true touch sensors could be run later using only the model’s predicted contact, and it still rotated objects in hand well—sometimes even better than using real fingertip sensors on hardware (because real sensors can be noisy, while the predicted signal matches sim-trained expectations).
Why it matters:
- Touch sensors can be expensive, fragile, or hard to install everywhere on a hand. Predicting contact from camera + proprioception is cheaper, easier to scale, and can be good enough for many tasks.
- This makes dexterous in-hand manipulation more practical outside of carefully controlled labs.
What does this mean for the future?
Implications and potential impact:
- Cheaper, more robust robot hands: Robots could get many of the benefits of touch without needing lots of delicate tactile hardware.
- Ready for the real world: The method works with a single wrist camera and built-in joint sensing—setups that are common on many robots—making real deployment more realistic.
- Better manipulation skills: With reliable contact estimates, robots can adjust their grasps, prevent slips, and handle varied objects more confidently.
Limitations and next steps:
- Right now, the system predicts only yes/no touch and only at a few fingertip spots. In the future, it could estimate richer signals (like force strength or slip) and cover more of the hand (including the palm).
- It was trained on in-hand manipulation; broader tasks (like tool use or assembly) need more diverse training.
- Using more data and exploring different vision backbones could make it even stronger, possibly leading to a general “foundation model” for pseudo-touch.
Takeaway
NoContactNoWorries shows that a robot can “sense” touch without physical touch sensors by smartly combining what it sees, what it feels about its own body, and how things change over a short time. This virtual touch is accurate, works on new objects, and is strong enough to control real robot hands doing tricky in-hand moves—opening the door to more affordable and scalable dexterous robots.
Knowledge Gaps
Knowledge gaps, limitations, and open questions
Below is a single, actionable list of what remains missing, uncertain, or unexplored in the paper.
- Scope of contact: only four distal fingertip sites are modeled; extension to dense contact maps (full finger links, palm, lateral finger surfaces) and multi-contact interactions remains unaddressed.
- Binary contact only: no estimation of continuous signals (normal force magnitude, shear, torque, slip, micro-vibration) that are crucial for force-sensitive or deformable-object tasks.
- Task generality: training and evaluation target in-hand reorientation; transfer to other manipulation regimes (tool use, assembly, precision insertion, deformables) is not studied.
- Robot generality: results are limited to the LEAP hand; portability to different hands (e.g., Allegro, Shadow), different kinematics, and different joint sensing quality is unknown.
- Camera configuration: only a single wrist-mounted RGB-D view is used; benefits and costs of multi-view, eye-in-hand + external views, or active viewpoint control to mitigate occlusion are unexplored.
- Occlusion handling: the model remains robust but still degrades under heavy occlusion; explicit 3D scene reasoning (e.g., volumetric fusion, TSDF/implicit fields, NeRF-style reconstructions) is not evaluated as a remedy.
- Domain adaptation: the contact predictor is trained on simulated labels and evaluated on hardware without real-contact fine-tuning; systematic sim-to-real adaptation (self-training, adversarial or feature alignment, generative augmentation) is not explored.
- Label fidelity: “ground-truth” in sim (PhysX contacts) and real (FSR threshold at 0.1 N) differ in definition and noise; the impact of label mismatch and strategies to calibrate/align contact definitions across domains remain open.
- Dataset scale and diversity: training data (5 shapes, 50 trajectories) is small and limited in materials, surface properties, and textures; scaling laws, saturation effects, and performance under broader object diversity (transparent, reflective, soft, high/low friction) are unknown.
- Material and sensor failure modes: RGB-D failure cases (e.g., transparent/black/reflective surfaces, specularities, depth dropouts near minimum range) are not systematically stress-tested.
- Temporal modeling limits: only short windows (T=8) and a single-layer, single-head causal transformer are used; the effect of longer horizons, larger-capacity temporal models, or sequence distillation for latency reduction is not quantified.
- Latency and compute portability: inference latency is measured on a high-end GPU; performance on embedded compute (Jetson, CPU-only controllers), real-time determinism, and trade-offs between accuracy and compute are not evaluated.
- Sensor timing: robustness to sensor synchronization errors and time-varying delays between camera frames, joint encoders, and controller commands is not analyzed.
- Proprioceptive features: the method uses positions and commanded positions; the value of joint velocities, accelerations, torques/motor currents, or impedance setpoints for contact inference remains untested.
- Command availability: reliance on q_com assumes access to low-level commanded joints; generalization to platforms without explicit commanded states (or with different control stacks) is unclear.
- Uncertainty estimation: predictions are used as hard observations without calibrated confidences; uncertainty-aware outputs, threshold optimization, and downstream controllers that account for prediction reliability are not explored.
- Class imbalance: contact/no-contact imbalance handling (e.g., focal loss, reweighting, calibrated thresholds per site) is not discussed; the impact on recall under rare-contact regimes is unknown.
- Failure analysis: qualitative and quantitative characterization of failure modes (false positives under shadows/near-contact, false negatives under occlusion/fast motion) is limited.
- Comparative modeling: beyond a kinematic-depth heuristic, comparisons to learning methods with explicit 3D reasoning, optical flow/motion cues, or physics-informed models are absent.
- Backbone choice: a frozen asymmetric RGB-D segmentation backbone is used; systematic benchmarking of alternative vision encoders, depth-only/point-based backbones, or joint 2D–3D pretraining is left for future work.
- Multimodal fusion design: only one cross-attention layer with query asymmetry is studied; ablations on deeper cross-attention stacks, multi-head designs, token pruning, and spatially conditioned querying per fingertip are not provided.
- Generalization to new controllers: policies generating the data are PPO-GRU based; robustness of the contact predictor to different control policies, joint tracking characteristics, or controller bandwidth changes is not examined.
- End-to-end policy training: policies are trained with oracle contacts and deployed with predicted contacts; co-training or fine-tuning policies to explicitly exploit pseudo-tactile signals (and mitigate train–test sensor shift) is not investigated.
- Closed-loop stability: the effect of occasional contact mispredictions on controller stability, safety (object drops, excessive forces), and recovery behaviors is not analyzed.
- Hybrid sensing: the paper suggests fusing pseudo-tactile with real tactile when available, but does not implement or evaluate fusion strategies (Bayesian filters, learned gating, redundancy management).
- Spatial grounding: fingertip-specific queries in the vision-only baseline are learned embeddings; explicit grounding via forward kinematics projections into image/depth and learned region-of-interest features is not evaluated as a potential improvement.
- Object knowledge: leveraging known object meshes or online shape estimation to sharpen contact inference is not explored.
- Real-world calibration drift: robustness to extrinsic/intrinsic camera calibration drift, hand joint bias, and mechanical wear is not measured; online calibration or self-correction strategies are not provided.
- Environmental robustness: sensitivity to lighting changes, background clutter beyond the randomized simulation, and camera motion blur is not systematically assessed.
- Multi-contact transitions: performance during rapid regrasping, rolling contacts, and simultaneous multi-fingertip events (with or without intermittent occlusion) is not separately analyzed.
- Site-specific performance: per-fingertip accuracy differences (e.g., thumb vs. ring finger), and their dependence on viewpoint/occlusion geometry, are not reported.
- Metrics beyond F1: the impact of precision/recall asymmetry on downstream control (e.g., false-positive aversion vs. false-negative tolerance) and cost-sensitive tuning are not studied.
- Safety and compliance: without true tactile forces, regulating contact forces to protect delicate objects or ensure human safety is not demonstrated; integrating pseudo-tactile estimates with compliant/impedance control remains open.
- Data collection strategy: the use of multiple independently trained policies reduces leakage, but the effect of policy-induced data bias and active data collection (e.g., contact-seeking trajectories) on predictor generalization is not explored.
- Continual learning: methods for on-device adaptation from unlabeled experience (e.g., temporal consistency, physics constraints, self-supervision) to handle wear-and-tear and environment shifts are not considered.
- Dataset and reproducibility: availability of code, models, and datasets, and standardized benchmarks for pseudo-tactile contact estimation, are not detailed; broader community baselines are needed.
Practical Applications
Practical Applications of “NoContactNoWorries: Estimating Contact through Vision and Proprioception for In‑Hand Dexterous Manipulation”
The paper introduces a transformer-based “virtual tactile sensor” that infers binary fingertip contact from wrist-mounted RGB‑D and robot proprioception, enabling contact-aware control without physical tactile hardware. Below are actionable applications derived from the findings, methods, and system design.
Immediate Applications
These can be deployed now with modest engineering effort, using a wrist RGB‑D camera, joint state access, and standard robotics stacks.
- Virtual tactile sensor for robot hands and grippers — Robotics, Manufacturing, Logistics
- What: Replace binary touch sensors (e.g., FSRs) with the model’s pseudo-contact for grasp acquisition, adjustment, and in-hand reorientation.
- Tools/Products/Workflows: ROS2 node/driver exposing a K‑dim binary contact topic; integration with existing controllers as a gating signal; pre-trained model for LEAP/Allegro/Shadow hands; Isaac/PhysX-based data collection scripts.
- Assumptions/Dependencies: Wrist RGB‑D with proper calibration and synchronization; access to current and commanded joint states; object/task distribution not too far from training (convex or moderately complex shapes generalize better than highly occluded non‑convex shapes); binary contact suffices for the task.
- Sim-to-real deployment for contact-aware policies — Software/AI, Robotics
- What: Train policies in simulation with oracle contact; deploy on hardware with pseudo-contact without retraining, reducing reliance on fragile tactile sensors.
- Tools/Products/Workflows: PPO/GRU or diffusion-policy stacks augmented with a contact channel; deployment script to swap tactile inputs with pseudo-contact.
- Assumptions/Dependencies: Control loop latency budgets (~20 Hz) and inference latency (<10 ms) sustained; policy trained expecting binary contact; camera placement to minimize severe occlusions.
- Retrofit contact sensing for existing robot platforms — Robotics Integrators, OEMs
- What: Add “touch” capability to hands (LEAP, Allegro, Shadow) and simple grippers by mounting a wrist RGB‑D and running the model.
- Tools/Products/Workflows: Mounting kits for RealSense D455; extrinsic/intrinsic calibration utilities; packaged model weights; quick-start SDK.
- Assumptions/Dependencies: Mechanical clearance to place camera within depth sensor’s valid range; EGOCAM vantage point comparable to training; availability of commanded joint targets.
- Low-cost instrumentation for research labs and education — Academia
- What: Enable contact-rich research without purchasing or maintaining tactile sensors; teach contact-aware RL/control with commodity cameras.
- Tools/Products/Workflows: Open-source training and inference code, datasets, and lab exercises; automated contact labeling for recorded trajectories.
- Assumptions/Dependencies: Standard desktop GPU or optimized CPU inference; reproducible camera placement; adoption of the provided pre-trained encoder/backbone.
- Offline contact labeling for datasets and benchmarking — Academia, Software Tooling
- What: Post-hoc annotation of contact events from video+joint logs to create supervision for new tasks and evaluate contact-rich behaviors.
- Tools/Products/Workflows: Batch labeling pipeline; integration with dataset formats (e.g., DROID, RoboSet); visualization dashboards showing per-finger contact timelines.
- Assumptions/Dependencies: Time-synchronized RGB‑D and joint logs; similarity between logging setup and the model’s training conditions.
- Teleoperation HUD and operator feedback — Industrial Robotics, Remote Handling
- What: Visual overlays indicating per-fingertip contact in operator UIs to improve situational awareness under occlusions.
- Tools/Products/Workflows: UI widgets overlaying contact states; haptic buzzers triggered by contact onset/offset; ROS bridge to teleop software.
- Assumptions/Dependencies: Stable streaming latency; calibrated camera; tasks where binary contact feedback aids decisions.
- Safety interlocks and compliance monitoring — Human–Robot Collaboration, Factory Safety
- What: Use contact detection as a safety cue (e.g., halt when unintended contacts occur or proceed when expected contacts are confirmed).
- Tools/Products/Workflows: Rule-based safety logic (state machine) subscribed to contact signals; logging for safety audits.
- Assumptions/Dependencies: Binary contact suffices for the safety logic; false positive/negative rates acceptable for the risk profile; validated in the deployment environment.
- Sensor redundancy and failover — Field Robotics, Maintenance
- What: Provide a fallback “virtual touch” pathway when tactile hardware fails, or run redundant checks to improve robustness.
- Tools/Products/Workflows: Sensor fusion node combining tactile and pseudo-contact; watchdog that switches to pseudo-contact on fault.
- Assumptions/Dependencies: Sufficient visibility and proprioceptive fidelity; calibrated fusion thresholds.
- Energy/utilities and inspection manipulators — Energy, Infrastructure
- What: Binary contact cues for valve turning, knob manipulation, or latch engagement where tactile hardware is impractical outdoors.
- Tools/Products/Workflows: Ruggedized wrist camera; weatherproof housing; contact-gated hybrid position/force control.
- Assumptions/Dependencies: Robustness to lighting/weather; occasional occlusion tolerance; availability of joint states and commands.
- Prosthetics research prototypes — Healthcare
- What: Provide low-cost binary contact feedback to myoelectric prostheses using on-hand cameras and joint encoders for exploratory trials.
- Tools/Products/Workflows: Prosthetic wrist camera integration; translating contact states to vibrotactile feedback; clinical research protocols.
- Assumptions/Dependencies: Device form factor supporting a camera; alignment between kinematics and visual field; IR/lighting constraints for depth sensing.
- ROS2/SDK productization for integrators — Software, Robotics Platforms
- What: A turnkey “Pseudo-Tactile SDK” with ROS2 nodes, calibration tools, and examples for common hands and grippers.
- Tools/Products/Workflows: Docker images; launch files; example controllers; Isaac Gym data-generation recipes.
- Assumptions/Dependencies: Platform-specific URDFs and kinematics; camera drivers; license alignment for third-party backbones.
Long-Term Applications
These require further research, scaling to broader domains, more diverse data, and/or expanded sensing targets (e.g., force, shear).
- Dense pseudo-tactile maps and force/slip estimation — Advanced Manufacturing, Assembly, Deformable Object Handling
- What: Extend beyond binary contact to infer per-patch contact pressure, shear, and torque; enable fine force control, slip-aware manipulation, insertion, and threading.
- Dependencies: Richer supervision (e.g., GelSight/DIGIT co-training), improved vision backbones, better occlusion handling, and real-world force ground truth.
- General-purpose “foundation model” for contact — Cross-Sector Robotics
- What: Pretrain on large-scale, diverse hand–object interactions for zero-shot pseudo-contact across tasks (tool use, assembly, household manipulation) and robots.
- Dependencies: Broad, labeled or self-supervised datasets; standardized camera and kinematic metadata; scalable training regimes.
- Robust multi-view or view-agnostic inference — Warehousing, Home Robotics
- What: Reduce reliance on a single occluded wrist view using multi-view cameras or learned view-invariance; maintain contact accuracy under severe occlusions.
- Dependencies: Lightweight multi-camera rigs or learned 3D features; calibration-free or self-calibrating methods; synchronization.
- Embedded, low-power, on-device inference — Wearables, Mobile Platforms
- What: Run pseudo-contact on edge SoCs or MCUs for portable hands, prosthetics, and mobile manipulators.
- Dependencies: Model compression (quantization, pruning), efficient backbones, hardware accelerators.
- Standardized safety certification and test protocols for virtual contact — Policy, Standards Bodies
- What: Define benchmarks and certification tests for pseudo-contact accuracy, latency, and failure modes in safety-critical HRC scenarios.
- Dependencies: Consensus metrics (e.g., F1 under occlusion), cross-lab datasets, regulatory engagement.
- Tactile–vision–proprioception fusion as a product tier — Robotics OEMs
- What: Offer hybrid systems that fuse sparse tactile hardware with pseudo-contact for higher reliability, gracefully degrading when hardware fails.
- Dependencies: Sensor fusion algorithms; calibration and health monitoring; lifecycle management.
- Cross-robot and cross-task transfer without retraining — Integrators, Platforms
- What: Zero/few-shot adaptation of pseudo-contact models across different hands, kinematics, and tasks through structured conditioning and meta-learning.
- Dependencies: Unified kinematic representations; scalable adaptation procedures; diverse pretraining.
- Human haptics and XR/AR gloves — Consumer Tech, Training
- What: Use camera+IMU glove pose and vision to infer contact events, triggering haptic actuators for immersive training and telepresence without instrumented surfaces.
- Dependencies: Accurate hand pose tracking; occlusion-tolerant vision; real-time pipeline on wearables.
- Fleet analytics and quality assurance — Operations, Manufacturing IT
- What: Cloud services aggregating pseudo-contact logs for process monitoring, anomaly detection, and continuous improvement of manipulation routines.
- Dependencies: Secure data pipelines; standardized logging; privacy and IP policies.
- Foundation datasets and benchmarks for pseudo-tactile sensing — Academia, Consortia
- What: Curate multi-institution datasets with synchronized RGB‑D, proprioception, commands, and tactile ground truth to drive community progress and fair comparisons.
- Dependencies: Shared protocols, licenses, and data schemas; funding and consortium coordination.
Notes on feasibility and cross-cutting assumptions:
- Performance depends on camera placement, calibration, and occlusion; non-convex objects and heavy occlusions reduce accuracy.
- The current method predicts binary contact at fixed fingertips; tasks needing force, shear, or dense contact require further R&D.
- Real-time constraints are met on desktop GPUs (~8 ms), but embedded deployment needs optimization.
- Access to both current and commanded joint states is assumed; controllers that don’t expose commands may need adaptation.
- Domain shift (lighting, materials, hand geometry) may require fine-tuning or domain randomization during training.
Glossary
- AdamW: An optimizer that decouples weight decay from the gradient update for more stable training. "trained using AdamW with a constant learning rate of 1 × 10-4"
- Back-projection: Converting depth pixels into 3D points in the camera frame. "The depth image is back-projected (using camera intrinsics) into a local point cloud"
- Binary cross-entropy (BCE) loss: A loss function for binary classification that penalizes incorrect probability estimates. "with a binary cross-entropy loss over all timesteps T"
- Camera extrinsics: Parameters describing the camera’s pose relative to another reference frame. "calibrated extrinsics"
- Camera frustum: The 3D volume defining what the camera can see. "camera frustum"
- Camera intrinsics: Parameters describing the camera’s internal calibration (e.g., focal length, principal point). "using camera intrinsics"
- Causal attention: Attention that only uses past (and current) information, preventing peeking into the future. "captures temporal dynamics with causal attention"
- Causal Transformer: A Transformer that applies causal masking for sequence modeling over time. "a causal Transformer encoder"
- Closed-loop manipulation: Control that uses feedback from the current state to adjust actions continuously. "under closed-loop manipulation they can correlate with contact events"
- Cross-attention: An attention mechanism where queries from one modality attend to keys/values from another. "through a cross-attention mechanism"
- Cumulative Rotation Angle (CRA): The total angle rotated over an episode, used as a task metric. "Cumulative Rotation Angle (CRA), measured in radians"
- Cumulative Rotation Reward (CRR): A reward accumulated from rotation performance, used for evaluation in simulation. "Cumulative Rotation Reward (CRR), derived from the object's angular velocity"
- DIGIT: A compact, high-resolution vision-based tactile sensor. "GelSight [6] and DIGIT [7] measure rich contact signals"
- Domain randomization: Randomly varying simulation parameters to improve transfer to the real world. "we apply targeted domain randomization to both dynamics and perception."
- Euclidean distance: The straight-line distance between two points in Euclidean space. "minimum Euclidean distance"
- Feedforward (control): A control component that anticipates required actions from desired motion without relying solely on feedback. "feedback and feedforward signals jointly inform action"
- Force-sensitive resistor (FSR): A tactile sensor whose resistance changes with applied force. "force-sensitive resistors (FSRs) attached to the LEAP Hand fingertips"
- Forward kinematics: Computing the position of robot parts from joint angles. "via forward kinematics"
- GelSight: A high-resolution vision-based tactile sensor using elastomer deformation. "GelSight [6] and DIGIT [7] measure rich contact signals"
- Gated Recurrent Unit (GRU): A recurrent neural network cell used for sequence modeling. "a Gated Recurrent Unit (GRU)-based policy"
- Intel RealSense D455: An RGB-D depth camera commonly used in robotics. "Intel RealSense D455 camera"
- Kinematic depth baseline: A heuristic baseline that predicts contact by combining kinematics with depth proximity. "Kinematic depth baseline"
- LEAP Hand: A low-cost, anthropomorphic robotic hand platform. "LEAP Hand"
- Multi-head attention: Parallel attention heads that jointly attend to information from different representation subspaces. "shared multi-head attention"
- Occlusion/self-occlusion: Visual blockage of scene elements by other objects or by the robot’s own body. "self-occlusion during finger-object interaction"
- Oracle (tactile) signal: The idealized ground-truth tactile signal used as a reference or during training. "replace oracle/tactile contact"
- PhysX: NVIDIA’s physics engine used for simulation of dynamics and contacts. "physics engine (PhysX)"
- Point cloud: A set of 3D points representing the geometry of surfaces. "local point cloud"
- Pose-conditioned attention: Attention that conditions visual features on the robot’s current and commanded poses. "pose-conditioned attention mechanism"
- Proportional–Derivative (PD) controller: A controller using proportional and derivative terms to track desired positions. "proportional-derivative (PD) controller"
- Proprioception: Internal sensing of the robot’s joint states and motions. "Vision and proprioception provide global scene context and internal state"
- Proximal Policy Optimization (PPO): A policy-gradient reinforcement learning algorithm with clipped objectives. "trained using Proximal Policy Optimization (PPO)"
- Ray casting: Tracing rays through a scene to determine visibility or intersections. "ray casting"
- RGB-D: Combined color (RGB) and depth sensing modality. "RGB-D inputs"
- Rollouts: Sequences of states, actions, and observations collected by executing a policy. "rollouts collected in simulation"
- Semantic segmentation backbone: A pretrained network for per-pixel labeling used as a feature extractor. "RGB-D semantic segmentation backbone"
- Sigmoid activation: A logistic function mapping real values to probabilities in [0,1]. "element-wise sigmoid activation"
- Sim-to-real transfer: Deploying models trained in simulation on real hardware. "sim-to-real transfer"
- Spearman rank correlation: A nonparametric measure of monotonic association between variables. "Spearman rank correlation"
- Temporal median filtering: A time-domain filter that uses the median across frames to reduce noise. "temporal median filtering"
- Transformer (architecture): A neural network based on self-attention for sequence and multimodal learning. "a transformer-based multimodal framework"
- Visuo-tactile: Integrating visual and tactile sensing modalities. "visuo-tactile occupancy"
- Zero-shot generalization: Generalizing to unseen objects or tasks without additional training. "used to evaluate zero-shot generalization"
Collections
Sign up for free to add this paper to one or more collections.