Visuotactile Grasp Affordance Network

Updated 10 July 2026

Visuotactile Grasp Affordance Networks are models that combine visual observations with tactile and force data to generate dense heatmaps for optimal grasp locations.
They employ multimodal fusion techniques, including CNNs, attention modules, and support-conditioned architectures, to enhance prediction accuracy and task-specific performance.
These networks facilitate closed-loop manipulation, improving handling of deformable and occluded objects through iterative tactile feedback and adaptive control strategies.

Searching arXiv for recent and foundational papers on visuotactile grasp affordance, multisensory affordance learning, and related grasping/manipulation frameworks. A visuotactile grasp affordance network is a grasp-affordance model that estimates where, and sometimes how, a robot should grasp by combining visual observation with tactile or force-related evidence. In the literature represented here, the closest direct instantiation is the Attention-Guided Network (AGN), a support-conditioned model that predicts a dense $88\times 88$ graspability map from a current RGB image, an affordance query, previous exploratory interactions, and force/torque and tactile signals (Liang et al., 2022). Surrounding work broadens the concept in three directions: task-conditioned grasp suitability rather than generic stability, tactile supervision or post-contact validation rather than vision-only inference, and closed-loop manipulation in which affordance prediction, exploration, and tactile assessment are mutually coupled.

1. Conceptual basis and historical framing

The modern notion of grasp affordance in this line of work is explicitly task-dependent. In the task-oriented metric framework of (Cavalli et al., 2019), affordance is not merely force closure or grasp robustness; it is a task score $F_T(O,G,U)$ over object $O$ , grasp $G$ , and use point $U$ , approximated by physically meaningful metrics such as robustness, rotational inertia, hand effort on impact, hand effort on hold, momentum discharge efficiency, force transmitted to use, and use local geometry. This formulation established a clear distinction between “stable enough to hold” and “suitable for a downstream action.”

A closely related shift appears in self-assessed affordance transfer. SAGAT defines grasp affordance as task-dependent grasp suitability for actions such as pouring, handover, or shaking, and reframes the problem as selecting a grasp configuration $g_p^*$ inside a visually detected affordance region $G^*$ so that the task policy succeeds (Ardón et al., 2020). “Self-assessment” in that framework means predicting, before execution, whether a grasp candidate is likely to support task completion by forward simulating task outcome, comparing the predicted effect with previously demonstrated successful effects, and assigning a confidence score.

Taken together, these works place visuotactile grasp affordance networks within a broader transition from appearance-based graspability to outcome-aware, task-aware, and contact-aware grasp reasoning. A plausible implication is that “affordance” in this literature denotes a relational property among object geometry, task semantics, contact state, and execution policy, rather than a fixed visual attribute.

2. Affordance representations

The dominant representation in visuotactile grasp-affordance work is a dense spatial field, usually a heatmap over image pixels. In AGN, the affordance map $M$ is a pixelwise probability map over the input image, and the pixel with highest value is interpreted as the best manipulation location for the queried affordance (Liang et al., 2022). For grasping specifically, the model predicts a dense $88\times88$ graspability heatmap indicating which image regions are likely to permit stable grasp.

Other visuotactile systems use related but not identical affordance targets. In cloth manipulation, the edge-grasp affordance network predicts a pixelwise heatmap over a depth image of a hanging towel, specialized to whether a candidate point corresponds to a true outer edge, a collision-free approach, and a single slideable layer (Sunil et al., 2022). In reactive in-air clothing manipulation, the affordance model again outputs a dense per-pixel map, but the supervision is derived first from simulated geometric side-grasp criteria and then from tactile outcome labels produced by a classifier during robot data collection (Sunil et al., 4 Sep 2025). For deformable-object grasp-state assessment, the target is not a heatmap but a 3-class state label—sliding, appropriate, excessive—computed from visual-tactile sequences during an executed grasp (Cui et al., 2020). That model does not localize grasp points, but it does learn a contact-state affordance signal that distinguishes insufficient force from safe grasping and over-force deformation.

Representative system	Modalities	Output
AGN (Liang et al., 2022)	RGB + force/torque + tactile + prior explorations	$1\times88\times88$ affordance heatmap
Edge grasp affordance (Sunil et al., 2022)	Depth + tactile supervision	Pixelwise edge-grasp heatmap
Reactive clothing affordance (Sunil et al., 4 Sep 2025)	Depth + tactile self-supervision	Dense per-pixel affordance map
C3D-VTFN (Cui et al., 2020)	Visual-tactile sequences	3-class grasp-state label

These representations differ in what “touch” contributes. In AGN, tactile and force/torque features are direct inputs to affordance inference. In the cloth systems, tactile sensing mainly supplies supervision and execution-time verification. In deformable grasp-state assessment, tactile is part of the fused inference input but the prediction concerns grasp condition rather than grasp location. This suggests that visuotactile affordance can enter at three levels: as an input modality, as a supervisory signal, or as an execution-time validator.

3. Network architectures and multimodal fusion strategies

The most explicit visuotactile grasp-affordance architecture in this set is AGN. It consists of a Current Query Block (CQB), a Previous Explorations Block (PEB), and an Up-Sampling Block (USB) (Liang et al., 2022). The CQB processes the current $F_T(O,G,U)$ 0 RGB image through a 3-layer CNN and combines it with a one-hot affordance query encoded by an MLP, producing a $F_T(O,G,U)$ 1 latent feature map. The PEB processes each previous exploration image with a CNN, applies a channel attention module supervised by the previous manipulation region, encodes force/torque and tactile inputs through an MLP, and fuses the visual and touch features into a 64-dimensional vector. Previous explorations are then split into positive and negative groups, averaged separately, and tiled into two $F_T(O,G,U)$ 2 prototype maps. The USB concatenates the current-query map with the positive and negative support maps and decodes them through three ConvTranspose/Conv2d groups to a final $F_T(O,G,U)$ 3 affordance heatmap. Architecturally, this is a support-conditioned, multimodal segmentation network rather than a grasp-pose regressor.

Not all relevant visuotactile work fuses touch in this manner. In in-hand 6D pose estimation, tactile sensing is converted into an object-surface point cloud and fused with RGB-D using DenseFusion-style dense local correspondence, with a separate global feature branch and per-point confidence prediction (Dikhale et al., 4 Jan 2026). Although the target is pose rather than affordance, the representation is directly transferable: tactile is treated as 3D contact geometry rather than as raw taxels or a single force vector. This suggests that a visuotactile grasp affordance network can plausibly operate in point-cloud space and reason over hidden surfaces and contact patches under gripper-induced occlusion.

An earlier alternative is early fusion in a shared volumetric frame. In multi-modal geometric learning, depth and tactile contacts are voxelized into a common $F_T(O,G,U)$ 4 occupancy grid and passed through a 3D CNN for full-object completion (Watkins et al., 2018). The model is not an affordance predictor, but it demonstrates that sparse tactile contacts can induce non-local updates in hidden-shape hypotheses. A plausible implication is that affordance prediction can be factorized into visuotactile geometry completion followed by an affordance head, or learned jointly with shared 3D features.

4. Supervision, datasets, and optimization

Supervision in visuotactile grasp-affordance work ranges from dense simulation labels to tactilely derived sparse labels. AGN uses a binary-cross-entropy-like loss for the current affordance map plus an MSE attention loss on prior exploration maps, trained with Adam, learning rate $F_T(O,G,U)$ 5, weight decay $F_T(O,G,U)$ 6, and 1000 epochs (Liang et al., 2022). The YCBUSR benchmark contains 2868 samples over 9 object classes with 5-fold cross-validation and seen/unseen evaluation. On the authors’ own grasping dataset, AUROC rises from $F_T(O,G,U)$ 7 and $F_T(O,G,U)$ 8 for the baseline to $F_T(O,G,U)$ 9 and $O$ 0 for AGN, and to $O$ 1 and $O$ 2 for AGN-T on seen and unseen objects, respectively. A tactile-only variant yields $O$ 3 on seen objects and $O$ 4 on unseen objects, which the authors interpret as evidence that distributed tactile sensing carries more affordance-relevant information than force/torque alone.

In cloth manipulation, simulation pretraining is also central, but the label semantics are different. The edge grasp affordance network is trained in simulation from geometric criteria including edge percentage, no collision, single layer, and reachability, using 200 cloth configurations rotated in $O$ 5 increments for 4800 depth images with corresponding affordance maps (Sunil et al., 2022). Real adaptation then uses roughly 3000 robot grasps in 15 hours, with labels supplied by a tactile temporal classifier that distinguishes edge, corner, fold, all fabric, and no fabric. The final Sim2Real + fine-tuned model achieves $O$ 6 precision@40 and $O$ 7 true edge grasp success rate in deployment.

The reactive clothing system extends that tactile-supervision paradigm. Its affordance U-Net is first pretrained on 300 unique cloth configurations rotated in $O$ 8 increments, for 3600 images total, using simulation labels derived from right-arm reachability, collision-free side approach, and a maximum of two cloth layers between gripper fingers (Sunil et al., 4 Sep 2025). Real-world fine-tuning then uses on the order of $O$ 9 grasp attempts, with tactile outcome labels supplied by a classifier trained on 350 grasps across approximately 20 shirts. The fine-tuning objective includes a neighboring-pixel weighted loss, spatial smoothness regularization, simulation-consistency regularization, and optimizer weight decay: $G$ 0

$G$ 1

$G$ 2

Offline evaluation on 125 human-labeled grasp points reports precision@80 of $G$ 3 for Sim2Real, $G$ 4 for Real2Real, and $G$ 5 for Sim fine-tuned on Real.

For deformable objects, supervision is obtained by extensive grasping and lifting experiments with different widths and forces on 16 deformable objects, yielding approximately 20,000 visual sequence samples of length 5 with corresponding tactile sequences (Cui et al., 2020). The C3D-VTFN, trained with cross-entropy using Adam at learning rate $G$ 6 and batch size 8, reports classification accuracy as high as $G$ 7 for sliding, appropriate, and excessive grasp states.

5. Closed-loop execution and application regimes

A defining feature of visuotactile grasp-affordance systems is that affordance prediction is often embedded in a sequential manipulation loop rather than used once as an open-loop detector. In AGN, the affordance map is part of the state in a POMDP formulation, together with a location map from an AlexNet-based classifier, and a DQN policy chooses moving direction among 8 actions and moving step size among 5 actions (Liang et al., 2022). The reward is

$G$ 8

where $G$ 9 is the number of movement steps. The best practical result is that AGN-T+RL with only 2 explorations outperforms non-RL methods even with 5 explorations, summarized by at least $U$ 0 grasping improvement and at least $U$ 1 pushing improvement over non-RL methods using 5 explorations.

In cloth edge manipulation, the affordance network proposes grasp points, but tactile sensing decides whether the attempted grasp truly captured a slideable edge (Sunil et al., 2022). Once an edge is grasped, a tactile pose-estimation network operating at 33 Hz estimates edge center and orientation, enabling local servoing along the edge until the adjacent corner is reached. The same work reports a $U$ 2 edge-grasp success rate for the visuotactile affordance network and shows that the affordance stage and tactile control stage are functionally distinct: the former selects a likely graspable edge region, whereas the latter keeps the edge centered and aligned during sliding.

The reactive garment system generalizes this pattern to highly occluded in-air clothing. Grasp is executed only if both dense correspondence confidence and grasp affordance exceed predefined thresholds; otherwise the robot rotates the garment and re-evaluates (Sunil et al., 4 Sep 2025). After execution, the same tactile classifier used for self-supervised affordance fine-tuning is reused online for real-time grasp validation, with reported accuracies of $U$ 3 on the right arm and $U$ 4 on the left arm. In folding experiments, 6 empty grasps out of 30 total grasps over 10 trials were caught by tactile classification and triggered recovery.

A broader but relevant execution setting is in-hand manipulation under heavy self-occlusion. The visuotactile pose-estimation system shows that under occlusion above $U$ 5, position error improves from $U$ 6 cm for vision-only to $U$ 7 cm for visuotactile estimation, and angular error improves from $U$ 8 to $U$ 9 (Dikhale et al., 4 Jan 2026). Although the target is pose rather than grasp affordance, the result directly supports the execution-time value of touch when the gripper hides the very surfaces most relevant to stable contact.

6. Limits, adjacent paradigms, and recurring misconceptions

A recurrent misconception is that any affordance-based grasp model is already visuotactile. Several recent systems are affordance-aware but remain purely visual or vision-language. Open-vocabulary task-oriented frameworks infer object relevance and part-level affordances from language and RGB or RGB-D, then constrain grasp generation to the inferred part or mask; this is the case for instruction-conditioned part grounding and affordance-constrained grasping in clutter (Tang et al., 2 Mar 2025), dense language-conditioned visual affordance maps with geometric execution via superquadrics (Ma et al., 2024), and zero-shot part-based heatmap construction with positive and negative task-conditioned regions (Tong et al., 25 Nov 2025). These systems are important front-ends for semantic grasping, but they do not model contact, force, slip, compliance, or tactile verification.

A second misconception is that affordance prediction must be a coarse heatmap over object parts. Recent visual-only work shows a trend toward more action-ready representations, including multi-keypoint affordance structures that encode functional finger, little finger, and wrist correspondences, and convert them analytically into dexterous hand pose through Keypoint-based Grasp matrix Transformation (Yang et al., 27 Feb 2025). Another line uses structured grasp taxonomy, contact semantics, and functional affordance as conditioning for dexterous grasp generation (Zhang et al., 3 Dec 2025), while diffusion-based semantic grasp synthesis introduces an intermediate affordance map $g_p^*$ 0 and a Distribution Adjustment Module to refine grasp latents under semantic and contact constraints (Wu et al., 9 Mar 2026). These are not tactile methods, but they indicate that affordance is increasingly treated as a contact structure rather than a generic saliency field.

The principal limitations of current visuotactile grasp-affordance systems are also consistent across domains. AGN is support-conditioned rather than one-shot, requires prior explorations, and outputs 2D contact-location-centric heatmaps rather than full grasp pose parameters (Liang et al., 2022). Cloth systems are specialized to side grasps and to narrow local objectives such as edge capture or graspability under layer-count constraints (Sunil et al., 2022). Tactile self-supervision can be noisy when the gripper accidentally catches cloth in front of the intended target, so positive tactile outcome does not always imply that the queried visual pixel was truly graspable (Sunil et al., 4 Sep 2025). Deformable-object grasp-state assessment is highly effective for force selection and online correction, but it does not propose grasp locations (Cui et al., 2020).

The field therefore exhibits a stable architectural decomposition. Vision supplies global geometry, object identity, or candidate regions; touch supplies local contact truth, hidden-surface evidence, or post-contact validation; affordance acts as the intermediary between them. This suggests that the mature form of a visuotactile grasp affordance network is not simply a two-branch fusion module, but a layered system in which task semantics, visual spatial priors, tactile contact evidence, and execution policy are represented separately and then coupled through dense maps, keypoints, or contact-centric latent variables.