---
title: Referring Grasp Affordance (RGA)
url: https://www.emergentmind.com/topics/referring-grasp-affordance-rga
type: topic
---

# Referring Grasp Affordance (RGA)

Referring Grasp Affordance (RGA) denotes a class of language-conditioned robotic grasping problems in which visual observations and a referring signal are used to identify not only the target object but also the grasp-relevant region or grasp family that should be selected for execution. In recent formulations, the output is often a pixel-wise affordance tensor over position and discretized orientation, such as \(Q_g \in \mathbb{R}^{H \times W \times N}\) or \(A \in \mathbb{R}^{H \times W \times N}\), from which a grasp is extracted by \(\arg\max\) over pixels and angle bins; in neighboring work, the same role is played by an affordance mask, a transferred contact point, or a behavioral manifold in relative pose space [2409.19457] [2509.08126] [2507.23734]. RGA is therefore best understood as an intermediate-level formulation between referring expression grounding and grasp execution: the referring signal specifies which object, part, or action-relevant region matters, and the grasping system converts that specification into an executable grasp under geometric, semantic, and task constraints.

## 1. Intellectual lineage and task boundary

RGA emerged from several previously separate lines of work. One line treated grasp affordance as a semantic reasoning problem. A knowledge-base formulation used predicates such as `hasShape`, `hasTexture`, `hasMaterial`, `canBeFound`, `hasAffordance`, `hasCategory`, and `graspRegion`, and modeled their relations with a Markov Logic Network so that grasp affordance prediction could return multiple hypotheses rather than a single grasp label [1906.09836]. Another line defined task-oriented affordance functions over object, grasp, and use point, \(F_T:(O,G,U)\mapsto \mathbb{R}\), so that grasp quality could be evaluated relative to beating, cutting, or picking rather than only by generic stability [1907.04761]. A third line incorporated environmental context into grasp-action affordance reasoning and reported that adding environment features improved average diagonal affordance-classification accuracy from \(92.57\%\) to \(96.81\%\) [1905.10610]. A fourth line selected task-suitable grasps by forward-simulating task execution and comparing simulated outcomes to prior successful outcomes, improving average task success from \(79.2\%\) to \(85.4\%\) [2007.02132].

These antecedents did not yet define a full language-grounded RGA problem. Their typical assumptions were that the object was already identified, that the task label was known, or that affordance categories were supplied by visual semantics or human teaching rather than by free-form language. Even so, they established several principles that remain central to RGA: grasp choice is task-dependent; multiple affordance hypotheses can coexist on the same object; and intermediate affordance representations are often more useful than direct one-shot grasp regression [1904.02530] [1906.09836] [1907.04761].

Modern RGA formulations make these principles explicit by placing language-conditioned affordance prediction at the center of the pipeline. A CLIP-based parameter-efficient framework defines RGA as prediction of pixel-wise grasp affordance maps \(Q_g \in \mathbb{R}^{H \times W \times N}\) from RGB, depth, and a referring expression, with grasp extraction by \((x^*,y^*,\theta^*)=\arg\max_{(x,y,\theta)}Q_g(x,y,\theta)\) and \(z=D(x^*,y^*)\) [2409.19457]. OGRG studies the same task under weak supervision and uses an affordance volume \(A \in \mathbb{R}^{H \times W \times N}\) together with a segment-then-grasp pipeline [2509.08126]. Large-scale affordance segmentation work, although not always naming the task RGA, frames the same core problem as predicting an instruction-conditioned affordance map that is then converted with depth into a 3D grasp proposal [2507.23734].

## 2. Affordance as an explicit intermediate representation

A defining feature of RGA research is the use of an explicit affordance representation between language and grasp execution. In the most direct formulations, this representation is a dense image-space tensor. The CLIP-based RGA model predicts \(Q_g \in \mathbb{R}^{H \times W \times N}\), where each of the \(N\) channels corresponds to a discrete grasp rotation angle; OGRG uses the same basic structure, \(A \in \mathbb{R}^{H \times W \times N}\), with \(N=6\) angle bins spaced by \(30^\circ\) and a fixed gripper width \(l^*\) in the weakly supervised setting [2409.19457] [2509.08126]. In these models, affordance is already action-shaped: it is not merely an object mask, but a score field over graspable locations and orientations.

A closely related representation is the instruction-conditioned affordance mask. RAGNet and AffordanceNet define a pixel-wise affordance map for the grasp-relevant region of the referred object, then construct a masked point set \(\hat{P}=P\otimes M\) and back-project each valid affordance pixel with depth into 3D using camera intrinsics and extrinsics [2507.23734]. AffordanceGrasp-R1 follows the same logic but inserts an intermediate box-and-point grounding stage, \((B,P)=F(I,T)\), then decodes an affordance mask \(\hat{M}=O(I;B,P)\), lifts it to a 3D subcloud \(\hat{S}\), and uses that subcloud to filter global-scene grasp candidates [2602.03547]. AffordGrasp, in the task-oriented grasping sense, also grounds a pixel-level affordance mask \(\mathcal{M}_{p^*}\) after GPT-4o infers the explicit task \(\mathcal{T}\), object \(\mathcal{O}\), optimal part \(p^*\), and corresponding affordance \(a^*\) from instruction and image [2503.00778].

Other papers use geometry-aligned affordance representations rather than image masks. AffordGrasp for human-hand synthesis introduces an intermediate pointwise affordance map \(P_a\) over the object point cloud \(P_g\), explicitly linking language semantics to object geometry before latent diffusion generates a MANO hand pose [2603.08021]. AffordDexGrasp defines a per-point affordance map \(\Omega=\{a_i\}_{i=1}^{M}\), constructed from grouped dexterous grasps that share intention, contact part, and grasp direction, and uses that map as the intermediate variable between language and dexterous grasp generation [2503.07360].

A distinct but highly relevant alternative is the behavioral manifold representation. Rather than predicting one best grasp pose, grasp affordance can be represented as a partition of relative pose space \(\mathcal{P}\) under a synthesis policy \(\mathcal{G}:\mathcal{P}\rightarrow\mathcal{C}\), where connected regions induce consistent grasp types after execution. On a mug, this produces separate manifolds for force-closure around the body, pinching around the body, force-closure around the handle, and pinching on the handle [2405.04188]. This is not a learned latent manifold; it is a geometric-behavioral manifold induced by kinematics, contact, and policy execution. For RGA, this matters because a phrase such as “grasp the mug by the handle” naturally selects a region or family of regions, not a single point estimate [2405.04188].

Taken together, these representations suggest that contemporary RGA increasingly treats affordance as an explicit intermediate modality rather than as an implicit byproduct of direct grasp regression. That inference is consistent with the repeated use of masks, pointwise affordance fields, and behaviorally induced manifolds as the bridge between semantic grounding and control [2405.04188] [2603.08021].

## 3. Referring signals and grounding mechanisms

The referring signal in RGA is not uniform across the literature. Some systems use free-form language, some use reasoning instructions, and some replace language with embodied gesture. What unifies them is a two-stage structure in which a coarse cue identifies the target and a finer cue specifies the intended affordance or grasp style.

Large-scale instruction-conditioned affordance segmentation provides the clearest language-grounded examples. RAGNet contains \(273\)k images, \(180\) categories, and \(26\)k reasoning instructions, and explicitly increases language difficulty by removing category names from hard instructions and replacing them with functional descriptions such as “I need something to drink coffee” or “Use to pierce and lift food” [2507.23734]. OGRG further emphasizes open-form expressions and duplicated object instances, and evaluates absolute and relative spatial language in scenes containing repeated objects, including phrases such as “bottom center dice” and “the tissue box that is to the upper right of the green cylinder green cup” [2509.08126]. AffordanceGrasp-R1 makes the grounding process explicit by training Qwen2.5-VL-7B with CoT cold-start and GRPO so that it predicts a bounding box \(B\) and point prompt \(P\), which are then decoded into an affordance mask by SAM 2 with LoRA; prompt consistency is enforced by discarding samples with \(\mathrm{IoU}(M,\hat{M})<0.6\) [2602.03547].

A second paradigm uses in-context reasoning over language and scene rather than direct dense grounding. AffordGrasp processes an instruction \(\mathcal{L}\) and image \(\mathcal{I}\) with GPT-4o to infer \(\mathcal{T}, \mathcal{O}, p^*, a^*=\text{GPT-4o}(\mathcal{L},\mathcal{I})\), then uses VLPart first to localize the object box \(\mathcal{B}_{O}\) and then to predict the affordance mask \(\mathcal{M}_{p^*}\) within the object crop [2503.00778]. This mediated grounding is not phrased as classical referring-expression segmentation; instead, the language first resolves task and object, then part and affordance.

A third paradigm substitutes gesture for language while preserving the computational structure of referring affordance grounding. GAT-Grasp takes a binocular visual observation \(O\), a pointing gesture \(P\), and a grasp gesture \(G\); the pointing ray determines a target crop \(I^T\), the grasp gesture retrieves similar human-object interaction exemplars from a memory bank \(\mathcal{M}=\{(G_{m,i}^{\diamond}, I_i^S, C_i^S)\}_{i=1}^{N}\), CLIP reranks source images by similarity to the target crop, and DIFT-based dense correspondence transfers the source contact point \(C^S\) to the target \(C^T\) [2503.06227]. This system achieved an average success rate of \(51.67\%\) across nine cluttered object-part conditions, outperforming GPT-4o, Qwen-VL, Robo-ABC, and RAM in that evaluation [2503.06227]. Although the referring signal is geometric rather than symbolic, the decomposition into coarse target localization and fine affordance disambiguation is directly transferable to RGA.

Language-conditioned human-hand synthesis occupies an intermediate position. AffordGrasp augments HO-3D, OakInk, GRAB, and AffordPose with generated instructions and predicts an affordance map \(P_a\) from \((P_g,I)\) before diffusion-based hand generation [2603.08021]. AffordDexGrasp uses GPT-4o as a pre-understanding stage to extract object category, intention, contacting part, and one of six canonical grasp directions, then rewrites the instruction into a compact sentence such as “use the mug from the left by contacting the handle” before predicting an affordance map and a dexterous grasp [2503.07360]. These systems are object-centric rather than scene-centric, but they show that part-sensitive and intent-sensitive grasp differentiation can be driven by textual conditioning [2603.08021] [2503.07360].

## 4. Grasp synthesis, ranking, and execution

Once a referring signal has been grounded into an affordance representation, the next question is how that representation constrains grasp generation. Three recurrent strategies appear in the literature: direct affordance argmax, candidate generation with affordance-aware ranking, and policy- or dynamics-aware optimization.

The most direct strategy is to interpret the affordance tensor itself as the grasp decision surface. In the CLIP-based RGA formulation, the selected grasp is \((x^*,y^*,\theta^*)=\argmax_{(x,y,\theta)}Q_g(x,y,\theta)\) with depth taken from the input depth image [2409.19457]. OGRG uses the same argmax rule on \(A(x,y,\theta)\), differing mainly in that affordance prediction is conditioned on the grounded object mask and trained with only single-pixel binary grasp labels [2509.08126]. These formulations are compact and deployment-oriented, but they also inherit the limitations of planar angle discretization and fixed-width or simplified gripper parameterization.

The second strategy separates grounding from grasp candidate generation. AffordanceGrasp-R1 generates 6D grasp candidates from the full scene point cloud with GA-Grasp, lifts the predicted affordance mask to a 3D target subcloud \(\hat{S}\), assigns each grasp a 3D IoU with the semantics-aligned target volume, keeps top candidates by IoU, then ranks by grasp confidence and applies NMS. In robot experiments on 10 language-conditioned grasping tasks, this design achieved \(80\%\) average success on easy instructions and \(72\%\) on hard instructions, outperforming AffordanceNet’s \(62\%\) and \(50\%\) in the same setting [2602.03547]. GAT-Grasp similarly combines contact-point transfer and human-hand-derived orientation \(R_h\) with candidate grasps from HGGD, selecting \(G^*=\arg\max_{G_i}\left(s_i-\lambda\|I-R_h^TR_i\|_F\right)\) so that physically plausible grasps are ranked by agreement with the intended grasp orientation [2503.06227]. Aff-Grasp constrains Contact-GraspNet with the predicted graspable affordance region and then executes task-specific motion primitives, reporting \(97.2\%\) correct affordance prediction and \(80.6\%\) successful grasping on seen objects in one robot evaluation [2408.10123].

The third strategy makes execution policy or dynamic stability part of the affordance itself. The behavioral-manifold framework defines a synthesis policy \(\mathcal{G}\) over relative pose space and evaluates grasp quality with a dynamic local-sensitivity metric \(\mu_s\) derived from object acceleration sensitivity to gripper-base acceleration. Static metrics such as \(\mu_g\), \(\mu_f\), and \(\mu_m\) are compared to \(\mu_s\), and the dynamic metric can favor grasps that are more resilient under manipulation while requiring lower torque. For the power drill, the paper reports \(v_c=136.6\), \(v_s=158.3\), \(\mu_c=0.53\), and \(\mu_s=0.87\) when comparing baseline and shape-informed synthesis, indicating both a larger viable basin and higher best-obtained quality under shape adaptation [2405.04188]. In task-oriented metric learning, affordance is similarly defined as a function of grasp and use point, \(F_T:(O,G,U)\mapsto\mathbb{R}\), with beating, cutting, and picking derived from primitive metrics such as epsilon quality, rotational inertia, force transmitted to use, and hand effort on hold [1907.04761]. These formulations are highly relevant to RGA because referring expressions often specify not only where to grasp but what the grasp must support afterward.

Human-hand grasp synthesis extends the same pattern beyond robotic grippers. AffordGrasp uses separate PointNet encoders for object geometry and affordance geometry, latent diffusion conditioned on \((I,P_g,P_a)\), and a Distribution Adjustment Module to improve semantic consistency and physical realism without test-time optimization; it reports semantic accuracy gains such as \(80.08\%\) on OakInk and \(72.00\%\) on out-of-domain HO-3D [2603.08021]. AffordDexGrasp uses Affordance Flow Matching to generate a pointwise affordance map and Grasp Flow Matching to generate dexterous grasps, with test-time optimization that preserves affordance-consistent contact while reducing penetration [2503.07360]. These are not drop-in robotic RGA systems, but they show that affordance-mediated grasp generation scales to higher-dimensional embodiments.

## 5. Data regimes, supervision, and evaluation

RGA research is strongly shaped by annotation strategy. Fully supervised pipelines require dense affordance masks or dense grasp maps; weakly supervised pipelines use sparse point or success/failure labels; retrieval-based pipelines replace dense annotation with memory banks of prior interaction examples; simulation-heavy methods generate large volumes of supervisory data under controllable conditions.

| Resource or system | Scale or supervision | Primary RGA role |
|---|---|---|
| RAGNet [2507.23734] | \(273\)k images, \(180\) categories, \(26\)k reasoning instructions | Language-conditioned affordance segmentation benchmark |
| OGRG-RGA [2509.08126] | \(>16{,}000\) visual-language-grasp triplets; single-pixel grasp labels | Weakly supervised affordance learning |
| AED and Aff-Grasp [2408.10123] | 721 images, 13 objects, 8 affordance classes | Precise affordance segmentation and robot evaluation |
| AffordGrasp [2603.08021] | Instruction-augmented HO-3D, OakInk, GRAB, AffordPose | Language-conditioned human grasp generation |
| AffordDexGrasp [2503.07360] | 33 categories, 1536 objects, 1909 scenes, 43,504 dexterous grasps | Open-set language-guided dexterous affordance |

The benchmark most explicitly aligned with language-conditioned affordance grounding is RAGNet. On HANDAL, AffordanceNet reached gIoU \(60.3\) and cIoU \(60.8\); on GraspNet novel it reached \(45.6\) and \(33.2\); and on 10 real-robot grasping tasks it achieved an average success rate of \(70\), compared with \(32\) for GraspNet in that study [2507.23734]. AffordanceGrasp-R1 pushed the same line further, reporting average gIoU/cIoU of \(66.7/65.9\) across main subsets, \(67.4/69.0\) on reasoning subsets, and real-robot grasp success of \(80\%\) on easy and \(72\%\) on hard language-conditioned tasks [2602.03547].

Weak supervision remains a major theme. OGRG’s RGA setting uses only a single labeled grasp pixel on one angle slice with a binary \(\{0,1\}\) label, rather than dense grasp maps, and still reports average simulation grasp success of \(93.42\%\) on seen-background tests and \(70.8\%\) real-robot grasp success, with grounding accuracy of \(87.5\%\) [2509.08126]. The earlier CLIP-based parameter-efficient RGA framework reported \(88.56\%\) average simulation success with depth and \(75.0\%\) real-robot grasp success in its setting, again using angle-indexed affordance maps rather than dense rectangle supervision [2409.19457].

Data can also be harvested from interaction rather than manually labeled. GAT-Grasp builds a gesture-conditioned affordance memory from a subset of HOI4D plus manual data and performs “zero-shot affordance transfer without additional training” at the task level [2503.06227]. Aff-Grasp automatically derives graspable and functional affordance masks from EPIC-Kitchens and Ego4D via hand-object and tool-object interactions, then evaluates on AED, where Geometry-guided Affordance Transformer reaches mIoU \(68.62\), F1 \(81.09\), and accuracy \(83.51\), surpassing OOAL by \(15.9\%\) in mIoU [2408.10123]. Human-hand systems likewise reuse existing interaction corpora: AffordGrasp pseudo-labels OakInk and GRAB from AffordPose affordance classes, then generates instructions with Qwen [2603.08021], while AffordDexGrasp groups grasp data by intention, contact part, and direction to construct affordance labels [2503.07360].

Evaluation protocols reflect the dual nature of RGA. One family of metrics measures grounding quality, such as gIoU, cIoU, or oIoU for affordance masks and grounding masks [2507.23734] [2509.08126]. Another measures semantic correctness, such as ACC for whether the generated grasp matches the intended affordance class [2603.08021]. A third measures execution, usually by grasp success rate or task success on real robots [2408.10123] [2602.03547]. A persistent methodological point is that RGA systems are evaluated not only on whether they grasp successfully, but on whether they grasp the correct object, at the correct region, for the intended purpose.

## 6. Limitations, misconceptions, and open directions

A frequent misconception is that RGA is merely referring expression segmentation followed by off-the-shelf grasping. The literature does not support that reduction. Multiple papers argue that the target of prediction is not the whole object mask but the grasp-relevant region, part, or grasp family, and that downstream grasp ranking must account for semantics, task, or dynamic robustness rather than generic force closure alone [2507.23734] [2405.04188]. Conversely, not every affordance-aware grasping paper is a full RGA system. Several influential methods remain object-centric, task-centric, or embodiment-specific, without multi-object referential disambiguation.

The limits are explicit in many papers. The behavioral-manifold formulation has no natural language grounding, no part detector or semantic attribute model, no multi-object referent disambiguation, and no global arm reachability model; the manifold is also policy-dependent, so changing the synthesis controller changes the affordance landscape [2405.04188]. AffordGrasp for human-hand synthesis is strong on object-part- and action-conditioned generation, but remains mostly single-object and uses a 10-class affordance taxonomy rather than unrestricted language [2603.08021]. RAGNet and AffordanceNet provide hard function-based instructions, but the benchmark does not emphasize long relational chains or richer temporal task grounding [2507.23734]. AffordanceGrasp-R1 identifies two dominant failure modes: slight misalignment of the affordance region can suppress valid grasps during filtering, and grasps can still land on non-actionable parts or unstable edges [2602.03547].

Embodiment and action space remain major sources of fragmentation. Some systems output planar 4-DoF or 5-DoF grasps with discrete angle bins and fixed or lightly modeled gripper width [2409.19457] [2509.08126]. Others work with parallel-jaw 6D proposals filtered by affordance masks [2602.03547] [2408.10123]. Still others generate MANO hand parameters or dexterous-hand joint configurations rather than robot grasps [2603.08021] [2503.07360]. This suggests that “affordance” is the most transferable layer across embodiments, whereas the final grasp representation remains task- and hardware-specific.

Language coverage is also uneven. Gesture-based systems replace symbolic language with pointing and hand shape [2503.06227]. Open-vocabulary task-oriented systems can infer tasks from implicit instructions but do not explicitly evaluate full relational referring expressions such as “the mug behind the kettle” or “the left screwdriver with the black handle” [2503.00778]. OGRG directly targets duplicates and spatial relations, but its own conclusion notes planar grasps, fixed camera viewpoints, and tabletop environments with common household objects [2509.08126]. A plausible implication is that mature RGA systems will need to combine the large-scale, reasoning-based affordance supervision of segmentation benchmarks, the explicit intermediate affordance representations used in diffusion, retrieval, and manifold methods, and the task-aware ranking mechanisms developed in dynamic or simulation-based grasp selection [2507.23734] [2405.04188] [2007.02132].

The field’s strongest consensus is therefore architectural rather than algorithmic: referring signals should first be converted into a geometric affordance representation, and only then into a grasp. Whether that representation is an angle-indexed affordance volume, a pixel-wise part mask, a transferred contact point, or a behavioral manifold, it functions as the central abstraction that makes object reference, part grounding, and grasp execution commensurate.

Source: https://www.emergentmind.com/topics/referring-grasp-affordance-rga