Referring Grasp Affordance (RGA)

• Referring Grasp Affordance (RGA) is a language-conditioned robotic grasping framework that uses visual observations and referring signals to target specific graspable regions.
• It employs explicit intermediate affordance representations, such as pixel-wise maps or masks, to bridge semantic grounding and precise grasp execution.
• RGA research integrates multi-modal inputs and dynamic stability metrics to enhance task-specific grasp ranking and improve overall robotic grasp success.

Referring Grasp Affordance (RGA) denotes a class of language-conditioned robotic grasping problems in which visual observations and a referring signal are used to identify not only the target object but also the grasp-relevant region or grasp family that should be selected for execution. In recent formulations, the output is often a pixel-wise affordance tensor over position and discretized orientation, such as $Q_g \in \mathbb{R}^{H \times W \times N}$ or $A \in \mathbb{R}^{H \times W \times N}$, from which a grasp is extracted by $\arg\max$ over pixels and angle bins; in neighboring work, the same role is played by an affordance mask, a transferred contact point, or a behavioral manifold in relative pose space (Yu et al., 2024, Yu et al., 9 Sep 2025, Wu et al., 31 Jul 2025). RGA is therefore best understood as an intermediate-level formulation between referring expression grounding and grasp execution: the referring signal specifies which object, part, or action-relevant region matters, and the grasping system converts that specification into an executable grasp under geometric, semantic, and task constraints.

1. Intellectual lineage and task boundary

RGA emerged from several previously separate lines of work. One line treated grasp affordance as a semantic reasoning problem. A knowledge-base formulation used predicates such as hasShape, hasTexture, hasMaterial, canBeFound, hasAffordance, hasCategory, and graspRegion, and modeled their relations with a Markov Logic Network so that grasp affordance prediction could return multiple hypotheses rather than a single grasp label (Ardón et al., 2019). Another line defined task-oriented affordance functions over object, grasp, and use point, $F_T:(O,G,U)\mapsto \mathbb{R}$, so that grasp quality could be evaluated relative to beating, cutting, or picking rather than only by generic stability (Cavalli et al., 2019). A third line incorporated environmental context into grasp-action affordance reasoning and reported that adding environment features improved average diagonal affordance-classification accuracy from $92.57\%$ to $96.81\%$ (Ardón et al., 2019). A fourth line selected task-suitable grasps by forward-simulating task execution and comparing simulated outcomes to prior successful outcomes, improving average task success from $79.2\%$ to $85.4\%$ (Ardón et al., 2020).

These antecedents did not yet define a full language-grounded RGA problem. Their typical assumptions were that the object was already identified, that the task label was known, or that affordance categories were supplied by visual semantics or human teaching rather than by free-form language. Even so, they established several principles that remain central to RGA: grasp choice is task-dependent; multiple affordance hypotheses can coexist on the same object; and intermediate affordance representations are often more useful than direct one-shot grasp regression (Kasaei et al., 2019, Ardón et al., 2019, Cavalli et al., 2019).

Modern RGA formulations make these principles explicit by placing language-conditioned affordance prediction at the center of the pipeline. A CLIP-based parameter-efficient framework defines RGA as prediction of pixel-wise grasp affordance maps $Q_g \in \mathbb{R}^{H \times W \times N}$ from RGB, depth, and a referring expression, with grasp extraction by $$(x^*,y^*,\theta^*)=\arg\max_{(x,y,\theta)}Q_g(x,y,\theta)$$ and $A \in \mathbb{R}^{H \times W \times N}$0 (Yu et al., 2024). OGRG studies the same task under weak supervision and uses an affordance volume $A \in \mathbb{R}^{H \times W \times N}$1 together with a segment-then-grasp pipeline (Yu et al., 9 Sep 2025). Large-scale affordance segmentation work, although not always naming the task RGA, frames the same core problem as predicting an instruction-conditioned affordance map that is then converted with depth into a 3D grasp proposal (Wu et al., 31 Jul 2025).

2. Affordance as an explicit intermediate representation

A defining feature of RGA research is the use of an explicit affordance representation between language and grasp execution. In the most direct formulations, this representation is a dense image-space tensor. The CLIP-based RGA model predicts $A \in \mathbb{R}^{H \times W \times N}$2, where each of the $A \in \mathbb{R}^{H \times W \times N}$3 channels corresponds to a discrete grasp rotation angle; OGRG uses the same basic structure, $A \in \mathbb{R}^{H \times W \times N}$4, with $A \in \mathbb{R}^{H \times W \times N}$5 angle bins spaced by $A \in \mathbb{R}^{H \times W \times N}$6 and a fixed gripper width $A \in \mathbb{R}^{H \times W \times N}$7 in the weakly supervised setting (Yu et al., 2024, Yu et al., 9 Sep 2025). In these models, affordance is already action-shaped: it is not merely an object mask, but a score field over graspable locations and orientations.

A closely related representation is the instruction-conditioned affordance mask. RAGNet and AffordanceNet define a pixel-wise affordance map for the grasp-relevant region of the referred object, then construct a masked point set $A \in \mathbb{R}^{H \times W \times N}$8 and back-project each valid affordance pixel with depth into 3D using camera intrinsics and extrinsics (Wu et al., 31 Jul 2025). AffordanceGrasp-R1 follows the same logic but inserts an intermediate box-and-point grounding stage, $A \in \mathbb{R}^{H \times W \times N}$9, then decodes an affordance mask $\arg\max$0, lifts it to a 3D subcloud $\arg\max$1, and uses that subcloud to filter global-scene grasp candidates (Zhou et al., 3 Feb 2026). AffordGrasp, in the task-oriented grasping sense, also grounds a pixel-level affordance mask $\arg\max$2 after GPT-4o infers the explicit task $\arg\max$3, object $\arg\max$4, optimal part $\arg\max$5, and corresponding affordance $\arg\max$6 from instruction and image (Tang et al., 2 Mar 2025).

Other papers use geometry-aligned affordance representations rather than image masks. AffordGrasp for human-hand synthesis introduces an intermediate pointwise affordance map $\arg\max$7 over the object point cloud $\arg\max$8, explicitly linking language semantics to object geometry before latent diffusion generates a MANO hand pose (Wu et al., 9 Mar 2026). AffordDexGrasp defines a per-point affordance map $\arg\max$9, constructed from grouped dexterous grasps that share intention, contact part, and grasp direction, and uses that map as the intermediate variable between language and dexterous grasp generation (2503.07360).

A distinct but highly relevant alternative is the behavioral manifold representation. Rather than predicting one best grasp pose, grasp affordance can be represented as a partition of relative pose space $F_T:(O,G,U)\mapsto \mathbb{R}$0 under an overview policy $F_T:(O,G,U)\mapsto \mathbb{R}$1, where connected regions induce consistent grasp types after execution. On a mug, this produces separate manifolds for force-closure around the body, pinching around the body, force-closure around the handle, and pinching on the handle (Zechmair et al., 2024). This is not a learned latent manifold; it is a geometric-behavioral manifold induced by kinematics, contact, and policy execution. For RGA, this matters because a phrase such as “grasp the mug by the handle” naturally selects a region or family of regions, not a single point estimate (Zechmair et al., 2024).

Taken together, these representations suggest that contemporary RGA increasingly treats affordance as an explicit intermediate modality rather than as an implicit byproduct of direct grasp regression. That inference is consistent with the repeated use of masks, pointwise affordance fields, and behaviorally induced manifolds as the bridge between semantic grounding and control (Zechmair et al., 2024, Wu et al., 9 Mar 2026).

3. Referring signals and grounding mechanisms

The referring signal in RGA is not uniform across the literature. Some systems use free-form language, some use reasoning instructions, and some replace language with embodied gesture. What unifies them is a two-stage structure in which a coarse cue identifies the target and a finer cue specifies the intended affordance or grasp style.

Large-scale instruction-conditioned affordance segmentation provides the clearest language-grounded examples. RAGNet contains $F_T:(O,G,U)\mapsto \mathbb{R}$2k images, $F_T:(O,G,U)\mapsto \mathbb{R}$3 categories, and $F_T:(O,G,U)\mapsto \mathbb{R}$4k reasoning instructions, and explicitly increases language difficulty by removing category names from hard instructions and replacing them with functional descriptions such as “I need something to drink coffee” or “Use to pierce and lift food” (Wu et al., 31 Jul 2025). OGRG further emphasizes open-form expressions and duplicated object instances, and evaluates absolute and relative spatial language in scenes containing repeated objects, including phrases such as “bottom center dice” and “the tissue box that is to the upper right of the green cylinder green cup” (Yu et al., 9 Sep 2025). AffordanceGrasp-R1 makes the grounding process explicit by training Qwen2.5-VL-7B with CoT cold-start and GRPO so that it predicts a bounding box $F_T:(O,G,U)\mapsto \mathbb{R}$5 and point prompt $F_T:(O,G,U)\mapsto \mathbb{R}$6, which are then decoded into an affordance mask by SAM 2 with LoRA; prompt consistency is enforced by discarding samples with $F_T:(O,G,U)\mapsto \mathbb{R}$7 (Zhou et al., 3 Feb 2026).

A second paradigm uses in-context reasoning over language and scene rather than direct dense grounding. AffordGrasp processes an instruction $F_T:(O,G,U)\mapsto \mathbb{R}$8 and image $F_T:(O,G,U)\mapsto \mathbb{R}$9 with GPT-4o to infer $92.57\%$0, then uses VLPart first to localize the object box $92.57\%$1 and then to predict the affordance mask $92.57\%$2 within the object crop (Tang et al., 2 Mar 2025). This mediated grounding is not phrased as classical referring-expression segmentation; instead, the language first resolves task and object, then part and affordance.

A third paradigm substitutes gesture for language while preserving the computational structure of referring affordance grounding. GAT-Grasp takes a binocular visual observation $92.57\%$3, a pointing gesture $92.57\%$4, and a grasp gesture $92.57\%$5; the pointing ray determines a target crop $92.57\%$6, the grasp gesture retrieves similar human-object interaction exemplars from a memory bank $92.57\%$7, CLIP reranks source images by similarity to the target crop, and DIFT-based dense correspondence transfers the source contact point $92.57\%$8 to the target $92.57\%$9 (Wang et al., 8 Mar 2025). This system achieved an average success rate of $96.81\%$0 across nine cluttered object-part conditions, outperforming GPT-4o, Qwen-VL, Robo-ABC, and RAM in that evaluation (Wang et al., 8 Mar 2025). Although the referring signal is geometric rather than symbolic, the decomposition into coarse target localization and fine affordance disambiguation is directly transferable to RGA.

Language-conditioned human-hand synthesis occupies an intermediate position. AffordGrasp augments HO-3D, OakInk, GRAB, and AffordPose with generated instructions and predicts an affordance map $96.81\%$1 from $96.81\%$2 before diffusion-based hand generation (Wu et al., 9 Mar 2026). AffordDexGrasp uses GPT-4o as a pre-understanding stage to extract object category, intention, contacting part, and one of six canonical grasp directions, then rewrites the instruction into a compact sentence such as “use the mug from the left by contacting the handle” before predicting an affordance map and a dexterous grasp (2503.07360). These systems are object-centric rather than scene-centric, but they show that part-sensitive and intent-sensitive grasp differentiation can be driven by textual conditioning (Wu et al., 9 Mar 2026, 2503.07360).

4. Grasp synthesis, ranking, and execution

Once a referring signal has been grounded into an affordance representation, the next question is how that representation constrains grasp generation. Three recurrent strategies appear in the literature: direct affordance argmax, candidate generation with affordance-aware ranking, and policy- or dynamics-aware optimization.

The most direct strategy is to interpret the affordance tensor itself as the grasp decision surface. In the CLIP-based RGA formulation, the selected grasp is $96.81\%$3 with depth taken from the input depth image (Yu et al., 2024). OGRG uses the same argmax rule on $96.81\%$4, differing mainly in that affordance prediction is conditioned on the grounded object mask and trained with only single-pixel binary grasp labels (Yu et al., 9 Sep 2025). These formulations are compact and deployment-oriented, but they also inherit the limitations of planar angle discretization and fixed-width or simplified gripper parameterization.

The second strategy separates grounding from grasp candidate generation. AffordanceGrasp-R1 generates 6D grasp candidates from the full scene point cloud with GA-Grasp, lifts the predicted affordance mask to a 3D target subcloud $96.81\%$5, assigns each grasp a 3D IoU with the semantics-aligned target volume, keeps top candidates by IoU, then ranks by grasp confidence and applies NMS. In robot experiments on 10 language-conditioned grasping tasks, this design achieved $96.81\%$6 average success on easy instructions and $96.81\%$7 on hard instructions, outperforming AffordanceNet’s $96.81\%$8 and $96.81\%$9 in the same setting (Zhou et al., 3 Feb 2026). GAT-Grasp similarly combines contact-point transfer and human-hand-derived orientation $79.2\%$0 with candidate grasps from HGGD, selecting $79.2\%$1 so that physically plausible grasps are ranked by agreement with the intended grasp orientation (Wang et al., 8 Mar 2025). Aff-Grasp constrains Contact-GraspNet with the predicted graspable affordance region and then executes task-specific motion primitives, reporting $79.2\%$2 correct affordance prediction and $79.2\%$3 successful grasping on seen objects in one robot evaluation (Li et al., 2024).

The third strategy makes execution policy or dynamic stability part of the affordance itself. The behavioral-manifold framework defines an overview policy $79.2\%$4 over relative pose space and evaluates grasp quality with a dynamic local-sensitivity metric $79.2\%$5 derived from object acceleration sensitivity to gripper-base acceleration. Static metrics such as $79.2\%$6, $79.2\%$7, and $79.2\%$8 are compared to $79.2\%$9, and the dynamic metric can favor grasps that are more resilient under manipulation while requiring lower torque. For the power drill, the paper reports $85.4\%$0, $85.4\%$1, $85.4\%$2, and $85.4\%$3 when comparing baseline and shape-informed synthesis, indicating both a larger viable basin and higher best-obtained quality under shape adaptation (Zechmair et al., 2024). In task-oriented metric learning, affordance is similarly defined as a function of grasp and use point, $85.4\%$4, with beating, cutting, and picking derived from primitive metrics such as epsilon quality, rotational inertia, force transmitted to use, and hand effort on hold (Cavalli et al., 2019). These formulations are highly relevant to RGA because referring expressions often specify not only where to grasp but what the grasp must support afterward.

Human-hand grasp synthesis extends the same pattern beyond robotic grippers. AffordGrasp uses separate PointNet encoders for object geometry and affordance geometry, latent diffusion conditioned on $85.4\%$5, and a Distribution Adjustment Module to improve semantic consistency and physical realism without test-time optimization; it reports semantic accuracy gains such as $85.4\%$6 on OakInk and $85.4\%$7 on out-of-domain HO-3D (Wu et al., 9 Mar 2026). AffordDexGrasp uses Affordance Flow Matching to generate a pointwise affordance map and Grasp Flow Matching to generate dexterous grasps, with test-time optimization that preserves affordance-consistent contact while reducing penetration (2503.07360). These are not drop-in robotic RGA systems, but they show that affordance-mediated grasp generation scales to higher-dimensional embodiments.

5. Data regimes, supervision, and evaluation

RGA research is strongly shaped by annotation strategy. Fully supervised pipelines require dense affordance masks or dense grasp maps; weakly supervised pipelines use sparse point or success/failure labels; retrieval-based pipelines replace dense annotation with memory banks of prior interaction examples; simulation-heavy methods generate large volumes of supervisory data under controllable conditions.

Resource or system	Scale or supervision	Primary RGA role
RAGNet (Wu et al., 31 Jul 2025)	$85.4\%$8k images, $85.4\%$9 categories, $Q_g \in \mathbb{R}^{H \times W \times N}$0k reasoning instructions	Language-conditioned affordance segmentation benchmark
OGRG-RGA (Yu et al., 9 Sep 2025)	$Q_g \in \mathbb{R}^{H \times W \times N}$1 visual-language-grasp triplets; single-pixel grasp labels	Weakly supervised affordance learning
AED and Aff-Grasp (Li et al., 2024)	721 images, 13 objects, 8 affordance classes	Precise affordance segmentation and robot evaluation
AffordGrasp (Wu et al., 9 Mar 2026)	Instruction-augmented HO-3D, OakInk, GRAB, AffordPose	Language-conditioned human grasp generation
AffordDexGrasp (2503.07360)	33 categories, 1536 objects, 1909 scenes, 43,504 dexterous grasps	Open-set language-guided dexterous affordance

The benchmark most explicitly aligned with language-conditioned affordance grounding is RAGNet. On HANDAL, AffordanceNet reached gIoU $Q_g \in \mathbb{R}^{H \times W \times N}$2 and cIoU $Q_g \in \mathbb{R}^{H \times W \times N}$3; on GraspNet novel it reached $Q_g \in \mathbb{R}^{H \times W \times N}$4 and $Q_g \in \mathbb{R}^{H \times W \times N}$5; and on 10 real-robot grasping tasks it achieved an average success rate of $Q_g \in \mathbb{R}^{H \times W \times N}$6, compared with $Q_g \in \mathbb{R}^{H \times W \times N}$7 for GraspNet in that study (Wu et al., 31 Jul 2025). AffordanceGrasp-R1 pushed the same line further, reporting average gIoU/cIoU of $Q_g \in \mathbb{R}^{H \times W \times N}$8 across main subsets, $Q_g \in \mathbb{R}^{H \times W \times N}$9 on reasoning subsets, and real-robot grasp success of $$(x^*,y^*,\theta^*)=\arg\max_{(x,y,\theta)}Q_g(x,y,\theta)$$0 on easy and $$(x^*,y^*,\theta^*)=\arg\max_{(x,y,\theta)}Q_g(x,y,\theta)$$1 on hard language-conditioned tasks (Zhou et al., 3 Feb 2026).

Weak supervision remains a major theme. OGRG’s RGA setting uses only a single labeled grasp pixel on one angle slice with a binary $$(x^*,y^*,\theta^*)=\arg\max_{(x,y,\theta)}Q_g(x,y,\theta)$$2 label, rather than dense grasp maps, and still reports average simulation grasp success of $$(x^*,y^*,\theta^*)=\arg\max_{(x,y,\theta)}Q_g(x,y,\theta)$$3 on seen-background tests and $$(x^*,y^*,\theta^*)=\arg\max_{(x,y,\theta)}Q_g(x,y,\theta)$$4 real-robot grasp success, with grounding accuracy of $$(x^*,y^*,\theta^*)=\arg\max_{(x,y,\theta)}Q_g(x,y,\theta)$$5 (Yu et al., 9 Sep 2025). The earlier CLIP-based parameter-efficient RGA framework reported $$(x^*,y^*,\theta^*)=\arg\max_{(x,y,\theta)}Q_g(x,y,\theta)$$6 average simulation success with depth and $$(x^*,y^*,\theta^*)=\arg\max_{(x,y,\theta)}Q_g(x,y,\theta)$$7 real-robot grasp success in its setting, again using angle-indexed affordance maps rather than dense rectangle supervision (Yu et al., 2024).

Data can also be harvested from interaction rather than manually labeled. GAT-Grasp builds a gesture-conditioned affordance memory from a subset of HOI4D plus manual data and performs “zero-shot affordance transfer without additional training” at the task level (Wang et al., 8 Mar 2025). Aff-Grasp automatically derives graspable and functional affordance masks from EPIC-Kitchens and Ego4D via hand-object and tool-object interactions, then evaluates on AED, where Geometry-guided Affordance Transformer reaches mIoU $$(x^*,y^*,\theta^*)=\arg\max_{(x,y,\theta)}Q_g(x,y,\theta)$$8, F1 $$(x^*,y^*,\theta^*)=\arg\max_{(x,y,\theta)}Q_g(x,y,\theta)$$9, and accuracy $A \in \mathbb{R}^{H \times W \times N}$00, surpassing OOAL by $A \in \mathbb{R}^{H \times W \times N}$01 in mIoU (Li et al., 2024). Human-hand systems likewise reuse existing interaction corpora: AffordGrasp pseudo-labels OakInk and GRAB from AffordPose affordance classes, then generates instructions with Qwen (Wu et al., 9 Mar 2026), while AffordDexGrasp groups grasp data by intention, contact part, and direction to construct affordance labels (2503.07360).

Evaluation protocols reflect the dual nature of RGA. One family of metrics measures grounding quality, such as gIoU, cIoU, or oIoU for affordance masks and grounding masks (Wu et al., 31 Jul 2025, Yu et al., 9 Sep 2025). Another measures semantic correctness, such as ACC for whether the generated grasp matches the intended affordance class (Wu et al., 9 Mar 2026). A third measures execution, usually by grasp success rate or task success on real robots (Li et al., 2024, Zhou et al., 3 Feb 2026). A persistent methodological point is that RGA systems are evaluated not only on whether they grasp successfully, but on whether they grasp the correct object, at the correct region, for the intended purpose.

6. Limitations, misconceptions, and open directions

A frequent misconception is that RGA is merely referring expression segmentation followed by off-the-shelf grasping. The literature does not support that reduction. Multiple papers argue that the target of prediction is not the whole object mask but the grasp-relevant region, part, or grasp family, and that downstream grasp ranking must account for semantics, task, or dynamic robustness rather than generic force closure alone (Wu et al., 31 Jul 2025, Zechmair et al., 2024). Conversely, not every affordance-aware grasping paper is a full RGA system. Several influential methods remain object-centric, task-centric, or embodiment-specific, without multi-object referential disambiguation.

The limits are explicit in many papers. The behavioral-manifold formulation has no natural language grounding, no part detector or semantic attribute model, no multi-object referent disambiguation, and no global arm reachability model; the manifold is also policy-dependent, so changing the synthesis controller changes the affordance landscape (Zechmair et al., 2024). AffordGrasp for human-hand synthesis is strong on object-part- and action-conditioned generation, but remains mostly single-object and uses a 10-class affordance taxonomy rather than unrestricted language (Wu et al., 9 Mar 2026). RAGNet and AffordanceNet provide hard function-based instructions, but the benchmark does not emphasize long relational chains or richer temporal task grounding (Wu et al., 31 Jul 2025). AffordanceGrasp-R1 identifies two dominant failure modes: slight misalignment of the affordance region can suppress valid grasps during filtering, and grasps can still land on non-actionable parts or unstable edges (Zhou et al., 3 Feb 2026).

Embodiment and action space remain major sources of fragmentation. Some systems output planar 4-DoF or 5-DoF grasps with discrete angle bins and fixed or lightly modeled gripper width (Yu et al., 2024, Yu et al., 9 Sep 2025). Others work with parallel-jaw 6D proposals filtered by affordance masks (Zhou et al., 3 Feb 2026, Li et al., 2024). Still others generate MANO hand parameters or dexterous-hand joint configurations rather than robot grasps (Wu et al., 9 Mar 2026, 2503.07360). This suggests that “affordance” is the most transferable layer across embodiments, whereas the final grasp representation remains task- and hardware-specific.

Language coverage is also uneven. Gesture-based systems replace symbolic language with pointing and hand shape (Wang et al., 8 Mar 2025). Open-vocabulary task-oriented systems can infer tasks from implicit instructions but do not explicitly evaluate full relational referring expressions such as “the mug behind the kettle” or “the left screwdriver with the black handle” (Tang et al., 2 Mar 2025). OGRG directly targets duplicates and spatial relations, but its own conclusion notes planar grasps, fixed camera viewpoints, and tabletop environments with common household objects (Yu et al., 9 Sep 2025). A plausible implication is that mature RGA systems will need to combine the large-scale, reasoning-based affordance supervision of segmentation benchmarks, the explicit intermediate affordance representations used in diffusion, retrieval, and manifold methods, and the task-aware ranking mechanisms developed in dynamic or simulation-based grasp selection (Wu et al., 31 Jul 2025, Zechmair et al., 2024, Ardón et al., 2020).

The field’s strongest consensus is therefore architectural rather than algorithmic: referring signals should first be converted into a geometric affordance representation, and only then into a grasp. Whether that representation is an angle-indexed affordance volume, a pixel-wise part mask, a transferred contact point, or a behavioral manifold, it functions as the central abstraction that makes object reference, part grounding, and grasp execution commensurate.