Task-Conditioned Grasp Affordance Region

Updated 5 July 2026

The paper establishes a structured, task-conditioned spatial representation that guides robotic grasping by integrating semantic intent with geometric affordance masks and heatmaps.
It utilizes multimodal inputs such as images and language to predict task-relevant regions, yielding higher grasp success rates in cluttered and real-world environments.
Methods range from 2D heatmaps to 3D keypoints, balancing coarse semantic localization with the precision needed for dexterous or bimanual manipulation.

Searching arXiv for the cited papers to ground the article in current literature. To manipulate objects in a task-relevant manner, a robot must determine not only whether an object is graspable, but which region of the object should be grasped for the current task. A task-conditioned grasp affordance region is the spatial region—represented variously as a 2D heatmap, a pixel-level mask, a 3D point-cloud subset, a learned patch, or a set of contact keypoints—that is appropriate for grasping because it supports the intended action rather than merely satisfying geometric grasp stability. Across recent work, this notion serves as an intermediate representation linking language, semantics, scene context, and geometry to executable grasps, and it is contrasted explicitly with a single best grasp hypothesis or with generic object-level graspability (Tong et al., 25 Nov 2025, Ardón et al., 2019).

1. Conceptual scope and formal meaning

A task-conditioned grasp affordance region is defined in the literature as a task-specific subset of an object or scene that should be grasped because of the intended manipulation objective. In OVAL-Grasp, the problem is formulated from an image $I \in \mathbb{R}^{H \times W \times C}$ and a language task $t$ , with the goal of producing an end-effector pose $g \in SE(3)$ that lands on a part of object $x$ in a way that supports task execution; the central representation is a heatmap $H$ that localizes the task-relevant affordance region and is later used to score candidate grasps (Tong et al., 25 Nov 2025). AffordGrasp defines the same concept as the part of the scene that the robot should grasp because of the current task, not merely because it is geometrically graspable, and grounds this through a part-level affordance mask $M_{p^*}$ derived from task reasoning and visual grounding (Tang et al., 2 Mar 2025). AffordanceGrasp-R1 similarly treats the target region as an instruction-conditioned affordance mask that identifies the object part or region relevant to the instruction, such as the handle of a hammer or the graspable body of a mug (Zhou et al., 3 Feb 2026).

Earlier work frames the concept in terms of a grasp affordance region rather than a single pose. “Learning Grasp Affordance Reasoning through Semantic Relations” distinguishes a single grasp hypothesis from a grasp affordance region, defining the latter as a spatially extended patch on the object surface that supports a particular task-conditioned action such as pour, hand over, or stack (Ardón et al., 2019). “Towards Affordance Prediction with Vision via Task Oriented Grasp Quality Metrics” formalizes task-oriented affordance as a function over triples $(O,G,U)$ , where object, grasp, and use point are jointly scored for a task $T$ , inducing a high-value subset of grasp and use configurations that can be interpreted as a task affordance region (Cavalli et al., 2019).

A recurring distinction is that the same object may induce different valid affordance regions under different tasks. OVAL-Grasp gives examples in which a mug may be grasped differently for “pick up the mug,” “bring the mug to drink,” and “scan the barcode,” and explicitly treats desirable and undesirable parts separately (Tong et al., 25 Nov 2025). Learning Precise Affordances from Egocentric Videos likewise separates graspable affordance from functional affordance, so that task-oriented manipulation can predict both where the robot should grasp and which part should be used for the downstream action (Li et al., 2024). This suggests that “task-conditioned grasp affordance region” is best understood not as a static object attribute, but as a task-indexed relation between semantic intent, actionable object part, and grasp execution.

2. Representational forms of the affordance region

Recent systems differ primarily in how they represent the task-conditioned region. OVAL-Grasp represents it as a 2D heatmap over the image, initialized to zero, augmented with whole-object support, increased on desirable part segments, decreased on undesirable part segments, scaled to $[0,255]$ , and smoothed with a $3 \times 3$ Gaussian blur (Tong et al., 25 Nov 2025). In that formulation, brighter regions correspond to areas the robot should prefer and darker regions correspond to areas to avoid. AffordGrasp represents the region as a progression from structured reasoning output $t$ 0, to object box $t$ 1, to masked image $t$ 2, to affordance mask $t$ 3, then to a filtered point cloud and final 6D grasp pose $t$ 4 (Tang et al., 2 Mar 2025). AffordanceGrasp-R1 similarly uses a binary pixel mask $t$ 5 and its projected 3D subcloud $t$ 6, which becomes the semantics-aligned target volume for grasp filtering (Zhou et al., 3 Feb 2026).

Point-cloud-native approaches move the representation directly into 3D. “Learning 6-DoF Task-oriented Grasp Detection via Implicit Estimation and Visual Affordance” predicts one 3D affordance heatmap per task label over the partial object point cloud, with

$t$ 7

so task conditioning is represented as a task-indexed scalar field over points rather than a 2D segmentation (Chen et al., 2022). AnchorDP3 operationalizes affordance as the set of 3D points belonging to task-critical objects in the point cloud, extracted from simulator-rendered semantic masks and used to anchor keyposes such as Pre-Grasp, Target Grasp Open, and Target Grasp Closed (Zhao et al., 24 Jun 2025).

Other work uses more structured spatial abstractions. The semantic-relational framework of 2019 learns prototypical grasping patches and maps them to 3D geometry, representing the optimal patch $t$ 8 as a set of 3-D points $t$ 9 with dominant plane $g \in SE(3)$ 0, centroid $g \in SE(3)$ 1, and orientation $g \in SE(3)$ 2 (Ardón et al., 2019). Multi-Keypoint Affordance Representation replaces coarse regions with three task-conditioned contact keypoints $g \in SE(3)$ 3, $g \in SE(3)$ 4, and $g \in SE(3)$ 5, linked to wrist and finger contact structure so that the region becomes a sparse geometric constraint on dexterous posture rather than a broad mask (Yang et al., 27 Feb 2025). Task-Aware Bimanual Affordance Prediction uses grid cells selected by a VLM on object crops; those cells are projected into 3D and used to filter global grasps, so the affordance region is a selected subset of object-space cells rather than a dense part mask (Hahne et al., 9 Apr 2026).

The representational choice determines how tightly the region constrains action. A plausible implication is that heatmaps and masks are well suited to coarse semantic localization and grasp ranking, whereas keypoints, patches, and anchor poses are better aligned with dexterous or bimanual constraints because they encode stronger geometric structure (Yang et al., 27 Feb 2025, Hahne et al., 9 Apr 2026).

3. Semantic conditioning: language, context, and reasoning

A central development in this area is the shift from fixed-label affordance prediction to open-vocabulary or reasoning-based semantic decomposition. OVAL-Grasp uses a LLM $g \in SE(3)$ 6 to identify object $g \in SE(3)$ 7 and decompose it into desirable parts and undesirable parts, with GPT-4o used in the main system; examples include handle, lid, tab, label, and barcode (Tong et al., 25 Nov 2025). AffordGrasp uses GPT-4o to convert an implicit instruction $g \in SE(3)$ 8 and image $g \in SE(3)$ 9 into explicit task, object, part, and affordance: $x$ 0 thereby grounding implicit instructions such as “I am thirsty” or “I need to tighten screws” into part-level affordance reasoning (Tang et al., 2 Mar 2025). AffordanceGrasp-R1 uses an MLLM to infer a bounding box $x$ 1 and point $x$ 2 from image-instruction pairs,

$x$ 3

and explicitly avoids direct mask generation from the MLLM because raw mask generation is described as noisy and unstable (Zhou et al., 3 Feb 2026).

The use of semantics is older than the foundation-model era. The Markov Logic Network framework of 2019 encodes attributes such as shape, texture, material, category, location, grasp affordance, and grasp region as logical predicates, then infers a probability distribution over grasp-affordance / grasp-region pairs by Gibbs sampling from the MLN posterior (Ardón et al., 2019). “Reasoning on Grasp-Action Affordances” further expands semantic conditioning to environmental context, using shape, texture, categorical, and environment attributes in a knowledge-base graph to infer affordance groups such as to eat, to contain, and to hand over, then constraining grasp-region selection accordingly (Ardón et al., 2019).

Several recent systems emphasize that the task signal may be multimodal. GAT-Grasp does not begin from language or category priors, but from two gestures: a pointing gesture $x$ 4 that localizes a coarse search region, and a grasp gesture $x$ 5 that refines where to grasp and how to orient the gripper (Wang et al., 8 Mar 2025). GauTOAO asks, for an object already in hand and a natural-language task, which part is the task-relevant affordance region, using a prompt that forces the model to output the specific object part most useful for task completion (Wang et al., 2024). Affordance2Action distinguishes explicit part prompts from task-reasoning instructions and treats the mapping from manipulation intent to functional part as the core grounding problem (Liu et al., 2 Jun 2026).

These methods converge on the idea that task conditioning is fundamentally a semantic disambiguation problem. The region is not merely “on the object”; it is inferred from what the robot is supposed to achieve, what part must remain unobstructed, what part should be contacted, and, in some methods, which part should explicitly be avoided (Tong et al., 25 Nov 2025, Appius et al., 2024).

4. From affordance region to grasp selection and execution

Once localized, the affordance region must be coupled to grasp generation. OVAL-Grasp instantiates the grasp proposal model $x$ 6 as ContactGraspNet, which generates candidate 6-DoF grasps $x$ 7. Each candidate is scored by sampling the heatmap at the reprojected gripper contact point and at a point nearest the extended gripper z-axis: $x$ 8 after which grasps are sorted by total score and the highest-scoring grasp is executed (Tong et al., 25 Nov 2025). The inclusion of $x$ 9 explicitly discourages grasps whose approach axis would obstruct task-relevant regions.

AffordGrasp takes a masking-and-filtering route. After grounding the affordance mask $H$ 0, it converts the depth image into a partial-view point cloud, applies the affordance mask so grasp generation only considers points inside the task-conditioned affordance region, and then uses AnyGrasp to generate candidates $H$ 1 with grasp pose parameterization

$H$ 2

The selected grasp favors high confidence and closeness to the centroid $H$ 3 of the affordance region (Tang et al., 2 Mar 2025). AffordanceGrasp-R1 instead generates grasps from the full-scene point cloud using GA-Grasp, then defines a semantics-aligned subcloud

$H$ 4

and uses 3D IoU matching between each candidate gripper volume and the target volume induced by $H$ 5 to filter and rank candidates (Zhou et al., 3 Feb 2026).

Earlier 3D point-cloud work uses affordance more as a refinement prior. The 2022 implicit-estimation framework produces coarse 6-DoF candidates with an implicit estimation network, predicts a task-specific affordance heatmap over the point cloud, selects the top-100 affordance points, computes the L2 distance between each coarse grasp center and these points,

$H$ 6

and fuses this with the coarse grasp score,

$H$ 7

to obtain refined task-conditioned grasps (Chen et al., 2022). The 2019 task-oriented grasp-quality framework formalizes this more generally as

$H$ 8

so the affordance region is the high-scoring subset of grasp/use hypotheses under a task-specific metric (Cavalli et al., 2019).

For dexterous or bimanual manipulation, the affordance region can directly determine posture or arm assignment. Multi-Keypoint Affordance Representation constructs object and hand frames from task-conditioned keypoints and computes the relative transformation

$H$ 9

thereby turning affordance geometry into grasp transformation parameters $M_{p^*}$ 0 (Yang et al., 27 Feb 2025). Task-Aware Bimanual Affordance Prediction filters AnyGrasp’s global grasp pool so that only grasps whose center point lies inside the selected 3D region are retained, and in coordinated mode searches for a grasp pair $M_{p^*}$ 1 satisfying

$M_{p^*}$ 2

which binds affordance localization to arm allocation and inter-gripper safety (Hahne et al., 9 Apr 2026).

5. Empirical behavior, generalization, and robustness

The empirical literature repeatedly associates better affordance localization with better task-oriented grasping. On 20 objects with 3 tasks each, OVAL-Grasp reports 95.0% part selection and 78.3% grasp success, compared with 60.0% and 56.7% for GraspGPT and 73.3% and 66.7% for ShapeGrasp; in cluttered scenes, OVAL-Grasp reports 80.0% part selection, versus 26.7% for GraspGPT and 46.7% for ShapeGrasp (Tong et al., 25 Nov 2025). The same paper reports that OVAL-Grasp successfully identifies and segments the correct object part 95% of the time and grasps the correct actionable area 78.3% of the time in real-world experiments with the Fetch mobile manipulator, including visually defined parts and partial occlusions (Tong et al., 25 Nov 2025).

AffordGrasp reports stronger performance in clutter when grasping is explicitly restricted to affordance regions. In cluttered simulation scenes, average GSR is 0.77 for AffordGrasp and 0.54 for ThinkGrasp, with particularly large gains for screwdriver (0.76 versus 0.16) and wine glass (0.52 versus 0.00); in real-world experiments, AffordGrasp reports 0.83, compared with 0.73 for ThinkGrasp, 0.61 for AnyGrasp, and 0.14 for RAM (Tang et al., 2 Mar 2025). AffordanceGrasp-R1 reports 66.7 gIoU / 65.9 cIoU average on segmentation benchmarks, versus 53.4 / 51.1 for AffordanceNet, and in zero-shot robot experiments achieves 80% average success under easy instructions and 72% average success under hard instructions, compared with 62% and 50% for AffordanceNet (Zhou et al., 3 Feb 2026).

Mask-based affordance prediction from egocentric data also shows strong downstream value. Geometry-guided Affordance Transformer achieves mIoU 68.62, F1 81.09, and Accuracy 83.51 on AED, improving over OOAL by +15.9% mIoU, +12.39% F1, and +17.72% Accuracy; in robotic evaluation, Aff-Grasp reports 97.2% affordance prediction, 80.6% successful grasp, and 65.3% successful interaction on the accuracy evaluation, with robustness and unseen-object evaluations also reported (Li et al., 2024). In the dexterous setting, GAAF-Dex reports KLD = 1.459, NSS = 1.242, and SIM = 0.327 for functional affordance grounding, with 74.65% overall average precision for gesture prediction (Yang et al., 2024). The multi-keypoint CMKA/KGT framework reports KLD improves by 45.35%, SIM improves by 54.19%, and NSS improves by 101.63% on the FAH dataset, and about 40% on average improvement in functional grasp success rate over GAAF-Dex in real robotic tasks (Yang et al., 27 Feb 2025).

Historical results already showed the same trend in different formalizations. The MLN-based affordance-reasoning paper reports mean AUC 0.84 for its MLN knowledge base in zero-shot grasp-affordance prediction, with mean Hausdorff distance below 0.1 for all affordance regions in patch similarity tests and on-robot success around 95.2% for cubic, 91.6% for cylindrical, 82.6% for irregular, and 95.8% for spherical objects (Ardón et al., 2019). “Reasoning on Grasp-Action Affordances” reports 81.3% average accuracy on object affordance reasoning and 88% grasp-region coincidence with Cornell grasp labels in zero-shot evaluation, while “Task-Aware Robotic Grasping by evaluating Quality Diversity Solutions through Foundation Models” reports 76.4% weighted IoU, 91.1% precision, and 76.4% recall for predicted task-conditioned grasp regions against human survey maps (Ardón et al., 2019, Appius et al., 2024).

Taken together, these results suggest that affordance-region quality is often the dominant bottleneck in task-oriented grasping when object parts are semantically defined, visually subtle, partially occluded, or task-dependent.

6. Variants, misconceptions, and open directions

A common misconception is that a task-conditioned grasp affordance region is simply a synonym for a grasp point or for a generic “graspable part.” The literature rejects that interpretation. The 2019 MLN framework explicitly models a distribution over affordance-region labels rather than a single fixed grasp point (Ardón et al., 2019). OVAL-Grasp includes both positive and negative evidence in the heatmap, so the representation identifies where the robot should grasp and where it should avoid grasping for the same task (Tong et al., 25 Nov 2025). Learning Precise Affordances from Egocentric Videos distinguishes graspable from functional affordances, showing that the region to hold and the region to use are often different and both may be required for successful task execution (Li et al., 2024).

Another misconception is that affordance grounding is reducible to coarse part segmentation. Task-Aware Bimanual Affordance Prediction argues that coarse part labels such as “handle” are often insufficient because the optimal contact subregion depends on the task and on what the other hand is doing (Hahne et al., 9 Apr 2026). Affordance2Action further emphasizes that manipulation in clutter is often one-to-many: a single task may correspond to either one functional region or multiple valid functional regions, and multi-region outputs should be evaluated with set-based metrics such as

$M_{p^*}$ 3

rather than a single-mask IoU (Liu et al., 2 Jun 2026). This suggests that “the affordance region” is sometimes more accurately a set of valid regions.

The main methodological axes now visible in the literature are clear. One axis concerns reasoning substrate: logical-semantic knowledge bases and hand-designed task metrics in earlier work (Ardón et al., 2019, Cavalli et al., 2019), versus LLM/VLM-based open-vocabulary reasoning in recent systems (Tong et al., 25 Nov 2025, Tang et al., 2 Mar 2025, Zhou et al., 3 Feb 2026). A second axis concerns spatial representation: heatmaps and masks (Tong et al., 25 Nov 2025, Li et al., 2024), point-cloud subregions and 3D heatmaps (Chen et al., 2022), keypoints (Yang et al., 27 Feb 2025), keyposes (Zhao et al., 24 Jun 2025), and affordance-transferred contact points from human-object interaction memory (Wang et al., 8 Mar 2025). A third axis concerns action coupling: some methods use the region to re-rank generic grasp proposals (Tong et al., 25 Nov 2025), others to restrict the candidate set (Tang et al., 2 Mar 2025), define geometric frames (Yang et al., 27 Feb 2025), or guide downstream manipulation policies as explicit spatial priors (Liu et al., 2 Jun 2026).

Limitations are also consistent across papers. Failures arise when the semantic module hallucinates the wrong part decomposition or when segmentation misidentifies or over-segments parts under occlusion (Tong et al., 25 Nov 2025). Mask-based filtering can suppress valid grasps if small affordance regions are misaligned (Zhou et al., 3 Feb 2026). Several systems remain dependent on curated datasets, simulator access, or pretrained modular components rather than end-to-end closed-loop learning (Ardón et al., 2019, Zhao et al., 24 Jun 2025, Wang et al., 2024). Orientation is not always explicitly optimized in task-aware selection for antipodal grippers (Appius et al., 2024), whereas dexterous methods emphasize that coarse region localization alone cannot constrain posture adequately (Yang et al., 27 Feb 2025).

Overall, the field has moved from viewing affordance as a generic graspability score to treating it as a structured, task-indexed spatial representation that mediates between semantics and grasp synthesis. A plausible implication is that future work will continue to unify three requirements already visible in separate strands of the literature: open-vocabulary semantic reasoning, precise scene-level or object-part localization, and direct geometric constraints on single-arm, dexterous, or bimanual execution (Tong et al., 25 Nov 2025, Yang et al., 27 Feb 2025, Hahne et al., 9 Apr 2026).