GCNGrasp-VP: Task-Oriented Robotic Grasping

• The paper presents a novel integration of task-oriented grasp evaluation with an affordance-guided view planner to mitigate occlusion of key object regions.
• It leverages a segmentation-style architecture that decouples global object-task encoding from local grasp querying, ensuring constant-time inference.
• The framework uses a weakly supervised affordance loss to align grasp statistics with high-confidence regions, guiding efficient active view planning.

GCNGrasp-VP is a task-oriented robotic grasping framework that integrates a fast grasp evaluator with active view planning to address a specific failure mode of task-oriented manipulation: the task-relevant local region of an object may be occluded in the initial observation. The framework couples GCNGrasp-v2, which jointly predicts task-oriented grasp compatibility and a dense affordance field on an observed object point cloud, with the Affordance-guided View Planner (Affordance-VP), which uses that field as a task-aware information gain signal for camera motion without requiring scene reconstruction (Tong et al., 17 Jun 2026). Its central premise is that, for task-oriented grasping, visibility of the whole object is insufficient; the robot must expose and reason over the semantically appropriate local region, such as a pan handle, a cup side compatible with drinking, or a brush handle rather than the bristles.

1. Problem setting and conceptual scope

GCNGrasp-VP is formulated for task-oriented grasping rather than task-agnostic grasping. In task-agnostic settings, many stable grasps on an object may be acceptable. In task-oriented settings, the grasp must be both mechanically valid and functionally compatible with a task instruction $I$. Occlusion is therefore harmful at the level of the task-relevant local region, not merely at the level of whole-object recognition or generic graspability (Tong et al., 17 Jun 2026).

This framing distinguishes the method from two established lines of work. First, prior task-oriented grasping methods commonly assume that the task-relevant region is already visible, or they use affordance predictions only to filter currently visible grasp candidates. Second, prior next-best-view methods for grasping are typically geometry-driven or scene-uncertainty-driven, using reconstruction uncertainty, entropy, or mutual information over an explicit scene model. GCNGrasp-VP argues that such planners can prioritize visually complex but task-irrelevant regions; the paper’s canonical example is a brush, for which uncertainty-driven planning may emphasize the bristles even when the handle is the relevant interaction region.

The framework therefore targets a narrower but practically important objective: reveal the correct local region quickly, with minimal camera motion, and without the reconstruction overhead associated with uncertainty-based active perception. The underlying assumption is that feasible viewpoints are constrained by gravity to a compact hemisphere above the object, so a greedy planner can often succeed with only one additional observation.

2. GCNGrasp-v2: shared object-task encoding and local grasp querying

The grasping component, GCNGrasp-v2, is an architectural revision of GCNGrasp-v1. The original formulation jointly encoded object, task, and candidate grasp:

$$h_1 = \text{GCN}(PN_{\text{down}}([X, g]), I) \in \mathbb{R}^{C},$$

$y = \text{MLP}(h_1) \in \{0,1\},$

where $X \in \mathbb{R}^{N \times 3}$ is the object point cloud, $I$ is the task instruction, and $g \in \mathbb{R}^{6 \times 3}$ is a candidate grasp represented by six control points. This design requires recomputing the expensive object-task-grasp encoding for every grasp candidate.

GCNGrasp-v2 changes the representation into a segmentation-style architecture. It first computes a global object-task embedding independent of candidate grasps,

$$h_2 = \text{GCN}(PN_{\text{down}}(X), I) \in \mathbb{R}^{C},$$

then upsamples it to pointwise features,

$$[X', F] = PN_{\text{up}}(X, h_2) \in \mathbb{R}^{N' \times (3 + C')}.$$

For each grasp $g$, the model performs a local query around the six control points:

$$C = \text{ContactQuery}(g, X', k) \in \mathbb{N}^{6 \times k},$$

$$h_1 = \text{GCN}(PN_{\text{down}}([X, g]), I) \in \mathbb{R}^{C},$$0

$$h_1 = \text{GCN}(PN_{\text{down}}([X, g]), I) \in \mathbb{R}^{C},$$1

Operationally, task-oriented grasp representation becomes a combination of global object-task semantics and local contact-region features around a 6-control-point grasp parameterization. Object geometry is encoded with PointNet++, while semantic relations between object categories and tasks are modeled by a graph convolutional network over a knowledge graph.

The paper describes the resulting inference complexity as constant time, $$h_1 = \text{GCN}(PN_{\text{down}}([X, g]), I) \in \mathbb{R}^{C},$$2, because the dominant object-task encoding is computed once and then reused across all candidates (Tong et al., 17 Jun 2026). A precise interpretation is that the backbone cost is no longer repeated per grasp, even though local querying still depends on the number of candidates. This distinction is important: the method does not remove candidate-wise computation entirely, but it changes the scaling bottleneck.

3. Affordance field prediction and weak supervision

A second output head predicts an affordance field over the object point cloud. In this formulation, the affordance field is a dense set of task-conditioned scores indicating which surface regions support task-relevant interaction. It is not supervised by explicit part masks. Instead, it is derived from task-oriented grasp statistics and is intended to localize visible surface regions enriched with valid task-oriented grasps (Tong et al., 17 Jun 2026).

From the pointwise task-oriented features $$h_1 = \text{GCN}(PN_{\text{down}}([X, g]), I) \in \mathbb{R}^{C},$$3, the model predicts affordance scores by another set-abstraction stage and an MLP:

$$h_1 = \text{GCN}(PN_{\text{down}}([X, g]), I) \in \mathbb{R}^{C},$$4

$$h_1 = \text{GCN}(PN_{\text{down}}([X, g]), I) \in \mathbb{R}^{C},$$5

followed by upsampling to the original point cloud,

$$h_1 = \text{GCN}(PN_{\text{down}}([X, g]), I) \in \mathbb{R}^{C},$$6

The supervision target is a representative point $$h_1 = \text{GCN}(PN_{\text{down}}([X, g]), I) \in \mathbb{R}^{C},$$7 computed from positive and negative task-oriented grasps. For an object-task pair with $$h_1 = \text{GCN}(PN_{\text{down}}([X, g]), I) \in \mathbb{R}^{C},$$8 candidate grasps, let

• $$h_1 = \text{GCN}(PN_{\text{down}}([X, g]), I) \in \mathbb{R}^{C},$$9,
• $y = \text{MLP}(h_1) \in \{0,1\},$0 be the grasp centers,
• $y = \text{MLP}(h_1) \in \{0,1\},$1 be the binary task labels.

The selected index is defined by

$y = \text{MLP}(h_1) \in \{0,1\},$2

and the representative point is

$y = \text{MLP}(h_1) \in \{0,1\},$3

This chooses the grasp center closest to the centroid of positive grasps and farthest from the centroid of negative grasps. The affordance loss then constrains the weighted centroid of predicted affordance scores to match $y = \text{MLP}(h_1) \in \{0,1\},$4:

$y = \text{MLP}(h_1) \in \{0,1\},$5

Training uses a joint loss,

$y = \text{MLP}(h_1) \in \{0,1\},$6

with

$y = \text{MLP}(h_1) \in \{0,1\},$7

The paper states that $y = \text{MLP}(h_1) \in \{0,1\},$8 is small because $y = \text{MLP}(h_1) \in \{0,1\},$9 is only an approximate target. This weak supervision is explicitly described as noisy, so the affordance head functions as a lightly weighted auxiliary signal rather than a direct part-segmentation objective.

Conceptually, the affordance field serves two roles. It regularizes the grasp model toward localization of task-relevant regions, and it provides the task-aware signal used by the active view planner. That coupling is the key structural novelty of the framework.

4. Affordance-guided view planning

Affordance-VP uses the predicted affordance field to choose the next camera pose on a feasible hemisphere above the object. It does not construct TSDF or ESDF volumes, occupancy maps, or uncertainty maps. Instead, it computes a task-aware objective directly from the observed point cloud and the predicted affordance scores (Tong et al., 17 Jun 2026).

The planner proceeds in three stages. First, after upsampling $X \in \mathbb{R}^{N \times 3}$0 to $X \in \mathbb{R}^{N \times 3}$1, it extracts high-confidence affordance points above the 90th percentile, clusters them with DBSCAN, and selects the largest cluster as the target region:

$X \in \mathbb{R}^{N \times 3}$2

$X \in \mathbb{R}^{N \times 3}$3

Second, candidate viewpoints $X \in \mathbb{R}^{N \times 3}$4 are sampled over the feasible hemisphere, for example by farthest point sampling. For each candidate position $X \in \mathbb{R}^{N \times 3}$5, the camera is oriented toward the object centroid $X \in \mathbb{R}^{N \times 3}$6:

$X \in \mathbb{R}^{N \times 3}$7

$X \in \mathbb{R}^{N \times 3}$8

$X \in \mathbb{R}^{N \times 3}$9

Third, the next-best-view objective is minimized:

$I$0

$I$1

with weights chosen by Bayesian optimization,

$I$2

The orientation term encourages the view direction to align with the high-affordance target region. For point $I$3,

$I$4

and

$I$5

The occlusion term penalizes lines of sight that are blocked by non-target obstacle points $I$6. For target point $I$7 and obstacle point $I$8,

$I$9

$g \in \mathbb{R}^{6 \times 3}$0

$g \in \mathbb{R}^{6 \times 3}$1

To preserve real-time performance, a cylindrical query discards obstacle points far from the relevant line of sight before these distances are computed. The elevation term,

$g \in \mathbb{R}^{6 \times 3}$2

discourages degenerate top-down views that would otherwise be favored by pure information-gain maximization.

The planner’s “information gain” is therefore not Shannon information gain over a latent reconstructed scene. It is a task-aware surrogate: improve visibility of the predicted high-affordance region as efficiently as possible.

5. Training data, supervision, and experimental protocol

The primary dataset for task-oriented grasp prediction is TaskGrasp. Evaluation is reported under object instance generalization and task generalization. For view-planning experiments, the paper additionally constructs a multi-view observation dataset with four object-task pairs: pan–pour, watering can–dispense, brush–brush, and cup–drink. Inter-frame depth alignment is performed with DepthAnything3 to reduce sensor noise (Tong et al., 17 Jun 2026).

Training details reported for GCNGrasp-v2 are explicit: 200 epochs on $g \in \mathbb{R}^{6 \times 3}$3 NVIDIA RTX 4090 GPUs, requiring about 3 hours. Testing, including all task-oriented grasping models and the planner, is performed on a single NVIDIA RTX 3090 GPU. The experimental frontend is standardized across planners in order to isolate the effect of view planning. Target object segmentation uses GroundedSAM, task-agnostic grasp candidate generation uses ContactGraspNet, the planner differs by method, and the downstream grasp evaluator is uniformly GCNGrasp-v2.

This protocol is methodologically important because it attributes performance gains to the planning stage rather than to changes in the candidate generator or evaluator. It also clarifies the role of GCNGrasp-v2 within the full system: the same shared-backbone model is used both to score task-oriented grasps and to generate the affordance field that guides the active camera motion.

The paper does not report the exact latent dimensions $g \in \mathbb{R}^{6 \times 3}$4, $g \in \mathbb{R}^{6 \times 3}$5, and $g \in \mathbb{R}^{6 \times 3}$6, the exact value of $g \in \mathbb{R}^{6 \times 3}$7, the exact number of candidate viewpoints, the exact robot platform, or the exact camera model in the provided technical summary. These omissions do not alter the high-level architecture, but they matter for exact reproduction.

6. Empirical behavior, limitations, and relation to adjacent methods

On TaskGrasp with complete shape input, GCNGrasp-v2 improves over GCNGrasp-v1 on most metrics, and the variant with the affordance head remains competitive or best depending on the split (Tong et al., 17 Jun 2026). For object instance generalization, GCNGrasp-v1 reports $g \in \mathbb{R}^{6 \times 3}$8, $g \in \mathbb{R}^{6 \times 3}$9, and $$h_2 = \text{GCN}(PN_{\text{down}}(X), I) \in \mathbb{R}^{C},$$0; GCNGrasp-v2 reports $$h_2 = \text{GCN}(PN_{\text{down}}(X), I) \in \mathbb{R}^{C},$$1, $$h_2 = \text{GCN}(PN_{\text{down}}(X), I) \in \mathbb{R}^{C},$$2, and $$h_2 = \text{GCN}(PN_{\text{down}}(X), I) \in \mathbb{R}^{C},$$3; and GCNGrasp-v2 + affordance reports $$h_2 = \text{GCN}(PN_{\text{down}}(X), I) \in \mathbb{R}^{C},$$4, $$h_2 = \text{GCN}(PN_{\text{down}}(X), I) \in \mathbb{R}^{C},$$5, and $$h_2 = \text{GCN}(PN_{\text{down}}(X), I) \in \mathbb{R}^{C},$$6, with affordance peak error $$h_2 = \text{GCN}(PN_{\text{down}}(X), I) \in \mathbb{R}^{C},$$7. For task generalization, GCNGrasp-v1 reports $$h_2 = \text{GCN}(PN_{\text{down}}(X), I) \in \mathbb{R}^{C},$$8, $$h_2 = \text{GCN}(PN_{\text{down}}(X), I) \in \mathbb{R}^{C},$$9, and $$[X', F] = PN_{\text{up}}(X, h_2) \in \mathbb{R}^{N' \times (3 + C')}.$$0; GCNGrasp-v2 reports $$[X', F] = PN_{\text{up}}(X, h_2) \in \mathbb{R}^{N' \times (3 + C')}.$$1, $$[X', F] = PN_{\text{up}}(X, h_2) \in \mathbb{R}^{N' \times (3 + C')}.$$2, and $$[X', F] = PN_{\text{up}}(X, h_2) \in \mathbb{R}^{N' \times (3 + C')}.$$3; and GCNGrasp-v2 + affordance reports $$[X', F] = PN_{\text{up}}(X, h_2) \in \mathbb{R}^{N' \times (3 + C')}.$$4, $$[X', F] = PN_{\text{up}}(X, h_2) \in \mathbb{R}^{N' \times (3 + C')}.$$5, and $$[X', F] = PN_{\text{up}}(X, h_2) \in \mathbb{R}^{N' \times (3 + C')}.$$6, with $$[X', F] = PN_{\text{up}}(X, h_2) \in \mathbb{R}^{N' \times (3 + C')}.$$7. With partial-view input, degradation is reported as small on TaskGrasp because the dataset views are fairly ideal; the affordance version yields $$[X', F] = PN_{\text{up}}(X, h_2) \in \mathbb{R}^{N' \times (3 + C')}.$$8, $$[X', F] = PN_{\text{up}}(X, h_2) \in \mathbb{R}^{N' \times (3 + C')}.$$9, and $g$0 for object instance generalization, and $g$1, $g$2, and $g$3 for task generalization.

The efficiency claims are central. As the number of grasp candidates increases from 25 to 150, the baseline inference time rises from about $g$4 s to more than $g$5 s and baseline memory rises to nearly $g$6 GB, whereas GCNGrasp-v2 remains below $g$7 s and below $g$8 GB. At the full-pipeline level, excluding $g$9 s of preprocessing, GCNGrasp-v2 inference is $$C = \text{ContactQuery}(g, X', k) \in \mathbb{N}^{6 \times k},$$0 s, Affordance-VP planning is $$C = \text{ContactQuery}(g, X', k) \in \mathbb{N}^{6 \times k},$$1 s, GauSS-MI is $$C = \text{ContactQuery}(g, X', k) \in \mathbb{N}^{6 \times k},$$2 s, and Active-NGF is $$C = \text{ContactQuery}(g, X', k) \in \mathbb{N}^{6 \times k},$$3 s. The phrase “millisecond-level computational latency” therefore refers to the core decision components rather than the entire sensing pipeline.

The most distinctive empirical result is that one planned view adjustment often nearly saturates performance. Starting from a single view, mean AP is $$C = \text{ContactQuery}(g, X', k) \in \mathbb{N}^{6 \times k},$$4 for pan/pour, $$C = \text{ContactQuery}(g, X', k) \in \mathbb{N}^{6 \times k},$$5 for watering can/dispense, $$C = \text{ContactQuery}(g, X', k) \in \mathbb{N}^{6 \times k},$$6 for brush/brush, and $$C = \text{ContactQuery}(g, X', k) \in \mathbb{N}^{6 \times k},$$7 for cup/drink. After one view adjustment, Affordance-VP reaches $$C = \text{ContactQuery}(g, X', k) \in \mathbb{N}^{6 \times k},$$8, $$C = \text{ContactQuery}(g, X', k) \in \mathbb{N}^{6 \times k},$$9, $$h_1 = \text{GCN}(PN_{\text{down}}([X, g]), I) \in \mathbb{R}^{C},$$00, and $$h_1 = \text{GCN}(PN_{\text{down}}([X, g]), I) \in \mathbb{R}^{C},$$01, compared with $$h_1 = \text{GCN}(PN_{\text{down}}([X, g]), I) \in \mathbb{R}^{C},$$02, $$h_1 = \text{GCN}(PN_{\text{down}}([X, g]), I) \in \mathbb{R}^{C},$$03, $$h_1 = \text{GCN}(PN_{\text{down}}([X, g]), I) \in \mathbb{R}^{C},$$04, and $$h_1 = \text{GCN}(PN_{\text{down}}([X, g]), I) \in \mathbb{R}^{C},$$05 for GauSS-MI, and $$h_1 = \text{GCN}(PN_{\text{down}}([X, g]), I) \in \mathbb{R}^{C},$$06, $$h_1 = \text{GCN}(PN_{\text{down}}([X, g]), I) \in \mathbb{R}^{C},$$07, $$h_1 = \text{GCN}(PN_{\text{down}}([X, g]), I) \in \mathbb{R}^{C},$$08, and $$h_1 = \text{GCN}(PN_{\text{down}}([X, g]), I) \in \mathbb{R}^{C},$$09 for Active-NGF. After two adjustments, performance fluctuates: Affordance-VP reports $$h_1 = \text{GCN}(PN_{\text{down}}([X, g]), I) \in \mathbb{R}^{C},$$10, $$h_1 = \text{GCN}(PN_{\text{down}}([X, g]), I) \in \mathbb{R}^{C},$$11, $$h_1 = \text{GCN}(PN_{\text{down}}([X, g]), I) \in \mathbb{R}^{C},$$12, and $$h_1 = \text{GCN}(PN_{\text{down}}([X, g]), I) \in \mathbb{R}^{C},$$13, which the paper attributes to accumulated noise in multi-view feature fusion. In real-world single-object trials after one planned movement, initial success is $$h_1 = \text{GCN}(PN_{\text{down}}([X, g]), I) \in \mathbb{R}^{C},$$14 for pan/pour, $$h_1 = \text{GCN}(PN_{\text{down}}([X, g]), I) \in \mathbb{R}^{C},$$15 for watering can/dispense, $$h_1 = \text{GCN}(PN_{\text{down}}([X, g]), I) \in \mathbb{R}^{C},$$16 for brush/brush, and $$h_1 = \text{GCN}(PN_{\text{down}}([X, g]), I) \in \mathbb{R}^{C},$$17 for cup/drink; after one move, Affordance-VP achieves $$h_1 = \text{GCN}(PN_{\text{down}}([X, g]), I) \in \mathbb{R}^{C},$$18, $$h_1 = \text{GCN}(PN_{\text{down}}([X, g]), I) \in \mathbb{R}^{C},$$19, $$h_1 = \text{GCN}(PN_{\text{down}}([X, g]), I) \in \mathbb{R}^{C},$$20, and $$h_1 = \text{GCN}(PN_{\text{down}}([X, g]), I) \in \mathbb{R}^{C},$$21, outperforming GauSS-MI and Active-NGF across the four tasks.

Several clarifications are important for interpretation. First, the affordance field is not equivalent to a mask-supervised affordance segmentation model; it is weakly supervised by a representative point derived from grasp statistics. Second, “constant-time” inference should be understood as shared-backbone inference whose dominant cost is independent of grasp count, not as a literal elimination of all candidate-wise computation. Third, the absence of reconstruction does not imply the absence of occlusion reasoning; occlusion enters explicitly through the target-region selection and the point-cloud-based occlusion term.

The principal limitations are also explicit. Affordance supervision is weak and noisy, so affordance peaks can shift away from the true grasp region, causing low-information camera motions. Real-world validation is limited to single-object scenarios and four task-object pairs. Large-scale validation in cluttered multi-object scenes therefore remains open. These limitations are method-specific rather than incidental: the framework is strongest when local task-relevant visibility matters, a small number of camera moves is acceptable, and fast online replanning is preferred over full scene reconstruction.

Within the broader grasping literature, GCNGrasp-VP occupies a distinct position. Relative to GtG 2.0, which performs graph-based local scoring of grasp hypotheses in clutter using gripper-centric graphs with inside and outside contextual points, GCNGrasp-VP addresses partial visibility through active, task-aware view planning rather than through implicit geometric reasoning around externally generated candidates (Moghadam et al., 5 May 2025). Relative to GP-net, which produces camera-frame 6-DoF grasp proposals from a single depth image under flexible viewpoints, GCNGrasp-VP is not merely viewpoint-flexible; it actively selects a new viewpoint on the basis of task-conditioned affordance predictions, whereas GP-net remains single-view and dependent on visible grasp contacts (Konrad et al., 2022). In that sense, GCNGrasp-VP is best understood as a task-aware active perception system whose technical contribution lies in making view planning practical through a shared grasp-affordance backbone.