AFFORD2ACT: Affordance-guided Robotic Manipulation

• The paper introduces AFFORD2ACT, a framework that leverages affordance filtering, semantic keypoint distillation, and transformer-based policy learning to achieve efficient visuomotor control.
• The method employs a three-stage pipeline to distill a compact 38-dimensional state from action-relevant RGB image features, reducing computational overhead.
• Experimental results demonstrate an 82% success rate on unseen objects and diverse tasks, highlighting its data efficiency and robust generalization.

Searching arXiv for the AFFORD2ACT paper and closely related affordance-based manipulation work to ground the article in current literature. AFFORD2ACT is an affordance-guided framework for automatic keypoint selection in vision-based robotic manipulation. It addresses the problem of extracting a compact, semantically meaningful, and task-relevant visual state from RGB images while avoiding dense image or point-cloud representations that are computationally heavy and often entangle irrelevant background features. The framework follows a three-stage pipeline—affordance filtering, category-level keypoint construction, and transformer-based policy learning with embedded gating—and yields a compact 38-dimensional state policy that can be trained in 15 minutes, operates in real time without proprioception or dense representations, and achieves an 82% success rate on unseen objects, novel categories, backgrounds, and distractors across diverse real-world manipulation tasks (Singh et al., 1 Oct 2025).

1. Problem formulation and conceptual basis

AFFORD2ACT is motivated by a representation bottleneck in visuomotor learning. Dense RGB, depth, and point-cloud policies can be expressive, but they are high-dimensional and computationally heavy, and they often overfit to textures, lighting, clutter, and distractors rather than object function. Sparse 2D keypoints offer an alternative because they can capture object motion and geometry with much lower dimensionality, but earlier keypoint-based methods either rely on manual or heuristic selection or couple keypoint selection tightly to specific tasks, limiting scalability and semantic consistency across object instances and categories (Singh et al., 1 Oct 2025).

The framework adopts affordances as the organizing principle for representation learning. In this setting, affordances are the parts of objects that invite particular actions such as “hold,” “cut_with,” “pour,” “brush_with,” “stir,” or “kick.” AFFORD2ACT uses an affordance prompt and a single RGB image to localize actionable regions, then distills a minimal set of semantic 2D keypoints from those regions, and finally trains a compact policy over those keypoints. This design makes the representation explicitly task-relevant while remaining lightweight (Singh et al., 1 Oct 2025).

A common misconception is that AFFORD2ACT is a dense affordance-grounding system or a scene-level segmentation method. It is instead a sparse keypoint-distillation framework: affordance maps are used as a scaffold for selecting functional points, not as the final control representation. That distinction is central to the method’s compactness and data efficiency (Singh et al., 1 Oct 2025).

2. Affordance filtering and semantic keypoint distillation

The first stage, affordance filtering, takes a first-frame image $I_1$ and a natural-language affordance prompt $A$, and uses an open-vocabulary affordance model to produce an affordance heatmap

$H = f_{\text{aff}}(I_1, A).$

This heatmap is converted into a binary affordance mask by thresholding with a quantile: $$M(x, y) = \mathbb{1}\big[ H(x, y) \geq \tau_q(H) \big].$$ The support region $\Omega(M)$ defines the image area judged relevant to the specified action. In the paper’s examples, this mechanism localizes regions such as a mug handle, knife blade, brush bristles, or the part of an object relevant for pouring (Singh et al., 1 Oct 2025).

The second stage constructs category-level semantic keypoints inside the affordance region. On a reference frame $I_1^r$, dense DINO features are computed,

$$F_r = \phi(I_1^r) \in \mathbb{R}^{H \times W \times D},$$

and a small set of anchor points is sampled or clustered within $\Omega(M)$: $$\mathcal{A}_r = \left\{ (p_i^r,\, \mathbf{f}_i^r) \right\}_{i=1}^{N}, \quad p_i^r \in \Omega(M),\ \mathbf{f}_i^r = F_r(p_i^r).$$ These anchors are intended to represent functional parts of the affordance region rather than arbitrary image landmarks (Singh et al., 1 Oct 2025).

Cross-instance correspondence is established on a new demonstration frame $I_1^t$ by matching DINO features inside the current affordance mask. For each reference anchor, AFFORD2ACT computes cosine similarity

$A$0

and selects the maximizer

$A$1

Because the search is constrained by the affordance mask, correspondence is biased toward action-relevant parts rather than visually similar but functionally irrelevant regions. The text prompt therefore influences both region selection and downstream keypoint identity: changing the prompt changes the functional region and therefore the semantic anchors (Singh et al., 1 Oct 2025).

Temporal consistency is then imposed with CoTracker: $A$2 In the reported experiments, the policy uses $A$3 keypoints, comprising 15 object keypoints and 4 end-effector keypoints. The result is a sparse trajectory representation in which each keypoint index is intended to correspond to the same functional part across frames and across object instances, including unseen categories (Singh et al., 1 Oct 2025).

3. Transformer policy and embedded gating

The policy stage consumes the tracked keypoints rather than dense visual features. At each timestep, each keypoint $A$4 is embedded by a small network $A$5, producing a token sequence

$A$6

These tokens are processed by a transformer block,

$A$7

so that each keypoint is interpreted in the context of the others. This contextualization is meant to capture relations among functional parts, such as handle–rim structure in pouring or blade–handle structure in cutting (Singh et al., 1 Oct 2025).

AFFORD2ACT then applies an embedded gating mechanism to weight the importance of keypoints. Given transformer outputs $A$8, the gating network computes

$A$9

The pooled representation $H = f_{\text{aff}}(I_1, A).$0 forms the compact state passed to the action head. The paper states that this policy state is 38-dimensional and that the action head is a two-layer MLP producing end-effector delta pose and a gripper scalar. No proprioception is used in the AFFORD2ACT policy input (Singh et al., 1 Oct 2025).

The paper also defines an effective number of keypoints,

$H = f_{\text{aff}}(I_1, A).$1

which quantifies how many keypoints are effectively used at a given timestep. Empirically, $H = f_{\text{aff}}(I_1, A).$2 is much smaller than 19, indicating that the controller typically relies on only a small subset of semantically selected points during each phase of the task. This is a crucial distinction from uniform sparse representations, which retain sparsity but not necessarily semantic selectivity (Singh et al., 1 Oct 2025).

Training is by behavior cloning from teleoperated demonstrations. The paper describes the pipeline as keypoint embedding, transformer, gating, and action prediction from keypoint trajectories, emphasizing that the low-dimensional state and lightweight network permit rapid training—approximately 15 minutes per task—while maintaining real-time execution (Singh et al., 1 Oct 2025).

4. Experimental setting and empirical performance

The experimental platform is a UR3e robotic arm with a single side-mounted Intel RealSense D435i camera. AFFORD2ACT uses only single-view RGB demonstrations. Six real-world tasks are evaluated: Hold, Cut, Brush, Pour, Stir, and Kick. Each task is associated with an affordance prompt or prompt combination, such as “hold,” “cut_with,” “brush_with,” “pour,” “stir,” or “kick.” Data collection uses 40 teleoperated demonstrations per task, with randomized object positions (Singh et al., 1 Oct 2025).

Evaluation is based primarily on real-world success rate over 20 episodes per method and task, with separate consideration of seen and unseen instances. The benchmark also includes semantic trajectory metrics that score intermediate sub-goals, allowing failures to be localized to particular phases such as grasp, alignment, or final task execution. Generalization tests cover unseen instances from the same category, unseen categories, background and lighting changes, and static and dynamic distractors (Singh et al., 1 Oct 2025).

Across the six tasks, AFFORD2ACT is reported to achieve an 82% success rate on unseen objects, novel categories, backgrounds, and distractors. It outperforms RGB-BC and RGB-D-BC baselines, a UAD-based dense affordance representation, and the Pi0 foundation model fine-tuned on the same data. The paper attributes Pi0’s weaker performance partly to its training assumptions, including a two-camera setup with a wrist camera, whereas AFFORD2ACT operates from a single side-view camera (Singh et al., 1 Oct 2025).

The framework’s data-efficiency results are central to its characterization. In scaling experiments with 100%, 50%, and 25% of the 40 demonstrations, AFFORD2ACT maintains strong performance even with as few as 10 demonstrations, whereas RGB and RGB-D baselines degrade substantially. The paper interprets this as evidence that sparse, affordance-grounded keypoints reduce sample complexity relative to dense visual representations (Singh et al., 1 Oct 2025).

5. Ablations, robustness properties, and failure modes

The ablation study isolates the contribution of semantic keypoint selection and gating. AFFORD2ACT is compared against several automatic keypoint-selection baselines, including YOLO + DINO, YOLO + Uniform Grid, DINO only, SAM + DINO, and hand-picked P3PO keypoints. All methods use the same downstream policy with 19 keypoints for fairness. The reported conclusion is that YOLO-based methods degrade in clutter and on unseen categories, DINO-only correspondence is insufficiently constrained, and SAM-based variants remain dependent on accurate segmentation, whereas AFFORD2ACT matches or exceeds hand-picked performance without manual labeling and is substantially stronger than automatic baselines in cross-category generalization (Singh et al., 1 Oct 2025).

Prompt robustness is another notable property. Replacing prompts with synonyms such as “pick” for “hold,” “mix” for “stir,” or “sweep” for “brush_with” yields essentially the same affordance regions and keypoints, and the learned policies generalize to these synonym prompts at test time. This suggests that the text-conditioning stage is not narrowly tied to a single lexical form (Singh et al., 1 Oct 2025).

The embedded gating mechanism is decisive. Removing gating and assigning equal weight to all tokens causes overall success rate to drop from 86% to 52%. The paper describes the degradation as especially severe in cluttered scenes, where equal weighting encourages the policy to attend to irrelevant keypoints. This is consistent with the intended role of gating as a learned selector over semantically filtered points rather than a mere pooling layer (Singh et al., 1 Oct 2025).

The paper also reports a two-stage long-horizon extension on the Stir task. A vision-LLM planner detects completion of a sub-goal such as grasp completion, after which half of the keypoints are re-anchored with a new affordance prompt for the next phase. This two-stage scheme reaches 90% success on Stir, illustrating that the representation can be extended to multi-phase tasks when keypoint selection is allowed to change across subgoals (Singh et al., 1 Oct 2025).

The main limitations are explicit. Keypoints are fixed at inference unless an additional re-anchoring mechanism is introduced; the system uses only a single external RGB camera; it depends on the quality of the open-vocabulary affordance model; and it assumes a static viewpoint with manageable motion blur and occlusion. Reported failure modes include inaccurate affordance masks on unusual or heavily occluded tools, grasp slips late in tasks such as pour or cut, and confusion caused by equal-weight token pooling when gating is removed (Singh et al., 1 Oct 2025).

6. Position within affordance-to-action research

AFFORD2ACT occupies a specific point in the affordance-to-action design space. It differs from scene-level affordance grounding systems such as Affordance2Action, which center on task-conditioned functional-region masks in cluttered real scenes and use those masks as spatial priors for manipulation policies (Liu et al., 2 Jun 2026). AFFORD2ACT instead converts affordance cues into a sparse set of semantic 2D keypoints, making the control state low-dimensional and directly suitable for lightweight behavior cloning (Singh et al., 1 Oct 2025).

It also differs from real-time affordance detection systems such as YOLOA, which jointly predict object classes, bounding boxes, and dense affordance heatmaps as a “what–where–how” perception front-end (Ji et al., 3 Dec 2025). AFFORD2ACT does not aim to solve joint detection and affordance prediction in the image as a standalone perception problem. Its objective is representation distillation for control: prompt-conditioned affordance regions are an intermediate device for constructing functional keypoints, not the final output consumed by the robot policy (Singh et al., 1 Oct 2025).

Relative to Afford-VLA, which internalizes task-conditioned affordance masks as an action-aligned visual planning interface within a vision-language-action model (Wang et al., 22 May 2026), AFFORD2ACT is markedly more modular and minimalist. The policy does not operate on multimodal token sequences from a large VLA; it operates on sparse keypoint tokens produced by a decoupled affordance-and-correspondence pipeline. This makes it far lighter, but also means it inherits the limitations of fixed keypoint layouts unless explicit re-anchoring is added (Singh et al., 1 Oct 2025).

At a broader level, AFFORD2ACT can be situated between part-level affordance discovery and explicit affordance-aware planning. PartAfford learns part-level affordances from 3D objects under weak supervision (Xu et al., 2022), while ADAPT targets dynamic, unspecified affordance constraints in long-horizon embodied planning (Chen et al., 16 Apr 2026). AFFORD2ACT is narrower in scope than either of these: it is a real-robot manipulation framework that uses affordances to select sparse 2D action-centric features for efficient visuomotor control (Singh et al., 1 Oct 2025).

This positioning explains both its strengths and its boundaries. Its strengths are compactness, fast training, semantic interpretability of control inputs, and strong real-world generalization under limited data. Its boundaries are equally clear: it is not a dense scene-grounding benchmark, not a full “what–where–how” perception stack, not a 3D part-affordance learner, and not a dynamic affordance monitor for long-horizon planners. Within its intended domain, however, it provides a concrete demonstration that text-conditioned affordances can be operationalized as a pipeline from prompt to affordance mask to semantic keypoints to lightweight control policy (Singh et al., 1 Oct 2025).