Affordance-R1: Multimodal Affordance Grounding

• The paper introduces Affordance-R1, a unified framework that maps images and language instructions to task-specific object regions using an explicit chain-of-thought reasoning process.
• It employs GRPO-based reinforcement learning within a multimodal architecture to jointly optimize spatial localization and semantic affordance labeling.
• Empirical results demonstrate significant improvements in gIoU and zero-shot generalization over baselines, validating its integrated reasoning and grounding approach.

Affordance-R1 is an affordance grounding framework for multimodal LLMs that maps an image and a natural-language instruction to task-relevant object regions and affordance labels, while making its decision process explicit through a Chain-of-Thought-style reasoning trace. It is described as the first unified affordance grounding framework that integrates cognitive CoT guided Group Relative Policy Optimization (GRPO) within a reinforcement learning paradigm, and it is trained exclusively via reinforcement learning with GRPO and without explicit reasoning data. In this formulation, affordance grounding is not treated as a purely perceptual segmentation problem, but as a joint problem of localizing where an instructed action can be carried out and reasoning about why that region is appropriate (Wang et al., 8 Aug 2025).

1. Conceptual lineage and definitional setting

The background of Affordance-R1 lies in a long-running shift in affordance theory. In Gibson’s ecological account, affordances are relational properties of the environment: what the world offers an organism in terms of action. Gibson described affordances as “invariant characteristics” that are directly and pre-cognitively perceived, often summarized by the identity $affordances = perceptions$. Norman’s design-oriented reinterpretation replaced this directness with “perceived affordances,” emphasizing visibility, clarity, and learnability in interfaces rather than invariant physical meaning (Matei, 2020).

Digital and communicative settings complicated the Gibsonian picture further. The same on-screen feature can support multiple socially learned meanings, so digital affordances are not well captured by a one-to-one mapping from object form to action. One proposed resolution is to treat on-screen affordances as symbols, with the on-screen feature as signifier and the learned action possibility as signified; in that view, affordance research becomes a semiotic and linguistic enterprise rather than a theory of direct pickup alone (Matei, 2020).

Parallel work in cognitive science and robotics recast affordances in explicitly computational terms. One reinforcement-learning account models affordance formation in an MDP $(S,A,T,R,\gamma)$ and treats affordances as learned associations between perceptual features and motor policies driven by success and failure signals, with higher-level action categories such as “press” and “slide” emerging through supervised categorization of successful rollouts (Liao et al., 2021). Another line, framed through Computational Rationality, defines internal affordances by confidence and predicted utility inside an internal model of the world, making affordance perception a decision-making process rather than a direct-perception reflex (Liao et al., 16 Jan 2025).

Within visual robotics, a unified formulation has also been proposed: from visual observations, a task description, and an end-effector model, a predictor returns the action, the object, the interaction regions, and the end-effector pose; in the common single-image case this reduces to $f(I,T)\to\{c,b,M,P\}$, where $c$ is an affordance label, $b$ a bounding box, $M$ a per-pixel affordance mask, and $P$ a 6-DoF pose (Apicella et al., 8 May 2025). Affordance-R1 inherits this broader move toward unified, task-conditional affordance prediction, but specializes it to multimodal reasoning and open-world grounding (Wang et al., 8 Aug 2025).

A recurrent source of confusion in the literature is the conflation of affordance with object functionality or goal-related action. In egocentric annotation work, affordance is defined as hand-centered, animal-relative, and goal-irrelevant, whereas mechanical action is tool-centered and goal verbs such as “turn off tap” describe end states rather than invariant low-level motor acts (Yu et al., 2022). Affordance-R1 operates on task-conditioned affordance grounding, not on an unrestricted inventory of object functions.

2. Formal task definition and affordance objective

Affordance-R1 poses affordance grounding as a mapping from an input image $I$ and an instruction $\mathcal{T}$ to a subset of region–action pairs. Let $\mathcal{R}=\{r_1,\dots,r_M\}$ denote candidate regions or object parts in the image, and let $(S,A,T,R,\gamma)$0 denote affordance types such as “openable,” “pressable,” or “graspable.” The goal is to learn a function

$(S,A,T,R,\gamma)$1

that returns the correct region–action pairs for the instructed interaction (Wang et al., 8 Aug 2025).

The framework is motivated by two limitations attributed to prior methods. Purely perceptual approaches train segmenters on human-object interaction masks but lack explicit reasoning, and they tend to fail on images out of domain. Vanilla multimodal LLMs can be prompted to ground regions via special tokens, but without a verifiable reasoning trace their generalization remains brittle (Wang et al., 8 Aug 2025). Affordance-R1 therefore makes the reasoning process part of the policy output.

Central to the method is the affordance reward:

$(S,A,T,R,\gamma)$2

where $(S,A,T,R,\gamma)$3 is the current state and $(S,A,T,R,\gamma)$4 is the generated continuation. The format reward encourages the structure $$\langle\text{think}\rangle \dots \langle/\text{think}\rangle\; \langle\text{rethink}\rangle \dots \langle/\text{rethink}\rangle\; \langle\text{answer}\rangle \dots \langle/\text{answer}\rangle$$. The perception reward includes an IoU reward of $(S,A,T,R,\gamma)$5 if each predicted box has IoU $(S,A,T,R,\gamma)$6 with ground truth, an $(S,A,T,R,\gamma)$7 reward of $(S,A,T,R,\gamma)$8 if box or point coordinates are within 10 pixels, and a box-count reward ensuring that all true affordance areas are proposed. The cognition reward checks that the action label’s embedding matches the ground-truth affordance type, with a word2vec cosine threshold $(S,A,T,R,\gamma)$9 (Wang et al., 8 Aug 2025).

This formulation makes the framework simultaneously structural, spatial, and semantic. A response is rewarded not only for identifying the correct region, but also for presenting a valid reasoning scaffold and naming the appropriate affordance type (Wang et al., 8 Aug 2025).

3. GRPO-based reinforcement learning

Affordance-R1 uses Group Relative Policy Optimization rather than an actor-critic design. In each training iteration, the method samples a batch of questions $f(I,T)\to\{c,b,M,P\}$0, draws $f(I,T)\to\{c,b,M,P\}$1 candidate continuations $f(I,T)\to\{c,b,M,P\}$2, computes rewards $f(I,T)\to\{c,b,M,P\}$3, and normalizes them within the group:

$f(I,T)\to\{c,b,M,P\}$4

The update objective is

$f(I,T)\to\{c,b,M,P\}$5

where

$f(I,T)\to\{c,b,M,P\}$6

The paper also notes an equivalent RL view,

$f(I,T)\to\{c,b,M,P\}$7

while emphasizing that GRPO sidesteps explicit value estimation through group-wise advantage ranking (Wang et al., 8 Aug 2025).

Training is purely reinforcement learning, with no supervised fine-tuning on reasoning chains. The reported settings are AdamW, weight-decay $f(I,T)\to\{c,b,M,P\}$8, KL coefficient $f(I,T)\to\{c,b,M,P\}$9, initial learning rate $c$0, batch size $c$1 images per step, and $c$2 continuations each. The discount factor is $c$3, total training is approximately $c$4 steps per epoch for $c$5 epochs, and the GRPO clip parameter follows the standard settings in DeepSeek-Math, with $c$6 (Wang et al., 8 Aug 2025).

A notable design choice is that reward is assigned only to completed responses. The framework therefore treats reasoning, grounding, and labeling as a single end-to-end policy output rather than as separately supervised subtasks (Wang et al., 8 Aug 2025).

4. Multimodal architecture and the ReasonAff dataset

Affordance-R1 is built on a multimodal LLM such as Qwen2.5-VL-7B and operates in two stages. In Stage I, the model receives the raw image, tokenized by a vision backbone, together with the instruction and a fixed prompt template enforcing the $c$7, $c$8, and $c$9 slots. It alternates between attending to visual tokens and generating textual reasoning. In the final answer block it emits bounding boxes $b$0, point coordinates $b$1, and matched affordance labels $b$2. In Stage II, those boxes and points are used as prompts to an off-the-shelf segmenter such as SAM2-Large, which refines them into pixel-accurate masks $b$3 (Wang et al., 8 Aug 2025).

The framework’s dataset, ReasonAff, was constructed by rewriting the Instruct-Part dataset with GPT-4o. The reported statistics are 48 object categories, 30 affordance types, and rich contextual instructions that go beyond explicit prompts such as “point to the handle” to implicit tasks such as “How would you open the oven to reheat food?” Every example is paired with a binary affordance mask and its bounding box, together with a human-like reasoning Q&A instruction (Wang et al., 8 Aug 2025).

ReasonAff is important because the method is trained without explicit reasoning-chain supervision. The dataset provides task-oriented region and action supervision, while the reasoning behavior itself is induced through the reward design and GRPO optimization rather than by imitation of gold CoT traces (Wang et al., 8 Aug 2025).

The stored reasoning trace is described as explicit and verifiable. This differs from prior grounding systems in which the localization output may be inspectable but the rationale for selecting a region is not (Wang et al., 8 Aug 2025).

5. Empirical results, zero-shot generalization, and emergent reasoning

On the in-domain ReasonAff benchmark, Affordance-R1 reports $b$4, $b$5, $b$6, and $b$7, surpassing Seg-Zero $b$8, Vision Reasoner $b$9, and all other baselines by large margins. In zero-shot out-of-domain evaluation, it reports $M$0 on UMD versus Seg-Zero $M$1 and Vision Reasoner $M$2, and $M$3 on AGD20K versus Seg-Zero $M$4 and Vision Reasoner $M$5 (Wang et al., 8 Aug 2025).

The ablation study attributes incremental gains to specific reward components. Adding the rethinking reward yields approximately $M$6 gIoU, the recognition reward adds approximately $M$7 gIoU, and the Box-Num reward adds approximately $M$8 gIoU (Wang et al., 8 Aug 2025). These results are consistent with the claim that the framework’s performance depends not only on spatial supervision but also on structural reasoning and semantic affordance recognition.

A distinctive reported property is emergent reasoning. Although no supervised CoT was provided, the model is said to routinely generate plausible multi-step traces at test time. For the prompt “How to reheat the soup?”, the example reasoning sequence first searches for a heatable container such as a microwave or stove, then narrows to the microwave door handle, and finally marks the corresponding openable region (Wang et al., 8 Aug 2025). The paper interprets this as evidence that the model is learning “why” and “where” jointly rather than merely memorizing visual templates.

Qualitative tests on web images and EPIC-KITCHENS scenes are described as confirming robust, open-world generalization (Wang et al., 8 Aug 2025). A plausible implication is that the method’s reward structure encourages transfer across novel object appearances because it rewards relational consistency between instruction, reasoning scaffold, region localization, and affordance type rather than a fixed visual pattern alone.

6. Position within the broader affordance-reasoning landscape

Affordance-R1 belongs to a rapidly diversifying family of systems that connect affordance prediction with reasoning, language, and embodied action, but adjacent models optimize different outputs. AffordanceGrasp-R1 combines a CoT cold-start strategy with GRPO for reasoning-based affordance segmentation, then generates grasp candidates from the global scene point cloud and filters them using instruction-conditioned affordance masks; it reports improved gIoU/cIoU on HANDAL, HANDAL+, GraspNet-seen, GraspNet-novel, and 3DOI, along with real-robot grasping gains over AffordanceNet (Zhou et al., 3 Feb 2026). AFUN moves beyond 2D grounding by predicting a task-conditional functional mask and a 3D post-contact motion curve from a single RGB-D observation and a language task description, reporting mean gIoU/cIoU improvements of $M$9 across 8 test sets from 4 benchmarks and real-world deployment without finetuning for robot embodiment or task-specific heuristics (Wang et al., 1 Jun 2026).

Other contemporaneous systems emphasize different forms of structure. A knowledge-guided vision-language approach stores affordances in a typed knowledge graph, retrieves object classes from TypeDB, performs open-vocabulary detection with GLIP, applies human-in-the-loop relabeling in CLIP space, and uses Scallop for spatial verification; in a door-opening scenario, its nearest-neighbor relabeling and spatial constraints raise mAP@0.5 from under $P$0 to over $P$1 on fine-grained affordance classes (Burghouts et al., 2024). YOLOA instead addresses the joint “what–where–how” problem in real time through a YOLOv11-style dual-branch detector refined during training by an LLM Adapter, reporting $P$2 mAP on ADG-Det / IIT-Heat and inference speeds up to $P$3 FPS, with a lightweight variant up to $P$4 FPS (Ji et al., 3 Dec 2025).

These comparisons clarify what Affordance-R1 is and is not. It is not primarily a real-time detector, a grasp planner, or a 3D motion predictor. It is a reasoning-centered affordance grounding framework whose core contribution is the integration of GRPO-based RL with explicit multimodal CoT-style affordance reasoning (Wang et al., 8 Aug 2025).

The paper also identifies its own limitations. RL stability can be brittle if reward components are noisy; ReasonAff still covers only 48 objects and 30 affordances; and current segmentation may fail on very small or highly occluded parts. Proposed extensions include a model-in-the-loop data engine using Affordance-R1 and GPT-4o as a verifier, integration with embodied simulators or real robots, and extension from 2D to 3D affordance grounding by adding depth or point-cloud tokens (Wang et al., 8 Aug 2025).

A broader methodological issue remains reproducibility. A survey of visual affordance research argues that fragmented definitions, datasets, and metrics make comparison unfair and motivates a unified formulation together with an “Affordance Sheet” documenting datasets, model artefacts, experimental setup, performance measures, and robot validation (Apicella et al., 8 May 2025). This suggests that Affordance-R1 should be understood not only as a specific model, but also as part of a larger effort to standardize how affordance reasoning systems are specified, evaluated, and connected to physical robot action.