ReasonAff: Affordance Reasoning Dataset

Updated 4 July 2026

ReasonAff is a high-quality affordance-centric reasoning dataset that combines fine-grained affordance masks with implicit, reasoning-based instructions for multimodal LLM tuning.
It transforms direct affordance labels into context-rich, GPT-4o rewritten prompts, enabling robust reinforcement learning and chain-of-thought inference.
The dataset boosts open-domain generalization and outperforms traditional segmentation models by integrating spatial, semantic, and reward-based supervision.

ReasonAff is the affordance-centric reasoning dataset introduced alongside "Affordance-R1: Reinforcement Learning for Generalizable Affordance Reasoning in Multimodal LLM" (Wang et al., 8 Aug 2025). It is built on top of Instruct-Part, preserves fine-grained affordance masks, and replaces short, direct affordance instructions with implicit, reasoning-based instructions intended for multimodal LLM instruction tuning. In that formulation, the dataset is not merely a collection of affordance regions; it is a training and evaluation substrate for models that must infer functional object regions from context-rich queries such as task-oriented or question-like prompts, and it is explicitly tied to reinforcement-learning-based chain-of-thought-style affordance grounding (Wang et al., 8 Aug 2025).

1. Definition and conceptual scope

ReasonAff is described as “a high-quality affordance-centric reasoning dataset” and as “a high-quality affordance dataset ReasonAff for MLLM-based instruction-tuning, which is crucial for embodied perception and reasoning” (Wang et al., 8 Aug 2025). Its defining design choice is to preserve the fine-grained affordance masks inherited from Instruct-Part while rewriting the linguistic side of the task so that affordance grounding becomes an implicit reasoning problem rather than a direct part-name localization problem.

This makes ReasonAff distinct from earlier 2D affordance datasets such as UMD, IIT-AFF, ADE-Af, PAD, PADv2, AGD20K, and Instruct-Part, which are described as “overly simplistic, lack real-world contextual complexity, and are specifically tailored for training visual segmentation models … making them unsuitable for MLLM instruction tuning” (Wang et al., 8 Aug 2025). In the comparison summarized in the Affordance-R1 paper, ReasonAff is the only entry marked as supporting diversity, reasoning, and Q&A, and it covers 48 object categories and 30 affordance categories (Wang et al., 8 Aug 2025).

The shift is methodological as much as representational. Earlier datasets primarily support direct grounding from explicit affordance labels or part references to segmentation masks. ReasonAff instead asks a model to recover functional regions from prompts framed around use, goal, or interaction. The dataset therefore sits at the intersection of affordance segmentation, multimodal instruction following, and reason-structured inference.

2. Construction pipeline and annotation scheme

ReasonAff is constructed from Instruct-Part. It inherits the source images, the binary affordance masks, and the underlying affordance categories, but rewrites the instruction field using GPT-4o (Wang et al., 8 Aug 2025). The explicit motivation is that the original Instruct-Part instructions were “too direct and simple,” with many sentences having “consistent structures” and many being “completely identical,” which was judged likely to limit the reasoning ability of the model.

The rewriting pipeline takes three inputs to GPT-4o: an exocentric affordance image from Instruct-Part, a Human-Object-Interaction image corresponding to the same affordance, and the original instruction (Wang et al., 8 Aug 2025). The HOI image is supplied “to alleviate hallucination issues and avoid identical instructions to enhance diversity.” The resulting outputs are implicit, reasoning-based instructions grounded in daily tasks and intended to avoid leakage of the explicit affordance label.

ReasonAff also computes geometric supervision directly from the inherited affordance masks. For each binary mask, it derives a bounding box

$(x_1, y_1, x_2, y_2)$

and a centroid point

$(x_p, y_p),$

where the box is defined by the leftmost-topmost and rightmost-bottommost pixels and the centroid is the center of mass of the mask pixels (Wang et al., 8 Aug 2025). These are derived annotations rather than new manual labels, but they are central to the subsequent reinforcement-learning reward.

A single ReasonAff example therefore consists conceptually of an image $\mathbf{I}$ , a rewritten implicit instruction $\mathbf{T}$ , one or more affordance masks, and for each mask a bounding box, centroid, and affordance label. The language side of the dataset is intentionally more varied than the source corpus: the paper reports a word cloud analysis and violin-plot comparisons showing that ReasonAff instructions are longer and lexically richer than the original instructions (Wang et al., 8 Aug 2025).

3. Task formulation and representation

In Affordance-R1, ReasonAff is used to formalize affordance grounding as reasoning-conditioned region prediction. The core task is to predict affordance areas from an image–instruction pair:

$\mathcal{A}_{\text{ff}} = \mathcal{F}(\mathbf{T}, \mathbf{I}),$

where $\mathcal{F}$ is the affordance reasoning model (Wang et al., 8 Aug 2025).

Operationally, the reasoning model first predicts boxes and points:

$(\{\mathbf{B}_i, \mathbf{P}_i\})_{i=1}^N = \mathcal{F}(\mathbf{I}, \mathbf{T}),$

with

$\mathbf{B}_i = (x_{1,i}, y_{1,i}, x_{2,i}, y_{2,i}), \qquad \mathbf{P}_i = (x_{p,i}, y_{p,i}),$

and these are then passed to a segmentation model $\mathcal{M}$ , specified as SAM2-Large, to obtain final masks:

$\mathbf{A}_i = \mathcal{M}(\mathbf{B}_i, \mathbf{P}_i).$

This decomposes affordance grounding into a reasoning stage and a segmentation stage (Wang et al., 8 Aug 2025).

ReasonAff does not provide explicit chain-of-thought annotations. Instead, the training prompt enforces a structured output with three tagged sections:

> ...
<rethink> ... </rethink>
<answer> ... </answer>

The <think> section is a free-form “Thinking Process Here,” the <rethink> section is a second-pass reconsideration, and the <answer> section contains the final structured answer with predicted boxes and points in a predefined format (Wang et al., 8 Aug 2025). The dataset thus supports chain-of-thought-style affordance reasoning indirectly: reasoning is not labeled, but the task is designed so that successful policy learning benefits from producing such reasoning.

This distinction is important. In ReasonAff, the supervised signal is spatial and semantic rather than textual-rational. The reasoning trace is an emergent policy artifact shaped by format and correctness rewards rather than a gold explanation sequence.

4. Role in Affordance-R1 and reinforcement learning

ReasonAff is the training dataset for Affordance-R1’s GRPO-based reinforcement learning, and the Affordance-R1 paper states that the model is “trained exclusively via reinforcement learning with GRPO and without explicit reasoning data” (Wang et al., 8 Aug 2025). The policy is initialized from Qwen2.5-VL-7B and samples multiple candidate outputs for each ReasonAff example.

Given a question $(x_p, y_p),$ 0, the model samples $(x_p, y_p),$ 1 candidate outputs and computes a scalar reward for each. The group-relative advantage is

$(x_p, y_p),$ 2

and the GRPO objective is written as

$(x_p, y_p),$ 3

The paper reports a total batch size of 8, a sampling number of 8 per training step, RL steps of 750, and 5 epochs (Wang et al., 8 Aug 2025).

The reward is decomposed into three parts: format reward, perception reward, and affordance recognition reward. Format reward checks the presence and correctness of <think>, <rethink>, and <answer>. Perception reward uses the ReasonAff-derived boxes and points: an IoU reward gives 1 if $(x_p, y_p),$ 4, an L1 reward gives 1 if the coordinate error is below 10, and a Box-Num reward encourages the correct number of affordance areas (Wang et al., 8 Aug 2025). The affordance recognition reward uses word2vec similarity between predicted affordance text and the ground-truth affordance label:

$(x_p, y_p),$ 5

ReasonAff’s importance in this pipeline is therefore twofold. First, it supplies the image–instruction distribution on which the policy is optimized. Second, its inherited masks and derived geometry make it possible to define rule-based rewards without supervised chain-of-thought labels.

5. Empirical performance and generalization

Affordance-R1 is evaluated on ReasonAff using gIoU, cIoU, Precision@50, and Precision@50:95. On ReasonAff, the reported results are:

gIoU: 67.41
cIoU: 62.72
P@50:95: 74.50
P@50: 55.22

These exceed the reported baselines, including Seg-Zero and Vision Reasoner, on all four metrics (Wang et al., 8 Aug 2025). The performance table also places ReasonAff in a regime where reasoning-enabled MLLM methods outperform both classical segmentation baselines and non-reasoning multimodal models.

The same paper attributes cross-dataset gains to ReasonAff’s instruction design. When training on Instruct-Part and evaluating out of domain, the reported UMD scores are gIoU 44.37, cIoU 38.06, P@50:95 26.24, and P@50 47.13; training on ReasonAff yields gIoU 49.85, cIoU 42.24, P@50:95 34.08, and P@50 53.35 (Wang et al., 8 Aug 2025). On AGD20K, training on Instruct-Part gives KLD 10.79, SIM 0.30, and NSS 0.89, whereas training on ReasonAff gives KLD 9.73, SIM 0.36, and NSS 0.98 (Wang et al., 8 Aug 2025). The reported interpretation is that reasoning-based instructions improve open-world and out-of-domain generalization relative to the original direct-instruction dataset.

Ablation results further isolate the role of ReasonAff-compatible rewards. With rethinking, recognition, and Box-Num rewards all removed, the model scores gIoU 60.58 and cIoU 51.94. Adding rethinking raises performance to gIoU 63.04 and cIoU 56.33; adding recognition yields gIoU 65.25 and cIoU 61.22; using all three gives the full 67.41 gIoU and 62.72 cIoU configuration (Wang et al., 8 Aug 2025). These results indicate that the dataset is not only a source of supervision but also the basis for reward engineering that couples reasoning structure, semantic affordance identity, and spatial grounding.

6. Relation to broader reason-centered research

Although ReasonAff is a concrete affordance dataset, its underlying design belongs to a broader research movement that treats reasoning as the production of explicit, inspectable reasons rather than opaque answers. In logical language reasoning, "Reasoning is about giving reasons" introduces the Representation of the Logical Structure (RLS), an intermediate representation in which logical atoms and rules are made explicit and reasoning is delegated to a symbolic solver (Shah et al., 20 Aug 2025). In preference-based reinforcement learning, ReCouPLe uses natural-language rationales as causal projection axes so that reward differences are explained through articulated reasons rather than spurious correlates (Hwang et al., 5 Mar 2026). In mathematical evaluation, ReasonEval argues that reasoning quality must be measured beyond final-answer accuracy through validity and redundancy of intermediate steps (Xia et al., 2024). In retrieval-augmented generation, "Reason and Verify" combines explicit rationale generation with statement-level faithfulness verification (Khan et al., 10 Mar 2026). In text-guided image editing evaluation, ReasonEdit builds a dataset of chain-of-thought critiques and trains reward models over logicality, accuracy, and usefulness of evaluative reasoning itself (Chen et al., 8 May 2026).

ReasonAff differs from these systems in modality and supervision. It is not a logical parser, a reward model for preferences, or a general reasoning judge. Its specialty is embodied affordance grounding from implicit instructions. Yet the family resemblance is strong: explicit structure, inspectable intermediate outputs, and reward or evaluation functions that privilege reasons over uninterpreted scores. A plausible implication is that ReasonAff should be read not only as a dataset name but also as an instance of a general shift toward reason-centered multimodal supervision.

7. Limitations and significance

ReasonAff’s immediate significance lies in converting affordance segmentation into an MLLM-compatible reasoning problem. It retains the precise mask supervision of a segmentation dataset while rewriting the linguistic side into implicit, context-rich instructions. This combination enables reinforcement learning without explicit reasoning annotations and supports emergent test-time reasoning in the resulting policy (Wang et al., 8 Aug 2025).

Its limitations follow from the same design. The reasoning traces are not ground-truth annotations; they are enforced by output format and downstream rewards. The dataset inherits its spatial labels from Instruct-Part, so its coverage remains bounded by that source corpus. Its measured gains are strongest in the Affordance-R1 framework, where boxes, points, and affordance labels are directly converted into reward signals. More broadly, ReasonAff does not solve the general problem of open-world affordance reasoning in unconstrained language, but rather supplies a structured benchmark and training substrate for that direction.

Within the affordance literature, ReasonAff is notable because it introduces diversity, reasoning, and Q&A into a setting previously dominated by direct grounding datasets (Wang et al., 8 Aug 2025). Within the wider reason-centered literature, it is a multimodal example of the principle that explicit intermediate structure—whether logical atoms, rationales, or affordance-oriented thought traces—can support stronger generalization, richer diagnostics, and more transparent model behavior.