Collective Affordance Sets: Shared Action Insights

• Collective Affordance Sets are structured representations that treat affordances as shared, context-dependent action possibilities instead of isolated object properties.
• They are used to model scene-level action profiles, human–object interplay, and function-based mappings, delivering insights into scene categorization and collaborative frameworks.
• Applications span graded action profiles, latent embedding transfer, and augmented reality for human–robot interactions, highlighting cross-cultural and multi-agent affordance reasoning.

Collective affordance sets are structured representations in which affordances are treated not as isolated properties of single objects, but as reusable, comparable, and often shared relations distributed across objects, scenes, agents, or contexts. In contemporary research, the notion appears in several mathematically distinct forms: as graded action profiles for scene categories, as shared verb embeddings reused across human–object interactions, as culturally grounded mappings from functions to diverse artifacts, as shared reachability regions in human–robot collaboration, and as ternary or rough relations over actors, objects, and environments (Greene et al., 2014, Hou et al., 2021, Nwatu et al., 2 Dec 2025, Gruszczynski et al., 4 Dec 2025).

1. Conceptual and formal characterizations

One major line of work represents collective affordance sets as high-dimensional action profiles. In scene categorization, each scene category $c$ is represented by a vector $\mathbf{a}_c = (a_{c1}, a_{c2}, \dots, a_{c,227})$, where $a_{ck}$ is the proportion of participants endorsing action $k$ as something that “could typically be done” in that scene. In this formulation, what matters is not a single action such as sitting or cooking, but the full pattern over 227 actions; the collective affordance set of a scene category is therefore a graded set representation over an action vocabulary (Greene et al., 2014).

A second formulation treats affordances as shared latent units that can be recombined across objects. In affordance transfer learning for human–object interaction detection, verbs are treated as affordance categories, and the collective affordance set is written as $\mathcal{A} = \bigcup_{v=1}^{N_v} \mathcal{A}_v$, where $\mathcal{A}_v = \{\mathbf{a}_{v,k}\}_{k=1}^{S_v}$ is the set of affordance embeddings for verb $v$. Human–object interactions are represented compositionally as $[\mathbf{a};\mathbf{o}]$, explicitly separating affordance and object features so that a shared affordance representation can be transferred to new objects (Hou et al., 2021).

A third formulation is function-centric and relational. In the Culture Affordance Atlas, functions are defined as the affordances of an object, and the atlas implements a relation $R \subseteq \mathcal{F} \times \mathcal{O}$ between functions and objects. For a function $f$, its collective affordance set is $\mathbf{a}_c = (a_{c1}, a_{c2}, \dots, a_{c,227})$0, that is, the culturally diverse set of objects documented to realize that function (Nwatu et al., 2 Dec 2025).

A fourth formulation is logical. “Towards a logic of affordances” models an affordance as a ternary relation $\mathbf{a}_c = (a_{c1}, a_{c2}, \dots, a_{c,227})$1, relating actors, objects, and environments; rough affordances are represented by a pair $\mathbf{a}_c = (a_{c1}, a_{c2}, \dots, a_{c,227})$2 of lower and upper approximations induced by indiscernibility classes over those three domains (Gruszczynski et al., 4 Dec 2025). A related but decision-theoretic perspective appears in computational rationality, where an internal affordance is represented as the tuple $\mathbf{a}_c = (a_{c1}, a_{c2}, \dots, a_{c,227})$3; here, the relevant set is the agent’s current set of internal action possibilities, evaluated under bounded optimality (Liao et al., 16 Jan 2025).

Taken together, these formulations show that “collective affordance sets” is not a single canonical data structure. The common thread is that affordances are modeled as shared, set-valued structure: profiles over actions, clusters of transferable embeddings, many-to-many function–object relations, or formally approximated relations over actor–object–environment triples.

2. Scene-level affordance geometry

The scene-categorization literature provides one of the clearest empirical instantiations of collective affordance sets. Using 311 scene categories and 227 ATUS-derived actions, a $\mathbf{a}_c = (a_{c1}, a_{c2}, \dots, a_{c,227})$4 affordance matrix was constructed and cosine distance was used to measure dissimilarity between scene categories in affordance space. The resulting affordance-based similarity matrix predicted human scene categorization better than models based on objects, CNN features, semantics, or coarse superordinate labels (Greene et al., 2014).

Model	Correlation with human similarity	Brief note
Affordance-based similarity	0.50	Approximately two-thirds of noise ceiling
CNN-based perceptual similarity	0.39	Best non-affordance visual model
Object-based similarity	0.33	LabelMe-derived object presence
Semantic distance	0.27	WordNet path-based
Superordinate model	0.25	Indoor / urban outdoor / natural outdoor

The noise ceiling was approximately $\mathbf{a}_c = (a_{c1}, a_{c2}, \dots, a_{c,227})$5. A combined model with all nine feature spaces explained $\mathbf{a}_c = (a_{c1}, a_{c2}, \dots, a_{c,227})$6 of the variance in the human similarity pattern, while the top-three model using affordances, objects, and CNN features explained $\mathbf{a}_c = (a_{c1}, a_{c2}, \dots, a_{c,227})$7. Within that explained variance, affordances dominated: affordance similarity independently explained $\mathbf{a}_c = (a_{c1}, a_{c2}, \dots, a_{c,227})$8 of the total variance in the human matrix, or $\mathbf{a}_c = (a_{c1}, a_{c2}, \dots, a_{c,227})$9 of the variance explained by all models combined, whereas CNN similarity independently explained $a_{ck}$0 and object similarity $a_{ck}$1 (Greene et al., 2014).

The qualitative organization of the human similarity matrix is also consistent with a collective affordance-set account. Hierarchical clustering revealed not only subordinate clusters such as bamboo forest, woodland, and rainforest, but also broader clusters aligned with activity classes such as sports, cutting across indoor/outdoor and manmade/natural boundaries. This supports the claim that scenes can be treated as points in an affordance space whose geometry is more predictive of human categorization than object inventories or low-level visual features (Greene et al., 2014).

3. Object-, part-, and HOI-level collective affordances

At the level of human–object interaction, affordance transfer learning makes the collective nature of affordances explicit. The model decouples HOI representations into affordance and object features, using $a_{ck}$2 for real interactions and $a_{ck}$3 for composite interactions formed by pairing affordance features from HOI images with object features from external datasets such as COCO and Objects365. The framework learns a finite verb vocabulary—117 verbs in HICO-DET and 21 in HOI-COCO—as a shared affordance set, and it materializes that set at inference time through an affordance feature bank $a_{ck}$4. Empirically, ATL (COCO)$a_{ck}$5 reached $a_{ck}$6 Full and $a_{ck}$7 Rare on HICO-DET, and ATL (COCO) reached $a_{ck}$8 on the Unseen category in novel-object HOI detection; for object affordance recognition, ATL (HOI+COCO) achieved $a_{ck}$9 mAP on COCO Val2017 and $k$0 mAP on novel classes (Hou et al., 2021).

Part-level discovery work replaces dense affordance supervision with object-level affordance sets. PartAfford defines a global affordance category set $k$1 and gives each object $k$2 only an object-level affordance set $k$3. Slot attention, reconstruction, and a Hungarian-matched set loss are then used to discover part-affordance correspondences without part-level affordance labels during training. The dataset contains 24 affordance categories shared among $k$4 objects, and the full model achieved mean IoU / AP of $k$5 on the “sittable” group, $k$6 on “support”, and $k$7 on “openable” (Xu et al., 2022).

AffordPose shifts the emphasis from parts to interaction realizations. It provides 641 object meshes across 13 categories, 8 hand-centered affordances, and 26,712 affordance-driven hand–object interactions, with each object having 1–5 affordances. The dataset explicitly defines per-affordance pose sets through MANO parameters, and its statistical analyses show both common structure and diversity within an affordance class through representative hands, per-joint standard deviations, and finger-contact probabilities. In downstream evaluation, using all pose parameters yielded $k$8 affordance classification accuracy and $k$9 localization IoU, while affordance-conditioned generation reached a mean affordance accuracy of $\mathcal{A} = \bigcup_{v=1}^{N_v} \mathcal{A}_v$0 (Jian et al., 2023).

Affordance grounding work in multimodal LLMs adds explicit reasoning over shared affordances across object categories. Affordance-R1 defines $\mathcal{A} = \bigcup_{v=1}^{N_v} \mathcal{A}_v$1, where the model predicts affordance regions from image and instruction, and optimizes reasoning with GRPO using format, perception, and recognition rewards. ReasonAff is built from 48 object categories and 30 affordance categories, and the model improves OOD affordance grounding on UMD and AGD20K, reaching $\mathcal{A} = \bigcup_{v=1}^{N_v} \mathcal{A}_v$2 gIoU on UMD and $\mathcal{A} = \bigcup_{v=1}^{N_v} \mathcal{A}_v$3 gIoU on AGD20K. This suggests that chain-of-thought training can induce reusable affordance schemas shared across heterogeneous objects, even though the paper does not introduce an explicit symbolic set representation for them (Wang et al., 8 Aug 2025).

4. Function-centric and cross-cultural collective affordance sets

The Culture Affordance Atlas redefines affordance organization around culturally grounded function rather than object identity. Functions are defined as “the affordances of an object,” and the atlas maps 38,479 Dollar Street images and 270 filtered topics into a hierarchical structure of 7 high-level categories, 46 functions, 288 unique objects, and 367 documented object–function entries. The central relation is many-to-many: many distinct objects realize the same function, and many objects belong to multiple functions. Examples include “sleeping,” which groups Western bed, charpai, mat, sleeping bag, bedspread/bedding, and floor mat, and “cleaning teeth,” which groups toothbrush, chewing stick, fingers, charcoal, tooth powder, tooth soap, salt, and clay/brick powder (Nwatu et al., 2 Dec 2025).

This atlas is explicitly evidential and comparative. Each object–function pair has at least one ethnographic or scholarly citation, 98% of them from the eHRAF World Cultures database. The need for a function-centric reorganization is quantified by the fact that, on average, $\mathcal{A} = \bigcup_{v=1}^{N_v} \mathcal{A}_v$4 of Dollar Street topics do not match the actual object in the image. In CLIP association tests, the similarity–income slope is approximately $\mathcal{A} = \bigcup_{v=1}^{N_v} \mathcal{A}_v$5 for topic prompts but approximately $\mathcal{A} = \bigcup_{v=1}^{N_v} \mathcal{A}_v$6 for function prompts; in retrieval, the median reduction in the high–low income recall gap is $\mathcal{A} = \bigcup_{v=1}^{N_v} \mathcal{A}_v$7 for CLIP and 11 percentage points for SigLIP2; and for combined topic+function prompts, the slope of similarity versus income is approximately $\mathcal{A} = \bigcup_{v=1}^{N_v} \mathcal{A}_v$8 (Nwatu et al., 2 Dec 2025).

In this formulation, collective affordance sets are cross-cultural equivalence classes of objects organized by shared function rather than shared appearance. This suggests a direct connection between collective affordance sets and inclusive dataset design: a function-centric representation can preserve culturally specific object diversity while still defining a common semantic space for recognition and retrieval (Nwatu et al., 2 Dec 2025).

5. Shared affordances across agents and embodiments

In human–robot collaboration, collective affordance sets are often spatialized. Shared affordance-awareness via augmented reality defines a human affordance set $\mathcal{A} = \bigcup_{v=1}^{N_v} \mathcal{A}_v$9 and a robot affordance set $\mathcal{A}_v = \{\mathbf{a}_{v,k}\}_{k=1}^{S_v}$0, each represented as a voxel grid of reachable states in a shared world frame established through Azure Spatial Anchors. Human affordances are updated online from observed arm reach and kinematics, whereas robot affordances are precomputed from manipulability ellipsoids. The system then reasons over these sets in Algorithm 1: if the robot cannot reach a target object but the object lies in $\mathcal{A}_v = \{\mathbf{a}_{v,k}\}_{k=1}^{S_v}$1, it queries the human to move it into the robot’s affordable area; if both agents target the same reachable object, the robot replans; if neither set contains the target according to the current model, the robot queries the human to refine the estimate (Moore et al., 2023). A plausible interpretation is that human-only, robot-only, and shared regions correspond respectively to $\mathcal{A}_v = \{\mathbf{a}_{v,k}\}_{k=1}^{S_v}$2, $\mathcal{A}_v = \{\mathbf{a}_{v,k}\}_{k=1}^{S_v}$3, and $\mathcal{A}_v = \{\mathbf{a}_{v,k}\}_{k=1}^{S_v}$4.

Cross-embodied affordance transfer generalizes this idea from spatial sets to latent equivalence classes. The framework represents an affordance instance as $\mathcal{A}_v = \{\mathbf{a}_{v,k}\}_{k=1}^{S_v}$5, then blends action, effect, and object encodings into a common latent $\mathcal{A}_v = \{\mathbf{a}_{v,k}\}_{k=1}^{S_v}$6. Affordance Equivalence is the name given to the resulting shared affordance representation spanning multiple agents and objects. In insertability, graspability, and rollability experiments, the latent space clusters equivalent affordances across different objects, actions, and robots, and the same latent can be decoded into agent-specific action trajectories, enabling cross-embodiment transfer and direct imitation (Aktas et al., 2024).

A decision-theoretic account of collective affordances appears in computational rationality. There, an internal affordance is represented as $\mathcal{A}_v = \{\mathbf{a}_{v,k}\}_{k=1}^{S_v}$7, and collective affordances arise when agents model the internal environments of others in order to infer each other’s confidence and capabilities for joint action. The paper explicitly gives “lifting a heavy object that requires two or more people to coordinate their actions” as the paradigm case and argues that hypothetical motion trajectories can be extended from single-agent simulation to joint-action prediction (Liao et al., 16 Jan 2025).

Across these works, collective affordance sets are not merely unions of individual capabilities. They are shared structures that support allocation, transfer, and coordination: reachable-state sets in AR-mediated collaboration, latent equivalence classes across embodiments, or group-level inferred action possibilities under bounded rationality.

6. Reasoning, uncertainty, and unresolved constraints

Affordance reasoning benchmarks make collective affordance sets explicit as candidate sets to be pruned by evidence. Affordance20Q constructs 1,009 games over 454 objects and 59 affordances; in each game, the questioner receives a candidate affordance set of 8 affordances and must identify the target without ever seeing the object’s identity, using only yes/no questions about physical properties. The evaluation defines a belief distribution $\mathcal{A}_v = \{\mathbf{a}_{v,k}\}_{k=1}^{S_v}$8 over the candidate set and measures information gain by $\mathcal{A}_v = \{\mathbf{a}_{v,k}\}_{k=1}^{S_v}$9. Human success rate is $v$0 in 10.7 average turns, whereas the best closed-source LLM reaches $v$1 in 18.5 turns. KARI, the KB-Anchored Rule Induction pipeline, improves open-source LLMs by up to 15.2 points; its rules cover the target affordance in $v$2 of games, yielding an average $v$3-point gain when the target is covered and an average $v$4-point effect when only distractors are covered (Jiang et al., 12 Jun 2026).

Logical work formalizes uncertainty over collective affordance sets through rough-set machinery. In the ternary relation framework $v$5, lower approximation $v$6 contains triples whose entire indiscernibility block $v$7 is included in $v$8, while upper approximation $v$9 contains triples whose block intersects $[\mathbf{a};\mathbf{o}]$0. Modal operators such as $[\mathbf{a};\mathbf{o}]$1 and $[\mathbf{a};\mathbf{o}]$2 then characterize, respectively, environments where some or all actor–object pairs from $[\mathbf{a};\mathbf{o}]$3 afford the relation. This yields definite and possible collective affordance sets under coarse-grained knowledge rather than full observability (Gruszczynski et al., 4 Dec 2025).

Current implementations remain strongly bounded by their representational assumptions. In ATL, affordances are equated with the verb vocabulary of HICO-DET or HOI-COCO, so affordances outside that label space cannot be represented (Hou et al., 2021). In the Culture Affordance Atlas, the taxonomy contains 46 functions and is explicitly described as a “foundational layer” rather than a comprehensive ontology (Nwatu et al., 2 Dec 2025). AffordPose uses 8 hand-centered affordances and assigns at most one affordance label per part (Jian et al., 2023). Affordance20Q excludes affordances that are not deducible from physical-property dimensions alone, such as microwave heating via a magnetron (Jiang et al., 12 Jun 2026). Shared affordance-awareness in AR simplifies affordances to reachability regions (Moore et al., 2023). The logic-of-affordances framework remains extensional and does not yet formalize higher-order group interaction or dynamic learning (Gruszczynski et al., 4 Dec 2025).

The literature therefore points toward several open directions already identified within the individual works: temporal and social affordances in scene understanding, multi-step and multi-object activities in culturally grounded datasets, bi-manual and human–robot cooperative affordances in hand interaction corpora, richer human capability models in shared AR workspaces, and broader KB coverage plus multimodal grounding for rule-based affordance reasoning (Greene et al., 2014, Nwatu et al., 2 Dec 2025, Jian et al., 2023, Moore et al., 2023, Jiang et al., 12 Jun 2026). A plausible synthesis is that the next stage of research will require explicit representations that can jointly encode shared categories, uncertainty, dynamics, and group structure, rather than treating collective affordance sets only as static label vocabularies or fixed prototype banks.