---
title: Affordance Graphs in Robotic Reasoning
url: https://www.emergentmind.com/topics/affordance-graphs
type: topic
---

# Affordance Graphs in Robotic Reasoning

Affordance graphs are graph-based representations in which affordances are modeled as structured relations among objects, object parts, actions, effects, and contextual constraints rather than as isolated labels attached to single objects. In the literature considered here, the term covers several distinct but related constructions: binary action–object graphs for zero-shot human–object interaction recognition, knowledge-base graphs for grasp-affordance reasoning, scene graphs whose nodes or edges encode manipulation relations, graphs over compound objects for effect prediction and planning, part-aware 3D scene graphs, and relational graphs that lift semantic part relations into metric poses in \(SE(3)\) for articulated manipulation [2009.01039][1906.09836][2309.10426][2110.14137][2503.07909][2606.27036].

## 1. Conceptual scope

A recurring premise across this body of work is that affordance is fundamentally relational. In zero-shot HOI recognition, the relation is whether an action can be performed on an object, encoded as a binary bipartite affordance graph between action classes and object classes [2009.01039]. In robotic grasp reasoning, the relation links object attributes, category, location, affordance, and grasp region inside a probabilistic knowledge base graph learned with a Markov Logic Network [1906.09836]. In compound-object manipulation, the relation is extended from single objects or object pairs to a current compound \(T\), a candidate object \(o\), and predicted effects \((E_1,E_2,E_3)\), explicitly formalized as
\[
A = (T, o, (E_1, E_2, E_3)).
\]
That formulation makes the affordance graph an effect-predictive representation over assemblies of arbitrary size [2309.10426].

This relational view also sharpens the distinction between affordance, functionality, and goal-directed action. The egocentric annotation work on EPIC-KITCHENS argues that affordance should be treated as hand-centered, animal-relative, and goal-irrelevant, and should therefore be separated from tool-mediated mechanical action and from high-level verb labels such as “turn off” or “take” [2206.05424][2302.03292]. The same distinction appears in image-based scene reasoning, where whether an action such as sit, run, or grasp is appropriate depends not only on the target object but on surrounding objects, physical obstacles, danger, and social norms [1712.07576]. This suggests that an affordance graph is most informative when it represents not only capability, but also the conditions under which that capability is actionable, forbidden, unstable, or contextually transformed.

## 2. Major representational families

The literature contains several recurrent graph schemas.

| Graph family | Core structure | Representative papers |
|---|---|---|
| Action–object feasibility graph | Bipartite graph between action and object classes | [2009.01039] |
| Semantic knowledge-base graph | Entities and weighted semantic relations inferred by MLN | [1906.09836] |
| Scene/object relation graph | Object-instance nodes with spatial or manipulation edges | [1712.07576][2110.14137] |
| Compound-object graph | Nodes are constituent objects; edges encode assembly order | [2309.10426] |
| Part-aware scene graph | Object nodes linked to functional interactive elements | [2503.07909] |
| Embodied symbolic affordance graph | Verb \(\rightarrow\) property \(\rightarrow\) object two-hop graph | [2512.04231] |
| Relational 6D affordance graph | Primary part, anchor part, and kinematic relation in \(SE(3)\) | [2606.27036] |

In the action–object setting, the graph is explicitly undirected and bipartite, with adjacency matrix \(\mathbf{M} \in \{0,1\}^{|\mathcal{O}| \times |\mathcal{A}|}\), where \(m_{jk}=1\) means object \(o_j\) affords action \(a_k\) [2009.01039]. This is the most compact affordance-graph form: feasibility is encoded directly as graph connectivity, and action similarity is induced through distance-2 neighborhoods over shared object nodes.

In probabilistic semantic reasoning, the graph becomes a weighted relational KB. “Learning Grasp Affordance Reasoning through Semantic Relations” represents objects, shapes, textures, materials, categories, locations, affordances, and grasp regions as entities connected by first-order predicates such as `hasShape`, `hasAffordance`, and `graspRegion`, with rule templates weighted by an MLN [1906.09836]. Here the graph is not only a container of relations; it is also the substrate for inference over multiple affordance–region hypotheses.

Scene-grounded robotic formulations shift the graph from class space to instance space. In “Learning to Act Properly,” the graph nodes are object instances or stuff regions in an image and edges encode spatial adjacency, with a Spatial GGNN propagating context to predict whether an action is positive, firmly negative, or an exception requiring explanation and consequence generation [1712.07576]. In “Relationship Oriented Affordance Learning through Manipulation Graph Construction,” the Manipulation Relationship Graph \(G=(V,E)\) is a directed, typed, weighted graph over object instances, where edges encode manipulation relations such as scoop, pour, cut, contain, wipe, and dump [2110.14137].

Two later directions increase representational granularity. “FunGraph” augments a 3D scene graph with functional interactive elements such as handles, knobs, switches, buttons, and pedals, connected to object nodes by directed `has-part` edges whose attributes carry affordance labels such as Foot Push, Tip Push, Rotate, Pinch Pull, Hook Pull, Hook Turn, and Key Press [2503.07909]. “Multi-Object Graph Affordance Network” instead represents a compound object as
\[
G=(N,E), \quad N=\{n_1,\dots,n_k\}, \quad E=\{e_1,\dots,e_{k-1}\},
\]
with directed edges following sequential placement from top to bottom and self-connections at all nodes. This graph is neither fully connected nor symbolic; it is an ordered relational state over an evolving assembly [2309.10426].

A complementary neuro-symbolic line uses explicit semantic composition. CRAFT-E constructs a two-hop bipartite graph with verb nodes, property nodes, and object nodes, with weighted \(v \rightarrow p\) and \(p \rightarrow o\) edges and affordance energy aggregated over \(v \rightarrow p \rightarrow o\) paths [2512.04231]. RelAfford6D advances this idea for articulated manipulation by representing a primary interacting part, a physical anchor, and an action/kinematic relation, then instantiating the semantic graph as metric poses in \(SE(3)\) [2606.27036].

## 3. Grounding graph structure from perception and data

Affordance graphs are grounded from a wide range of perceptual and annotation pipelines. One recurring strategy is to infer object- or part-level graph nodes from segmentation or detection, and then populate attributes from learned visual encoders. In MOGAN, each single object is observed as a \(32 \times 32\) normalized depth image; a pretrained autoencoder with linear layers \(256,256,64\) and latent dimension \(4\) yields a 6-dimensional feature vector after appending the maximum and minimum depth values, and these 6D vectors become graph node attributes [2309.10426]. In image-based scene reasoning, object-instance masks and cropped ResNet-50 features initialize node states, while adjacency is defined by boundary contact in the segmentation map [1712.07576].

Part-centric grounding has become increasingly important. OVAL-Prompt decomposes open-vocabulary affordance localization into task \(\rightarrow\) object \(\rightarrow\) part name \(\rightarrow\) mask, using GPT-4 to infer the relevant object and part and VLPart to segment the part; if the part query fails, the system reprompts for alternative names such as “cup rim” instead of “cup top” [2404.11000]. FunGraph similarly uses YOLO-Worldv8.2 for object detection, RT-DETR for functional element detection, SAM2 for segmentation, CLIP features for semantics, and GPT-4o for contextual label refinement such as converting a generic Hook Pull into “freezer handle” [2503.07909]. In RelAfford6D, SAM3 produces masks for the inferred semantic parts and FoundationPose lifts those parts into 6D poses \(\mathbf{T}_P^*\) and \(\mathbf{T}_A^*\), with BundleSDF providing reconstructed object meshes [2606.27036].

The data side is equally heterogeneous. The egocentric annotation papers define affordance labels as a combination of goal-irrelevant motor action and grasp type, and introduce mechanical action for object–object interaction in tool-use scenarios [2206.05424][2302.03292]. A4Bench organizes affordance evaluation into constitutive affordance, driven by intrinsic properties such as shape, size, and material, and transformative affordance, covering misleading, time-dependent, cross-cultural, and individual-specific cases [2506.00893]. 3DAffordSplat contributes a multimodal affordance corpus over point clouds and 3D Gaussian Splatting, where each Gaussian instance is represented as
\[
\mathcal{G}=\{\mathbf{m},\mathbf{s},\mathbf{r},o,\mathbf{c}\},
\]
and the target is a binary affordance mask \(\mathcal{M}\in\{0,1\}^N\) over Gaussian primitives [2504.11218].

This variety of grounding mechanisms implies that “affordance graph” is not tied to a single sensory substrate. The graph may be built from RGB regions, depth patches, point clouds, 3D Gaussian primitives, egocentric clips, or symbolic attribute labels, provided that the resulting structure supports affordance-relevant relations.

## 4. Inference, learning, and planning

The computational role of the graph varies from probabilistic reasoning to message passing, search, transport, and analytical control. In the MLN-based grasp system, the graph defines a probability distribution over possible worlds:
\[
P(X=x) = \frac{1}{Z}\exp\Big(\sum_{i=1}^n w_i f_i(x_{\{i\})\Big),
\]
with weights learned generatively by maximizing pseudo-log-likelihood and inference approximated by Gibbs sampling [1906.09836]. In zero-shot HOI recognition, a GCN over the affordance graph yields external action and object embeddings, while the graph also supplies pseudo-labels for unseen actions and a margin-based regularizer that makes graph-neighboring actions closer in latent space [2009.01039]. In “Learning to Act Properly,” the GGNN uses adjacency-based message passing and GRU-style updates to propagate physical and social scene context over object nodes [1712.07576].

For robotic manipulation, forward prediction and planning often become the central function. MOGAN passes the compound-object graph through two `GCNConv` layers, pools mean and maximum node embeddings, and decodes relational outcomes \((E_1,E_2,E_3)\) between a candidate new object and each object already in the compound. Those predicted effects then act as a transition model inside a tree search over object-ordering decisions, with \(E_3\) pruning collapse branches and \(E_1,E_2\) supporting height or occlusion objectives [2309.10426]. “Object-agnostic Affordance Categorization via Unsupervised Learning of Graph Embeddings” instead constructs Activity Graphs and object-centered Activity Graphlets from depth-informed qualitative spatial relations and Allen-style temporal relations, embeds them with graph2vec, and performs hierarchical clustering so that affordance categories emerge from graph similarity rather than being specified a priori [2304.05989].

Several recent systems use graph reasoning as one term in a larger embodied objective. CRAFT-E defines graph-based affordance energy over weighted verb–property–object paths and combines it with CLIP-based visual-language alignment and grasp energy in a joint score
\[
E(v,r_i)=\alpha E_{\text{grasp}}+\beta E_{\text{aff}}+\gamma E_{\text{align}},
\]
with \(\alpha=\beta=\gamma=1.0\), selecting the ROI with minimum total energy [2512.04231]. GRAFT uses part-based graphs, source-node masses \(\mu_{s,i}=\gamma^{d_i}V_i\) rooted at the interacted part, and Unbalanced Fused Gromov–Wasserstein matching to retrieve a source instance, align functional parts, and transfer contact points through point-wise correspondence [2606.25241]. RelAfford6D takes a different route: the graph edge \(a\) acts as a kinematic manifold selector, and the grounded relation is converted into revolute or prismatic motion families such as
\[
\mathcal{M}_{\text{rev}}(q)
\quad \text{or} \quad
\mathcal{M}_{\text{pris}}(q),
\]
followed by analytical trajectory synthesis and closed-loop replanning [2606.27036]. This is a particularly explicit example of an affordance graph functioning as a control representation rather than merely a perceptual prior.

## 5. Applications and empirical evidence

Affordance graphs have been evaluated in zero-shot recognition, scene-level action reasoning, manipulation planning, open-vocabulary localization, multimodal 3D grounding, and embodied robot execution. In zero-shot HOI recognition, the bipartite affordance-graph model improves HICO unseen-only mAP from 7.50 to 10.20 under the best reported setting and shows sensitivity to graph completeness, dropping as the fraction of retained graph edges decreases from \(1.0\) to \(0.6\) [2009.01039]. In scene-dependent image reasoning, Spatial GGNN achieves the strongest relationship prediction among the compared models, for example reaching \(0.745/0.461\) mAcc/mAcc-E for sit and \(0.730/0.452\) for run with ground-truth segmentation [1712.07576].

Robotic manipulation results are especially strong where the graph directly encodes action structure. MOGAN succeeds in \(283/300\) simulated planning tasks over six task types and succeeds in \(28\) of \(30\) real-world tallest/shortest trials with a 7-DOF UR10 and Robotiq 3-Finger Adaptive Gripper [2309.10426]. AR-Net with the Manipulation Relationship Graph reaches \(88.89\%\) on task relationship recognition and \(73.33\%\) on task completion across physical experiments, outperforming segmentation-only and detection-plus-knowledge-graph baselines [2110.14137]. RelAfford6D reports that removing the anchor and tracking only the primary-part pose reduces success to \(48.5\%\), compared with \(74.4\%\) for the full system, directly supporting the graph’s relational design [2606.27036].

In language-grounded object selection, CRAFT-E shows that an explicit verb–property–object graph can remain competitive with black-box models while exposing intermediate reasoning paths. On the static benchmark under Perfect Grasp + GT Detect, CRAFT-E reaches \(0.6495\), and in real-world robot trials it reaches \(0.4667\) with the GraspNet backend and \(0.4000\) with the KpNet backend, the highest average grasp success rate across the two backends [2512.04231]. OVAL-Prompt, without any finetuning, achieves a weighted F-score average of \(0.711\) on UMD affordance localization, with particularly strong performance on cut (\(0.823\)) and pound (\(0.809\)); its ablation shows that direct VLM-only affordance prompting yields only \(0.011\) average, while reprompting with alternative part names raises performance from \(0.392\) to \(0.711\) [2404.11000].

Multimodal 3D settings show a similar pattern. AffordSplatNet on 3DAffordSplat reaches mIoU \(30.25\) in the seen setting and \(17.31\) in the unseen setting, outperforming transferred point-cloud baselines; CMSA improves unseen generalization even though it can slightly reduce seen mIoU [2504.11218]. GRAFT improves affordance generalization to ASR \(0.85\), NSS \(0.82\), and DTM \(0.02\), and reaches \(81.25\%\) success in SAPIEN floating-gripper evaluation, substantially above semantic-retrieval baselines [2606.25241].

Benchmarks also reveal the limits of current graph and non-graph systems. A4Bench shows that even the best MLLM, Gemini-2.0-Pro, reaches only \(18.05\%\) overall exact match accuracy, far below human performance at \(85.34\%\) best and \(81.25\%\) worst, with particularly weak performance on transformative affordance categories [2506.00893]. This indicates that explicit relational structure remains an open need rather than a solved problem.

## 6. Boundaries, misconceptions, and open directions

A central misconception in the literature is to equate any scene graph with affordance reasoning to a full affordance graph. The 3D scene-graph work on commonsense affordance estimation explicitly states that its method is best understood as a 3D scene graph augmented with inferred affordance distributions, not as a graph where actions, effects, or task conditions are first-class graph entities [2409.05392]. FunGraph makes a similar distinction: it moves significantly toward an affordance graph by attaching affordance-bearing part nodes and `has-part` relations, but it still lacks explicit action outcomes, preconditions, temporal execution structure, and articulated-state effects [2503.07909]. This suggests that the defining issue is not whether a graph contains objects and affordance labels, but whether it encodes actionable relations rich enough for inference and control.

Another persistent boundary concerns what should count as affordance at all. The egocentric annotation papers argue that verb labels often collapse together hand–object affordance, tool-mediated functionality, and goal semantics; their introduction of mechanical action as a separate object–object relation is therefore a conceptual clarification, not merely a labeling convenience [2206.05424][2302.03292]. A4Bench extends this pressure by showing that constitutive affordance alone is insufficient: misleading, time-dependent, cross-cultural, and individual-specific affordances require context-conditioned, state-conditioned, and agent-conditioned relations [2506.00893]. The benchmark’s distinction implies that static object–action edges are only one layer of a fuller affordance graph.

The papers also converge on several technical limitations. Many systems still use narrow relation vocabularies: MRG has six relation classes, FunGraph seven functional labels, and CECI only containment-style scene edges [2110.14137][2503.07909][2409.05392]. Several methods rely on coarse or externally supplied part structure, as in GRAFT’s manual part decomposition or OVAL-Prompt’s dependence on detector-friendly part names [2606.25241][2404.11000]. Others remain weak on temporal composition and state transitions: Activity Graphlets capture episodes and temporal relations, but not higher-order event structure such as pourable or throwable transitions; FunGraph and commonsense-3DSG approaches do not model effects or long-horizon manipulation plans [2304.05989][2503.07909][2409.05392]. RelAfford6D addresses articulated constraints directly, but its graph remains a compact two-node relational topology rather than a general multi-object affordance program [2606.27036].

Taken together, these limitations point toward a next stage for affordance graphs: graphs with explicit object, part, action, effect, state, agent, and context nodes; typed temporal and causal edges; metric grounding in \(SE(3)\); uncertainty-aware or probabilistic inference; and mechanisms for online update from interaction outcomes. The current literature already provides most of the ingredients—bipartite feasibility structure, semantic KB reasoning, compound-object graph dynamics, part-level 3D grounding, transport-based correspondence, and analytic kinematic execution—but these ingredients are still distributed across distinct graph families rather than unified in a single representation [2009.01039][1906.09836][2309.10426][2504.11218][2606.25241][2606.27036].

Source: https://www.emergentmind.com/topics/affordance-graphs