Affordance Graphs in Robotic Reasoning

• Affordance Graphs are graph-based representations that encode relational affordances among objects, parts, actions, effects, and contextual constraints.
• They have been applied in zero-shot human–object interaction recognition, robotic grasp reasoning, and compound-object effect prediction.
• These graphs integrate diverse perceptual inputs and probabilistic inference methods to guide planning, control, and dynamic scene reasoning.

Affordance graphs are graph-based representations in which affordances are modeled as structured relations among objects, object parts, actions, effects, and contextual constraints rather than as isolated labels attached to single objects. In the literature considered here, the term covers several distinct but related constructions: binary action–object graphs for zero-shot human–object interaction recognition, knowledge-base graphs for grasp-affordance reasoning, scene graphs whose nodes or edges encode manipulation relations, graphs over compound objects for effect prediction and planning, part-aware 3D scene graphs, and relational graphs that lift semantic part relations into metric poses in $SE(3)$ for articulated manipulation (Sarullo et al., 2020, Ardón et al., 2019, Girgin et al., 2023, Tang et al., 2021, Rotondi et al., 10 Mar 2025, Zhang et al., 25 Jun 2026).

1. Conceptual scope

A recurring premise across this body of work is that affordance is fundamentally relational. In zero-shot HOI recognition, the relation is whether an action can be performed on an object, encoded as a binary bipartite affordance graph between action classes and object classes (Sarullo et al., 2020). In robotic grasp reasoning, the relation links object attributes, category, location, affordance, and grasp region inside a probabilistic knowledge base graph learned with a Markov Logic Network (Ardón et al., 2019). In compound-object manipulation, the relation is extended from single objects or object pairs to a current compound $T$, a candidate object $o$, and predicted effects $(E_1,E_2,E_3)$, explicitly formalized as

$A = (T, o, (E_1, E_2, E_3)).$

That formulation makes the affordance graph an effect-predictive representation over assemblies of arbitrary size (Girgin et al., 2023).

This relational view also sharpens the distinction between affordance, functionality, and goal-directed action. The egocentric annotation work on EPIC-KITCHENS argues that affordance should be treated as hand-centered, animal-relative, and goal-irrelevant, and should therefore be separated from tool-mediated mechanical action and from high-level verb labels such as “turn off” or “take” (Yu et al., 2022, Yu et al., 2023). The same distinction appears in image-based scene reasoning, where whether an action such as sit, run, or grasp is appropriate depends not only on the target object but on surrounding objects, physical obstacles, danger, and social norms (Chuang et al., 2017). This suggests that an affordance graph is most informative when it represents not only capability, but also the conditions under which that capability is actionable, forbidden, unstable, or contextually transformed.

2. Major representational families

The literature contains several recurrent graph schemas.

Graph family	Core structure	Representative papers
Action–object feasibility graph	Bipartite graph between action and object classes	(Sarullo et al., 2020)
Semantic knowledge-base graph	Entities and weighted semantic relations inferred by MLN	(Ardón et al., 2019)
Scene/object relation graph	Object-instance nodes with spatial or manipulation edges	(Chuang et al., 2017, Tang et al., 2021)
Compound-object graph	Nodes are constituent objects; edges encode assembly order	(Girgin et al., 2023)
Part-aware scene graph	Object nodes linked to functional interactive elements	(Rotondi et al., 10 Mar 2025)
Embodied symbolic affordance graph	Verb $\rightarrow$ property $\rightarrow$ object two-hop graph	(Chen et al., 3 Dec 2025)
Relational 6D affordance graph	Primary part, anchor part, and kinematic relation in $SE(3)$	(Zhang et al., 25 Jun 2026)

In the action–object setting, the graph is explicitly undirected and bipartite, with adjacency matrix $$\mathbf{M} \in \{0,1\}^{|\mathcal{O}| \times |\mathcal{A}|}$$, where $m_{jk}=1$ means object $T$0 affords action $T$1 (Sarullo et al., 2020). This is the most compact affordance-graph form: feasibility is encoded directly as graph connectivity, and action similarity is induced through distance-2 neighborhoods over shared object nodes.

In probabilistic semantic reasoning, the graph becomes a weighted relational KB. “Learning Grasp Affordance Reasoning through Semantic Relations” represents objects, shapes, textures, materials, categories, locations, affordances, and grasp regions as entities connected by first-order predicates such as hasShape, hasAffordance, and graspRegion, with rule templates weighted by an MLN (Ardón et al., 2019). Here the graph is not only a container of relations; it is also the substrate for inference over multiple affordance–region hypotheses.

Scene-grounded robotic formulations shift the graph from class space to instance space. In “Learning to Act Properly,” the graph nodes are object instances or stuff regions in an image and edges encode spatial adjacency, with a Spatial GGNN propagating context to predict whether an action is positive, firmly negative, or an exception requiring explanation and consequence generation (Chuang et al., 2017). In “Relationship Oriented Affordance Learning through Manipulation Graph Construction,” the Manipulation Relationship Graph $T$2 is a directed, typed, weighted graph over object instances, where edges encode manipulation relations such as scoop, pour, cut, contain, wipe, and dump (Tang et al., 2021).

Two later directions increase representational granularity. “FunGraph” augments a 3D scene graph with functional interactive elements such as handles, knobs, switches, buttons, and pedals, connected to object nodes by directed has-part edges whose attributes carry affordance labels such as Foot Push, Tip Push, Rotate, Pinch Pull, Hook Pull, Hook Turn, and Key Press (Rotondi et al., 10 Mar 2025). “Multi-Object Graph Affordance Network” instead represents a compound object as

$T$3

with directed edges following sequential placement from top to bottom and self-connections at all nodes. This graph is neither fully connected nor symbolic; it is an ordered relational state over an evolving assembly (Girgin et al., 2023).

A complementary neuro-symbolic line uses explicit semantic composition. CRAFT-E constructs a two-hop bipartite graph with verb nodes, property nodes, and object nodes, with weighted $T$4 and $T$5 edges and affordance energy aggregated over $T$6 paths (Chen et al., 3 Dec 2025). RelAfford6D advances this idea for articulated manipulation by representing a primary interacting part, a physical anchor, and an action/kinematic relation, then instantiating the semantic graph as metric poses in $T$7 (Zhang et al., 25 Jun 2026).

3. Grounding graph structure from perception and data

Affordance graphs are grounded from a wide range of perceptual and annotation pipelines. One recurring strategy is to infer object- or part-level graph nodes from segmentation or detection, and then populate attributes from learned visual encoders. In MOGAN, each single object is observed as a $T$8 normalized depth image; a pretrained autoencoder with linear layers $T$9 and latent dimension $o$0 yields a 6-dimensional feature vector after appending the maximum and minimum depth values, and these 6D vectors become graph node attributes (Girgin et al., 2023). In image-based scene reasoning, object-instance masks and cropped ResNet-50 features initialize node states, while adjacency is defined by boundary contact in the segmentation map (Chuang et al., 2017).

Part-centric grounding has become increasingly important. OVAL-Prompt decomposes open-vocabulary affordance localization into task $o$1 object $o$2 part name $o$3 mask, using GPT-4 to infer the relevant object and part and VLPart to segment the part; if the part query fails, the system reprompts for alternative names such as “cup rim” instead of “cup top” (Tong et al., 2024). FunGraph similarly uses YOLO-Worldv8.2 for object detection, RT-DETR for functional element detection, SAM2 for segmentation, CLIP features for semantics, and GPT-4o for contextual label refinement such as converting a generic Hook Pull into “freezer handle” (Rotondi et al., 10 Mar 2025). In RelAfford6D, SAM3 produces masks for the inferred semantic parts and FoundationPose lifts those parts into 6D poses $o$4 and $o$5, with BundleSDF providing reconstructed object meshes (Zhang et al., 25 Jun 2026).

The data side is equally heterogeneous. The egocentric annotation papers define affordance labels as a combination of goal-irrelevant motor action and grasp type, and introduce mechanical action for object–object interaction in tool-use scenarios (Yu et al., 2022, Yu et al., 2023). A4Bench organizes affordance evaluation into constitutive affordance, driven by intrinsic properties such as shape, size, and material, and transformative affordance, covering misleading, time-dependent, cross-cultural, and individual-specific cases (Wang et al., 1 Jun 2025). 3DAffordSplat contributes a multimodal affordance corpus over point clouds and 3D Gaussian Splatting, where each Gaussian instance is represented as

$o$6

and the target is a binary affordance mask $o$7 over Gaussian primitives (Wei et al., 15 Apr 2025).

This variety of grounding mechanisms implies that “affordance graph” is not tied to a single sensory substrate. The graph may be built from RGB regions, depth patches, point clouds, 3D Gaussian primitives, egocentric clips, or symbolic attribute labels, provided that the resulting structure supports affordance-relevant relations.

4. Inference, learning, and planning

The computational role of the graph varies from probabilistic reasoning to message passing, search, transport, and analytical control. In the MLN-based grasp system, the graph defines a probability distribution over possible worlds: $o$8 with weights learned generatively by maximizing pseudo-log-likelihood and inference approximated by Gibbs sampling (Ardón et al., 2019). In zero-shot HOI recognition, a GCN over the affordance graph yields external action and object embeddings, while the graph also supplies pseudo-labels for unseen actions and a margin-based regularizer that makes graph-neighboring actions closer in latent space (Sarullo et al., 2020). In “Learning to Act Properly,” the GGNN uses adjacency-based message passing and GRU-style updates to propagate physical and social scene context over object nodes (Chuang et al., 2017).

For robotic manipulation, forward prediction and planning often become the central function. MOGAN passes the compound-object graph through two GCNConv layers, pools mean and maximum node embeddings, and decodes relational outcomes $o$9 between a candidate new object and each object already in the compound. Those predicted effects then act as a transition model inside a tree search over object-ordering decisions, with $(E_1,E_2,E_3)$0 pruning collapse branches and $(E_1,E_2,E_3)$1 supporting height or occlusion objectives (Girgin et al., 2023). “Object-agnostic Affordance Categorization via Unsupervised Learning of Graph Embeddings” instead constructs Activity Graphs and object-centered Activity Graphlets from depth-informed qualitative spatial relations and Allen-style temporal relations, embeds them with graph2vec, and performs hierarchical clustering so that affordance categories emerge from graph similarity rather than being specified a priori (Toumpa et al., 2023).

Several recent systems use graph reasoning as one term in a larger embodied objective. CRAFT-E defines graph-based affordance energy over weighted verb–property–object paths and combines it with CLIP-based visual-language alignment and grasp energy in a joint score

$(E_1,E_2,E_3)$2

with $(E_1,E_2,E_3)$3, selecting the ROI with minimum total energy (Chen et al., 3 Dec 2025). GRAFT uses part-based graphs, source-node masses $(E_1,E_2,E_3)$4 rooted at the interacted part, and Unbalanced Fused Gromov–Wasserstein matching to retrieve a source instance, align functional parts, and transfer contact points through point-wise correspondence (Lin et al., 23 Jun 2026). RelAfford6D takes a different route: the graph edge $(E_1,E_2,E_3)$5 acts as a kinematic manifold selector, and the grounded relation is converted into revolute or prismatic motion families such as

$(E_1,E_2,E_3)$6

followed by analytical trajectory synthesis and closed-loop replanning (Zhang et al., 25 Jun 2026). This is a particularly explicit example of an affordance graph functioning as a control representation rather than merely a perceptual prior.

5. Applications and empirical evidence

Affordance graphs have been evaluated in zero-shot recognition, scene-level action reasoning, manipulation planning, open-vocabulary localization, multimodal 3D grounding, and embodied robot execution. In zero-shot HOI recognition, the bipartite affordance-graph model improves HICO unseen-only mAP from 7.50 to 10.20 under the best reported setting and shows sensitivity to graph completeness, dropping as the fraction of retained graph edges decreases from $(E_1,E_2,E_3)$7 to $(E_1,E_2,E_3)$8 (Sarullo et al., 2020). In scene-dependent image reasoning, Spatial GGNN achieves the strongest relationship prediction among the compared models, for example reaching $(E_1,E_2,E_3)$9 mAcc/mAcc-E for sit and $A = (T, o, (E_1, E_2, E_3)).$0 for run with ground-truth segmentation (Chuang et al., 2017).

Robotic manipulation results are especially strong where the graph directly encodes action structure. MOGAN succeeds in $A = (T, o, (E_1, E_2, E_3)).$1 simulated planning tasks over six task types and succeeds in $A = (T, o, (E_1, E_2, E_3)).$2 of $A = (T, o, (E_1, E_2, E_3)).$3 real-world tallest/shortest trials with a 7-DOF UR10 and Robotiq 3-Finger Adaptive Gripper (Girgin et al., 2023). AR-Net with the Manipulation Relationship Graph reaches $A = (T, o, (E_1, E_2, E_3)).$4 on task relationship recognition and $A = (T, o, (E_1, E_2, E_3)).$5 on task completion across physical experiments, outperforming segmentation-only and detection-plus-knowledge-graph baselines (Tang et al., 2021). RelAfford6D reports that removing the anchor and tracking only the primary-part pose reduces success to $A = (T, o, (E_1, E_2, E_3)).$6, compared with $A = (T, o, (E_1, E_2, E_3)).$7 for the full system, directly supporting the graph’s relational design (Zhang et al., 25 Jun 2026).

In language-grounded object selection, CRAFT-E shows that an explicit verb–property–object graph can remain competitive with black-box models while exposing intermediate reasoning paths. On the static benchmark under Perfect Grasp + GT Detect, CRAFT-E reaches $A = (T, o, (E_1, E_2, E_3)).$8, and in real-world robot trials it reaches $A = (T, o, (E_1, E_2, E_3)).$9 with the GraspNet backend and $\rightarrow$0 with the KpNet backend, the highest average grasp success rate across the two backends (Chen et al., 3 Dec 2025). OVAL-Prompt, without any finetuning, achieves a weighted F-score average of $\rightarrow$1 on UMD affordance localization, with particularly strong performance on cut ($\rightarrow$2) and pound ($\rightarrow$3); its ablation shows that direct VLM-only affordance prompting yields only $\rightarrow$4 average, while reprompting with alternative part names raises performance from $\rightarrow$5 to $\rightarrow$6 (Tong et al., 2024).

Multimodal 3D settings show a similar pattern. AffordSplatNet on 3DAffordSplat reaches mIoU $\rightarrow$7 in the seen setting and $\rightarrow$8 in the unseen setting, outperforming transferred point-cloud baselines; CMSA improves unseen generalization even though it can slightly reduce seen mIoU (Wei et al., 15 Apr 2025). GRAFT improves affordance generalization to ASR $\rightarrow$9, NSS $\rightarrow$0, and DTM $\rightarrow$1, and reaches $\rightarrow$2 success in SAPIEN floating-gripper evaluation, substantially above semantic-retrieval baselines (Lin et al., 23 Jun 2026).

Benchmarks also reveal the limits of current graph and non-graph systems. A4Bench shows that even the best MLLM, Gemini-2.0-Pro, reaches only $\rightarrow$3 overall exact match accuracy, far below human performance at $\rightarrow$4 best and $\rightarrow$5 worst, with particularly weak performance on transformative affordance categories (Wang et al., 1 Jun 2025). This indicates that explicit relational structure remains an open need rather than a solved problem.

6. Boundaries, misconceptions, and open directions

A central misconception in the literature is to equate any scene graph with affordance reasoning to a full affordance graph. The 3D scene-graph work on commonsense affordance estimation explicitly states that its method is best understood as a 3D scene graph augmented with inferred affordance distributions, not as a graph where actions, effects, or task conditions are first-class graph entities (Saucedo et al., 2024). FunGraph makes a similar distinction: it moves significantly toward an affordance graph by attaching affordance-bearing part nodes and has-part relations, but it still lacks explicit action outcomes, preconditions, temporal execution structure, and articulated-state effects (Rotondi et al., 10 Mar 2025). This suggests that the defining issue is not whether a graph contains objects and affordance labels, but whether it encodes actionable relations rich enough for inference and control.

Another persistent boundary concerns what should count as affordance at all. The egocentric annotation papers argue that verb labels often collapse together hand–object affordance, tool-mediated functionality, and goal semantics; their introduction of mechanical action as a separate object–object relation is therefore a conceptual clarification, not merely a labeling convenience (Yu et al., 2022, Yu et al., 2023). A4Bench extends this pressure by showing that constitutive affordance alone is insufficient: misleading, time-dependent, cross-cultural, and individual-specific affordances require context-conditioned, state-conditioned, and agent-conditioned relations (Wang et al., 1 Jun 2025). The benchmark’s distinction implies that static object–action edges are only one layer of a fuller affordance graph.

The papers also converge on several technical limitations. Many systems still use narrow relation vocabularies: MRG has six relation classes, FunGraph seven functional labels, and CECI only containment-style scene edges (Tang et al., 2021, Rotondi et al., 10 Mar 2025, Saucedo et al., 2024). Several methods rely on coarse or externally supplied part structure, as in GRAFT’s manual part decomposition or OVAL-Prompt’s dependence on detector-friendly part names (Lin et al., 23 Jun 2026, Tong et al., 2024). Others remain weak on temporal composition and state transitions: Activity Graphlets capture episodes and temporal relations, but not higher-order event structure such as pourable or throwable transitions; FunGraph and commonsense-3DSG approaches do not model effects or long-horizon manipulation plans (Toumpa et al., 2023, Rotondi et al., 10 Mar 2025, Saucedo et al., 2024). RelAfford6D addresses articulated constraints directly, but its graph remains a compact two-node relational topology rather than a general multi-object affordance program (Zhang et al., 25 Jun 2026).

Taken together, these limitations point toward a next stage for affordance graphs: graphs with explicit object, part, action, effect, state, agent, and context nodes; typed temporal and causal edges; metric grounding in $\rightarrow$6; uncertainty-aware or probabilistic inference; and mechanisms for online update from interaction outcomes. The current literature already provides most of the ingredients—bipartite feasibility structure, semantic KB reasoning, compound-object graph dynamics, part-level 3D grounding, transport-based correspondence, and analytic kinematic execution—but these ingredients are still distributed across distinct graph families rather than unified in a single representation (Sarullo et al., 2020, Ardón et al., 2019, Girgin et al., 2023, Wei et al., 15 Apr 2025, Lin et al., 23 Jun 2026, Zhang et al., 25 Jun 2026).