AffordanceVLA: Affordance-Aware Vision-Language-Action

• The paper introduces a modular framework using Which2Act, Where2Act, and How2Act to ground vision-language semantics into actionable affordance forecasts.
• AffordanceVLA is defined by its spatially and semantically conditioned affordance maps that direct accurate object manipulation in robotic systems.
• The approach employs a three-stage curriculum and automated data augmentation to ensure robust transfer from perception to embodied control.

AffordanceVLA denotes an affordance-centered vision–language–action formulation in which task-oriented intermediate representations mediate between a pretrained vision–language backbone and a manipulation policy. In the paper "AffordanceVLA: A Vision-Language-Action Model Empowering Action Generation through Affordance-Aware Understanding" (Yu et al., 4 Jun 2026), this mediation is organized as structured affordance forecasting through three components—Which2Act, Where2Act, and How2Act—which respectively model object-centric grounding, 2D interaction localization, and 3D geometric reasoning before action generation. In a broader research sense, the term also names a family of affordance-aware robotic systems that use language and perception to infer what object or part to manipulate, where to interact, and how to execute the action (Tang et al., 2 Mar 2025).

1. Scope and conceptual identity

AffordanceVLA is motivated by what the 2026 work identifies as a structural mismatch between vision-LLM semantic spaces and embodied control policies. Pretrained VLMs are optimized for semantic alignment between images and text, whereas robotic control requires precise perception–action mappings in physical space. The proposed remedy is to insert structured affordance forecasting as an intermediate representation that is spatially grounded, semantically conditioned, and action-coupled (Yu et al., 4 Jun 2026).

Within the broader literature, “affordance” is consistently task-conditioned rather than purely object-intrinsic. A knife may expose a handle for grasping and a blade for cutting; a cup may expose a handle for hanging and an opening for pouring; the same object can therefore induce different actionable regions depending on the instruction or manipulation stage (Kong et al., 17 May 2026). This task dependence is one of the main reasons affordance-aware VLAs differ from generic visuomotor policies: they do not merely identify objects, but attempt to localize functional interaction regions and translate them into control-relevant structure.

A common misconception is that affordance-aware VLA necessarily implies a single end-to-end policy network. The current literature includes several distinct realizations: modular pipelines in which a VLM reasons and a separate geometric module acts, architectures that internalize affordance as an explicit visual planning channel, systems that inject affordance into visual features through training-time alignment, and unified models that place affordance forecasting between understanding and action (Tang et al., 2 Mar 2025).

2. Core representational scheme

In the 2026 AffordanceVLA formulation, the overall computation is staged as

$$h_t^{und} = \mathcal{M}_{und}(O_t, l), \qquad \hat{A}_t = \mathcal{M}_{gen}(h_t^{und}), \qquad \hat{a}_{t:t+k} = \mathcal{M}_{act}(h_t^{und}, \hat{A}_t, s_t),$$

where $O_t$ is the RGB observation, $l$ is the instruction, $s_t$ is the proprioceptive state, $\mathcal{M}_{und}$ is the Understanding Expert, $\mathcal{M}_{gen}$ is the Affordance Generation Expert, and $\mathcal{M}_{act}$ is the Action Expert (Yu et al., 4 Jun 2026). This formulation makes affordance neither a post hoc explanation nor a purely auxiliary supervision target; it is the explicit interface between perception and control.

The three affordance components serve complementary functions.

Component	Function	Supervised target
Which2Act	Object-centric grounding that suppresses distractions	Visual latent prediction
Where2Act	2D interaction localization	Affordance map estimation
How2Act	3D geometric reasoning	3D shape latent and 10-DoF layout

Which2Act predicts a target-region visual latent rather than a class label. The ground-truth crop is encoded by a frozen Flux VAE into $z_q \in \mathbb{R}^{C \times H \times W}$, with the main configuration using $C=16$, $H=W=32$, and total latent dimension $O_t$0. The prediction loss is

$O_t$1

This forces the affordance tokens to reconstruct the appearance of the task-relevant object or part rather than the full scene, thereby reducing clutter sensitivity (Yu et al., 4 Jun 2026).

Where2Act estimates a dense 2D affordance map $O_t$2 over the image. The supervision target $O_t$3 is generated from an interaction point and a SAM mask, then optimized with pixelwise BCE:

$O_t$4

This turns language-conditioned interaction localization into a dense prediction problem rather than a sparse point regression problem (Yu et al., 4 Jun 2026).

How2Act provides explicit 3D priors through two branches. One branch predicts a 3D shape latent with a denoising objective,

$O_t$5

and the other regresses a 10-DoF layout vector containing rotation, scale, and translation using Smooth L1 loss,

$O_t$6

The result is a representation that encodes both local interaction geometry and global object placement (Yu et al., 4 Jun 2026).

3. Architecture and optimization

AffordanceVLA implements these representations through a Mixture-of-Transformer architecture with three specialized experts: an Understanding Expert, an Affordance Generation Expert, and an Action Expert. The inter-expert attention pattern is explicitly unidirectional—Understanding $O_t$7 Affordance $O_t$8 Action—while each expert retains bidirectional self-attention internally. This Unidirectional Understanding–Affordance–Action design prevents action gradients from overwriting semantic and affordance representations, while still allowing the policy to consume both (Yu et al., 4 Jun 2026).

The Affordance Generation Expert contains separate token groups for Which2Act, Where2Act, and How2Act. These token groups are not independent heads in isolation; they are jointly refined inside the same transformer stack and can cross-attend to one another. The paper’s block-wise-token ablation disables this interaction and degrades performance, which suggests that object-centric grounding, 2D localization, and 3D geometry are not merely parallel supervision targets but mutually informative subspaces of the same affordance representation (Yu et al., 4 Jun 2026).

Training follows a three-stage curriculum. Stage I is general affordance grounding pre-training on AGD20K, RefSpatial, and PRISM, with the vision encoder, Understanding Expert, and Action Expert frozen, and the objective

$O_t$9

where

$l$0

with $l$1 and $l$2. Stage II jointly trains affordance and action on InternData-A1 with

$l$3

using $l$4 and $l$5. Stage III performs target-task post-training on LIBERO, CALVIN, DROID subsets, and in-house real trajectories, with the affordance weight reduced to $l$6 so that action adaptation dominates while affordance remains a regularizer (Yu et al., 4 Jun 2026).

This training strategy implies a specific theory of transfer. Stage I aligns affordance tokens with VLM semantics; Stage II couples that affordance space to robotic action; Stage III adapts the integrated representation to the target embodiment and benchmark. A plausible implication is that affordance forecasting is being used not only as a prediction target but as a curriculum for stabilizing representation transfer from generic VLMs into embodied control.

4. Automated affordance supervision and data generation

A central practical problem for affordance-aware VLAs is the scarcity of dense affordance labels in robotic datasets. AffordanceVLA addresses this through a robust automated data augmentation pipeline rather than manual per-frame annotation (Yu et al., 4 Jun 2026).

For Stage I, the method relies on affordance-rich external corpora such as AGD20K, RefSpatial, and PRISM. For Stages II and III, it synthesizes affordance supervision from robot trajectories. The pipeline begins with keyframe extraction from robot joint velocities and gripper state, then uses Claude Opus-4.5 to decompose the original instruction into atomic sub-instructions aligned to those keyframes. Per-keyframe affordance queries are handled with VLMs and grounding models, including Qwen3-VL, a fine-tuned RexOmni model, SAM, and SAM-3D; object boxes, interaction points, masks, 3D shapes, and layout signals are then converted into Which2Act, Where2Act, and How2Act targets (Yu et al., 4 Jun 2026).

This design places AffordanceVLA in a broader methodological shift away from hand-labeled affordance masks and toward synthetic supervision generated by foundation models. Related work explores several alternative forms of this idea. AffordVLA aligns intermediate VLA visual features to dense affordance features from a zero-shot affordance teacher built from Qwen3-VL and a SAM3-based module, and removes the teacher entirely at inference (Kong et al., 17 May 2026). Afford-VLA internalizes affordance more explicitly by introducing learnable <AFF> tokens, decoding patch-level affordance masks inside the VLA, and using sparse Top-$l$7 straight-through pooling to convert them into action-conditioning embeddings (Wang et al., 22 May 2026). Chain-of-Affordance instead supervises structured object, grasp, spatial, and movement affordances as an intermediate reasoning chain that conditions a DiffusionVLA policy (Li et al., 2024).

Earlier systems expose the same trajectory of ideas in more modular forms. OVAL-Prompt uses GPT-4 plus VLPart for open-vocabulary affordance localization without any new training (Tong et al., 2024). AffordanceLLM uses LLaVA-style reasoning plus a mask decoder to generate 2D affordance heatmaps from action queries (Qian et al., 2024). NaturalVLM turns fine-grained instructions into point-cloud affordance maps and end-effector poses for multi-step manipulation (Xu et al., 2024). These systems are not identical to AffordanceVLA, but they establish the broader design principle that affordance can be used as a language-grounded control interface rather than a purely descriptive label.

5. Empirical position within affordance-aware VLA research

The AffordanceVLA paper reports strong performance across diverse manipulation scenarios in simulation and real-world settings, but the broader empirical case for affordance-centered intermediate representations is established most clearly by adjacent systems (Yu et al., 4 Jun 2026). Across multiple benchmarks, these systems consistently outperform direct instruction-to-action baselines or explicit-mask baselines that lack tightly integrated affordance reasoning.

System	Setting	Reported result
Afford-VLA (Wang et al., 22 May 2026)	LIBERO average	97.4%
Afford-VLA (Wang et al., 22 May 2026)	LIBERO-Plus average	78.1%
AffordVLA (Kong et al., 17 May 2026)	RoboTwin 2.0 average success	61.2 (Easy), 28.8 (Hard)
AffordVLA (Kong et al., 17 May 2026)	Real-world runtime	12.4 Hz, 80.4 ms
CoA (Li et al., 2024)	Real-world 7-task average success	85.5%
CoA (Li et al., 2024)	LIBERO average	79.8%
AffordGrasp (Tang et al., 2 Mar 2025)	Real-world average GSR	0.83
AffordGrasp (Tang et al., 2 Mar 2025)	Cluttered simulation average GSR	0.77

These results support several narrower conclusions. First, affordance-aware representations improve robustness under clutter, distractors, and pose variation. AffordGrasp raises cluttered-scene simulation grasp success from 0.54 for ThinkGrasp to 0.77, and real-world average grasp success to 0.83 (Tang et al., 2 Mar 2025). Second, affordance integration can improve policy performance without sacrificing runtime: AffordVLA preserves near-identical runtime to $l$8 at 12.4 Hz and 80.4 ms, whereas an explicit affordance-input baseline drops to 4.8 Hz and 206.9 ms (Kong et al., 17 May 2026). Third, action-aligned internal affordance pathways can outperform both generic VLAs and external affordance modules: Afford-VLA reaches 97.4% average success on LIBERO and 78.1% on LIBERO-Plus (Wang et al., 22 May 2026). Fourth, chain-structured affordance reasoning is especially effective for long-horizon and obstacle-sensitive behavior, with CoA improving real-world average success to 85.5% and outperforming OpenVLA, Octo, and vanilla DiffusionVLA (Li et al., 2024).

The cumulative implication is that AffordanceVLA is not an isolated architectural proposal but part of an empirical trend: affordance-aware intermediate structure is repeatedly associated with better generalization, stronger spatial precision, and, in some implementations, preserved inference efficiency.

6. Interpretive themes, misconceptions, and open problems

A recurring misconception is that affordance-aware understanding and control necessarily require explicit mask injection at inference. The recent literature now spans at least four distinct paradigms. Modular systems such as AffordGrasp and OVAL-Grasp perform VLM reasoning, open-vocabulary grounding, and geometry-based action generation as separate stages (Tang et al., 2 Mar 2025). Training-only supervision methods such as AffordVLA use affordance teachers to shape internal visual representations and then discard the teacher at deployment (Kong et al., 17 May 2026). Internal-planning approaches such as Afford-VLA make affordance an explicit mask-like internal variable that is generated and consumed inside the policy (Wang et al., 22 May 2026). Intervention frameworks such as Affordance Field Intervention instead leave the VLA frozen and use a 3D Spatial Affordance Field only when the robot enters a “memory trap” under distribution shift (Xu et al., 8 Dec 2025).

Another misconception is that affordance is purely 2D or purely semantic. The literature contains strong 3D and video-guided formulations. LMAffordance3D fuses language instructions, interaction images, and point clouds to predict probabilistic 3D affordance heatmaps, including full-view, partial-view, and rotation-view settings (Zhu et al., 7 Apr 2025). VAGNet introduces video-guided 3D affordance grounding, arguing that dynamic human–object interaction sequences provide functional supervision that static cues miss (Mao et al., 24 Feb 2026). These results suggest that future AffordanceVLA systems may need to integrate image-space localization, 3D shape priors, and temporal interaction evidence rather than relying on any single affordance modality.

The main open problems are likewise consistent across papers. Teacher-based systems depend on teacher quality and can inherit hallucinations or misgrounding (Kong et al., 17 May 2026). Explicit affordance-mask systems still depend on supervision quality and are often limited to 2D affordances (Wang et al., 22 May 2026). Modular grasping systems remain open-loop and can fail under unusual ambiguity or segmentation errors (Tang et al., 2 Mar 2025). Broader surveys of the design space also point to the need for long-horizon stage transitions, tighter coupling between affordance priors and action dynamics, richer modalities such as tactile or temporal affordances, and evaluation on more diverse robots and environments (Li et al., 2024).

Taken together, AffordanceVLA marks a consolidation point in affordance-aware robotic foundation modeling. It names a specific 2026 architecture that formalizes Which2Act, Where2Act, and How2Act inside a unified VLA (Yu et al., 4 Jun 2026), but it also indexes a broader research program: replacing opaque direct visuomotor mappings with structured, grounded, instruction-conditioned affordance representations that better align semantic understanding with embodied action.