AffordVLA: Affordance-Enhanced Manipulation

Updated 13 July 2026

AffordVLA is an affordance-aware framework that enhances VLA models by aligning an intermediate visual layer with a frozen zero-shot teacher.
It employs implicit feature alignment using cosine similarity to emphasize key interaction regions like handles and blades, improving perceptual focus.
The approach boosts training efficiency and robustness in cluttered settings, achieving significant gains in both simulation and real-world tasks.

Searching arXiv for the exact AffordVLA paper and closely related affordance-aware VLA work. AffordVLA is an affordance-enhanced vision-language-action framework for robotic manipulation that internalizes manipulation-centric affordance perception into VLA visual representations through implicit representation alignment, rather than through explicit mask injection or an external affordance module at inference time (Kong et al., 17 May 2026). Its central claim is that standard VLA policies encode global object semantics effectively but do not reliably emphasize functional interaction regions such as handles, blades, openings, or contact surfaces, and that this weakness limits robustness in cluttered and unstructured environments. AffordVLA addresses this by using a frozen zero-shot affordance teacher during training to provide task-conditioned affordance visual features from RGB observations and language instructions, then aligning an intermediate visual layer of the VLA to those features while keeping the runtime policy architecture essentially unchanged.

1. Problem setting and design rationale

AffordVLA is motivated by a specific diagnosis of standard VLA models: they inherit visual backbones from pretrained VLMs that are optimized for global semantic understanding, which is effective for object recognition and language grounding but insufficient for manipulation-relevant perception (Kong et al., 17 May 2026). In the formulation presented by the paper, the critical question for manipulation is not only what object is present, but which part of that object should be interacted with for the current task. The examples given include a mug handle, knife blade, hammer head, and container opening.

The method also rejects a straightforward remedy, namely explicit affordance mask prediction followed by mask injection into the policy, on three grounds. First, explicit affordance methods usually require large-scale affordance labels. Second, they create inference-time dependence on an external affordance detector or segmentation module. Third, they introduce latency and overhead that can interfere with high-frequency control. AffordVLA therefore adopts a training-only affordance pathway: affordance supervision is used to reshape the VLA’s internal representation, but the teacher is removed entirely at deployment.

In one sentence, the method uses a frozen zero-shot affordance teacher to produce task-conditioned affordance visual features from the current RGB image and language instruction, and aligns an intermediate visual layer of the VLA to those teacher features with a cosine-similarity loss (Kong et al., 17 May 2026). This makes affordance a representational bias rather than an explicit runtime input.

2. System architecture

AffordVLA has three components: an affordance teacher, an understanding expert, and an action expert (Kong et al., 17 May 2026). Only the understanding expert is directly modified by affordance alignment during training; the action-generation pipeline remains standard.

Component	Role	Realization
Affordance teacher	Generates task-conditioned affordance supervision	Frozen zero-shot teacher
Understanding expert	Encodes image, language, and robot state	$\pi_{0.5}$ / PaliGemma-style backbone
Action expert	Generates continuous action chunks	Lightweight Transformer following $\pi_0$

The understanding expert is built from $\pi_{0.5}$ / PaliGemma-style components, with Gemma-2B as the LLM backbone and SigLIP-So400m as the visual encoder. Given language instruction $l_t$ , image observation $I_t$ , and robot state $s_t$ , the model computes visual, text, and state tokens,

$V_t = E_{\mathrm{vis}}(I_t), \qquad L_t = T_{\mathrm{txt}}(l_t), \qquad S_t = T_{\mathrm{state}}(s_t),$

and fuses them with the VLM Transformer: $C_t = F_{\mathrm{VLM}}([V_t, L_t, S_t]).$

The action expert follows $\pi_0$ and predicts a future chunk of continuous robot actions,

$A_t = [a_t, a_{t+1}, \ldots, a_{t+H-1}].$

Training uses conditional flow matching. The action chunk is corrupted with Gaussian noise,

$\pi_0$ 0

and the action loss is

$\pi_0$ 1

The paper also states that the action expert uses a fully bidirectional attention mask over action tokens so that all actions in the chunk can attend to one another.

This division of labor is significant because the affordance signal does not replace end-to-end action generation. Instead, it modifies the perceptual substrate on which the action model operates.

3. Zero-shot affordance teacher

The affordance teacher is the most distinctive supervisory component in AffordVLA (Kong et al., 17 May 2026). It is designed specifically to avoid annotated affordance datasets, and it is frozen throughout training. The teacher has two stages.

The first stage is a multimodal task parsing module. Given image $\pi_0$ 2 and instruction $\pi_0$ 3, a VLM such as Qwen3-VL parses the task into a part-level affordance prompt,

$\pi_0$ 4

with prompts such as “hammer handle” or “hammer head.” The important property is that $\pi_0$ 5 is a task-relevant functional concept rather than a generic object label.

The second stage is an open-vocabulary affordance perception module built on SAM3-style encoders. The paper describes this module as having a text encoder and image encoder sharing a semantic space, an affordance fusion encoder that injects prompt information into visual features through Transformer-based cross-modal interaction, and a multi-scale decoder plus Transformer decoder that output pixel-level affordance predictions. The teacher produces both an intermediate affordance representation and a pixel-level prediction: $\pi_0$ 6

A key design choice is that AffordVLA aligns the VLA to the intermediate feature $\pi_0$ 7, not to the final mask $\pi_0$ 8. The paper explicitly argues that the feature preserves both task semantics and local spatial structure, whereas the final mask mainly regularizes the teacher and verifies that the zero-shot predictions are sensible. In the paper’s notation, $\pi_0$ 9, where $\pi_{0.5}$ 0 is the number of flattened spatial tokens and $\pi_{0.5}$ 1 is the teacher feature dimension.

Implementation details reported for the teacher include approximately 0.8B parameters, visual input resolution $\pi_{0.5}$ 2, and 6 Transformer layers each in the fusion encoder and decoder.

4. Implicit feature alignment

The core mechanism of AffordVLA is implicit affordance representation alignment (Kong et al., 17 May 2026). To preserve token ordering in the autoregressive VLA, the teacher feature is first combined with positional embedding: $\pi_{0.5}$ 3

The target of alignment is not the final VLA output but an intermediate visual representation. Let $\pi_{0.5}$ 4 denote the visual representation from the $\pi_{0.5}$ 5-th layer of the understanding expert. The paper aligns the 12th layer out of 18. The stated rationale is that shallow layers are too local and their effect can be washed out later, whereas very deep layers are too abstract and modality-agnostic, with less visual structure; intermediate-deep layers provide a balance of spatial detail and semantic abstraction.

Because teacher and VLA features live in different spaces, the VLA feature is resized to the teacher’s spatial resolution with bilinear interpolation, normalized, and projected with a two-layer MLP: $\pi_{0.5}$ 6 After projection, $\pi_{0.5}$ 7, enabling direct comparison with $\pi_{0.5}$ 8.

The alignment objective is cosine similarity at each spatial location: $\pi_{0.5}$ 9 The full training objective is

$l_t$ 0

with $l_t$ 1.

The teacher is frozen; only the VLA/action model is updated. During training, AffordVLA therefore adds an alignment head and an auxiliary objective. During inference, the teacher is removed entirely, and the runtime pipeline remains the same as a standard VLA. This training–inference asymmetry is central to the method’s efficiency claim.

5. Empirical performance, efficiency, and ablations

AffordVLA is evaluated both as a teacher-driven affordance method and as a full manipulation policy (Kong et al., 17 May 2026). The teacher itself is tested on AGD20K Unseen, without training or finetuning on AGD20K, using instructions synthesized from labels with the template “region to <action> of the <object>.” It achieves KLD = 0.905, SIM = 0.496, and NSS = 1.906, which the paper identifies as best on KLD and NSS and second-best on SIM.

For policy evaluation, the main simulation benchmark is RoboTwin2.0, using five bimanual tasks: Beat Block Hammer, Place Bread Skillet, Place Bread Basket, Place Can Basket, and Place Cans Plasticbox. Each method is trained on only 50 disturbance-free Easy demonstrations. AffordVLA achieves 61.2% average SR on Easy and 28.8% average SR on Hard, which is reported as best overall. The paper further reports gains over DP3 by 14.8 points in Easy and over RDT by 18.6 points in Hard. The emphasis in the paper is that the advantage is especially meaningful under clutter, lighting variation, texture variation, and harder initial states.

The real-world evaluation uses a UR5 arm, Robotiq 2F-85 gripper, Kinect DK third-person camera, RealSense wrist camera, inference on an NVIDIA A100 server, and a local control client. Eight tasks are listed: pouring water, hanging a mug, cutting a banana with a knife, striking a block with a hammer, sweeping a table, wiping a stain, placing a marker into a pen holder, and sorting objects in clutter. Each task uses 50 real trajectories. The paper states that AffordVLA performs best overall across the eight tasks, with especially strong gains on tool-use tasks, cluttered scenes, handle localization, and target selection under distractions.

A major practical result concerns inference efficiency. On real-world tasks, the paper reports:

$l_t$ 2: 12.5 Hz, 80.2 ms
$l_t$ 3 explicit affordance input: 4.8 Hz, 206.9 ms
AffordVLA: 12.4 Hz, 80.4 ms

This supports the claim that explicit affordance injection slows the system substantially, whereas AffordVLA preserves essentially the original runtime because the teacher is training-only.

The ablations are aligned with the method’s central thesis. Removing the affordance alignment loss reduces performance across RoboTwin2.0 tasks, especially in the Hard setting. Attention visualization at the 12th layer shows that, without alignment, attention is diffuse over object, background, and distractors, whereas with alignment it concentrates on functional regions such as handles, openings, and contact points. The paper also reports that the aligned model needs about 5.2k fewer iterations to reach an average SR of 45%, indicating improved training efficiency. A further real-world observation is that directly feeding affordance masks into $l_t$ 4 is not consistently helpful and often performs worse than the original model, which the paper attributes to perturbing the pretrained VLM representation distribution.

6. Position within affordance-aware VLA research

AffordVLA occupies a specific position within a broader family of affordance-aware embodied models. Its distinguishing characteristic is implicit feature alignment: affordance is injected into an intermediate visual representation during training, but not represented as an explicit runtime mask or token sequence at inference (Kong et al., 17 May 2026).

Related methods span several distinct design choices:

Method	Affordance mechanism	Inference-time dependency
AffordVLA	Implicit alignment to frozen teacher features	No teacher at runtime
Afford-VLA	Internal affordance mask via <AFF> tokens and Top- $l_t$ 5 pooling	Internal affordance pathway retained
AffordanceVLA	Which2Act, Where2Act, How2Act in MoT architecture	Internal affordance experts retained

Afford-VLA formulates affordance as an explicit internal visual planning interface: it introduces learnable <AFF> tokens, decodes task-conditioned affordance masks, converts them to compact embeddings through hard Top- $l_t$ 6 patch pooling, and feeds those embeddings directly to the action model; joint training with a straight-through estimator allows action loss to refine the affordance pathway (Wang et al., 22 May 2026). In contrast, AffordVLA does not decode an internal mask for action conditioning and does not retain an affordance module at runtime.

AffordanceVLA introduces a more structured intermediate representation, decomposed into Which2Act, Where2Act, and How2Act, within a Mixture-of-Transformer architecture and a three-stage progressive training curriculum (Yu et al., 4 Jun 2026). Relative to that design, AffordVLA is narrower in scope: it focuses on reshaping an existing VLA representation rather than adding specialized affordance-generation experts and multi-branch forecasting objectives.

The broader affordance literature also includes upstream perception and grounding systems that are not full VLA policies. WorldAfford reframes affordance grounding as localization conditioned on natural-language instructions, using LLM reasoning, SAM, CLIP, and a modified LOCATE-style network (Chen et al., 2024). YOLOA addresses the “what–where–how” problem through a real-time affordance detection architecture with an LLM adapter, but is explicitly presented as a perception layer rather than an end-to-end action policy (Ji et al., 3 Dec 2025). These systems are complementary rather than directly comparable: they emphasize affordance perception and grounding, whereas AffordVLA uses affordance supervision to reshape policy representations.

Within the wider VLA landscape, adjacent work also targets weaknesses in perception–action coupling by different means. AVA-VLA shifts from a history-agnostic MDP perspective to a POMDP formulation with recurrent-state-guided Active Visual Attention (Xiao et al., 24 Nov 2025). Affordance Field Intervention adds a training-free 3D Spatial Affordance Field at inference time to help VLAs escape “Memory Trap” failures under distribution shift (Xu et al., 8 Dec 2025). Against these alternatives, AffordVLA represents the training-time, representation-level branch of affordance-aware VLA design: it improves spatially grounded manipulation without adding inference-time perception overhead, but it also inherits the limitations noted by its authors, including teacher noise, lack of direct action-level affordance integration, unverified long-horizon and multi-subgoal generalization, and limited cross-platform validation (Kong et al., 17 May 2026).