What Objects Enable, Not What They Are: Functional Latent Spaces for Affordance Reasoning

Abstract: Existing robot planning systems rely on appearance-based reasoning, where visual observations are encoded into latent spaces organized around object appearances (e.g., recognizing a "cart" based on how it looks). However, planning requires reasoning about task-relevant functionalities of objects (e.g., whether an object is "movable"), which appearance-based latent spaces do not capture. As a result, existing approaches struggle to generalize to novel robot-object interactions. We address this limited generalizability through affordance reasoning, enabling planning based on task-relevant object functionalities instead of appearance alone. We introduce A4D, which maps visual observations into a shared latent space structured around affordances (e.g., "movable"). By projecting visual observations into this functional latent space and measuring their proximity to affordances, A4D infers functionalities relevant to the observed object. Furthermore, we introduce an affordance discovery mechanism that expands the latent space to handle unseen scenarios where existing affordances are insufficient. A4D uses proximity in the functional latent space to quantify uncertainty in affordance inference and selectively triggers affordance discovery. We evaluate A4D across several planning tasks involving diverse and unseen affordances. A4D achieves 94% inference accuracy on existing affordances outperforming state-of-the-art approaches by over 15% points, improves new-affordance inference accuracy from 70% to over 90% with fewer than 10% of the original training data, and enables 100x faster inference. Code, videos, and data available at: https://A4Dance-reasoning.github.io.

• The paper introduces A4D, a system that constructs functional latent spaces for affordance inference, achieving 94% accuracy and a >100x runtime speedup.
• It employs calibrated uncertainty and few-shot learning to dynamically discover and label new affordances, effectively reducing annotation costs and preventing catastrophic forgetting.
• The approach outperforms standard vision-language models by over 15 percentage points, bridging the gap between visual perception and goal-conditioned robotic planning.

Functional Latent Spaces for Generalizable Affordance Reasoning in Robotics

Introduction

This work systematically advances affordance-based reasoning for robotic planning by reframing perception and interaction around the functionalities objects enable, rather than their visual identities. Standard latent visual recognition systems—constrained by semantic similarity in appearance—have demonstrated limited utility for robotic planning tasks requiring functional generalization across novel objects. This limitation emerges from the disconnect between appearance-based representations and the robot’s goal-conditioned requirements for interaction, such as whether a given object is movable, stackable, or supportable. The proposed system, Affordance-for-Discovery (A4D), introduces a structured functional latent space that enables inference and discovery of affordances from visual observations, supporting real-time, uncertainty-aware, and incrementally extensible reasoning. A4D explicitly couples perceptual representations with task-driven functions, thereby bridging gaps in both generalization and sample efficiency for adaptive robot planning.

Figure 1: A4D maps visual observations and affordances to a functional latent space.

Methodology: Constructing and Utilizing the Functional Latent Space

Affordance Axis Construction

A4D constructs a low-dimensional, interpretable representation within a pretrained joint vision-language embedding space (CLIP). For each affordance $a$, an axis is defined by the embeddings of positive property descriptions and their antonyms, yielding a 1D semantic direction. Visual embeddings of object images are projected onto this axis to provide a task-conditioned functional similarity score parameterized by the position along the axis. Fine-tuning with minimal labeled pairs further enhances the alignment between visual observations and functional axes.

Affordance Inference and Calibration

Inference is operationalized by projecting an object's visual embedding and thresholding its normalized position relative to the antonym and affordance endpoints. Midpoint or higher projections indicate positive inference of the affordance, yielding binary or scored predictions. Calibration using isotonic regression maps raw axis scores to calibrated probabilities, enabling robust quantification of uncertainty. This uncertainty quantification is critical for triggering further mechanisms in the framework, such as dynamic affordance discovery.

Figure 2: Overview of A4D: consists of affordance generation, discovery, inference, and labeling.

Affordance Discovery and Labeling

When calibration identifies high uncertainty or misalignment with the affordance memory (i.e., none of the current axes decisively explain observations), A4D invokes an affordance discovery mechanism. This process queries a vision-LLM (VLM) with task context to propose candidate affordances, which are then validated and labeled via high-performant VLM queries. This mechanism ensures coverage of novel and emergent affordances without exhaustive, manual annotations and without retraining from scratch.

Figure 3: A4D in two independent deployment scenarios. E.g. 1 (Move the cart'') selects an action based on Uncertainty. E.g. 2 (Climb the stairs'') triggers affordance discovery, adding Traversable.

Experimental Results

Accuracy and Real-Time Performance

A4D delivers 94% inference accuracy on base task affordances while outperforming the strongest baselines—including large VLMs—by over 15 percentage points on held-out test data. Notably, baseline vision-LLMs such as CLIP and BLIP, when used in a zero-shot configuration, remain close to random guessing for these functional distinctions, confirming the necessity of affordance-specific adaptation. On unseen object classes, A4D's accuracy exhibits only a mild reduction, demonstrating generalization of learned axes. Performance on previously unseen affordances drops without fine-tuning, consistent with the hypothesis that semantic axes alone lack sufficient expressivity for robust affordance recognition without targeted updates.

A critical practical result is A4D's runtime: affordance inference is achieved in around 22 ms per object, representing >100x speedup over the most efficient GPT-5 VLM baselines. This meets the computational constraints of closed-loop planning in robotics, contrasting strongly with the prohibitive latencies of on-demand VLM calls.

Affordance Discovery and Incremental Learning

The affordance discovery procedure is highly label-efficient: performance on held-out (previously unseen) affordances saturates to >92% with fewer than 16 labeled examples, representing an order-of-magnitude reduction in annotation cost. Few-shot updates ensure minimal catastrophic forgetting of prior learned functionality as the space expands, whereas large-scale updates introduce more pronounced interference with existing affordances.

Figure 4: Leave-one-out affordance acquisition and existing affordance forgetting. (L) Accuracy of new affordances vs. number of samples. (R) Average accuracy of seed affordances after adding.

Figure 5: Incremental Learning: Seed-Affordance forgetting vs New-Affordance acquisition.

Uncertainty-Guided VLM Fallback

The framework uses calibrated uncertainty to selectively invoke high-performing (but slow and expensive) VLM queries for low-confidence predictions. This uncertainty-gated hybrid achieves 93–97% overall accuracy while limiting VLM calls to <20% of encountered cases. The tradeoff between accuracy and VLM query rate can be smoothly controlled via the uncertainty threshold, allowing system designers to balance precision and computational cost.

Figure 6: Uncertainty-guided VLM trigger: Accuracy vs Query Frequency tradeoff. (L) Accuracy vs. fraction of VLM calls. (R) Percentage of VLM calls as the uncertainty threshold is swept.

Figure 7: Accuracy vs Query Frequency tradeoff on seen image classes.

End-to-End Robotic Planning Scenarios

A4D is demonstrated in downstream planning tasks: selection of objects for manipulation is driven by real-time low-uncertainty affordance inferences; when task requirements cannot be explained by the current affordance repertoire, uncertain inferences trigger the dynamic discovery and labeling of new functional axes (e.g., discovering “Traversable” when required to “Climb the stairs”), after which planning resumes using the expanded space without expensive queries.

Implications and Future Directions

This work reconfigures the foundation of scene understanding for robot policies from semantic categorization to explicit affordance-based representational spaces. Practically, the system's structure—comprising real-time inference, few-shot extensibility, and robust uncertainty quantification—addresses key bottlenecks in deploying robotic agents for open-world settings. The methodology provides a template for extending vision-LLMs as affordance-centric function extractors, rather than solely semantic classifiers. Strong empirical sample-efficiency and minimal catastrophic forgetting suggest that integrating constrastive axis-based adaptation into other architectures may be fruitful.

Nonetheless, several limitations are identified. The axes, once formed, do not adapt based on downstream task success or interactive robot feedback, and planner integration is currently ad hoc. The isotonic uncertainty calibration, while effective, lacks formal statistical guarantees, and the system as a whole is evaluated terminally by affordance inference accuracy rather than end-to-end control performance. Future research will involve on-policy adaptation of axes from rollout feedback, tighter integration of the affordance space with closed-loop planning algorithms, and formalization of uncertainty estimation for compositional task execution.

Conclusion

A4D establishes a scalable, interpretable, and efficient formalism for affordance reasoning in robotics, shifting perception and policy selection from appearance-based recognition to functional capability. Through functional latent spaces, uncertainty-aware discovery, and sample-efficient continual learning, the system enables real-time, extensible affordance inference with high empirical accuracy, surpassing traditional and VLM-based methods. The framework’s design offers a tractable path for bridging vision-language embedding architectures with adaptive robot autonomy, paving the way for task-driven interaction in open-world robot deployments.