AFUN: Towards an Affordance Foundation Model for Functionality Understanding

Abstract: Affordance understanding bridges visual perception and physical action, serving as an explainable interface for robot manipulation in open and unstructured real-world environments. Yet, building an affordance foundation model that not only understands where and how the interaction should happen, but also generalizes across diverse environments, objects, and tasks, remains a long-standing research challenge. Existing methods typically address only part of this challenge, either localizing task-relevant regions without specifying executable motion, or predicting motion but with limited scalability. In this paper, we present ourmodel, a step towards an affordance foundation model for functionality understanding. From a single RGB-D observation and a language task description, ourmodel predicts a task-conditional functional mask (where to interact) and a 3D post-contact motion curve (how to interact). To support open-world generalization, we build a large-scale standardized data pipeline that converts heterogeneous robot, human, simulation, and real-world scan data into a shared affordance schema with language, masks, and object-centric 3D motion labels. We evaluate ourmodel from three aspects: for affordance segmentation, ourmodel outperforms all baselines by a large margin across 8 test sets from 4 benchmarks, improving mean gIoU/cIoU by +23.9/+26.3; for contact-point prediction, it predicts substantially more accurate points, with a 12.7--61.3% hit-rate gain over the best baseline; and for 3D motion, it achieves the best performance on all three test sets. ourmodel can be deployed for real-world robot manipulation without finetuning for robot embodiment or using task-specific heuristics, demonstrating the ability to adapt to open-world affordance tasks. Project page: https://www.zhaoningwang.com/AFUN

• The paper introduces a unified model that simultaneously predicts actionable regions and corresponding 3D motion trajectories for robust affordance understanding.
• It employs a scalable multi-source data pipeline combining RGB-D, language prompts, and diverse simulation and real data to enhance segmentation and motion prediction.
• Empirical results show state-of-the-art improvements in segmentation metrics and real-world manipulation success, with significant gains over prior methods.

AFUN: Towards an Affordance Foundation Model for Functionality Understanding

Motivation and Problem Formulation

Achieving robust visual affordance understanding remains a primary challenge for embodied AI and robotics, as it sits at the intersection of perception and actionable manipulation across diverse, unstructured real-world settings. Prior methods lack scalability, either restricting affordance reasoning to localizing interaction regions without specifying executable motion, or outputting action trajectories with limited generalization. AFUN stands as a significant advancement, positing a unified affordance foundation model capable of open-world affordance understanding—predicting both where and how to interact with objects conditioned on RGB-D observations and natural language task prompts.

(AFUN overview in Figure 1 follows.)

Figure 1: Overview of the AFUN system, including large-scale data acquisition, joint segmentation and motion prediction, compositional generalization to open-world images, and direct deployment to real robot manipulation tasks.

Scalable Multi-Source Data Pipeline

A core contribution of AFUN is a highly scalable, standardized data curation pipeline extracting affordance supervision from heterogeneous sources: robot teleoperation logs, egocentric human videos, simulated tasks, and static scene scans. The pipeline addresses three critical modalities for each action interval: RGB-D observations, a language-annotated task prompt, a task-conditional segmentation mask of the interaction region, and a physically plausible, object-centric 3D post-contact trajectory.

The aggregation process begins with episode and interval segmentation, unifies data format and camera calibration, utilizes large vision-LLMs (Qwen3-VL) and SAM3 for manipulable-part localization and mask tracking, projects 2D mask tracks to 3D trajectories, and ultimately fits the resulting sparse tracks to smooth Bézier spline curves for parametric motion annotation. This cross-domain pipeline yields over 59k high-quality training samples after aggressive quality assurance.

Figure 2: The unified data pipeline integrates data from robots, human videos, simulation, and 3D scans, annotating each with task phrase, mask, and 3D motion trajectory.

This approach explicitly decouples hand/gripper kinematics from affordance-labeled object-centric post-contact dynamics, removing confounding pre-contact noise (see Figure 3 in the paper and Figure 4 for qualitative results).

Unified Model Architecture

AFUN comprises two tightly coupled modules: an instruction-conditional segmentation network, and a motion prediction module, both leveraging pretrained foundation backbones.

• MetaQuery-based VLM Conditioning: A frozen Qwen3-VL-8B VLM receives RGB input and instruction, yielding two sets of MetaQuery tokens—one for semantic segmentation and one for action motion reasoning.
• Task-Conditional Segmentation: The VLM’s segmentation tokens are mapped (via a small MLP) into SAM3’s language-conditioning space, which, combined with RGB, produces high-resolution affordance masks.
• 3D Point Feature Encoding: Depth is unprojected into point clouds, then encoded via a pretrained Sonata 3D feature extractor, providing dense geometric context.
• Trajectory Decoder: A transformer-based decoder, cross-attending to segment-level and geometric features, regresses control points for an anchored Bézier curve in 3D object space (see Figure 3).

  Figure 3: Architecture of AFUN: language and visual encoding, segmentation mask prediction, and 3D motion curve regression are tightly integrated.

This design allows joint mask and 3D motion inference in a single pass, maximizing reasoning transfer from large-scale vision-language pretraining.

Training Protocol

The optimization strategy proceeds in three stages:

1. MetaQuery–SAM3 Alignment: Initializing MetaQuery token sets and projection MLP by aligning the VLM and SAM3’s feature spaces with auxiliary text-image data, stabilizing subsequent training.
2. Affordance Segmentation Pre-training: Training only for mask prediction (motion head disabled) on four broad segmentation datasets (e.g., HOVA-500K, RAGNet, InstructPart, ReasonAFF) with SAM3’s detection/mask objectives.
3. Joint Segmentation–Motion Finetuning: Both heads co-trained on the curated multi-source dataset, with balanced loss terms for mask accuracy and curve regression (using sampled-point losses for Bézier curves, following Curve-GCN paradigms). All backbone parameters remain frozen to enable robust generalization.

Empirical Evaluation

Affordance Segmentation

AFUN exhibits state-of-the-art performance on eight test sets across four affordance segmentation benchmarks. Using generalized Intersection-over-Union (gIoU/cIoU), AFUN outperforms the strongest baseline (Affordance-R1) by over 23.9/26.3 points on mean gIoU/cIoU. It also achieves significant gains with fewer backbone parameters compared to larger LLM or VLM alternatives, supporting model and data efficiency claims.

Figure 4: AFUN produces highly accurate task-centric segmentation masks even for challenging and intent-dependent contact regions.

Contact Point Prediction

For contact-affordance tasks, AFUN leverages the pole of inaccessibility technique to designate canonical contact points within predicted masks. It achieves up to 61.3% higher point-in-affordance hit rates versus keypoint or mask-based prior works (A0, GLOVER++, VRB).

3D Motion Trajectory Prediction

On three diverse benchmarks (held-out AFUN, SceneFun3D, RoboMIND2), AFUN consistently surpasses recent state-of-the-art video, flow, or 3D point motion models in ADE, FDE, and contact-in-mask rates, even when competing baselines are granted AFUN’s mask predictions. On test splits with significant domain shift and unseen embodiment, AFUN maintains high kinematic plausibility for object-centric motion.

Figure 5: Qualitative motion results show AFUN’s ability to predict both the actionable region and smooth, physically valid motion curves, where prior methods often generate task-incoherent or erroneous predictions.

Generalization, Robustness, and Deployment

Ablations on LLM backbone, 3D encoder, and motion parameterization demonstrate the optimality of AFUN design choices. Qwen3-VL-8B, Sonata point features, and the distinct Bézier parameterization yield best-in-class segmentation and motion metrics.

Critically, AFUN can be deployed on real robotic manipulators (e.g., Franka arm) without embodiment-specific heuristics or finetuning. Mask and trajectory predictions are transformed into feasible grasp and execution plans (AnyGrasp pipeline), supporting robust manipulation over diverse categories from articulated microwaves to tool pickup.

Figure 6: Real-world Franka deployment: AFUN localizes actionable regions and synthesizes accurate post-contact trajectories required for successful manipulation.

Success rates for real-world manipulation tasks exceed 90% in contact-centric and articulation actions, confirming the model’s utility in physical robotics.

Limitations and Future Directions

While AFUN covers a broad distribution of tasks and objects, rare categories and novel articulation modes (e.g., spray bottles, sun visors) expose the limits of current dataset diversity and model transfer. Scaling the dataset further and refining generalization benchmarks remain essential open problems. Additionally, real-world deployment should always remain constrained by robust safety checks and supervised operation, as AFUN is a perception-action intermediary rather than a complete autonomous system.

Figure 7: Failure modes occur for categories with no close analogs in the training set, motivating further expansion of the dataset scope.

Conclusion

AFUN provides an integrated vision-language foundation model for functional affordance reasoning, fusing mask segmentation and 3D motion prediction in a single system. Its standardized data pipeline enables rigorous, scalable affordance supervision across diverse modalities, and its architecture achieves strong empirical and practical results on both simulated and real-robot benchmarks. As the field moves towards fully generalizable embodied AI, AFUN represents a tractable template for affordance-centric, open-world interaction grounded in actionable predictions.

---

Reference: "AFUN: Towards an Affordance Foundation Model for Functionality Understanding" (2606.02551)