O³Afford: One-Shot 3D Affordance

• O³Afford is a framework that grounds affordances between object pairs by generating dense 3D maps with one-shot learning.
• It employs semantic point clouds and joint-attention transformers to fuse geometric and visual cues for tasks like pouring, cutting, and hanging.
• Integration with LLMs enables constraint-based optimization, enhancing robotic planning and performance in both simulated and real-world settings.

O$^3$Afford—One-Shot 3D Object-to-Object Affordance Grounding—constitutes a generalizable framework for grounding physical affordances between pairs of objects in real-world robotic manipulation environments. Unlike traditional affordance methods that focus on single-object functionalities, O$^3$Afford grounds functional relations directly in 3D space, producing dense affordance maps on both source and target objects. These affordance predictions are integrated with LLMs for high-level reasoning and task-constrained manipulation, enabling robots to execute complex interactive tasks by leveraging both geometric and semantic cues.

1. Object-to-Object Affordance Grounding

O$^3$Afford targets the problem of predicting affordance maps for interacting object pairs in manipulation tasks, such as pouring (teapot to bowl), cutting (knife to apple), or hanging (mug to hook). The central task is to localize regions on both objects suitable for interaction, conditioned on the intended action category. This two-object formulation is operationally distinct from single-object approaches, which only highlight where an individual object could be grasped, pressed, or used, but do not identify complements for a given action.

This framework fundamentally links perception and action by requiring the system to encode spatial and geometric compatibility between two objects. Resulting affordance maps inform robotic planning by pinpointing optimal contact regions and interaction geometry.

2. Semantic-Geometric 3D Representation

The perception backbone in O$^3$Afford fuses semantics from foundation visual models with full 3D geometry:

• Semantic Point Clouds: Multi-view RGB-D observations of both source and target objects are processed through a vision foundation model (DINOv2), and semantic features are projected onto each object's 3D point cloud. Each point is embedded with both spatial coordinates $(x,y,z)$ and an $n$-dimensional semantic descriptor, yielding a unified "semantic point cloud."
• Feature Projection and Fusion: For every 3D point, RGB-D alignment projects features from multiple views. Feature fusion (e.g., via max- or average-pooling) ensures robustness to occlusions and viewpoint changes.
• Point Cloud Tokenization: Objects are further tokenized into point groups using farthest point sampling and $k$-nearest neighbors, which serve as input tokens to subsequent modules.

This representation encodes both geometry and object category semantics, supporting transfer to novel objects without retraining.

3. Joint-Attention Transformer Architecture

The object pair representations are processed in parallel streams and merged using a cross-object attention mechanism:

• Patch-based PointNet Encoder: Each object's point cloud tokens are embedded via a hierarchical PointNet.
• Joint-Attention Transformer Decoder: Cross-attention layers enable information flow between the source and target objects. For source tokens $Z^{(\text{src})}$ and target tokens $Z^{(\text{tgt})}$, joint attention is formalized by:

$$A^{(\text{src})} = \mathrm{CrossAttention}(Z^{(\text{src})}, Z^{(\text{tgt})}, Z^{(\text{tgt})})$$

$$A^{(\text{tgt})} = \mathrm{CrossAttention}(Z^{(\text{tgt})}, Z^{(\text{src})}, Z^{(\text{src})})$$

This dynamic interaction fuses semantic and geometric features from both objects, supporting prediction of affordance regions conditioned on specific object-to-object relationships.

• Output Affordance Maps: The transformer decoder predicts dense, per-point affordance probabilities $A^{(\text{src})}, A^{(\text{tgt})} \in [0,1]^N$ for each object’s point cloud.

4. Training Paradigm and Optimization

O$^3$Afford is trained under a one-shot regime: for each affordance (interaction) category, only a single example is provided during training. The binary cross-entropy loss for predicting each point’s affordance score is:

$$L_\mathrm{BCE} = -\frac{1}{N} \sum_{i=1}^{N} \left( y_i \log \hat{y}_i + (1-y_i)\log(1-\hat{y}_i) \right),$$

where $y_i$ is the ground-truth affordance label and $\hat{y}_i$ the predicted probability for point $i$.

The architecture’s patch-based encoding and cross-attention enhance robustness to occlusion and synthetic data sparsity—a critical property for real and simulated deployments.

5. LLM-Coupled Constraint-Based Manipulation

Affordance maps generated by O$^3$Afford are integrated with LLMs to bridge perception and robotic action:

• Constraint Function Generation: The LLM receives high-level task descriptors (e.g., “pouring,” “cutting”) and outputs Python-formulated constraint functions $S_i$, encoding geometric or physical manipulation requirements (e.g., alignment, orientation, collision avoidance).
• Optimization Pipeline: The robot seeks a transformation $T\in SE(3)$ of the source object that minimizes a weighted sum of LLM-defined constraints on the affordance maps:

$$\min_{T \in SE(3)} \sum_i \lambda_i S_i(P^{(\text{src})}, A^{(\text{src})}, P^{(\text{tgt})}, A^{(\text{tgt})}, T)$$

with $P^{(\cdot)}$ point clouds and $A^{(\cdot)}$ affordance probabilities.

• Autonomous Manipulation: This approach grounds high-level reasoning in explicit, spatially localized, perception-driven 3D constraints, supporting complex and compositional robotic tasks.

6. Experimental Evaluation

O$^3$Afford demonstrated strong performance in both simulated (SAPIEN) and real robotic (Franka Research 3 + Orbbec Femto Bolt) environments:

Metric	O$^3$Afford	Best Baseline
IOU	Higher	Lower
AUC	Higher	Lower
MAE	Lower	Higher

Benchmarks included object-pair tasks such as pouring, pressing, inserting, hanging, and cutting, with metrics including Intersection-over-Union (IOU), Mean Absolute Error (MAE), and AUC. In one-shot learning settings, O$^3$Afford achieved superior generalization and robustness, retaining predictive accuracy under up to 50% point cloud occlusion. Experimental results indicate improved task success rates in manipulation—in particular, tasks requiring precise, affordance-driven placement such as hanging or cutting showed the largest gains over baseline approaches.

7. Implications, Limitations, and Extensions

O$^3$Afford enables generalizable affordance reasoning in environments with limited annotated data, reducing the need for large-scale object-specific supervision. The approach scales to novel object categories and tasks, supports efficient transfer in unstructured domains, and bridges perceptual representations with high-level task reasoning via LLMs.

Current limitations include potential vulnerability to severe self-occlusion and challenging scenarios where sensor noise inhibits accurate 3D construction. Future extensions may incorporate language instructions more directly into the affordance prediction pipeline, and improvements in robustness to partial observations. The integration of semantic and geometric reasoning establishes a foundation for advanced mid-level robotic manipulation, automated tool use, and human–robot interaction in real-world domains.

---

O$^3$Afford thus advances the state-of-the-art in affordance grounding by combining semantic vision features, geometric comprehension, and language-based constraint generation within an end-to-end one-shot framework for object-to-object manipulation grounding (Tian et al., 7 Sep 2025).