CAST: Cross-modal Affordance Segmentation Transformer

• The paper introduces a novel transformer model that fuses rich 2D semantic cues with 3D geometric representations to overcome point cloud ambiguities.
• It employs a Cross-modal Affinity Transfer (CMAT) pre-training strategy that aligns patch-level 2D and 3D features for enhanced segmentation accuracy.
• The approach demonstrates significant improvements in functional part segmentation for applications in robotics, embodied AI, and AR.

The Cross-modal Affordance Segmentation Transformer (CAST) is a transformer-based architecture designed for fine-grained 3D affordance segmentation with multi-modal prompt awareness. CAST addresses the limitations of conventional 3D methods—which often struggle with the sparsity, geometric ambiguity, and lack of semantic boundaries inherent to point cloud data—by aligning and fusing informative semantic knowledge from large-scale 2D vision foundation models with 3D geometric representations. This enables precise functional part parsing of 3D objects conditioned on textual or visual prompts, such as explicit part names or paired exemplars, and establishes new state-of-the-art segmentation results for robotic manipulation, embodied AI, and AR contexts (Huang et al., 9 Oct 2025).

1. Motivation and Problem Formulation

3D affordance segmentation aims to partition an object’s point cloud into functionally meaningful regions that reflect potential interactions (e.g., segmenting a chair’s seat or a mug’s handle). Historically, 3D segmentation pipelines—often using point cloud encoders like PointNet or transformer variants—operate only on geometric structure, which can result in fuzzy functional boundaries and suboptimal generalization to novel parts, viewpoints, or occlusions.

CAST is motivated by two insights:

• Mature 2D Vision Foundation Models (VFMs, such as DINOv3) learn rich dense semantics, grouping image pixels according to both category and part-level cues.
• These semantic cues can be “lifted” from multi-view RGB renderings and transferred to the 3D domain to compensate for the weak semantic guidance in point-based models.

This leads to the design goal of a prompt-aware segmentation system that unifies multi-modal (language, vision) guidance with semantically structured 3D representation learning.

2. Cross-modal Affinity Transfer (CMAT) Pre-training

To enable rich, semantically grounded 3D representations, the CAST framework employs Cross-modal Affinity Transfer (CMAT) as a self-supervised pre-training stage:

• Multi-view Semantic Lifting: For each 3D object, a pre-trained 2D VFM (e.g., DINOv3) extracts dense semantic feature maps from a set of multi-view RGB renderings. Each 3D point in the cloud is backprojected to all rendered views; its corresponding 2D features are aggregated (e.g., pooled or averaged), yielding a lifted 2D semantic embedding for each point.
• Patch-level Representation: The point cloud is divided into patches, and for each patch $\mathcal{P}_j$, a patch-level 2D semantic feature $\bar{\mathbf{f}}_j^{2D}$ is computed by aggregating its points’ lifted embeddings:

$$\bar{\mathbf{f}}_j^{2D} = \frac{1}{|\mathcal{P}_j|} \sum_{\mathbf{p}_i \in \mathcal{P}_j} \mathbf{f}_i^{2D}$$

Analogous patch-level features $\bar{\mathbf{f}}_j^{3D}$ are computed from the 3D encoder.

• Cross-modal Affinity Matching: For both 2D and 3D features, affinity matrices are constructed:

$$\mathbf{A}_{jk}^{2D} = \frac{ \bar{\mathbf{f}}_j^{2D} \cdot \bar{\mathbf{f}}_k^{2D} }{ \| \bar{\mathbf{f}}_j^{2D} \| \| \bar{\mathbf{f}}_k^{2D} \| }$$

$$\mathbf{A}_{jk}^{3D} = \frac{ \bar{\mathbf{f}}_j^{3D} \cdot \bar{\mathbf{f}}_k^{3D} }{ \| \bar{\mathbf{f}}_j^{3D} \| \| \bar{\mathbf{f}}_k^{3D} \| }$$

The CMAT loss compels the 3D encoder to organize patch representations to reflect the semantic relational structure present in the 2D model by minimizing the affinity discrepancy:

$$\mathcal{L}_{\text{aff}} = \frac{1}{M^2} \sum_{j,k} (\mathbf{A}_{jk}^{3D} - \mathbf{A}_{jk}^{2D})^2$$

where $M$ is the number of patches.

• Joint Pre-training Objective: Three losses are combined:
  • A geometric reconstruction loss $\mathcal{L}_{\text{rec}}$ (predicting the centers of masked patches, as in masked autoencoding),
  • The affinity alignment loss $\mathcal{L}_{\text{aff}}$ as above,
  • A feature diversity loss $\mathcal{L}_{\text{div}}$ (KoLeo regularization) to prevent representational collapse:

  $$\mathcal{L}_{\text{div}} = -\frac{1}{M} \sum_{j} \log(d_j)$$

  where $d_j$ is the nearest-neighbor distance for patch $j$.

The overall pre-training loss:

$$\mathcal{L}_{\text{pretrain}} = \lambda_{\text{rec}} \mathcal{L}_{\text{rec}} + \lambda_{\text{aff}} \mathcal{L}_{\text{aff}} + \lambda_{\text{div}} \mathcal{L}_{\text{div}}$$

This results in a 3D transformer backbone with semantically structured latent space, suitable for functional part segmentation.

3. Prompt-driven Segmentation with Transformer Fusion

The task-specific CAST module fuses prompt encodings (text or vision) with the CMAT-pretrained geometric patch features:

• Prompt Embedding: A prompt is provided as either a language phrase (e.g., “handle”) via a text encoder (such as RoBERTa), or a visual exemplar (e.g., a cropped image of a functional part) via a VFM (e.g., DINOv3). These are projected into the same dimension as patch tokens and prepended to the sequence as prompt tokens. Learnable modality embeddings are used to mark each feature’s source.
• Co-attentional Transformer Blocks: Patch tokens (from the pre-trained 3D encoder) and prompt tokens are concatenated. A stack of co-attentional transformer blocks enables bidirectional interactions between the prompt and geometry representations, allowing prompt-driven reasoning and part localization.
• Segmentation Head: After fusion, an upsampling and MLP classification head produces dense segmentation maps, yielding a fine-grained, prompt-aware affordance mask over the 3D point cloud.

4. Transfer of 2D Semantic Knowledge to 3D Functional Boundaries

The core advantage of CAST is its explicit grounding of 3D features in the rich semantics of 2D VFMs:

• Point clouds are inherently sparse, noisy, and often ambiguous—pure geometry may not delineate, for example, the boundary between a mug’s handle and body.
• By aligning 3D representations with 2D features known to group semantically (via DINOv3), CAST introduces structured relationships and functional part awareness absent from purely geometric pre-training.
• This produces segmentations that track functional boundaries (e.g., handle, seat, button) despite geometric ambiguity or incomplete data.

A plausible implication is that further improvements may be possible by incorporating even broader 2D semantic sources (e.g., CLIP, SAM), or by developing more advanced multi-modal alignment strategies.

5. Experimental Performance and Ablations

Empirical validation on PIAD, PIADv2, and LASO datasets demonstrates the impact of CAST’s design choices:

Dataset	Prompt Type	Metric	SOTA (prev)	CAST
PIAD	Visual	SIM	0.590	0.725
PIADv2 Seen	Visual	aIoU	37.03%	44.88%
PIADv2 Unseen	Visual	aIoU	24.74%	30.07%
LASO Seen	Language	aIoU	17.7%	21.7%
LASO Unseen	Language	aIoU	12.2%	17.5%

• CMAT pretraining is essential: using only geometric reconstruction yields much weaker results, with affinity alignment (+5.2% aIoU gain) and further diversity regularization (+1.4% aIoU) cumulatively boosting segment quality.
• Qualitative results illustrate that CAST can accurately localize even small, functionally distinct regions, and generalizes robustly to unseen categories and ambiguous instances.

6. Applications and Implications

CAST is directly applicable in domains requiring prompt-aware functional perception:

• Robotics: Enables robots to identify, segment, and interact with actionable object parts, improving grasping, manipulation, or tool-use by providing explicit segmentation of interaction regions.
• Embodied AI: Facilitates agents’ ability to anchor functional concepts to geometry for contextually intelligent environmental interaction.
• Augmented/Virtual Reality: Supports real-time overlay of instructional or interactive elements on dynamically segmented object parts.
• The semantic transfer strategy underlying CAST is broadly relevant for robust 3D understanding, and suggests follow-on research in “unified” multi-modal segmentation, data-efficient 3D labeling, and more flexible prompt types (visual, language, or combined).

7. Broader Impact and Future Directions

CAST’s robust prompt-driven approach and cross-modal grounding form a foundation for advancing 3D functional scene understanding. The explicit unification of foundational 2D semantic cues and geometric reasoning provides a path for future work on:

• Generalized 3D task-driven perception,
• Open-vocabulary affordance segmentation,
• Improved transferability of 2D advances into 3D embodied contexts.

Ongoing challenges include handling extreme sparsity, semantic ambiguity in prompts, and scaling to full-scene or multi-object environments. However, the demonstrated pipeline establishes new baselines for prompt-aware 3D affordance segmentation and cross-modal learning in embodied AI (Huang et al., 9 Oct 2025).