WSAG: Weakly Supervised Affordance Grounding

• Weakly Supervised Affordance Grounding (WSAG) is the task of localizing action-enabling object regions using only weak labels such as image-level tags and sparse keypoints.
• It leverages methods like Class Activation Mapping, cross-view transfer, and contrastive learning to align affordance cues between exocentric and egocentric views for improved spatial precision.
• Recent models integrate self-supervised backbones, language-conditioned losses, and prototype clustering to generalize affordance detection across unseen object–action pairs.

Weakly Supervised Affordance Grounding (WSAG) is the problem of localizing object regions that enable, or afford, a specific action—such as “grasp,” “cut,” or “sit”—using supervision weaker than dense pixel-wise labels. Typical supervision includes only image-level action labels or sparse keypoints, commonly using collections of exocentric (third-person, interaction-in-progress) and egocentric (first-person, static object-centric) images. The central challenge is to robustly ground affordances under real-world conditions (occlusion, pose variation, ambiguous interaction cues) and to generalize to object–action pairs unseen in training, while minimizing manual annotation cost.

1. Problem Definition and Historical Foundations

WSAG formalizes the task: Given a set of images (often grouped as exocentric interaction frames and static egocentric views) and a set of affordance (action) labels $A = \{a_1,\dots,a_{N_c}\}$, learn a mapping from an input image $I$ and query $a$ to a spatial heatmap $H \in \mathbb{R}^{H\times W}$ that localizes the object region affording $a$, using only weak supervision. Early works (Srikantha et al., 2016, Sawatzky et al., 2017) targeted affordance segmentation with EM-based approaches leveraging sparse keypoints or image-level tags, employing DeepLab-style CNNs within iterative mask refinement schemes. These methods demonstrated that affordance segmentation is possible without dense labels, but suffered from limited generalization and spatial precision, especially under broad intra-class variation.

WSAG advances beyond classical affordance segmentation by operationalizing the Gibsonian notion of action possibility in a manner compatible with modern deep feature modeling, cross-domain transfer, and large video/image corpora.

2. Key Methodological Developments

Contemporary WSAG frameworks adopt several architectural and algorithmic strategies, often integrating cross-domain, prototype-based, or contrastive learning motifs:

• Class Activation Mapping (CAM): Shared classifiers and spatial maps derived from deep feature backbones highlight action-relevant image regions (Luo et al., 2022, Li et al., 2023). CAMs serve as the principal mechanism for converting classification supervision into spatial localization.
• Cross-View Transfer: Exocentric interaction images provide affordance cues (often occluded, cluttered, or biased by hand poses), while egocentric images enable inference under "object-at-rest" settings. Modern methods such as LOCATE (Li et al., 2023), Cross-View-AG (Luo et al., 2022), and selective contrastive approaches align object-part cues across these domains by mining invariances and learned correspondences.
• Prototypical and Pixel-Wise Contrast: Embedding-level objectives, including regional pooling, prototype clustering, and selective pixel/component alignment, encourage disambiguation of affordance-bearing parts versus background, and object-level versus part-level groundings (Moon et al., 11 Aug 2025).
• Contrastive, Part-Aware, and Language-Conditioned Losses: Recent systems include explicit contrastive learning over object-part relationships, leveraging language-derived interaction similarity maps via LLMs (Jang et al., 2024), and auxiliary regularizations aligning cross-modal (visual–textual) representations.

3. Representative Architectures and Learning Pipelines

A variety of pipeline designs instantiate these methodological elements:

• LOCATE Framework: Employs self-supervised ViT (DINO-ViT-S/16) as a backbone. Exocentric zones of interaction are detected via affordance-CAMs, clustered into prototype vectors (human, object-part, background), and the most relevant part-prototype is selected via a PartIoU-based metric with DINO self-attention serving as an object saliency prior. Egocentric affinity maps are guided by a cosine-embedding objective, directly encouraging regional similarity to exocentric affordance-part prototypes. Training alternates between exocentric and egocentric mini-batches, using only image-level affordance labels (Li et al., 2023).
• Cross-View Knowledge Transfer: Shared CNN backbones extract features from exocentric and egocentric images. Affordance Invariance Mining (AIM) reduces person-specific noise through low-rank dictionary factorization, before cross-view adaptive transfer and co-relation preserving losses enforce semantic alignment. Inference on egocentric targets utilizes CAM for spatial localization (Luo et al., 2022, Luo et al., 2022).
• Selective Contrastive Learning: Rather than rigidly enforcing per-part localization, these models adaptively balance prototypes at both object and part granularity, using CLIP-derived object affinity masks and pixel-level contrastive losses to suppress irrelevant regions and sharpen affinity for true affordance parts (Moon et al., 11 Aug 2025).
• Closed-Loop Transfer (LoopTrans): Introduces bidirectional knowledge flow: exocentric knowledge (CAM activations) is localized pixel-wise in egocentric images, and refined egocentric masks are distilled back to update exocentric activation heads through denoising distillation, enhancing robustness to domain gap and occlusion (Tang et al., 20 Oct 2025).
• Text-Conditioned and Part-Prior-Guided Approaches: Models such as INTRA (Jang et al., 2024) and WSAG-PLSP (Xu et al., 30 May 2025) integrate VLMs and open-vocabulary part-segmentation, generating affordance maps from arbitrary text queries (zero-shot), leveraging LLM-inferred similarity graphs and part-name mappings to improve cross-object, cross-affordance grounding.

4. Benchmark Datasets and Evaluation Protocols

The advancement of WSAG has been closely coupled to the availability of large-scale, fine-grained affordance datasets:

• AGD20K: 20K exocentric images, 3.7K egocentric images, 36 affordance categories spanning 50 object types, with dense heatmap labels in test splits. Supports “seen” and “unseen” object category generalization protocols (Luo et al., 2022).
• OPRA, EPIC-Kitchens: Large-scale video-action datasets used for early hotspot-based methods and current closed-loop approaches (Nagarajan et al., 2018).
• Auxiliary Datasets: IIT-AFF, CAD, UMD—used for cross-dataset transfer and zero-shot generalization studies (Jang et al., 2024, Xu et al., 30 May 2025).

Evaluation is typically via saliency-based metrics computed between predicted heatmaps and dense ground-truth: Kullback-Leibler divergence (KLD, ↓), Similarity (SIM, ↑), and Normalized Scanpath Saliency (NSS, ↑). Some works employ AUC-Judd and per-affordance, per-scale breakdowns for fine-grained comparative analysis.

5. Quantitative and Qualitative Performance

State-of-the-art WSAG models consistently surpass traditional saliency, strongly supervised, and weakly supervised object localization methods on affordance-heatmap benchmarks:

Model	KLD (↓)	SIM (↑)	NSS (↑)	Notes
LOCATE (Li et al., 2023)	1.405	0.372	1.157	AGD20K-unseen, DINO-VIT, PartIoU selection
Cross-View-AG+ (Luo et al., 2022)	1.765	0.279	0.882	Prior best; ResNet-50, AIM+CFT+ACP
LoopTrans (Tang et al., 20 Oct 2025)	1.247	0.403	1.315	Closed-loop domain transfer, denoising
SelectiveCL (Moon et al., 11 Aug 2025)	1.243	0.405	1.368	Prototypical/pixel contrast, CLIP guidance
PLSP (full) (Xu et al., 30 May 2025)	1.153	0.437	1.418	Part-level pseudo-labels from VLPart+SAM
INTRA (Jang et al., 2024)	-	-	-	Exocentric-only, zero-shot strong generalizability

Ablation studies consistently show that transferring regional part information and using refined part-prototype or contrastive objectives yield the largest performance gains, with class-activation-based baselines underfitting to affordance part details.

Qualitatively, state-of-the-art systems can differentiate fine-grained affordance cues (blade vs. handle on a knife), resolve affordance disambiguations in multi-label or occluded settings, and maintain localization precision under domain shifts (pixel art, novel objects, or zero-shot affordances).

6. Limitations and Open Problems

Despite substantial improvements, active research addresses persistent challenges:

• Reliance on Pretrained Backbones: Many WSAG pipelines depend on frozen, self-supervised feature backbones (e.g., DINO ViT), whose part awareness may be incomplete for unfamiliar or underrepresented object categories (Li et al., 2023).
• Noisy or Biased Interaction Cues: Both exocentric occlusion and variability in human–object contact introduce noise and “interaction bias,” handled via invariance mining, co-relation preservation, or cluster-based part filtering, but remain sources of error (Luo et al., 2022, Tang et al., 20 Oct 2025).
• Multi-Affordance and Multi-Object Scenes: Most models ground a single affordance per image; extension to multi-label images or scenes with overlapping affordance regions is an open direction (Li et al., 2023, Jang et al., 2024).
• Part Discovery and Pseudo-Label Quality: Heuristics for selecting the “object part” prototype or for generating part-level pseudo-labels impose threshold tuning or require part-name mappings, a source of system brittleness (Li et al., 2023, Xu et al., 30 May 2025).
• Generalization/Scalability: While foundation model integration greatly expands coverage (VLMs, SAM, VLPart), integrating such models with fine spatial grounding remains an active research topic (Xu et al., 30 May 2025, Jang et al., 2024).

7. Extensions, Applications, and Future Directions

Emerging WSAG research explores several extensions:

• Video-Based Transfer and Temporal Consistency: Incorporating temporal cues and consistency constraints in exocentric–egocentric alignment, relevant for embodied navigation or robotic manipulation (Nagarajan et al., 2018, Tang et al., 20 Oct 2025).
• Language-Conditioned Flexibility and Zero-Shot Generalization: Text-conditioned affordance grounding via VLMs unlocks zero-shot scene understanding, compositional affordances, and natural language interfaces (Jang et al., 2024, Xu et al., 30 May 2025).
• Robotic Manipulation and Grasp Affordance: WSAG is directly applied to attribute-based object grounding and robot grasping, demonstrating high grasp-success rates with only weak spatial annotations (Yu et al., 9 Sep 2025).
• 3D and Multimodal Integration: Ongoing work aims to unify 3D part representations, multimodal interaction cues (audio, haptics), and prompt-based region refinement via large models (e.g., SAM) for robust real-world deployment.

In summary, Weakly Supervised Affordance Grounding has advanced from EM-based keypoint learning to sophisticated, cross-modal, and contrastive pipelines that exploit exocentric–egocentric transfer, self-supervised features, and vision–LLMs. The field continues to expand its empirical coverage, improve spatial and semantic grounding, and generalize across objects, actions, and data modalities (Li et al., 2023, Luo et al., 2022, Moon et al., 11 Aug 2025, Jang et al., 2024, Xu et al., 30 May 2025, Tang et al., 20 Oct 2025).