AffordDex: Unified Finger-Specific Affordance Framework

• AffordDex is a unified framework for fine-grained, finger-specific affordance estimation that integrates vision transformers, semantic diffusion models, and geometry-driven post-processing.
• It employs a dual-branch architecture to generate both fine-grained contact maps and coarse hand-object interaction embeddings for robust grasp synthesis.
• The system achieves platform-agnostic dexterous manipulation with high success rates by leveraging weak supervision and kinematics-aware optimization.

AffordDex is a unified framework for fine-grained, finger-specific affordance estimation and dexterous grasp synthesis, leveraging vision transformer features, semantic priors from generative diffusion models, and geometry-aware post-processing for tool-oriented robotic manipulation. Developed to enable reliable functional grasping of diverse objects by multi-fingered hands without the need for per-platform grasp data collection, AffordDex represents a convergence of weakly supervised affordance extraction, semantic grounding, and kinematics-aware retargeting (Yang et al., 2024, Han et al., 13 Jan 2026).

1. Granularity-Aware Affordance Representation

AffordDex organizes affordance prediction into two hierarchical levels:

• Fine-grained affordance maps ($A_f$) identify precise object regions for fingertip contact required by functional tasks. These are derived from both exocentric (human-in-the-loop) and egocentric (object-only) images by extracting deep feature tensors using frozen Vision Transformer (ViT) backbones (e.g., DINO-ViT-S). Hand keypoints are detected (e.g., via MediaPipe or RTMPose), and the functional fingertip location $(x_0, y_0)$ is localized. Binary circular masks $M_f$ localize contact regions, with mask pooling yielding per-task fine-grained affordance "prototypes." Heatmaps $P_e^{(c)}$ in the ego branch capture predicted fine-grained affordances for each functional task category.
• Coarse-grained affordance maps ($A_c$) represent the broader hand-object interaction region, corresponding to the typical region contacted by the palm and fingers. LOCATE-style token clustering is used for prototype extraction in exocentric images, while egocentric features are pooled and mapped to coarse grasp class embeddings.

Supervision is transferred from exocentric to egocentric domains by minimizing a cosine margin loss between feature embeddings, with a separate concentration loss to enforce spatial peaking of fine-grained heatmaps. Final coarse gesture classes are supervised by cross-entropy on both intermediate and final classifier outputs.

2. Network Architecture and Semantic Feature Integration

AffordDex employs a dual-branch architecture, both sharing a frozen DINO-ViT-S backbone ($\phi$):

• The fine-grained branch processes part-level features with convolutional heads and multi-layer perceptrons, guided by exocentric-derived affordance prototypes and mask-pooling.
• The coarse branch encodes hand-object interaction regions, associates them with one of 14 manually defined grasp gesture classes, and predicts corresponding nominal joint configurations for the robot hand.

Alternatively, in recent FSAG variants (Han et al., 13 Jan 2026), frozen Stable Diffusion U-Nets provide multi-scale, multi-timestep "hyperfeatures" $A_g$ that encode semantic part and function priors. Feature aggregation weights $w_{\ell, t}$ and bottleneck projections $b_\ell$ fuse visual and text-prompt-based semantics, decoded into finger-specific 2D affordance fields.

3. Geometry Alignment and Kinematic Retargeting

AffordDex integrates 2D affordance predictions with scene geometry:

• 2D–3D fusion: Predicted 2D affordance maps $\hat H_k(u)$ are projected onto partial point clouds via camera geometry, producing 3D candidate contact points $(x_0, y_0)$0 and locally estimated surface normals $(x_0, y_0)$1.
• Approach curves: Each finger’s desired path is parameterized as $(x_0, y_0)$2, with waypoints for approach, contact, and stable hold.
• Kinematics-aware optimization: For each finger, a damped Gauss–Newton quadratic program finds joint increments, minimizing weighted residuals between fingertip forward kinematics $(x_0, y_0)$3 and affordance targets $(x_0, y_0)$4, subject to joint, coupling, and collision constraints:

$(x_0, y_0)$5

$(x_0, y_0)$6

$(x_0, y_0)$7

This approach supports kinematic diversity and underactuation, enabling cross-platform transfer without retraining.

4. Weak Supervision and Dataset Construction

AffordDex emphasizes data efficiency through weak supervision:

• Exocentric–egocentric supervision transfer: Image-level task labels from exocentric images supervise feature extraction in egocentric object-only views, eliminating dense pixel-wise annotation needs.
• FAH dataset (Yang et al., 2024): 6,000 images across 18 commonly used tools and 6 functional tasks encompass 14 coarse grasp classes. Three annotators provide averaged test masks for evaluation.
• Video demonstration alignment (Han et al., 13 Jan 2026): Representative keyframes ("object-only," "stable grasp") are selected for each sequence, supporting robust affordance extraction from minimal human data.

5. Quantitative Evaluation and Empirical Performance

AffordDex has demonstrated robust performance across synthetic and real-world conditions:

Model / Method	KLD↓	SIM↑	NSS↑	Grasp AP (%)	Grasp Success (bottle S/U)
Cross-view-AG	1.682	0.271	1.149	–	–
LOCATE	1.541	0.330	1.158	–	–
AffordDex (GAAF-Dex)	1.459	0.327	1.242	74.65	–
Ours (DINO)	3.301	0.473	5.016	–	–
Ours (SD)	2.491	0.551	5.518	–	100/85

AffordDex reduces KLD relative to LOCATE by 5.3%, increases NSS by 7.3%, and achieves an average precision (AP) of 74.65% across tool-task grasp prediction. In FSAG, using diffusion hyperfeatures further reduces KLD by ~25% over DINO and ~63% over CLIP, with corresponding improvements in SIM and NSS. Real-robot experiments (UR5 + Inspire 6-DOF hand) show high success rates, e.g., 100% ("Hold Kettle"), with cross-embodiment transfer maintaining success when changing from 12+5 to 16+5 DOF hands.

6. Limitations and Prospects

Known limitations include:

• Failure on slender objects: The open-loop hold phase can induce slippage for elongated geometries.
• Depth sensing noise: Failures arise for highly reflective or transparent objects, disrupting normal estimation and collision reasoning.
• Semantic ambiguity: Ambiguous or non-specific text prompts for diffusion models may mislocalize affordances.

Future extensions suggest the incorporation of closed-loop tactile/force feedback (impedance control around finger-specific waypoints), dynamic in-hand manipulation (affordance updates during finger gaiting), and multi-view geometry fusion. Leveraging video diffusion models for spatio-temporal affordance estimation is proposed as a further direction.

7. Significance and Relation to Prior Work

AffordDex bridges the gap between vision-based affordance understanding and actionable, platform-agnostic dexterous manipulation. Its unification of fine-grained functional localization, semantic priors, weakly supervised transfer, and geometry/kinematic optimization contrasts with prior work requiring fully supervised grasp datasets or hardware-specific training. Its results on standardized metrics (KLD, SIM, NSS), fine-grained grasp grounding, and real-world dexterous manipulation validate its advantages for tool use and generalizable grasping (Yang et al., 2024, Han et al., 13 Jan 2026).