Hand Identity Annotation Techniques

• Hand identity annotation is the process of assigning consistent labels (e.g., left/right, subject IDs) to hands in digital data using both manual and automated protocols.
• It employs diverse methods such as color-glove segmentation, deep learning detectors like YOLOv10, and multi-view 3D fusion to ensure accurate labeling.
• The resulting annotations are crucial for applications in biometric authentication, egocentric analysis, surgical guidance, and human-object interaction recognition.

Hand identity annotation refers to the process of assigning consistent, unambiguous identity labels to hands present in digital data—whether as pixels, bounding boxes, joint/keypoint sets, mesh instances, or higher-order descriptors. The scope ranges from coarse “left”/“right” classification to fine-grained user- or subject-level distinction, and the annotated identities may refer to the anatomical side, the role within an interaction, or the unique person associated with the hand. This process is foundational for biometric authentication, egocentric action analysis, human-object interaction (HOI) recognition, medical workflow analysis, and advanced multi-modal computer vision. Diverse methods, spanning manual protocols, color-coded segmentation, multi-view geometric correspondence, deep learning frameworks, and vision-LLMs, have been developed to address the different requirements and challenges across application domains.

1. Annotation Protocols and Labeling Schemes

Hand identity annotation is typically operationalized at three primary semantic levels:

1. Anatomical Side (Laterality) Annotation:
   • Each detected hand is labeled as “left” or “right,” either absolutely (relative to world or camera) or per individual. Automated laterality annotation in video can be performed with multi-task object detectors. For example, in surgical video, the YOLOv10-based framework attaches dedicated laterality classification heads predicting (left, right) probabilities for each hand bounding box, trained on frame-level COCO-style ground-truth with manually assigned side labels (Sun et al., 21 Feb 2026).
   • In RGB-D segmentation, color-coded gloves enable per-pixel left/right hand labeling, with thresholding in normalized HSV space (distinct color channels for left/right, e.g., orange and green gloves) (Bojja et al., 2017).
2. Subject or User Identity Annotation:
   • In egocentric or biometric data, each sample sequence or image is associated with one subject ID, often inherited from acquisition metadata (one camera per subject/session) and stored as a unique integer. The ARCTIC/H2O datasets for 3D hand-pose user identification employ this protocol, affixing ground-truth tuples (object, HOI verb, subject ID) to each temporally segmented clip, with tracking validity manually audited before ID assignment (Hamza et al., 20 Sep 2025).
   • In forensic or biometric settings, as in CLIP-HandID, every hand image is tagged with an index in $\{1,\dots,C\}$ that serves solely as a class label for model training and evaluation; no semantic names or textual descriptors are included (Baisa et al., 14 Jun 2025).
3. Spatio-Temporal Instance Association:
   • In multi-person, multi-hand scenes, robust hand identity annotation requires temporally consistent association of each hand to a unique person. TouchMap-OR achieves this by fusing metrically aligned MANO hand meshes across views, clustering palms, and associating them via minimal 3D joint distances to tracked bodies. Global assignment frameworks (e.g., Hungarian matching, greedy but temporally gated mapping) prevent identity switches over long sequences (Ktistakis et al., 17 May 2026).
   • Annotation completeness, precision, and consistency are monitored, culling invalid frames and ensuring per-frame or per-clip correctness through manual or semi-automated filtering.

2. Automated and Semi-Automated Annotation Pipelines

Contemporary hand identity annotation pipelines can be categorized by data modality and the level of supervision/automation:

• Color-Glove RGB-D Annotation (HandSeg):
  • Users wear two highly saturated, distinct color gloves. RGB and depth channels are factory-calibrated and aligned (e.g., Intel RealSense SR300 hardware). Initial hand masks are generated via HSV thresholding per glove, then refined using per-image GrabCut and a linear SVM exploiting pixelwise color and spatial features. These refined masks are projected onto the depth field, generating per-pixel labels (background, left hand, right hand) for hundreds of thousands of frames without manual editing. Erroneous frames (~10%) are removed after visualization (Bojja et al., 2017).
• Multi-Task Deep Detection for Laterality and Localization (YOLOv10 Surgical):
  • Anchor-based detectors produce joint hand localization and laterality labels per bounding box. Networks use a multi-task loss combining localization/objectness and cross-entropy for laterality, trained on large-scale manually annotated datasets with COCO JSON bounding box and side labels. Extensive data augmentation (flips, rotation, jitter, mosaic) enhances network robustness (Sun et al., 21 Feb 2026).
• Vision-Language and Pseudo-Token Methods (CLIP-HandID):
  • Identity indices are encoded via learnable continuous “pseudo-tokens” inserted into text prompts passed to a fixed CLIP text encoder. These are learned via a textual inversion module mapping image embeddings to pseudo-token slots, with joint supervised contrastive and classification losses aligning image and token representations (Baisa et al., 14 Jun 2025).
• Multi-View 3D Fusion for Spatio-Temporal Identity (TouchMap-OR):
  • Articulated hand meshes (MANO) are fitted per camera-view, metrically aligned with depth maps, and merged via spatio-temporal clustering. Hands are assigned to tracked persons using minimum 3D distances to joints, corrected over time with identity voting. Semantic 3D room models allow trajectories to be annotated with specific contact events and surface identities (Ktistakis et al., 17 May 2026).
• Direct Metadata Inheritance (Egocentric Gesture Datasets):
  • In egocentric video, each clip inherits the camera wearer’s subject ID from recording metadata; no further per-hand labeling is necessary provided each sequence records a unique operator (Tsutsui et al., 2020, Hamza et al., 20 Sep 2025).

3. Feature Spaces for Hand Identity Recognition

Annotation alone is insufficient for robust downstream hand identity recognition; appropriate feature engineering or representation learning is required.

• Handcrafted Spatio-Kinematic Descriptors:
  • The I2S framework for 3D hand-pose-based person ID extracts spatial (joint locations, distances), orientation (joint angles, palm normals), kinematic (velocity/acceleration time-series), frequency-domain (DFT/PSD), and inter-hand (spatial envelope, IHSE) features. These are aggregated over time using central tendency or range-sensitive pooling operators (e.g., Dispersion-Aware Central Tendency). Discriminative feature fusion (SOKI: spatial, orientation, kinematic, IHSE) achieves F1=99.56% for raw user ID (Hamza et al., 20 Sep 2025).
• Visual Cues and Representations:
  • In egocentric recognition, distinctiveness of 2D silhouette, 3D geometry, skin texture, and color are evaluated in isolation and combination by ablated CNNs, revealing that combined cues (3D geometry plus color/texture) offer the highest ID accuracy (up to 19.5% on EgoGesture data), while background masking is critical to prevent trivial scene overfitting (Tsutsui et al., 2020).
• CNN, Vision Transformer, and Metric Learning Feature Spaces:
  • Hand vein biometrics use CNN (ResNet, DenseNet), transformer (ViT, VPCFormer), and hybrid Siamese/triplet networks for extracting high-dimensional vein pattern embeddings, combined with center or triplet losses for identity discrimination. ROI alignment, multi-session consistency, and cross-sensor calibration are essential to minimize label drift and maximize reproducibility (Hemis et al., 2024).

4. Evaluation Metrics and Quantitative Results

Hand identity annotation quality impacts the reliability of downstream evaluation metrics. Established identity recognition tasks report:

• Classification Metrics:
  • F1-score, precision, recall, mean Average Precision (mAP), and cumulative match curve (CMC; Rank-1.
  • For user identification from hand pose (I2S), spatial descriptor achieves mean F1=99.64%, with SOKI fusion yielding pipeline F1=97.52% (Hamza et al., 20 Sep 2025).
  • In CLIP-HandID, fine-tuned Rank-1 accuracy reaches 94.9%, mAP=95.9% for right dorsal hand images (Baisa et al., 14 Jun 2025).
  • Egocentric hand ID achieves 19.5% subject classification accuracy on masked, multi-cue RGB-D input (random = 2%), underlining the subtlety of hand trait identification (Tsutsui et al., 2020).
• Verification and Biometric Metrics:
  • Equal Error Rate (EER) is standard in hand vein biometrics, e.g., EER of 0.01% for ResNet50+CBAM on MMCBNU_6000, 0.66% for Siamese CNN on PolyU palm vein (Hemis et al., 2024).
  • HAN-Surface tracking (TouchMap-OR) achieves identity attribution accuracy of 0.96 for real clinical sequences (Ktistakis et al., 17 May 2026).
• Annotation Completeness:
  • HandSeg annotates nearly 100% of valid pixels, discarding only ~10% of frames for gross failure (Bojja et al., 2017).
  • Real-time hand laterality/disambiguation in YOLOv10 achieves 67%-71% laterality accuracy per detected hand at 38 FPS (Sun et al., 21 Feb 2026).

5. Best Practices, Limitations, and Domain-Specific Considerations

• Best Practices:
  • For segmentation, employ quasi-invariant cues (e.g., saturated glove colors) and robust color+depth fusion with multi-stage refinement (Bojja et al., 2017).
  • Always mask out backgrounds and enforce consistent ROI extraction, especially in egocentric and biometric tasks (Tsutsui et al., 2020, Hemis et al., 2024).
  • Use multi-session acquisition with time gaps for stability in biometric datasets (Hemis et al., 2024).
  • For cross-domain or multi-person settings, integrate geometric correspondence (multi-view fusion, clustering, global assignment) to ensure identity consistency, minimizing switches (Ktistakis et al., 17 May 2026).
• Challenges and Limitations:
  • Fine-grained subject ID is sensitive to occlusions, cross-hand ambiguity, and background clutter; accuracy degrades rapidly outside the acquisition domain or with small per-class sample sizes (Sun et al., 21 Feb 2026, Tsutsui et al., 2020).
  • RGB/HSV mask thresholds are brittle to lighting changes; glove-based annotation fails with saturated color drift (Bojja et al., 2017).
  • Semantic identity mapping in multi-person 3D fusion requires careful initialization and error correction to avoid persistent mismatches (Ktistakis et al., 17 May 2026).
  • For biometric verification, intra- and inter-session variation (pose, blood flow) and cross-device normalization are persistent annotation challenges (Hemis et al., 2024).
• Application-Specific Guidelines:
  • In forensic and criminal investigation, hand images are annotated only with index-based codes to avoid privacy breaches; all evaluation is on de-identified data (Baisa et al., 14 Jun 2025).
  • In clinical environments, maintaining real-time annotation (≥30 FPS) and achieving robust laterality in cluttered video streams are prioritized over perfect per-frame consistency (Sun et al., 21 Feb 2026).
  • In research, public datasets typically store annotations as class-index labels or structured filenames encoding subject, hand side, finger, and session (Hemis et al., 2024).

6. Future Directions and Recommendations

• Development of more robust annotation pipelines integrating learned contact classifiers, advanced segmentation (e.g., fine-grained SAM, depth inpainting), and temporal smoothing will further reduce label errors in complex, real-world environments (Ktistakis et al., 17 May 2026).
• Adopting federated annotation and model sharing protocols can enable cross-institutional biometric studies without exposing raw data (Hemis et al., 2024).
• Synthesizing more diverse and realistic training data (GANs, simulation) is recommended to overcome the limited size and diversity of manually annotated datasets, especially under domain shifts or class imbalance (Hemis et al., 2024).
• Multi-modal annotation, fusing appearance (RGB), geometry (3D joints/meshes), and interaction context (object/action cues), is expected to drive improvements in end-to-end hand identity recognition, with implications for security, AR, healthcare, and forensic applications (Hamza et al., 20 Sep 2025, Baisa et al., 14 Jun 2025).

7. Comparative Summary of Approaches

Approach	Label Level	Core Pipeline Features
HandSeg (color gloves)	Side (L/R)	HSV threshold + GrabCut + SVM on RGB-D
YOLOv10 (surgical)	Side (L/R)	Anchor-based detection + multi-task laterality head
I2S (3D pose)	Subject	3D joint features + XGBoost cascade
TouchMap-OR	Subject/person	Multi-view mesh fusion, 3D assignment, contact events
CLIP-HandID	Subject	De-identified index, pseudo-token in vision-language
Hand vein biometrics	Subject	CNN/ViT on ROI, cross-entropy/EER, multi-session
Egocentric ID (gesture)	Subject	Per-clip ID, 2D/3D cues + background masking

All approaches demonstrate the centrality of annotation protocols, tailored feature extraction, supervised learning strategies, and domain-specific evaluation metrics to the development of accurate and robust hand identity annotation systems (Bojja et al., 2017, Hamza et al., 20 Sep 2025, Baisa et al., 14 Jun 2025, Sun et al., 21 Feb 2026, Ktistakis et al., 17 May 2026, Hemis et al., 2024, Tsutsui et al., 2020).