Hand Visibility Detector: Per-Keypoint Visibility Estimation for Hands

Abstract: Hand Pose Estimation (HPE) is a fundamental technology for various applications such as AR/VR and robotics. In these applications, the visibility of each hand joint in the image is crucial for assessing the reliability of estimation results under occlusion. However, most existing HPE methods output joint positions without explicitly indicating their visibility. Although some methods account for occlusion or visibility, visibility estimation has mainly been used as an auxiliary signal for improving pose estimation. To our knowledge, per-joint hand visibility estimation has not been systematically studied as a standalone task. In this work, we propose Hand Visibility Detector, a model for estimating the visibility of individual hand joints, and present the first systematic investigation of visibility estimation as an independent task. We show that leveraging the prior knowledge of HPE models pretrained on large-scale data as a backbone yields high performance in this task. We further demonstrate the utility of Hand Visibility Detector on a downstream task of 3D hand pose annotation via multi-view triangulation of 2D keypoints, showing that visibility-weighted triangulation reduces reprojection error. Our method is released as a ready-to-use package, and the code and demo are available at https://github.com/ryhara/hand_visibility_detector .

• The paper introduces a lightweight GAU-based detector that predicts visibility for all 21 MANO keypoints using frozen HaMeR or WiLoR hand-pose features.
• The method achieves 0.931 mAP and 0.896 F1 on HInt, outperforming ImageNet-pretrained baselines by 3.4 mAP points with only 0.83 million trainable parameters.
• Visibility-weighted multi-view triangulation reduces reprojection errors across DexYCB, HO3D, and H2O, with a maximum mean-error reduction of 10.1% on HO3D.

Hand Visibility Detector: Per-Keypoint Visibility Estimation for Hands

Problem Formulation and Motivation

“Hand Visibility Detector: Per-Keypoint Visibility Estimation for Hands” (2608.11574) formulates per-keypoint hand visibility estimation as an independent computer-vision task rather than as an auxiliary component of hand pose estimation. The distinction is technically important. Existing HPE systems generally produce 2D or 3D joint locations even when individual joints are occluded, truncated by the image boundary, or visually ambiguous. Their outputs therefore conflate two different quantities: the estimated position of a joint and the degree to which that joint is directly observable.

The paper defines visibility for each of the 21 MANO keypoints as a binary property: a joint is visible if it is neither occluded nor outside the image frame. The proposed model predicts a continuous confidence score in $[0,1]$ for every joint. This formulation is intended to provide an explicit reliability signal for downstream systems, including multi-view 3D annotation, robotic perception, AR/VR interaction, and uncertainty-aware pose tracking.

The central empirical claim is that large-scale HPE pretraining provides a substantially more suitable representation for visibility estimation than generic ImageNet pretraining, even though the visibility head itself is lightweight and trained on a comparatively modest manually annotated dataset.

Figure 1: Per-joint visibility scores distinguish directly observable hand joints from joints occluded or truncated in the image.

Architecture

The model receives a cropped hand image and consists of two components: a frozen ViT-based hand encoder and a trainable visibility head. The encoder is initialized from either HaMeR or WiLoR, both large-scale hand reconstruction systems trained to infer 3D hand structure under diverse imaging conditions. Its output tokens are rearranged into a spatial feature map of dimensions $16 \times 12 \times 1280$.

Freezing the encoder is a deliberate design choice. Visibility estimation requires reasoning about hand topology, inter-joint relationships, occlusion patterns, and image context. These capabilities are assumed to be encoded in the pretrained HPE representation. Training only the task-specific head also substantially reduces optimization cost and limits overfitting to the visibility annotations in HInt.

The visibility head follows the general structure of the RTMPose head. A $1 \times 1$ convolution compresses the feature dimensionality, after which the spatial tokens are processed by a Gated Attention Unit (GAU). The GAU is intended to model global dependencies across the image rather than make independent local decisions for each spatial region. A final projection produces 21 joint-specific logits, which are spatially averaged and transformed with sigmoid functions into visibility probabilities. Training uses binary cross-entropy over the 21 joints.

Figure 2: The pipeline combines a frozen pretrained HPE ViT backbone with a lightweight GAU-based per-joint visibility head.

The complete model contains 631 million parameters, but only 0.83 million parameters are trained, corresponding to 0.131% of the full network. Training takes approximately 2.5 hours on a single NVIDIA H200 GPU with roughly 10 GB of memory. This parameter-efficient configuration is relevant for practical deployment because it permits the addition of visibility prediction without modifying or retraining the underlying HPE system.

Dataset and Evaluation Protocol

Training and evaluation use HInt, a dataset containing manually annotated per-joint visibility labels across diverse in-the-wild imagery, including web images and egocentric video frames. The experimental split contains 25,273 training frames and 5,374 evaluation frames. The dataset is particularly important to the paper’s argument because earlier visibility-aware hand models were generally trained in controlled settings and were not evaluated as standalone visibility predictors under heterogeneous occlusion conditions.

The authors report mean average precision (mAP) and F1 score. mAP evaluates ranking quality over continuous visibility scores, while F1 evaluates binary predictions after thresholding at 0.5. The comparison includes the visibility estimator from Kim et al. and the estimator used in Contact4D, both of which rely on ImageNet-pretrained CNN representations rather than hand-specific HPE backbones.

The proposed model obtains an mAP of 0.931 and an F1 score of 0.896. The Kim et al. baseline reaches 0.895 mAP and 0.858 F1, while the Contact4D baseline reaches 0.897 mAP and 0.860 F1. Thus, the improvement is 3.4 mAP points over the strongest baseline, matching the paper’s principal quantitative claim.

Representation Transfer and Ablation Results

The backbone ablation provides the strongest evidence for the paper’s representation-learning hypothesis. Generic CNN backbones perform substantially worse: CSPNeXt-X obtains 0.800 mAP and 0.799 F1, while ResNet-152 obtains 0.796 mAP and 0.797 F1. A generic ViT-H improves performance to 0.838 mAP and 0.819 F1, and DINOv3 reaches 0.897 mAP and 0.866 F1. However, the hand-specific HaMeR and WiLoR encoders achieve 0.932 and 0.931 mAP, respectively, with both reaching approximately 0.896 F1.

This result indicates that model scale alone is insufficient. The generic DINOv3 representation is substantially larger than the hand-specific encoders, yet remains inferior. The relevant prior is not merely visual feature quality but structured knowledge of hand geometry, articulation, and occlusion acquired through HPE training.

The paper also reports an apparently counterintuitive result: fine-tuning the WiLoR backbone for visibility estimation reduces mAP from 0.931 to 0.622 and F1 from 0.896 to 0.704. The authors interpret this as evidence that task-specific fine-tuning corrupts useful pretrained representations. The claim is consequential but should be interpreted narrowly. The experiment fine-tunes under the stated training protocol and dataset scale; it does not establish that all forms of fine-tuning are harmful. More conservative adaptation strategies, such as low-rank adaptation, discriminative learning rates, partial unfreezing, stronger regularization, or multi-task objectives, could potentially avoid this degradation.

The visibility-head ablation further supports the architecture. Removing the GAU lowers mAP from 0.931 to 0.887 and F1 from 0.896 to 0.860. Replacing the proposed head with the linear head used by Kim et al. yields 0.905 mAP and 0.874 F1. The performance gap suggests that global spatial interaction is useful for visibility classification, particularly because occlusion status may depend on evidence distributed across the hand and its surrounding context rather than on local appearance alone.

Calibration and Qualitative Behavior

The F1 score is relatively insensitive to the binarization threshold. Performance peaks near 0.5 and remains above 0.88 between thresholds of approximately 0.3 and 0.7. This stability is favorable for deployment because it reduces the need for precise threshold calibration across operating conditions.

Figure 3: F1 performance remains stable across a broad range of visibility-score thresholds and peaks near 0.5.

The qualitative examples cover self-occlusion, image truncation, and occlusion by external objects. The proposed model produces the most reliable visibility assignments across these categories. In particular, it appears to distinguish a joint that is geometrically inferable from a joint that is directly observed, which is the conceptual distinction motivating the task.

Figure 4: Qualitative predictions for self-occlusion, image truncation, and object-induced occlusion.

Nevertheless, the evaluation remains primarily classification-oriented. The paper does not report calibration metrics such as expected calibration error, Brier score, reliability diagrams, or selective risk curves. Since the predicted values are subsequently used as triangulation weights, calibration quality may matter independently of mAP and F1. A predictor can rank joints correctly while producing scores whose magnitudes are not proportional to measurement reliability.

Visibility-Weighted Multi-View Triangulation

The downstream experiment evaluates whether visibility estimates improve automatic 3D hand annotation. The authors triangulate WiLoR 2D keypoints using DLT across three multi-view datasets: DexYCB, HO3D, and H2O. They compare unweighted triangulation, triangulation weighted by WiLoR’s per-view detector confidence, and triangulation weighted by the proposed per-joint visibility scores.

The distinction between detector confidence and joint visibility is important. A hand detector may assign a high confidence to an entire hand crop even when a particular finger or joint is severely occluded. Per-joint visibility provides a more granular reliability estimate and can suppress corrupted 2D keypoints selectively.

Visibility-weighted triangulation achieves lower median, mean, and interquartile-range reprojection errors on all three datasets. The largest gain occurs on HO3D, which has fewer views and substantial hand-object occlusion. There, the mean reprojection error decreases by 10.1%. This result supports the practical value of modeling visibility independently from pose coordinates: views containing unreliable observations of a particular joint contribute less to its 3D reconstruction.

Figure 5: Visibility weighting reduces triangulation reprojection errors across DexYCB, HO3D, and H2O.

The qualitative triangulation results show that weighting views according to joint visibility suppresses 2D estimates displaced by occlusion and improves the consistency of 3D joint reprojections.

Figure 6: Per-joint visibility weighting improves triangulated 3D locations in regions affected by occlusion and erroneous 2D detections.

The downstream experiment also exposes a limitation. The weighting function is treated as a direct operational use of the predicted visibility probability, but the paper does not investigate alternative robust estimators or weight transformations. For example, reliability-aware M-estimation, visibility-conditioned covariance prediction, or joint-specific uncertainty models could exploit the continuous scores more effectively than direct weighted DLT. The observed improvements nevertheless establish that even an uncomplicated weighting strategy produces measurable benefits.

Theoretical and Practical Implications

Theoretically, the paper separates observability from localization. This separation is valuable because pose estimators often infer occluded joints through learned kinematic priors, whereas downstream systems may need to know whether a prediction is supported by image evidence. A visibility detector therefore supplies a semantic uncertainty variable that is not recoverable from joint coordinates alone.

The results also provide evidence for task-relevant transfer from structured pretrained models. HaMeR and WiLoR outperform generic image representations despite similar or larger parameter counts elsewhere in the comparison. The result suggests that pretraining objectives encoding articulated geometry can transfer to auxiliary perceptual attributes such as visibility, even when those attributes were not explicitly supervised during pretraining.

Practically, the released package can be integrated into multi-view annotation pipelines, HPE confidence estimation, occlusion-aware tracking, and robotic manipulation systems. In AR/VR, per-joint visibility can be used to gate interaction logic or avoid treating hallucinated fingertip positions as direct observations. In dataset construction, it can identify views that should receive reduced influence during triangulation or manual-review prioritization.

The method also has implications for video-based inference. Temporal modeling could enforce visibility consistency, distinguish transient motion blur from persistent occlusion, and exploit visibility transitions as structured events. A video model could jointly predict visibility, keypoint uncertainty, and occlusion boundaries, potentially improving both temporal pose stability and automatic annotation.

Limitations and Future Directions

The study has several limitations. First, HInt provides the primary supervision and evaluation domain, so broader cross-dataset generalization remains insufficiently characterized. The downstream datasets demonstrate utility, but they are used mainly to evaluate triangulation rather than direct visibility classification with independent visibility labels.

Second, the definition of visibility merges physical occlusion and image truncation into a single negative class. These phenomena have different causes and may require different responses in downstream systems. A richer label space could distinguish self-occlusion, object occlusion, inter-hand occlusion, truncation, blur, and low-resolution ambiguity.

Third, the model relies on hand crops. Errors in the upstream detector may affect both the visible evidence and the coordinate frame, yet detector robustness is not analyzed systematically. Joint visibility prediction under imperfect or partial crops would be important for end-to-end deployment.

Finally, the reported fine-tuning degradation motivates but does not settle the question of adaptation. Future work should test parameter-efficient fine-tuning, multi-task learning with pose and visibility, uncertainty calibration, temporal architectures, and explicit probabilistic triangulation. A particularly useful extension would predict a per-joint observation covariance rather than a scalar visibility probability, enabling geometry-aware fusion across cameras.

Conclusion

“Hand Visibility Detector: Per-Keypoint Visibility Estimation for Hands” (2608.11574) establishes per-joint hand visibility estimation as a standalone task and demonstrates that frozen, hand-specific HPE backbones provide an effective basis for it. With only a 0.83M-parameter trainable head, the method achieves 0.931 mAP and 0.896 F1 on HInt, outperforming ImageNet-pretrained visibility baselines by 3.4 mAP points. Its downstream use in visibility-weighted triangulation reduces mean reprojection error by up to 10.1% across three multi-view datasets. The findings support explicit observability modeling as a useful complement to pose estimation, while leaving open important questions concerning calibration, cross-domain generalization, temporal consistency, and uncertainty-aware 3D reconstruction.