Hand Visibility Detector: Methods and Applications

• Hand Visibility Detectors are computer-vision systems that determine whether a hand, hand region, or anatomical keypoint is visually observable, supporting applications such as pose estimation, augmented reality, robotics, and driver monitoring.
• HVDs use methods ranging from HSV segmentation, motion analysis, depth masks, and body-guided detection to Transformer-based per-keypoint classifiers, with performance depending strongly on illumination, occlusion, scale, motion, and the selected visibility definition.
• Per-keypoint visibility estimation provides 21 joint-level probabilities and can improve multiview triangulation, while robust systems should separately report hand presence, visible regions, occlusion state, localization confidence, and temporal confidence.

A Hand Visibility Detector (HVD) is a computer-vision system that determines whether a hand, hand region, or individual hand keypoint is visually observable in an image or video. Depending on its formulation, visibility may mean the presence of a sufficiently coherent hand-shaped region, the detection of a hand instance, the in-view status of an entire hand, or the direct observability of each anatomical joint. These definitions are not interchangeable: a hand may be geometrically inferable but visually occluded, a hand may be visible while fingertip detection fails, and a pose estimator may predict a plausible location for a hidden joint. The paper "Hand Visibility Detector: Per-Keypoint Visibility Estimation for Hands" (Hara et al., 12 Aug 2026) treats visibility as an independent per-joint prediction task, whereas earlier systems generally use visibility as an intermediate signal for hand detection, pose estimation, interaction recognition, rendering, or temporal forecasting.

1. Definitions and scope

Hand visibility has several operational meanings in the literature.

Region-level visibility denotes the presence of a sufficiently large, connected, hand-like image region. The fingertip-detection pipeline in "Fingertip Detection: A Fast Method with Natural Hand" (Raheja et al., 2012) generates an HSV skin mask, a binary silhouette, a largest connected blob, and a cropped hand region. These intermediate outputs can support visibility decisions without requiring successful fingertip localization. A visible fist, partially occluded hand, motion-blurred hand, or hand with merged fingers may therefore satisfy a region-level visibility criterion while failing fingertip detection.

Instance-level visibility denotes the existence of a detected hand bounding box or segmentation mask. Driver-hand systems combine object-level detection with pixel-level refinement. A YOLO proposal followed by an illumination-aware skin classifier produces a refined hand mask (Siddharth et al., 2018). HandyNet uses depth-only Mask R-CNN-style detection and instance segmentation, treating a hand as visible when a detectable hand instance has a nonempty visible depth mask (Rangesh et al., 2018). Body-guided detection similarly identifies visible or partially visible hand regions inside detected human bodies (Li et al., 2019).

Field-of-view visibility denotes whether an entire hand is in or out of the egocentric camera’s view. EgoH4 predicts a binary visibility score separately for the left and right hands over observed frames, using the sentinel coordinate $(-1,-1)$ when a hand is not visible (Hatano et al., 11 Apr 2025). This formulation does not distinguish partial finger occlusion, self-occlusion, or physical occlusion by an object.

Per-keypoint visibility denotes whether each anatomical landmark is directly observable rather than merely inferable from a pose prior. The model in (Hara et al., 12 Aug 2026) predicts 21 probabilities, one for each MANO keypoint, with visibility defined as neither occluded nor outside the image frame. Its output is therefore distinct from the joint coordinates produced by an HPE model.

Geometric recoverability is a related but different concept. Multiview bootstrapping uses detector confidence, RANSAC inlier membership, reprojection error, and successful triangulation to assess whether a keypoint can be reliably reconstructed (Simon et al., 2017). A reprojected keypoint may be geometrically recoverable even when it is visually occluded in a particular camera.

2. Classical image-processing foundations

Early HVD formulations commonly derive visibility from segmentation and geometric evidence.

The pipeline in (Raheja et al., 2012) converts an RGB image to HSV, applies a skin-color filter, generates a binary silhouette, smooths it with an averaging filter, selects the largest connected component, estimates wrist and finger directions using horizontal and vertical intensity histograms, and crops the hand region. A component area $A_h$ can be normalized by image area $WH$:

$\rho_A=\frac{A_h}{WH}.$

A basic visibility rule is to declare a hand visible when $A_h$ or $\rho_A$ exceeds an empirically selected threshold. The paper does not specify HSV bounds, smoothing parameters, visibility thresholds, timing measurements, datasets, or visibility metrics. Its largest-blob assumption can fail when the face, arm, skin-colored background, another hand, or a non-hand object is larger than the hand.

Additional geometric quantities include bounding-box fill, connected-component dominance, contour compactness, centroid, and image-boundary contact. Boundary contact should not automatically reject a component because it may indicate either a partially visible hand or background clutter. The system can consequently distinguish full visibility, partial visibility, uncertainty, and non-visibility only through additional implementation rules rather than through a reported classifier.

Interest-point methods provide another classical source of evidence. "Feature Detection for Hand Hygiene Stages" (Bakshi et al., 2021) demonstrates contour extraction, convex-hull construction, Harris detection, Shi–Tomasi detection, SIFT keypoints, and centroid extraction for the pose “Rub hands palm to palm.” Harris and Shi–Tomasi identify localized intensity changes, while SIFT provides approximately scale- and rotation-invariant descriptors. The study remains preliminary: it does not define a visibility label, confidence score, threshold, occlusion policy, or validated hand-presence decision rule.

Optical multi-touch systems use a different classical representation. "A novel processing pipeline for optical multi-touch surfaces" (Ewerling, 2013) applies distortion correction, illumination normalization, region-of-interest extraction, MSER analysis, fingertip detection, hand distinction, tracking, and output messaging. Its component tree contains nested extremal regions in which bright fingertip peaks may have parent regions corresponding to fingers, palms, wrists, or arms. This structure can support hand visibility near a diffuse back-illuminated surface, but the system is fundamentally optimized for fingertips approaching or touching that surface rather than arbitrary free-space hands.

3. Appearance, motion, and region-level detectors

Modern region-level HVDs combine appearance models with object detection, motion, temporal context, or depth.

The egocentric detector in "Detecting Hands in Egocentric Videos: Towards Action Recognition" (Cartas et al., 2017) uses a two-stage pipeline:

$$\text{skin map}\rightarrow\text{arm/hand proposals}\rightarrow\text{CNN hand recognition}.$$

PERPIX models RGB, HSV, LAB, SIFT, ORB, and Gabor-based features to generate skin regions. Connected regions may contain hands, arms, two connected arms, or other skin-colored structures. Geometric heuristics split two-arm regions, estimate wrist lines, and generate hand proposals. A CaffeNet-derived CNN then rejects non-hand proposals. The method was evaluated on the UNIGE-HANDS dataset, including a manually annotated subset of 2,000 images containing more than 1,739 hands. With 150 training frames per setting, PERPIX achieved an overall true-positive rate of $0.863$ and true-negative rate of $0.815$. The paper reports average precision values of $20.01\%$ and $A_h$0 in different sections, an unresolved inconsistency.

"Egocentric Hand Detection Via Dynamic Region Growing" (Huang et al., 2017) uses motion residuals to generate hand-like seed superpixels. ORB correspondences between successive frames are separated into global camera-motion correspondences and residual hand-related correspondences using RANSAC homography estimation. SLIC superpixels containing local peaks of residual motion become seeds. If no reliable seeds are generated, the frame is treated as a “no hand” frame and region growing is skipped.

Candidate superpixels are scored using four cues: HSV-histogram contrast, spatial proximity to seeds, temporal position consistency, and appearance continuity. The scores are combined with weights $A_h$1 and $A_h$2. Dynamic region growing stops when the best adjacent-superpixel score falls below a fraction $A_h$3 of the preceding reference score, and components smaller than $A_h$4 pixels are removed. The reported best ADL setting is $A_h$5 and $A_h$6. The method achieves approximately 32 fps, with GTEA F-scores of $A_h$7, $A_h$8, and $A_h$9 for coffee, tea, and peanut, respectively, and an EgoHands IoU of $WH$0. Its principal limitation is semantic ambiguity: moving objects, clothing, sleeves, arms, and hand-colored background regions may produce hand-like motion or appearance.

Driver-hand detection combines semantic object detection and local pixel evidence. "Driver Hand Localization and Grasp Analysis: A Vision-based Real-time Approach" (Siddharth et al., 2018) uses YOLO proposals, NMS, and a pixel-based skin classifier with ten global illumination models. YOLO detections with confidence at least $WH$1 are refined inside candidate boxes using RGB, HSV, and SIFT features. The resulting mask is used for localization, false-positive suppression, and grasp analysis. On the VIVA Hands Detection Dataset, the combined system achieves average precision and recall of $WH$2 and $WH$3 for L1 hands and $WH$4 and $WH$5 for L2 hands, at up to 35 fps. The method does not define a separate visibility class, calibrated visibility probability, partial-visibility label, or temporal model.

HandyNet replaces RGB appearance dependence with depth. It uses a single-channel $WH$6 depth image, cross-bilateral depth in-painting, a ResNet-50/FPN-based RPN, hand box regression, instance masks, and handheld-object classification (Rangesh et al., 2018). It is trained using masks generated through green gloves, red wristbands, registered RGB-depth images, connected-component analysis, Otsu depth splitting, and 3D centroid-distance merging. The system obtains class-agnostic test AP of $WH$7, $WH$8 of $WH$9, and approximately 15 Hz inference. It can segment visible hand surfaces under partial occlusion but cannot infer a completely hidden hand from a single frame.

4. Learned detectors, body context, and temporal aggregation

Learned detectors improve semantic discrimination and small-hand localization by incorporating context.

DID-Net uses two linked Faster R-CNN-style detectors (Li et al., 2019). BodyDetector processes the full image and detects persons, hands, and faces. PartsDetector then searches inside selected body regions using enlarged body-conditioned feature representations. This coarse-to-fine design increases the effective resolution of small hands and suppresses irrelevant background. On the Human-Parts dataset, which contains 14,962 images and 43,752 hand annotations, DID-Net achieves hand AP of $\rho_A=\frac{A_h}{WH}.$0, person AP of $\rho_A=\frac{A_h}{WH}.$1, face AP of $\rho_A=\frac{A_h}{WH}.$2, and overall mAP of $\rho_A=\frac{A_h}{WH}.$3 at IoU $\rho_A=\frac{A_h}{WH}.$4. The model detects visible or partly occluded regions but does not directly classify visibility states or estimate invisible hand extent.

Long-term hand detection addresses temporary disappearance and dramatic appearance change through temporal accumulation. "Fast Hand Detection in Collaborative Learning Environments" (Teeparthi et al., 2021) applies Faster R-CNN at one frame per second, converts detections into binary regions, and accumulates them over 12-second windows:

$\rho_A=\frac{A_h}{WH}.$5

ISODATA clustering and small-region removal then produce temporally persistent hand regions. The method improves AP from $\rho_A=\frac{A_h}{WH}.$6 to $\rho_A=\frac{A_h}{WH}.$7 at IoU $\rho_A=\frac{A_h}{WH}.$8, reports an approximately 78.8% reduction in false-positive detections, and operates at approximately four times real time overall. It does not explicitly distinguish fully visible, partially occluded, fully occluded, and absent hands. A projected region may remain after a hand leaves the view, and nearby hands may merge.

Functional hand-use recognition demonstrates that visibility is not equivalent to interaction. "Recognizing Hand Use and Hand Role at Home After Stroke from Egocentric Video" (Tsai et al., 2022) evaluates hand-object interaction and stabilizer/manipulator roles using a random forest, SlowFast, and a Hand Object Detector. The Hand Object Detector, which models hand-object contact, achieves macro MCC of $\rho_A=\frac{A_h}{WH}.$9 for more-affected hands, $A_h$0 for less-affected hands, and $A_h$1 for both hands combined. Hand-role classification remains close to chance despite superficially higher F1 and accuracy because stabilization dominates the data. The study demonstrates the importance of separating hand visibility, contact, functional use, and bilateral role.

EgoH4 adds a visibility head to a diffusion Transformer for egocentric 3D hand forecasting (Hatano et al., 11 Apr 2025). It predicts left- and right-hand in-view status over observed frames using binary cross-entropy, with visibility loss weight $A_h$2 and reprojection-loss weight $A_h$3. Visibility supervision improves forecasting, especially for out-of-view observations, but the paper reports no standalone visibility accuracy, precision, recall, F1, AUROC, or calibration. Its output is an observation-time field-of-view classifier, not a per-joint visual-occlusion detector.

5. Per-keypoint and geometry-aware visibility

Per-keypoint visibility explicitly separates direct image evidence from pose inference.

Multiview bootstrapping uses detector confidence thresholds, RANSAC, reprojection error, and inlier counts to generate reliable keypoint labels (Simon et al., 2017). A keypoint requires at least three inlier views, with a nominal RANSAC threshold of approximately four pixels. These criteria measure geometric consistency and recoverability, not necessarily direct visual visibility. A keypoint hidden in one view may still receive a valid reprojection label from other views.

VA-NeRF introduces 3D visibility-aware feature fusion for interacting hands (Huang et al., 2024). For a query point $A_h$4 and mesh vertices $A_h$5 and $A_h$6, visibility indicators $A_h$7, $A_h$8, and $A_h$9 determine how local image features, mesh features, symmetric cross-hand features, and global features are fused. The model uses MANO meshes and a visibility-guided discriminator producing a pixel-wise visibility map. Visibility is binary internally, but the map is used primarily to improve novel-view synthesis rather than as a standalone detector. Adding visibility-aware attention improves PSNR from $\rho_A$0 to $\rho_A$1, SSIM from $\rho_A$2 to $\rho_A$3, and reduces LPIPS from $\rho_A$4 to $\rho_A$5 in the reported ablation.

TexHOI models a different visibility quantity: hand-induced environmental-light occlusion on an object surface (Aggarwal et al., 7 Jan 2025). A MANO hand is approximated with 108 posed spheres, projected onto spherical-Gaussian illumination lobes, and used to estimate an illumination-weighted occlusion value $\rho_A$6. The visible-light fraction is $\rho_A$7, and the incident illumination is modeled as:

$\rho_A$8

This is not a binary camera-space hand mask. It is a surface-point- and lighting-dependent estimate of how much environmental illumination is blocked by the hand.

The dedicated model in (Hara et al., 12 Aug 2026) formulates HVD as 21 simultaneous binary classification problems. Given a cropped RGB hand image, a frozen HaMeR or WiLoR ViT backbone produces a spatial feature map of size $\rho_A$9. A trainable visibility head compresses channels to 256, tokenizes the spatial features, applies a Gated Attention Unit, maps the representation to 21 channels, and performs spatial average pooling followed by sigmoid activation. The head contains approximately 0.83 million trainable parameters.

For visibility labels $$\text{skin map}\rightarrow\text{arm/hand proposals}\rightarrow\text{CNN hand recognition}.$$0 and predictions $$\text{skin map}\rightarrow\text{arm/hand proposals}\rightarrow\text{CNN hand recognition}.$$1, the loss is mean binary cross-entropy:

$$\text{skin map}\rightarrow\text{arm/hand proposals}\rightarrow\text{CNN hand recognition}.$$2

Training uses 25,273 HInt frames and evaluation uses 5,374 frames. The hand box is expanded by 1.25, resized to $$\text{skin map}\rightarrow\text{arm/hand proposals}\rightarrow\text{CNN hand recognition}.$$3, and center-cropped to $$\text{skin map}\rightarrow\text{arm/hand proposals}\rightarrow\text{CNN hand recognition}.$$4. With a frozen WiLoR backbone, the model achieves mAP $$\text{skin map}\rightarrow\text{arm/hand proposals}\rightarrow\text{CNN hand recognition}.$$5 and F1 $$\text{skin map}\rightarrow\text{arm/hand proposals}\rightarrow\text{CNN hand recognition}.$$6. It outperforms Kim et al. and Contact4D, which achieve mAP values of $$\text{skin map}\rightarrow\text{arm/hand proposals}\rightarrow\text{CNN hand recognition}.$$7 and $$\text{skin map}\rightarrow\text{arm/hand proposals}\rightarrow\text{CNN hand recognition}.$$8, respectively. Fine-tuning WiLoR reduces mAP to $$\text{skin map}\rightarrow\text{arm/hand proposals}\rightarrow\text{CNN hand recognition}.$$9, while removing the GAU reduces mAP to $0.863$0, indicating the importance of preserving the hand-specific representation and modeling global spatial dependencies.

6. Applications, evaluation, and failure modes

HVD outputs support interaction systems, augmented and virtual reality, robotics, driver monitoring, hand hygiene analysis, multiview annotation, and inverse rendering.

For multiview 3D annotation, per-joint visibility scores can weight triangulation:

$0.863$1

The dedicated detector in (Hara et al., 12 Aug 2026) reduces reprojection error relative to unweighted and whole-view detection-confidence-weighted triangulation on DexYCB, HO3D, and H2O. The largest reported improvement is a 10.1% reduction in mean reprojection error on HO3D.

Evaluation must match the visibility definition. Region-level systems require box or mask IoU, precision, recall, F-score, AP, and temporal stability. Keypoint-level systems require mAP, F1, per-joint confusion matrices, calibration, and breakdowns by occlusion type and joint identity. Field-of-view systems require hand-side accuracy and explicit treatment of partial truncation. A system intended for geometric visibility should additionally report reprojection error, triangulation stability, or surface-point visibility error.

Common failure modes recur across formulations:

• Illumination variation: HSV and skin models can fail under shadows, overexposure, colored lighting, or white-balance changes. Illumination normalization and multiple appearance models improve but do not eliminate this problem.
• Skin-color ambiguity: faces, forearms, sleeves, beige backgrounds, and vehicle components can be mistaken for hands.
• Occlusion: self-occlusion, object occlusion, inter-hand occlusion, and truncation may remove direct evidence while leaving pose priors capable of producing plausible but unreliable joint estimates.
• Motion dependence: motion-residual detectors may miss stationary hands or accept moving non-hand objects.
• Scale and resolution: small hands may disappear during segmentation or produce insufficient local features; body-conditioned detectors address this through higher-resolution crops.
• Temporal persistence errors: accumulation can bridge missed detections but can also preserve stale regions or merge nearby hands.
• Pose and identity ambiguity: fists, folded hands, palm-to-palm contact, closed hands, and caregiver hands challenge hand counting and role assignment.
• Definition mismatch: contact is not functional use, field-of-view status is not visual observability, and geometric recoverability is not direct visibility.

A robust HVD should therefore expose separate outputs rather than a single binary flag:

$0.863$2

The most direct current formulation is the per-keypoint model in (Hara et al., 12 Aug 2026), which treats visibility as the primary target. Classical segmentation, body-conditioned detection, depth-based instance masks, multiview geometry, temporal aggregation, and visibility-aware rendering remain complementary: they provide region evidence, geometric constraints, temporal persistence, or physically meaningful occlusion estimates that a per-keypoint classifier alone does not provide.