MaskHOI: 3D Hand-Object Interaction Estimation

• MaskHOI is a masked pre-training framework designed for 3D hand-object interaction estimation, using region-specific mask ratios and skeleton-guided sampling.
• It employs a ViT-based masked autoencoder to reconstruct occluded image regions, thereby inferring 3D geometric cues from monocular RGB data.
• The framework integrates masked SDF-driven multimodal learning to couple image features with 3D supervision, resulting in improved performance on HO3D and DexYCB benchmarks.

Searching arXiv for the specified paper to ground the article with the official record. arxiv_search.query({"search_query":"id:(Xie et al., 18 Jul 2025)","max_results":5,"sort_by":"submittedDate","sort_order":"descending"}) MaskHOI is a masked pretraining framework for 3D hand-object interaction (HOI) estimation from monocular RGB input, introduced in "MaskHOI: Robust 3D Hand-Object Interaction Estimation via Masked Pre-training" (Xie et al., 18 Jul 2025). It targets a setting in which a single image $I \in \mathbb{R}^{3 \times H \times W}$ contains a hand manipulating an object, and the goal is to recover 3D hand pose and shape, such as MANO parameters $(\theta_h, \beta_h)$, together with 6D object pose, namely rotation $R_o \in SO(3)$ and translation $t_o \in \mathbb{R}^3$. The framework is designed to address two central difficulties identified for monocular HOI estimation: geometric ambiguity caused by the absence of explicit depth cues, and severe occlusions arising from mutual hand-object occlusion as well as self-occlusion of articulated hands and non-convex objects. Its defining components are a Masked Autoencoder (MAE)-driven pretraining strategy, a Region-specific Mask Ratio Allocation, and a Masked Signed Distance Field (SDF)-driven multimodal learning mechanism (Xie et al., 18 Jul 2025).

1. Problem setting and technical motivation

MaskHOI is situated in 3D HOI estimation, where precise joint poses of hands and objects must be inferred from monocular RGB observations (Xie et al., 18 Jul 2025). In this setting, monocular input provides only indirect cues for 3D structure, so the recovery of depth, articulation, and relative spatial arrangement is underconstrained. The problem is further complicated by the interaction regime itself: fingers may be hidden behind the object, object parts may be hidden behind the hand, and hand articulation can produce substantial self-occlusion.

The method is motivated by the observation that missing image evidence is not merely a nuisance variable but a structural property of HOI data. The framework therefore treats reconstruction under masking as a means of inducing geometric priors and occlusion robustness. A central premise is that the encoder should be trained to infer missing spatial and structural information rather than depend only on visible local appearance. The paper further argues that hands and rigid objects should not be treated symmetrically during masking because human hands exhibit far greater geometric complexity than rigid objects (Xie et al., 18 Jul 2025).

This suggests that a uniform masking policy is poorly matched to HOI scenes. In particular, a hand patch omitted during pretraining may remove information about articulated topology and kinematic structure, whereas a similarly sized omission on a rigid object may be less structurally disruptive. MaskHOI operationalizes this asymmetry through region-specific masking and skeleton-driven guidance.

2. MAE-based pretraining formulation

The core pretraining backbone is a ViT-style MAE architecture (Xie et al., 18 Jul 2025). The input image is partitioned into $N$ non-overlapping patches and embedded into tokens $X \in \mathbb{R}^{N \times D}$. A masking module selects a subset $X'$ of size $N' < N$, while masked tokens are replaced by a shared mask token. A shared transformer encoder $E$ processes only the unmasked tokens, and two lightweight decoders, $g_h$ and $(\theta_h, \beta_h)$0, reconstruct the full token set for hand and object regions separately.

The reconstruction branch is expressed through reconstructed embeddings

$(\theta_h, \beta_h)$1

Using the ground-truth patch embeddings $(\theta_h, \beta_h)$2, the reconstruction loss on masked tokens is

$(\theta_h, \beta_h)$3

where $(\theta_h, \beta_h)$4 denotes the set of masked indices (Xie et al., 18 Jul 2025).

Within this formulation, the mask-then-reconstruct objective is not only a denoising task. The stated purpose is to force the encoder to infer missing content so that learned representations become robust to occlusion and carry 3D structural cues. Because the decoders are specialized for hand and object regions, the reconstruction process is also implicitly aligned with the heterogeneity of HOI scenes, in which deformable articulated anatomy and rigid geometry coexist.

3. Region-specific masking and skeleton-guided hand masking

A distinctive component of MaskHOI is the Region-specific Mask Ratio Allocation, which replaces uniform random masking with differential mask ratios for hand, object, and background regions (Xie et al., 18 Jul 2025). Let $(\theta_h, \beta_h)$5 denote the set of patches assigned to region $(\theta_h, \beta_h)$6 by a 2D segmentation mask, and let $(\theta_h, \beta_h)$7. The per-region mask ratio is defined as

$(\theta_h, \beta_h)$8

where $(\theta_h, \beta_h)$9 is a balancing hyper-parameter, with the example $R_o \in SO(3)$0 given in the description. The allocation is chosen so that $R_o \in SO(3)$1, implying that fewer hand patches are masked at once.

The rationale is explicitly tied to geometric complexity: hand regions are treated as harder to reconstruct than rigid object regions, so their masking ratio is reduced. This is intended to balance learning difficulty rather than maximize masking uniformly. In effect, the masking policy becomes semantically conditioned on the content of the scene.

On top of region-specific mask rates, MaskHOI adds skeleton-driven hand masking guidance (Xie et al., 18 Jul 2025). The hand keypoint skeleton is treated as a tree, and structured occlusion patterns are sampled. The paper lists several patterns: an entire finger may be masked by selecting all patches covering one kinematic chain; only fingertip patches may be masked; or random subsets may be selected by breadth-first or depth-first traversal over the skeleton. This is designed to simulate real-world occlusion configurations such as fingertips hidden behind objects and to force the encoder to infer global hand structure from partial evidence.

The ablation evidence indicates that this masking design matters materially. On HO3D without synthetic data, the baseline with global random masking reports MJE $R_o \in SO(3)$2, STMJE $R_o \in SO(3)$3, PAMJE $R_o \in SO(3)$4, OME $R_o \in SO(3)$5, and ADD-S $R_o \in SO(3)$6. Adding Region-specific mask ratio (RMRA) changes these to MJE $R_o \in SO(3)$7, STMJE $R_o \in SO(3)$8, PAMJE $R_o \in SO(3)$9, OME $t_o \in \mathbb{R}^3$0, and ADD-S $t_o \in \mathbb{R}^3$1. Adding skeleton-guided sampling (HSMS) yields MJE $t_o \in \mathbb{R}^3$2, STMJE $t_o \in \mathbb{R}^3$3, PAMJE $t_o \in \mathbb{R}^3$4, OME $t_o \in \mathbb{R}^3$5, and ADD-S $t_o \in \mathbb{R}^3$6 (Xie et al., 18 Jul 2025). The paper also states that applying skeleton guidance $t_o \in \mathbb{R}^3$7 of the time during pretraining gave the best trade-off between realism of occlusion and leakage of structure prior.

4. Masked SDF-driven multimodal learning

To augment 2D reconstruction with 3D geometric supervision, MaskHOI introduces a Masked Signed Distance Field-driven multimodal learning mechanism (Xie et al., 18 Jul 2025). For a closed surface $t_o \in \mathbb{R}^3$8, representing a hand or object mesh, the SDF is defined as

$t_o \in \mathbb{R}^3$9

During pretraining, 3D query points $N$0 are randomly sampled in a bounding volume around the HOI scene. Each point $N$1 is projected to the image plane by camera intrinsics $N$2, producing $N$3, and the corresponding multi-scale feature $N$4 is fetched from the decoded tokens. A small MLP head predicts

$N$5

Supervision is applied only to those queries whose projected patches were masked, which the paper terms self-masking 3D SDF prediction.

The SDF regression objective is

$N$6

The stated purpose of this design is to enable the encoder to perceive the global geometric structure of hands and objects beyond the 2D image plane, thereby overcoming inherent limitations of monocular input and alleviating self-occlusion issues (Xie et al., 18 Jul 2025).

A potential misconception is that MaskHOI is solely an image reconstruction method. The inclusion of SDF prediction makes it explicitly multimodal in the sense used by the paper: image-space features are coupled to 3D geometric supervision, and the encoder is trained to hallucinate 3D geometry where the image has been masked. The ablation study supports the relevance of this component. Starting from HOISDF, adding vanilla MAE initialization gives MJE $N$7, STMJE $N$8, PAMJE $N$9, OME $X \in \mathbb{R}^{N \times D}$0, and ADD-S $X \in \mathbb{R}^{N \times D}$1, whereas adding Geometric-aware Multi-modal (GML) yields MJE $X \in \mathbb{R}^{N \times D}$2, STMJE $X \in \mathbb{R}^{N \times D}$3, PAMJE $X \in \mathbb{R}^{N \times D}$4, OME $X \in \mathbb{R}^{N \times D}$5, and ADD-S $X \in \mathbb{R}^{N \times D}$6 (Xie et al., 18 Jul 2025).

5. Network architecture, supervision, and training pipeline

The encoder $X \in \mathbb{R}^{N \times D}$7 is a Vision Transformer, with ViT-Base given as an example, operating only on unmasked tokens (Xie et al., 18 Jul 2025). The decoders $X \in \mathbb{R}^{N \times D}$8 and $X \in \mathbb{R}^{N \times D}$9 are each lightweight 4-layer transformers that upsample to the full set of $X'$0 tokens. DPT-style heads extract multi-scale feature maps $X'$1 and $X'$2 from the decoded tokens.

MaskHOI incorporates several supervision branches. For 2D tasks on $X'$3 and $X'$4, it uses hand segmentation and keypoint heatmap prediction with standard MSE/CE losses denoted $X'$5. For 3D geometry, the SDF head is supervised by $X'$6. For parameter regression, object 6D pose $X'$7 is regressed by an MLP head with

$X'$8

Hand MANO parameters $X'$9 are regressed by a small transformer head with

$N' < N$0

together with vertex-level and joint-level MANO losses (Xie et al., 18 Jul 2025).

The overall objective is written as

$N' < N$1

Typical weights are given as $N' < N$2, $N' < N$3, $N' < N$4–$N' < N$5, $N' < N$6, and $N' < N$7.

Training proceeds in two stages (Xie et al., 18 Jul 2025). In the pretraining stage, region-specific masking and skeleton masking are applied together with SDF and 2D self-supervision and parameter regression. The optimization setup is AdamW with $N' < N$8, 50k steps, cosine decay, and warm-up 1k. In the fine-tuning stage on HO3D and DexYCB, the pretrained encoder replaces the backbone of a downstream HOI estimator such as HOISDF, and end-to-end training is performed using the complete supervised loss.

6. Empirical performance, ablations, and interpretation

The reported experiments compare MaskHOI with state-of-the-art methods on DexYCB and HO3D v2, with all metrics defined as “lower is better” unless marked otherwise (Xie et al., 18 Jul 2025). On the DexYCB split, HOISDF reports MJE $N' < N$9, PAMJE $E$0, OCE $E$1, MCE $E$2, and ADD-S $E$3, whereas MaskHOI reports MJE $E$4, PAMJE $E$5, OCE $E$6, MCE $E$7, and ADD-S $E$8. On HO3D v2, HOISDF reports MJE $E$9, STMJE $g_h$0, PAMJE $g_h$1, OME $g_h$2, and ADD-S $g_h$3, while MaskHOI reports MJE $g_h$4, STMJE $g_h$5, PAMJE $g_h$6, OME $g_h$7, and ADD-S $g_h$8. Under co-training with synthetic data, HOISDF* reports MJE $g_h$9, STMJE $(\theta_h, \beta_h)$00, PAMJE $(\theta_h, \beta_h)$01, OME $(\theta_h, \beta_h)$02, and ADD-S $(\theta_h, \beta_h)$03, whereas MaskHOI* reports MJE $(\theta_h, \beta_h)$04, STMJE $(\theta_h, \beta_h)$05, PAMJE $(\theta_h, \beta_h)$06, OME $(\theta_h, \beta_h)$07, and ADD-S $(\theta_h, \beta_h)$08 (Xie et al., 18 Jul 2025).

The paper highlights several observations from these results. MaskHOI reduces hand MJE by approximately $(\theta_h, \beta_h)$09 mm on HO3D and object ADD-S by approximately $(\theta_h, \beta_h)$10 mm on DexYCB. It also states that gains are consistent across aligned metrics such as PAMJE and STMJE and across object metrics including OCE, MCE, and ADD-S. When synthetic data is used, MaskHOI* still outperforms HOISDF* by $(\theta_h, \beta_h)$11–$(\theta_h, \beta_h)$12 mm on hand and object metrics.

The component ablations on HO3D without synthetic data clarify the role of each design choice (Xie et al., 18 Jul 2025). The HOISDF baseline reports MJE $(\theta_h, \beta_h)$13, STMJE $(\theta_h, \beta_h)$14, PAMJE $(\theta_h, \beta_h)$15, OME $(\theta_h, \beta_h)$16, and ADD-S $(\theta_h, \beta_h)$17. Adding vanilla MAE initialization yields MJE $(\theta_h, \beta_h)$18, STMJE $(\theta_h, \beta_h)$19, PAMJE $(\theta_h, \beta_h)$20, OME $(\theta_h, \beta_h)$21, and ADD-S $(\theta_h, \beta_h)$22. Adding Geometric-aware Multi-modal (GML) gives MJE $(\theta_h, \beta_h)$23, STMJE $(\theta_h, \beta_h)$24, PAMJE $(\theta_h, \beta_h)$25, OME $(\theta_h, \beta_h)$26, and ADD-S $(\theta_h, \beta_h)$27. Adding Direct Param. Regression (DPR) gives MJE $(\theta_h, \beta_h)$28, STMJE $(\theta_h, \beta_h)$29, PAMJE $(\theta_h, \beta_h)$30, OME $(\theta_h, \beta_h)$31, and ADD-S $(\theta_h, \beta_h)$32. Adding Hybrid Structure-aware Mask Strat. gives MJE $(\theta_h, \beta_h)$33, STMJE $(\theta_h, \beta_h)$34, PAMJE $(\theta_h, \beta_h)$35, OME $(\theta_h, \beta_h)$36, and ADD-S $(\theta_h, \beta_h)$37.

These results help dispel two common simplifications. First, generic MAE initialization alone is not presented as sufficient for strong HOI performance, since the ablation shows mixed effects relative to the HOISDF baseline. Second, uniform global random masking is not equivalent to the proposed masking scheme, since region-specific and skeleton-guided masking alter the hand-object trade-offs and produce the strongest hand-side gains in the reported ablations. The paper’s concluding interpretation is that both tailored masking and SDF-driven multimodal objectives are critical to MaskHOI’s gains (Xie et al., 18 Jul 2025).