MagicHOI: Monocular 3D Hand-Object Reconstruction

• MagicHOI is a 3D reconstruction framework that leverages diffusion-based novel view synthesis to regularize unseen object regions in monocular interaction videos.
• It initializes hands and objects separately using pretrained estimators and structure-from-motion, then aligns them with a visibility-aware weighting strategy and contact constraints.
• Quantitative results on HO3D-v3 demonstrate that MagicHOI outperforms prior methods in reconstruction fidelity, producing complete, artifact-free 3D models.

Searching arXiv for MagicHOI and closely related hand-object reconstruction work. MagicHOI is a method for reconstructing hands and objects from short monocular interaction videos, even under limited viewpoint variation, with the central premise that large-scale novel view synthesis diffusion models provide object supervision that can regularize unseen object regions during hand interactions (Wang et al., 7 Aug 2025). It is designed for real-world settings in which fixed camera viewpoints and static grips leave parts of the object unobserved, a regime where template-free methods that assume full object visibility often fail and template-based methods remain dependent on object templates (Wang et al., 7 Aug 2025). In the broader HOI literature, this method is distinct from the similarly named Magic-HOI dataset, which was introduced for open-world HOI detection and multi-modal prompt-based training rather than hand-object reconstruction (Yang et al., 2024).

1. Problem setting and methodological scope

MagicHOI addresses accurate 3D hand-object reconstruction from short monocular video clips in settings with severe hand occlusion and incomplete visual coverage of the manipulated object (Wang et al., 7 Aug 2025). The target scenario is not a controlled multi-view capture setup, but realistic interaction footage in which the object may remain partially hidden throughout the clip because the camera viewpoint changes little and the grasp itself occludes critical geometry (Wang et al., 7 Aug 2025).

The method is positioned against two limitations identified in prior RGB-based reconstruction pipelines. Most RGB-based hand-object reconstruction methods rely on object templates, while template-free methods typically assume full object visibility (Wang et al., 7 Aug 2025). MagicHOI instead operates in a template-free regime and introduces a prior-driven mechanism for completing unobserved regions without discarding direct image evidence. This is the basis for the characterization that it bridges the gap between pure geometry-driven methods, which cannot hallucinate unseen content, and pure prior-driven methods, which often ignore direct observations (Wang et al., 7 Aug 2025).

2. Pipeline architecture and initialization

The pipeline begins with independent initialization of the hand and the object (Wang et al., 7 Aug 2025). For the hand, a pretrained hand pose estimator such as HaMeR predicts MANO parameters: hand pose $\theta$, shape $\beta$, rotation $\mathbf{R}_h$, and translation $\mathbf{t}_h$ (Wang et al., 7 Aug 2025). For the object, Structure-from-Motion, for example using HLoc, estimates object camera poses together with object depth via multi-view stereo (Wang et al., 7 Aug 2025).

A reference frame is then selected to build an object-only reference image $I^{\text{ref}}$ using off-the-shelf segmentation and inpainting, for example InpaintAnything, to remove hand occlusions (Wang et al., 7 Aug 2025). This reference is passed to Zero-1-to-3, a diffusion-based novel view synthesis model with frozen weights, together with the reference camera pose and a target novel view (Wang et al., 7 Aug 2025). The synthesized object views serve as priors for views that were not observed in the monocular clip. To make these priors compatible with the reconstructed scene geometry, MagicHOI estimates a rigid transformation that aligns the object’s SfM frame to the canonical frame of the NVS model (Wang et al., 7 Aug 2025).

This staged design is central to the method’s operating principle. Observed geometry is anchored by monocular reconstruction cues, while unseen geometry is constrained through synthesized novel views generated by a large-scale category-agnostic prior (Wang et al., 7 Aug 2025).

3. Implicit object representation and visibility-aware diffusion regularization

The object is represented as an implicit neural Signed Distance and Texture Field,

$$f_{\psi_o}(\mathbf{x}) : \mathbb{R}^3 \to \mathbb{R} \times \mathbb{R}^3,$$

which maps a 3D point $\mathbf{x}$ to its signed distance and color (Wang et al., 7 Aug 2025). Volumetric rendering is used during optimization to generate images from this representation (Wang et al., 7 Aug 2025).

For observed object regions, MagicHOI uses a standard photometric RGB loss $\mathcal{L}_{\text{RGB}}$, a segmentation loss $\mathcal{L}_{\text{segm}}$, and a smoothness loss $\mathcal{L}_{\text{smooth}}$ on surface normals (Wang et al., 7 Aug 2025). For unobserved or occluded regions, it introduces a Score Distillation Sampling loss based on the NVS diffusion model,

$\beta$0

where $\beta$1 is a noisy novel view rendered from $\beta$2, $\beta$3 is known noise, $\beta$4 is the NVS denoiser, and $\beta$5 is the conditioning embedding composed of the reference image and camera poses (Wang et al., 7 Aug 2025).

A technical novelty of MagicHOI is a visibility-aware weighting strategy that modulates the influence of this diffusion prior (Wang et al., 7 Aug 2025). After initial optimization, a coarse 3D visibility grid $\beta$6 records which voxels are visible in any observed view (Wang et al., 7 Aug 2025). For a sampled novel view, the visibility ratio is computed as

$\beta$7

and the NVS loss weight is defined as

$\beta$8

Lower visibility therefore yields a larger NVS loss weight, so the prior acts primarily where evidence is missing or ambiguous (Wang et al., 7 Aug 2025).

The resulting object loss is

$\beta$9

This formulation regularizes unseen object regions while reducing the risk that the diffusion prior degrades observed regions (Wang et al., 7 Aug 2025).

4. Hand-object alignment through visible contact constraints

MagicHOI supplements object reconstruction with explicit hand-object alignment based on visible contact constraints (Wang et al., 7 Aug 2025). Earlier contact strategies are described as unreliable for occluded or poorly reconstructed surfaces, so MagicHOI restricts the supervision to visible and reliable contact points (Wang et al., 7 Aug 2025).

The extraction procedure projects fingertip vertices into the image, retains only those inside the hand mask and not occluded by other fingers, and traces rays from those hand vertices to locate corresponding contact points on the object (Wang et al., 7 Aug 2025). The subsequent optimization keeps the object fixed and adjusts only the hand translation $\mathbf{R}_h$0 and scale $\mathbf{R}_h$1 (Wang et al., 7 Aug 2025). The hand loss is

$\mathbf{R}_h$2

Each term has a specific geometric role: $\mathbf{R}_h$3 enforces reliable spatial hand-object alignment, $\mathbf{R}_h$4 preserves consistency between 2D hand keypoint projections and the input images, $\mathbf{R}_h$5 encourages temporal coherence, and $\mathbf{R}_h$6 penalizes hand mesh penetration into the object (Wang et al., 7 Aug 2025). This design makes the contact constraint selective rather than global, which is particularly important in monocular clips with persistent occlusion.

5. Quantitative performance and ablation evidence

On the HO3D-v3 dataset, MagicHOI reports the best values among the listed methods for both object reconstruction and hand-object alignment metrics (Wang et al., 7 Aug 2025). The reported results are: CD $\mathbf{R}_h$7, F5 $\mathbf{R}_h$8, F10 $\mathbf{R}_h$9, MPJPE $\mathbf{t}_h$0, CDh $\mathbf{t}_h$1, and RS $\mathbf{t}_h$2 (Wang et al., 7 Aug 2025). In the same evaluation, iHOI reports CD $\mathbf{t}_h$3, F5 $\mathbf{t}_h$4, F10 $\mathbf{t}_h$5, MPJPE $\mathbf{t}_h$6, CDh $\mathbf{t}_h$7, and RS $\mathbf{t}_h$8; DiffHOI reports CD $\mathbf{t}_h$9, F5 $I^{\text{ref}}$0, F10 $I^{\text{ref}}$1, MPJPE $I^{\text{ref}}$2, CDh $I^{\text{ref}}$3, and RS $I^{\text{ref}}$4; EasyHOI reports CD $I^{\text{ref}}$5, F5 $I^{\text{ref}}$6, F10 $I^{\text{ref}}$7, MPJPE $I^{\text{ref}}$8, CDh $I^{\text{ref}}$9, and RS $$f_{\psi_o}(\mathbf{x}) : \mathbb{R}^3 \to \mathbb{R} \times \mathbb{R}^3,$$0; HOLD reports CD $$f_{\psi_o}(\mathbf{x}) : \mathbb{R}^3 \to \mathbb{R} \times \mathbb{R}^3,$$1, F5 $$f_{\psi_o}(\mathbf{x}) : \mathbb{R}^3 \to \mathbb{R} \times \mathbb{R}^3,$$2, F10 $$f_{\psi_o}(\mathbf{x}) : \mathbb{R}^3 \to \mathbb{R} \times \mathbb{R}^3,$$3, MPJPE $$f_{\psi_o}(\mathbf{x}) : \mathbb{R}^3 \to \mathbb{R} \times \mathbb{R}^3,$$4, CDh $$f_{\psi_o}(\mathbf{x}) : \mathbb{R}^3 \to \mathbb{R} \times \mathbb{R}^3,$$5, and RS $$f_{\psi_o}(\mathbf{x}) : \mathbb{R}^3 \to \mathbb{R} \times \mathbb{R}^3,$$6 (Wang et al., 7 Aug 2025).

The ablation evidence isolates the contribution of the prior and the weighting strategy (Wang et al., 7 Aug 2025). Using only observed RGB supervision leads to incomplete object shapes in unobserved areas (Wang et al., 7 Aug 2025). NVS-only reconstructions can hallucinate implausible object geometry in observed regions (Wang et al., 7 Aug 2025). Joint RGB+NVS optimization without visibility-aware weighting produces smoother, more complete shapes than either source alone, but may still introduce artifacts (Wang et al., 7 Aug 2025). The full method, with visibility weighting, is reported to produce the most accurate, artifact-free reconstructions with correct object scale (Wang et al., 7 Aug 2025).

Qualitatively, the method is reported to complete unseen or occluded object parts more faithfully and realistically than prior methods, preserve hand-object spatial relationships without typical gaps, penetrations, or distortions, and generalize to diverse, in-the-wild internet video clips, including bimanual and heavily occluded scenes (Wang et al., 7 Aug 2025).

6. Relation to adjacent HOI research and naming disambiguation

A recurring source of confusion is the similarity between the names MagicHOI and Magic-HOI. Magic-HOI denotes a large-scale HOI dataset that gathers six existing datasets into a unified label space, forming over 186K images with 2.4K objects, 1.2K actions, and 20K HOI interactions for open-world HOI detection and MP-HOI training (Yang et al., 2024). MagicHOI, by contrast, is a monocular 3D hand-object reconstruction method that leverages novel view synthesis diffusion priors and visible contact constraints (Wang et al., 7 Aug 2025).

The distinction also extends to task formulation. Hoi3DGen is a framework for generating high-quality textured meshes of human-object interaction from text through an end-to-end text-to-3D pipeline (Sharma et al., 12 Mar 2026). OpenHOI targets open-world 3D hand-object interaction synthesis and generates long-horizon manipulation sequences for novel objects guided by free-form language commands (Zhang et al., 25 May 2025). GenHOI is a lightweight augmentation to pretrained video generation models for object-consistent hand-object interaction video synthesis, introducing Head-Sliding RoPE and a two-level spatial attention gate (Huang et al., 6 Mar 2026). EJIM addresses text-guided HOI generation through explicit joint-level interaction modeling with Mamba (Huang et al., 29 Mar 2025), while MP-HOI studies text-driven 3D HOI motion generation through multimodal priors, a modality-aware Mixture-of-Experts model, and cascaded diffusion with interaction supervision (Wang et al., 11 Feb 2026).

These neighboring lines of work indicate that HOI research now spans detection, reconstruction, motion generation, video synthesis, and text-to-3D asset creation. Within that landscape, MagicHOI is specifically a reconstruction framework for short monocular interaction videos, with its main contribution residing in how it integrates a frozen novel view synthesis diffusion prior into an inverse problem and tempers that prior through visibility-aware weighting and visible contact constraints (Wang et al., 7 Aug 2025).