ArtHOI-Wild: In-the-Wild 3D HOI Benchmark

• ArtHOI-Wild is an open-vocabulary 3D human-object interaction benchmark built from real-world single images.
• It overcomes limitations of indoor and synthetic datasets by incorporating diverse object categories and detailed 2D HOI annotations.
• The benchmark employs a coarse-to-fine annotation pipeline with Gaussian-based optimization to enhance accurate 3D contact and interaction reasoning.

ArtHOI-Wild is an open-vocabulary, in-the-wild 3D human-object interaction benchmark associated in the literature here with Open3DHOI, a dataset and annotation pipeline for reconstructing fine-grained 3D humans, objects, and their interactions from single images (Wen et al., 20 Mar 2025). It was introduced to address a central limitation of prior 3D HOI research: existing benchmarks are primarily indoor, often multi-view, and restricted to a small set of object categories, whereas large-scale 2D HOI resources already contain diverse, real-world, open-vocabulary interactions but lack 3D structure. In that sense, ArtHOI-Wild denotes a shift from category-limited indoor reconstruction toward single-image, real-image, open-vocabulary 3D HOI understanding, while remaining distinct from related uses of “wild” in training-free HOI recognition and zero-shot articulated HOI synthesis (Ai, 2024).

1. Definition and positioning

ArtHOI-Wild corresponds here to Open3DHOI, which is presented as the first in-the-wild, open-vocabulary 3D HOI dataset based on real-world images (Wen et al., 20 Mar 2025). Its stated role is to serve as a test set / benchmark rather than a large-scale training corpus. The benchmark is designed to better represent real-world HOIs “with any objects and interactions,” and to support future open-vocabulary 3D HOI research.

The benchmark is motivated by the limitations of earlier 3D HOI datasets. BEHAVE, InterCap, IMHOI, and PROX-S are described as mostly indoor and often based on multi-view RGB-D sequences. WildHOI and 3DIR are in-the-wild, but they have few object categories and rely on fixed / unreal CAD models, which limits realism and category diversity. By contrast, the new benchmark is constructed from real 2D HOI imagery and recent single-image reconstruction methods, with the explicit goal of supporting generalization to real-world scenes with a wide range of objects (Wen et al., 20 Mar 2025).

A concise comparison reported in the source material is shown below.

Dataset	Objects	Actions
BEHAVE	10	no action labels
InterCap	10	no action labels
WildHOI	8	no action labels
3DIR	21	17 actions
PROX-S	40	17 actions
Ours	133	120 actions

The benchmark also includes richer annotation types, specifically human pose, object pose, contact, and 2D HOI annotations. This suggests that ArtHOI-Wild is intended not only for object placement or body fitting, but for the broader study of spatial interaction structure, contact reasoning, and semantic HOI understanding.

2. Data sources and annotation pipeline

The dataset is built through a coarse-to-fine manual annotation system on top of existing 2D HOI datasets and modern reconstruction tools (Wen et al., 20 Mar 2025). Source images are selected from HAKE-Large with 12k+ images, SWIG-HOI with 3k+ images, plus some web-collected images, forming an initial database of 15k+ images.

For each image, coarse reconstruction first estimates the human using OSX and the object using InstantMesh. Because in-the-wild images often contain severe occlusion, the pipeline applies occlusion completion using amodal completion and Stable Diffusion 1.5 for inpainting to obtain a complete object image before reconstruction. A rough 3D spatial relationship is then estimated by combining ZoeDepth, human and object masks, depth point clouds, and sampled mesh point clouds, from which the system infers rough position, size, and placement of the human-object pair.

The source material emphasizes that this rough reconstruction is useful for annotation but often contains mesh collision, inaccurate scales, inaccurate positions, and object pose errors from InstantMesh. The fine annotation stage therefore introduces two dedicated tools. The filtering tool asks annotators to inspect projected SMPL-X meshes, render objects from six viewpoints, and correct masks manually when inpainting mask completion fails. The pipeline also splits the SMPL-X body into 34 regions to annotate contact areas at the body-part level rather than the vertex level. The 3D interaction tool combines a Blender-based tool for object translation, rotation, and scale adjustment with a web-based fine annotation tool based on ImageNet3D, where multi-view projections help annotators judge 3D interaction more accurately (Wen et al., 20 Mar 2025).

After manual filtering and annotation, the retained subset contains 2.5k+ images. The final dataset contains 370 3D human-object pairs, 2,561 objects in 133 categories, and 3,671 interactions in 120 categories. Only single-person images are used in the final dataset, except that 63 images contain multiple objects interacting with one person and are split into separate HOI pairs. The overall pass rate from the 15k source images is 17%, a figure explicitly presented as evidence of the difficulty of reconstructing 3D HOIs from in-the-wild images (Wen et al., 20 Mar 2025).

3. Scale, diversity, and annotation semantics

The benchmark’s scale is characterized by 2.5k+ final annotated images, 2,561 objects, 133 object categories, 3,671 interactions, and 120 action categories (Wen et al., 20 Mar 2025). The source material further notes that the object distribution spans many unusual categories, including food and animals, which are rare in previous 3D HOI datasets. Object categories are organized using a WordNet-style taxonomy, and object size varies greatly across categories and even within the same category.

The paper computes object size using the volume function from Trimesh, then takes the cube root as size. It also states that the dataset includes abnormal HOIs because it originates from real HOI images, giving standing on a chair as an example. A plausible implication is that the benchmark does not restrict itself to canonical affordance patterns; instead, it preserves the irregularity of real-world imagery, which is consistent with the “in-the-wild” designation.

The resource also provides a template library with 58 object categories and 212 templates. The source explicitly states that this library is not used to build Open3DHOI, but is mentioned as useful for future work. This distinction matters because the benchmark is constructed from reconstructed real images rather than template instantiation.

A common misconception is that an in-the-wild 3D HOI dataset must necessarily be multi-view or sensor-based to be reliable. The benchmark’s design explicitly takes the opposite route: it lifts real 2D HOI images into 3D using modern reconstruction tools and then relies on coarse-to-fine manual correction. Another misconception is that open-vocabulary here means unconstrained automatic labeling; in fact, the pipeline is annotation-heavy and filtered aggressively, as indicated by the 17% pass rate (Wen et al., 20 Mar 2025).

4. Gaussian-HOI optimizer and reconstruction formulation

The principal reconstruction method associated with the benchmark is a training-free 3D HOI optimizer based on 3D Gaussian Splatting, referred to as the HOI-Gaussian optimizer or Gaussian-HOI optimizer (Wen et al., 20 Mar 2025). Its stated purpose is to reconstruct the spatial interactions between humans and objects while learning contact regions.

The method models the human and object as Gaussian sets, with human Gaussians $g_h$, object Gaussians $g_o$, and combined HOI Gaussians $g_{hoi}$. The rendering equation given in the source is

$$C=\sum_{n=1}^{|\mathcal{N}|} c_n \alpha_n^{\prime} \prod_{j=1}^{n-1}\left(1-\alpha_j^{\prime}\right),$$

with

$$\alpha_n^{\prime}=\alpha_n \exp \left(-\frac{1}{2}\left(\boldsymbol{x}^{\prime}-\boldsymbol{\mu}_n^{\prime}\right)^{\top} \boldsymbol{\Sigma}_n^{\prime-1}\left(\boldsymbol{x}^{\prime}-\boldsymbol{\mu}_n^{\prime}\right)\right).$$

Human Gaussians are initialized from SMPL-X vertices, object Gaussians from object mesh vertices, and the method follows GauHuman for human pose refinement and LBS offsets while introducing a learnable object pose parameter $W_{obj}$. The combined interaction Gaussian is defined through transformed human and object Gaussians: $$p^{hoi}=p^h \oplus p^o,\quad \Sigma^{hoi}=\Sigma^h \oplus \Sigma^o$$ with

$p^h=R^h p^{h'}+t^h,\quad p^o=R^o s^o p^{o'}+t^o$

and

$$\Sigma^h=R^h \Sigma^{h'} R^{h\top},\quad \Sigma^o=R^o \Sigma^{o'} R^{o\top}.$$

A key novelty is the use of Gaussian opacity and depth ordering to infer likely contact regions. The source describes the intuition as follows: regions where human and object occlude each other tend to have lower opacity in rendering; pixels not visible due to occlusion in the original image imply likely interaction areas; and points on the back side relative to the camera are initialized with very low opacity, so front-facing visible regions dominate rendering. The resulting per-Gaussian contact interaction score is

$$c=w_{\alpha} \cdot {Norm}(\alpha^h)+w_d \cdot d_C(p^h, p^o)^h.$$

In practice, opacity is first used to find candidate contact areas, which are then narrowed further using a distance threshold between human and object (Wen et al., 20 Mar 2025).

The loss is separated into rendering loss and HOI loss. The rendering term uses L1, L2, SSIM, and LPIPS, and is computed separately for $g_o$0, $g_o$1, and $g_o$2: $g_o$3 The HOI term consists of contact loss $g_o$4, collision loss $g_o$5, and ordinal depth loss $g_o$6: $g_o$7 and the final objective is

$g_o$8

The optimizer runs for 160 iterations: the first 100 iterations use only rendering loss, and later iterations add HOI loss. The source explains this staged optimization as a way to first align image appearance and then refine interaction geometry (Wen et al., 20 Mar 2025).

5. Tasks, evaluation protocol, and reported performance

The main reconstruction benchmark is conducted on the Open3DHOI test set, with the explicit note that the method is training-free and that object meshes before manual annotation are given (Wen et al., 20 Mar 2025). For fairness, both the proposed method and PHOSA initialize human body parameters from the dataset, and PHOSA is used as the principal baseline because the dataset is open-vocabulary and category-specific training would be unfair.

The evaluation reports object pose metrics—Scale, Translation (cm), Rotation, and Chamfer Distance—and introduces an interaction metric, the Collision-Contact score

$g_o$9

where $g_{hoi}$0 is human-object collision, $g_{hoi}$1 is the Chamfer distance between each human body part and the object, and $g_{hoi}$2 is the object mesh size.

On object pose metrics, the reported numbers are:

Method	Scale	Translation (cm)	Rotation	Chamfer
PHOSA	0.39	77.79	0.95	49.1
Ours w/o HOI Loss	0.25	38.66	0.45	16.9
Ours	0.16	38.44	0.41	19.3

On collision/contact metrics, the reported numbers are:

Method	$g_{hoi}$3	Collision	Contact
PHOSA	0.431	0.105	0.326
Coarse Recon	0.248	0.083	0.165
Ours Gs only	0.287	0.136	0.151
Gs depth	0.216	0.080	0.136
Gs colli	0.189	0.046	0.143
Gs depth colli	0.188	0.045	0.143
Gs depth colli cont	0.181	0.053	0.128

The interpretation given in the source is that Gaussian optimization alone mainly improves appearance alignment, while depth, collision, and contact losses are necessary for better interaction quality; the contact loss improves contact quality, though it may slightly increase collision in some cases (Wen et al., 20 Mar 2025).

Beyond reconstruction, the benchmark proposes 3D HOI Understanding and HOI Pose Chat as new tasks. For 3D HOI understanding, PointLLM-7B is queried with prompts such as “What is the action between the person and the [object]?” and evaluated with Top-1 accuracy. The reported results are 0.47 for Action / object prompt, 0.20 for Action without object, 0.32 for Object prompt, and 0.31 for Object without action. For contact prediction, adding 3D information improves results from Micro F1 0.6118 to 0.6207, Hamming Loss 0.0874 to 0.0844, and Jaccard Index 0.4303 to 0.4561. For ChatPose, the reported errors are MPJPE 103.6, MPVPE 131.2 with Action, MPJPE 105.2, MPVPE 133.5 with w/o Action, and MPJPE 103.4, MPVPE 130.9 with Action + Object. The paper explicitly concludes that current general 3D/LLM systems are still weak at precise 3D HOI reasoning (Wen et al., 20 Mar 2025).

6. Relation to adjacent “wild” HOI settings

The term ArtHOI-Wild can be confused with other recent HOI directions that also emphasize out-of-distribution generalization, test-time-only inference, or zero-shot behavior. The source material distinguishes at least two nearby but non-identical lines of work.

First, the paper “An analysis of HOI: using a training-free method with multimodal visual foundation models when only the test set is available, without the training set” studies a deliberately training-free HOI recognition regime in which only the test dataset is available (Ai, 2024). That work is framed as relevant to an ArtHOI-Wild-like setting because it asks what can be done when there is only test set and no training data at all, using multimodal visual foundation models such as CLIP, BLIP, and BLIP-2. Its core task is not 3D reconstruction but prompt-based recognition of the verb in $g_{hoi}$4 triplets. The reported pattern is that GT pairing performs best, GT-R degrades, and GroundingDINO degrades further; for blip2_coco_vitH/14@364px, the corresponding full mAP values are 49.56, 38.86, and 19.71. The interpretation given is that current foundation models exhibit some open-vocabulary HOI capability but are not robust enough to solve HOI recognition in the wild without training when pairing quality deteriorates (Ai, 2024).

Second, the paper “ArtHOI: Articulated Human-Object Interaction Synthesis by 4D Reconstruction from Video Priors” introduces ArtHOI, a zero-shot articulated human-object interaction synthesis framework based on 4D reconstruction from video priors (Huang et al., 4 Mar 2026). That work is close in spirit to a wild setting because it uses monocular video priors and emphasizes zero-shot generalization, but it explicitly does not introduce a dedicated ArtHOI-Wild benchmark. Instead, it targets articulated-object scenes such as opening fridges, cabinets, and microwaves, and frames the problem as inverse rendering from monocular video. Its key components are flow-based part segmentation and a decoupled reconstruction pipeline in which object articulation is reconstructed first and human motion is then synthesized conditioned on object states. The paper reports improvements in X-CLIP, Contact%, Foot Sliding, Penetration%, and articulation rotation error relative to baselines, but also notes limitations including single-part articulated objects, tracking failures, long temporal sequences, and a fixed camera assumption (Huang et al., 4 Mar 2026).

These adjacent works help delimit the meaning of ArtHOI-Wild. In the present context, it refers primarily to an open-vocabulary in-the-wild 3D HOI benchmark from single images rather than to a test-only recognition protocol or a zero-shot articulated video synthesis framework. At the same time, all three directions share a common research pressure: they reduce reliance on conventional closed-world supervision and expose the fragility of current systems under open-vocabulary, noisy, or real-world conditions.

7. Limitations, scope, and significance

The benchmark’s limitations are explicitly acknowledged in the source material (Wen et al., 20 Mar 2025). Annotation is expensive and difficult, as reflected by the 17% pass rate. Current reconstruction still fails in hard occlusion cases, especially under severe body self-occlusion and when objects lie in ambiguous occluded regions, which makes contact assignment difficult. The annotation pipeline also depends heavily on strong 3D reconstruction tools, so future gains are tied to progress in more robust 3D human/object reconstruction and image-to-3D systems. In addition, the PointLLM and ChatPose experiments indicate that current large models remain insufficient for fine-grained 3D HOI reasoning.

These limitations are important for interpreting the benchmark correctly. ArtHOI-Wild is not a claim that in-the-wild 3D HOI has been solved; rather, it is a curated, manually filtered, open-vocabulary test set that exposes the difficulty of the problem. A plausible implication is that its value lies less in scale alone than in the combination of real-image diversity, explicit contact annotations, and evaluation settings that make category-specific training less informative.

Within HOI research more broadly, the benchmark’s significance is that it provides a practical path from rich 2D HOI corpora to 3D evaluation on real-world images (Wen et al., 20 Mar 2025). It thereby shifts the center of gravity of 3D HOI research away from narrow indoor setups and toward in-the-wild, category-diverse, open-vocabulary human-object interaction understanding and reconstruction. In conjunction with the training-free HOI recognition results of (Ai, 2024) and the articulated 4D reconstruction perspective of (Huang et al., 4 Mar 2026), ArtHOI-Wild can be understood as part of a larger movement toward evaluating HOI systems under weaker supervision, greater object diversity, and more realistic interaction structure.