---
title: 'ArtHOI-Wild: In-the-Wild 3D HOI Benchmark'
url: https://www.emergentmind.com/topics/arthoi-wild
type: topic
---

# ArtHOI-Wild: In-the-Wild 3D HOI Benchmark

ArtHOI-Wild is an open-vocabulary, in-the-wild 3D human-object interaction benchmark associated in the literature here with **Open3DHOI**, a dataset and annotation pipeline for reconstructing fine-grained 3D humans, objects, and their interactions from single images [2503.15898]. It was introduced to address a central limitation of prior 3D HOI research: existing benchmarks are primarily indoor, often multi-view, and restricted to a small set of object categories, whereas large-scale 2D HOI resources already contain diverse, real-world, open-vocabulary interactions but lack 3D structure. In that sense, ArtHOI-Wild denotes a shift from category-limited indoor reconstruction toward single-image, real-image, open-vocabulary 3D HOI understanding, while remaining distinct from related uses of “wild” in training-free HOI recognition and zero-shot articulated HOI synthesis [2408.05772].

## 1. Definition and positioning

ArtHOI-Wild corresponds here to **Open3DHOI**, which is presented as the **first in-the-wild, open-vocabulary 3D HOI dataset based on real-world images** [2503.15898]. Its stated role is to serve as a **test set / benchmark** rather than a large-scale training corpus. The benchmark is designed to better represent real-world HOIs “with any objects and interactions,” and to support future **open-vocabulary** 3D HOI research.

The benchmark is motivated by the limitations of earlier 3D HOI datasets. **BEHAVE**, **InterCap**, **IMHOI**, and **PROX-S** are described as mostly **indoor** and often based on **multi-view RGB-D sequences**. **WildHOI** and **3DIR** are in-the-wild, but they have **few object categories** and rely on **fixed / unreal CAD models**, which limits realism and category diversity. By contrast, the new benchmark is constructed from real 2D HOI imagery and recent single-image reconstruction methods, with the explicit goal of supporting generalization to real-world scenes with a wide range of objects [2503.15898].

A concise comparison reported in the source material is shown below.

| Dataset | Objects | Actions |
|---|---:|---:|
| BEHAVE | 10 | no action labels |
| InterCap | 10 | no action labels |
| WildHOI | 8 | no action labels |
| 3DIR | 21 | 17 actions |
| PROX-S | 40 | 17 actions |
| Ours | 133 | 120 actions |

The benchmark also includes richer annotation types, specifically **human pose, object pose, contact, and 2D HOI annotations**. This suggests that ArtHOI-Wild is intended not only for object placement or body fitting, but for the broader study of spatial interaction structure, contact reasoning, and semantic HOI understanding.

## 2. Data sources and annotation pipeline

The dataset is built through a **coarse-to-fine manual annotation system** on top of existing 2D HOI datasets and modern reconstruction tools [2503.15898]. Source images are selected from **HAKE-Large** with **12k+ images**, **SWIG-HOI** with **3k+ images**, plus some web-collected images, forming an initial database of **15k+ images**.

For each image, coarse reconstruction first estimates the human using **OSX** and the object using **InstantMesh**. Because in-the-wild images often contain severe occlusion, the pipeline applies **occlusion completion** using **amodal completion** and **Stable Diffusion 1.5** for **inpainting** to obtain a complete object image before reconstruction. A rough 3D spatial relationship is then estimated by combining **ZoeDepth**, human and object masks, depth point clouds, and sampled mesh point clouds, from which the system infers rough **position, size, and placement** of the human-object pair.

The source material emphasizes that this rough reconstruction is useful for annotation but often contains **mesh collision**, **inaccurate scales**, **inaccurate positions**, and **object pose errors from InstantMesh**. The fine annotation stage therefore introduces two dedicated tools. The **filtering tool** asks annotators to inspect projected SMPL-X meshes, render objects from six viewpoints, and correct masks manually when inpainting mask completion fails. The pipeline also splits the **SMPL-X body into 34 regions** to annotate **contact areas** at the body-part level rather than the vertex level. The **3D interaction tool** combines a **Blender-based tool** for object **translation, rotation, and scale** adjustment with a **web-based fine annotation tool** based on **ImageNet3D**, where multi-view projections help annotators judge 3D interaction more accurately [2503.15898].

After manual filtering and annotation, the retained subset contains **2.5k+ images**. The final dataset contains **370 3D human-object pairs**, **2,561 objects** in **133 categories**, and **3,671 interactions** in **120 categories**. Only **single-person** images are used in the final dataset, except that **63 images** contain multiple objects interacting with one person and are split into separate HOI pairs. The overall pass rate from the 15k source images is **17%**, a figure explicitly presented as evidence of the difficulty of reconstructing 3D HOIs from in-the-wild images [2503.15898].

## 3. Scale, diversity, and annotation semantics

The benchmark’s scale is characterized by **2.5k+ final annotated images**, **2,561 objects**, **133 object categories**, **3,671 interactions**, and **120 action categories** [2503.15898]. The source material further notes that the object distribution spans many unusual categories, including **food and animals**, which are rare in previous 3D HOI datasets. Object categories are organized using a **WordNet-style taxonomy**, and object size varies greatly across categories and even within the same category.

The paper computes object size using the **volume function from Trimesh**, then takes the **cube root** as size. It also states that the dataset includes **abnormal HOIs** because it originates from real HOI images, giving **standing on a chair** as an example. A plausible implication is that the benchmark does not restrict itself to canonical affordance patterns; instead, it preserves the irregularity of real-world imagery, which is consistent with the “in-the-wild” designation.

The resource also provides a **template library** with **58 object categories** and **212 templates**. The source explicitly states that this library is **not used to build Open3DHOI**, but is mentioned as useful for future work. This distinction matters because the benchmark is constructed from reconstructed real images rather than template instantiation.

A common misconception is that an in-the-wild 3D HOI dataset must necessarily be multi-view or sensor-based to be reliable. The benchmark’s design explicitly takes the opposite route: it lifts real 2D HOI images into 3D using modern reconstruction tools and then relies on coarse-to-fine manual correction. Another misconception is that open-vocabulary here means unconstrained automatic labeling; in fact, the pipeline is annotation-heavy and filtered aggressively, as indicated by the **17%** pass rate [2503.15898].

## 4. Gaussian-HOI optimizer and reconstruction formulation

The principal reconstruction method associated with the benchmark is a **training-free 3D HOI optimizer** based on **3D Gaussian Splatting**, referred to as the **HOI-Gaussian optimizer** or **Gaussian-HOI optimizer** [2503.15898]. Its stated purpose is to reconstruct the spatial interactions between humans and objects while learning contact regions.

The method models the human and object as Gaussian sets, with human Gaussians \( g_h \), object Gaussians \( g_o \), and combined HOI Gaussians \( g_{hoi} \). The rendering equation given in the source is
\[
C=\sum_{n=1}^{|\mathcal{N}|} c_n \alpha_n^{\prime} \prod_{j=1}^{n-1}\left(1-\alpha_j^{\prime}\right),
\]
with
\[
\alpha_n^{\prime}=\alpha_n  \exp \left(-\frac{1}{2}\left(\boldsymbol{x}^{\prime}-\boldsymbol{\mu}_n^{\prime}\right)^{\top} \boldsymbol{\Sigma}_n^{\prime-1}\left(\boldsymbol{x}^{\prime}-\boldsymbol{\mu}_n^{\prime}\right)\right).
\]

Human Gaussians are initialized from **SMPL-X vertices**, object Gaussians from **object mesh vertices**, and the method follows **GauHuman** for human pose refinement and LBS offsets while introducing a learnable object pose parameter \( W_{obj} \). The combined interaction Gaussian is defined through transformed human and object Gaussians:
\[
p^{hoi}=p^h \oplus p^o,\quad \Sigma^{hoi}=\Sigma^h \oplus \Sigma^o
\]
with
\[
p^h=R^h p^{h'}+t^h,\quad p^o=R^o s^o p^{o'}+t^o
\]
and
\[
\Sigma^h=R^h \Sigma^{h'} R^{h\top},\quad \Sigma^o=R^o \Sigma^{o'} R^{o\top}.
\]

A key novelty is the use of **Gaussian opacity** and depth ordering to infer likely contact regions. The source describes the intuition as follows: regions where human and object occlude each other tend to have **lower opacity** in rendering; pixels not visible due to occlusion in the original image imply likely interaction areas; and points on the back side relative to the camera are initialized with **very low opacity**, so front-facing visible regions dominate rendering. The resulting per-Gaussian **contact interaction score** is
\[
c=w_{\alpha} \cdot {Norm}(\alpha^h)+w_d \cdot d_C(p^h, p^o)^h.
\]
In practice, opacity is first used to find candidate contact areas, which are then narrowed further using a **distance threshold** between human and object [2503.15898].

The loss is separated into **rendering loss** and **HOI loss**. The rendering term uses **L1**, **L2**, **SSIM**, and **LPIPS**, and is computed separately for \( g^h \), \( g^o \), and \( g^{hoi} \):
\[
\mathcal{L}_r=w_h \mathcal{L}_r^h+w_o  \mathcal{L}_r^o+w_{hoi} \mathcal{L}_r^{hoi}.
\]
The HOI term consists of **contact loss** \( \mathcal{L}_{cont} \), **collision loss** \( \mathcal{L}_{colli} \), and **ordinal depth loss** \( \mathcal{L}_{depth} \):
\[
\mathcal{L}_{hoi} = \mathcal{L}_{cont}  + \mathcal{L}_{colli} + \mathcal{L}_{depth},
\]
and the final objective is
\[
\mathcal{L} = w_r \cdot \mathcal{L}_{r} + w_{hoi} \cdot \mathcal{L}_{hoi}.
\]

The optimizer runs for **160 iterations**: the first **100 iterations** use only rendering loss, and later iterations add HOI loss. The source explains this staged optimization as a way to first align image appearance and then refine interaction geometry [2503.15898].

## 5. Tasks, evaluation protocol, and reported performance

The main reconstruction benchmark is conducted on the **Open3DHOI test set**, with the explicit note that the method is **training-free** and that object meshes before manual annotation are given [2503.15898]. For fairness, both the proposed method and **PHOSA** initialize human body parameters from the dataset, and **PHOSA** is used as the principal baseline because the dataset is open-vocabulary and category-specific training would be unfair.

The evaluation reports object pose metrics—**Scale**, **Translation (cm)**, **Rotation**, and **Chamfer Distance**—and introduces an interaction metric, the **Collision-Contact score**
\[
C_o^2=\operatorname{Sig}(\operatorname{Colli}(h,o))+\operatorname{Sig}(\sum_i^p(\operatorname{Cont}(i) / Size )),
\]
where \( \operatorname{Colli}(h,o) \) is human-object collision, \( \operatorname{Cont}(i) \) is the Chamfer distance between each human body part and the object, and \( Size \) is the object mesh size.

On object pose metrics, the reported numbers are:

| Method | Scale | Translation (cm) | Rotation | Chamfer |
|---|---:|---:|---:|---:|
| PHOSA | 0.39 | 77.79 | 0.95 | 49.1 |
| Ours w/o HOI Loss | 0.25 | 38.66 | 0.45 | 16.9 |
| Ours | 0.16 | 38.44 | 0.41 | 19.3 |

On collision/contact metrics, the reported numbers are:

| Method | \(C_o^2\) | Collision | Contact |
|---|---:|---:|---:|
| PHOSA | 0.431 | 0.105 | 0.326 |
| Coarse Recon | 0.248 | 0.083 | 0.165 |
| Ours Gs only | 0.287 | 0.136 | 0.151 |
| Gs depth | 0.216 | 0.080 | 0.136 |
| Gs colli | 0.189 | 0.046 | 0.143 |
| Gs depth colli | 0.188 | 0.045 | 0.143 |
| Gs depth colli cont | 0.181 | 0.053 | 0.128 |

The interpretation given in the source is that Gaussian optimization alone mainly improves appearance alignment, while depth, collision, and contact losses are necessary for better interaction quality; the contact loss improves contact quality, though it may slightly increase collision in some cases [2503.15898].

Beyond reconstruction, the benchmark proposes **3D HOI Understanding** and **HOI Pose Chat** as new tasks. For 3D HOI understanding, **PointLLM-7B** is queried with prompts such as “What is the action between the person and the [object]?” and evaluated with **Top-1 accuracy**. The reported results are **0.47** for **Action / object prompt**, **0.20** for **Action without object**, **0.32** for **Object prompt**, and **0.31** for **Object without action**. For contact prediction, adding 3D information improves results from **Micro F1 0.6118** to **0.6207**, **Hamming Loss 0.0874** to **0.0844**, and **Jaccard Index 0.4303** to **0.4561**. For **ChatPose**, the reported errors are **MPJPE 103.6, MPVPE 131.2** with **Action**, **MPJPE 105.2, MPVPE 133.5** with **w/o Action**, and **MPJPE 103.4, MPVPE 130.9** with **Action + Object**. The paper explicitly concludes that current general 3D/LLM systems are still weak at precise 3D HOI reasoning [2503.15898].

## 6. Relation to adjacent “wild” HOI settings

The term ArtHOI-Wild can be confused with other recent HOI directions that also emphasize out-of-distribution generalization, test-time-only inference, or zero-shot behavior. The source material distinguishes at least two nearby but non-identical lines of work.

First, the paper **“An analysis of HOI: using a training-free method with multimodal visual foundation models when only the test set is available, without the training set”** studies a deliberately **training-free** HOI recognition regime in which only the test dataset is available [2408.05772]. That work is framed as relevant to an **ArtHOI-Wild-like setting** because it asks what can be done when there is **only test set and no training data at all**, using multimodal visual foundation models such as **CLIP**, **BLIP**, and **BLIP-2**. Its core task is not 3D reconstruction but prompt-based recognition of the verb in \(\langle human, object, verb \rangle\) triplets. The reported pattern is that **GT** pairing performs best, **GT-R** degrades, and **GroundingDINO** degrades further; for **blip2_coco_vitH/14@364px**, the corresponding **full mAP** values are **49.56**, **38.86**, and **19.71**. The interpretation given is that current foundation models exhibit some open-vocabulary HOI capability but are not robust enough to solve HOI recognition in the wild without training when pairing quality deteriorates [2408.05772].

Second, the paper **“ArtHOI: Articulated Human-Object Interaction Synthesis by 4D Reconstruction from Video Priors”** introduces **ArtHOI**, a **zero-shot articulated human-object interaction synthesis framework** based on **4D reconstruction from video priors** [2603.04338]. That work is close in spirit to a wild setting because it uses **monocular video priors** and emphasizes **zero-shot generalization**, but it explicitly **does not introduce a dedicated ArtHOI-Wild benchmark**. Instead, it targets articulated-object scenes such as **opening fridges**, **cabinets**, and **microwaves**, and frames the problem as inverse rendering from monocular video. Its key components are **flow-based part segmentation** and a **decoupled reconstruction pipeline** in which object articulation is reconstructed first and human motion is then synthesized conditioned on object states. The paper reports improvements in **X-CLIP**, **Contact%**, **Foot Sliding**, **Penetration%**, and articulation rotation error relative to baselines, but also notes limitations including **single-part articulated objects**, **tracking failures**, **long temporal sequences**, and a **fixed camera assumption** [2603.04338].

These adjacent works help delimit the meaning of ArtHOI-Wild. In the present context, it refers primarily to an **open-vocabulary in-the-wild 3D HOI benchmark from single images** rather than to a test-only recognition protocol or a zero-shot articulated video synthesis framework. At the same time, all three directions share a common research pressure: they reduce reliance on conventional closed-world supervision and expose the fragility of current systems under open-vocabulary, noisy, or real-world conditions.

## 7. Limitations, scope, and significance

The benchmark’s limitations are explicitly acknowledged in the source material [2503.15898]. **Annotation is expensive and difficult**, as reflected by the **17%** pass rate. **Current reconstruction still fails in hard occlusion cases**, especially under severe body self-occlusion and when objects lie in ambiguous occluded regions, which makes contact assignment difficult. The annotation pipeline also depends heavily on strong 3D reconstruction tools, so future gains are tied to progress in more robust 3D human/object reconstruction and image-to-3D systems. In addition, the PointLLM and ChatPose experiments indicate that current large models remain insufficient for fine-grained 3D HOI reasoning.

These limitations are important for interpreting the benchmark correctly. ArtHOI-Wild is not a claim that in-the-wild 3D HOI has been solved; rather, it is a curated, manually filtered, open-vocabulary test set that exposes the difficulty of the problem. A plausible implication is that its value lies less in scale alone than in the combination of real-image diversity, explicit contact annotations, and evaluation settings that make category-specific training less informative.

Within HOI research more broadly, the benchmark’s significance is that it provides a practical path from rich 2D HOI corpora to 3D evaluation on real-world images [2503.15898]. It thereby shifts the center of gravity of 3D HOI research away from narrow indoor setups and toward **in-the-wild, category-diverse, open-vocabulary human-object interaction understanding and reconstruction**. In conjunction with the training-free HOI recognition results of [2408.05772] and the articulated 4D reconstruction perspective of [2603.04338], ArtHOI-Wild can be understood as part of a larger movement toward evaluating HOI systems under weaker supervision, greater object diversity, and more realistic interaction structure.

Source: https://www.emergentmind.com/topics/arthoi-wild