ArtHOI: Articulated Human-Object Interaction

• ArtHOI is a benchmark and set of frameworks focusing on artistic, few-shot, open-vocabulary human–object interaction reasoning and 4D reconstruction.
• It incorporates diverse methods including Bongard-HOI, HOICLIP, and video diffusion models to evaluate compositional and context-dependent HOI synthesis.
• The approach emphasizes explicit geometric reasoning, zero-shot generation, and foundation-model optimization to achieve realistic articulated interactions.

In the cited literature, “ArtHOI” appears in several closely related senses: as a proposed benchmark direction for artistic or stylized human–object interactions, and as the title of two 2026 frameworks for articulated 4D human–object interaction synthesis and reconstruction (Jiang et al., 2022, Huang et al., 4 Mar 2026, Wang et al., 26 Mar 2026). This suggests a shared scope centered on human–object interaction under few-shot, open-vocabulary, articulated, or monocular constraints, rather than a single fixed task definition.

1. Conceptual origins in HOI reasoning

A direct antecedent of ArtHOI is Bongard-HOI, a benchmark for few-shot visual reasoning about human–object interactions using natural images. It imports two core properties from classical Bongard problems: few-shot concept learning and context-dependent reasoning. Each problem contains 6 positive images, 6 negative images, and a query, and the underlying concept is an HOI triplet $c=\langle s,a,o\rangle$ with $s=\text{person}$. Positives and negatives are constructed with hard negatives that keep the object category fixed and change only the action label, so mere object recognition is insufficient (Jiang et al., 2022).

The same source explicitly frames this design as relevant to “ArtHOI”: in artistic or stylized images, HOIs can be rare, unusual, or stylized; fixed HOI labels or large training sets for every $(\text{verb}, \text{object})$ pair are inadequate; and the same pose may need to be interpreted as “hold”, “throw”, or “offer” depending on context. Bongard-HOI is therefore described as a testbed for the kind of human-like, concept-level HOI reasoning needed for an ArtHOI benchmark built from paintings, comics, or other non-standard imagery (Jiang et al., 2022).

2. Few-shot, compositional, and context-dependent evaluation

Bongard-HOI formalizes each instance as a 2-way, 6-shot problem. The support set is $S=P\cup N$, where $P$ contains 6 positive examples and $N$ contains 6 negative examples, and the query is $(I_q,y_q)$ with $y_q\in\{0,1\}$. The benchmark also defines four compositional test regimes: seen action, seen object; seen action, unseen object; unseen action, seen object; and unseen action, unseen object. These splits directly measure whether a model can recombine known verbs and objects into new HOI concepts (Jiang et al., 2022).

Quantitatively, the benchmark exposes a large human–machine gap. The HOITrans oracle reaches 62.46% average accuracy; the best meta-learning model, Meta-Baseline with ImageNet pretraining, reaches 55.82% with detector boxes and 58.30% with ground-truth boxes; and non-episodic baselines such as CNN-Baseline and WReN-BP remain at or near chance. By contrast, MTurk amateurs achieve 91.42% average accuracy, with 87–95% across all four test splits and little drop on unseen action/object variants (Jiang et al., 2022).

A recurrent implication for ArtHOI follows directly from these results: standard HOI detection training does not furnish the context-dependent, few-shot concept induction required when interactions are sparse, stylized, or compositionally novel. The same query image can be positive in one Bongard problem and negative in another, depending on the support-set-defined concept, and current models generally fail at this contextual reinterpretation (Jiang et al., 2022).

3. Detection and structured prediction foundations

Several HOI detection methods provide the 2D inference substrate on which broader ArtHOI systems can build. HOICLIP extracts prior knowledge from CLIP through an interaction decoder, a knowledge integration block that fuses cross-attention from CLIP and detector features, a verb classifier built from visual semantic arithmetic, and a training-free enhancement based on global HOI predictions. It reports 34.69 mAP on HICO-Det, 63.5 AP on V-COCO, and +4.04 mAP over GEN-VLKT in the NF-UC zero-shot setting (Ning et al., 2023).

FreeA addresses the annotation bottleneck by generating latent HOI labels without manual HOI supervision. Its pipeline consists of Candidate Image Construction (CIC), Human–Object Potential Interaction Mining (PIM), and Human–Object Interaction Inference (HII); it aligns human–object crops with HOI text templates, applies a Prior Knowledge-based Mask, and amplifies correlated verbs via Interaction Correlation Matching. In no-label mode it reaches 16.96 mAP on HICO-DET and 30.82 $AP^{S1}_{role}$ on V-COCO (Liu et al., 2024).

HOI-IDiff reframes HOI detection as a generative structured prediction problem by representing the output for each human–object pair as an HOI image $I^{hoi}\in\mathbb{R}^{H\times W\times 2}$, where $s=\text{person}$0 is the number of object categories and $s=\text{person}$1 is the number of interaction categories. It introduces a customized multinomial diffusion process and slice patchification tailored to these tensors, and reports 47.71 mAP default and 50.56 mAP known-object on HICO-DET, as well as 73.4 and 76.1 on V-COCO (Hui et al., 23 Mar 2025).

Taken together, these methods suggest that ArtHOI is not only a reconstruction problem. It also depends on robust HOI detection under zero-shot, weak-label, and generative structured-prediction regimes.

4. Open-vocabulary 3D HOI and interaction representations

Open3DHOI extends HOI into open-vocabulary in-the-wild 3D reconstruction. The dataset contains 2,561+ final images, 2,561 object instances, 133 object categories, 3,671 interaction instances, 120 distinct interaction/action labels, 370 unique 3D human-object pairs, and 34 annotated SMPL-X contact parts. Its accompanying Gaussian-HOI optimizer is training-free and category-agnostic: it initializes human Gaussians from SMPL-X, object Gaussians from single-image mesh reconstruction, and optimizes rendering, contact, collision, and ordinal depth losses. On object pose metrics it improves PHOSA from 0.39 / 77.79 / 0.95 / 49.1 to 0.16 / 38.44 / 0.41 / 19.3 for scale, translation, rotation, and Chamfer distance, and reduces the interaction metric $s=\text{person}$2 from 0.431 to 0.181 (Wen et al., 20 Mar 2025).

At a finer interaction scale, CHOIR introduces a coarse, versatile and fully differentiable field for hand–object interaction. It combines object BPS unsigned distances, distances from BPS points to 32 MANO anchors, and 32 multivariate Gaussians that encode dense contact maps. The associated JointDiffusion model is used for both refinement and synthesis, increasing the contact F1 score by 5% for refinement and decreasing the sim. displacement by 46% for synthesis (Morales et al., 2024).

A supervised precursor for articulated full-body interaction is CHAIRS, a motion-captured f-AHOI dataset with 46 participants, 81 articulated and rigid sittable objects, 1390 sequences, and 17.3 hours of interaction. In the known-geometry setting, its interaction-aware model improves Chamfer distance from 95.40 mm for CHORE to 72.30 mm, and reduces translation error from 87.58 mm to 66.23 mm (Jiang et al., 2022).

5. Zero-shot 4D generation and long-horizon synthesis

AnchorHOI treats text-driven 4D HOI generation as a zero-shot problem and distills priors from both image diffusion models and video diffusion models. Its two central intermediates are anchor NeRFs, which encode static interaction composition, and anchor keypoints, which encode body motion and contact. The method assumes no paired text–HOI training data and reports GPT-4V preference of over 70% in 4D contact and >50% overall, while user studies rate outputs at approximately 4.7–4.9 out of 5 across semantic alignment, contact realism, penetration, motion quality, and overall quality (Dai et al., 16 Dec 2025).

ARDHOI addresses the long-sequence regime. It learns continuous 16-frame HOI tokens with a contrastive VAE, models context with a 27-layer Mamba2 encoder, and predicts the next token with an MLP-based denoiser in an autoregressive diffusion process. On OMOMO it reaches FID 0.826, improving over HOI-Diff’s 1.075; on BEHAVE it reaches FID 1.872; and it reduces average inference time per sentence to 1.25 s with 260.9 B FLOPs (Geng et al., 21 Mar 2025).

These generative lines clarify an important boundary condition for ArtHOI. Zero-shot HOI generation from diffusion priors is feasible, but rigid-object assumptions and weak contact reasoning remain limiting factors unless the model includes explicit interaction-aware intermediates or structured latent priors.

6. ArtHOI as zero-shot articulated 4D synthesis from video priors

The 2026 framework “ArtHOI: Articulated Human-Object Interaction Synthesis by 4D Reconstruction from Video Priors” formulates articulated HOI synthesis as a 4D reconstruction problem from monocular video priors. A video diffusion model first generates a monocular RGB sequence from text, and the method then reconstructs a full 4D scene with a human SMPL-X body, an articulated object represented by 3D Gaussians, and time-varying object-part transforms. Its two key designs are flow-based part segmentation, which uses CoTracker together with SAM or SAM2 to separate dynamic from static regions and derive quasi-static binding near the articulation boundary, and a decoupled reconstruction pipeline that first recovers object articulation and then refines human motion conditioned on the reconstructed object states. On articulated scenes it reports X-CLIP 0.244, Contact 75.64%, Penetration 0.08%, and articulated-object rotation errors of 6.71° mean, 21.41° max, and 0.58° min; for a 60–120 frame video the total runtime is about 30 minutes on an A6000 GPU, including video generation (Huang et al., 4 Mar 2026).

This formulation explicitly targets a weakness of earlier zero-shot HOI systems: most were confined to rigid-object manipulation and lacked explicit 4D geometric reasoning. ArtHOI instead reconstructs articulation, contact, and temporal coherence from the generated 2D video itself, using differentiable rendering, tracking consistency, temporal smoothness, kinematic contact, and collision losses (Huang et al., 4 Mar 2026).

7. ArtHOI as monocular 4D reconstruction from foundation-model priors

A second 2026 work, “ArtHOI: Taming Foundation Models for Monocular 4D Reconstruction of Hand-Articulated-Object Interactions,” addresses a complementary setting: reconstructing a hand manipulating an unknown articulated object from a single monocular RGB video, without pre-scanned geometry or category templates. The framework integrates SAM2, Video-Depth-Anything, UniDepthV2, DiffuEraser, HunYuan3D, FoundationPose, CoTracker3, PartField, WiLoR, and Qwen-VL-Max. Its central contributions are Adaptive Sampling Refinement (ASR), which optimizes a normalized object mesh’s metric scale and pose by sampling candidate scales around a coarse estimate and selecting the best silhouette IoU, and an MLLM-guided hand-object alignment stage that converts contact reasoning and finger identities into constraints for hand/object mesh composition. The method also introduces ArtHOI-RGBD and ArtHOI-Wild. On CD Drive it reports CD 3.33 mm, MSSD 9.71 mm, F10 96.01, and F5 78.75; its HOI alignment reaches $s=\text{person}$3 values of 0.029 on ArtHOI-RGBD, 0.022 on RSRD, and 0.039 on ArtHOI-Wild; and ASR achieves 100% success across all datasets (Wang et al., 26 Mar 2026).

The same work also makes clear why foundation-model priors alone are insufficient. HunYuan3D produces normalized meshes with unknown metric scale; monocular depth is noisy; FoundationPose performance degrades when mesh and depth are inconsistent; and independently reconstructed hand and object meshes lead to floating contact or penetration. ArtHOI resolves these mismatches through explicit optimization, but the full pipeline on a 150-frame sequence still takes about 1 hour, with part motion optimization dominating runtime (Wang et al., 26 Mar 2026).

A recurrent misconception is that stronger closed-vocabulary detectors or larger diffusion priors are sufficient for ArtHOI. The literature instead shows that standard HOI detection training does not yield context-dependent few-shot concept induction (Jiang et al., 2022), rigid-object zero-shot generation does not solve articulated interaction synthesis (Huang et al., 4 Mar 2026), and foundation-model priors remain physically inconsistent unless refined by geometry-aware optimization (Wang et al., 26 Mar 2026). This suggests that ArtHOI is best understood as a research program organized around compositional reasoning, open-vocabulary geometry, explicit contact modeling, and temporally coherent articulation.