Human-Object Interaction Learning

• Human-Object Interaction Learning is a multidisciplinary field that models dynamic engagements between humans and objects using spatial, temporal, and contextual reasoning.
• It employs methodologies such as keypoint detection, mixed supervision, and language grounding to create accurate interaction representations across images, 3D, and 4D domains.
• Recent advances focus on overcoming challenges like long-tail distributions, occlusion, and negative-pair explosion by integrating multimodal reasoning and continual learning strategies.

Searching arXiv for recent and foundational HOI/HOIL papers to ground the article. Searching arXiv for "human-object interaction learning" and related open-vocabulary / mixed-supervision / compositional / 4D HOI works. Human-Object Interaction Learning (HOIL) denotes the study of how models acquire representations of human–object relations, from the classical task of predicting localized triplets such as $\langle$human, verb, object$\rangle$ to broader settings that require compositional generalization, open-vocabulary reasoning, temporal modeling, 3D or 4D contact understanding, and physics-based control. In image-based formulations, HOIL is usually instantiated as HOI detection, where a prediction is correct only when both human and object localizations are correct and the interaction label is correct; more recent work expands the paradigm to include mixed supervision, open-world adaptation, multimodal chain-of-thought reasoning, and embodied imitation (Wang et al., 2024, Zhang et al., 15 Aug 2025). Taken together, these lines of work show a field moving from instance-centric pair scoring toward spatially grounded interaction representations, semantics-aware classifiers, vision–language prompting, and contact-aware dynamical models (Wang et al., 2020, Wen et al., 30 Nov 2025).

1. Historical development and scope

Early work that would now be recognized as HOIL emphasized fine-grained event structure rather than large-scale image benchmarks. “Fine-grained Event Learning of Human-Object Interaction with LSTM-CRF” modeled 3D motion-capture sequences as structured tuples $$(\text{Subject}, \text{Object}, \text{Locative}, \text{Preposition}, \text{Verb})$$, combined an LSTM with a tree-CRF, and reduced invalid outputs from $20\%$ to $3\%$, improving precision from $39\%$ for LSTM-W to $43\%$ for LSTM-CRF (Do et al., 2017). This formulation already exposed themes that remain central: temporal slicing, output-structure constraints, and the need to separate plausible from implausible human–object events.

The mainstream image-based literature subsequently standardized HOI detection as the prediction of localized interaction triplets from still images. The survey “A Review of Human-Object Interaction Detection” organizes this literature around two-stage methods, one-stage end-to-end methods, zero-shot learning, weakly supervised learning, and the use of large-scale LLMs, with HICO-DET and V-COCO as the dominant benchmarks (Wang et al., 2024). Within that taxonomy, HOIL initially referred mostly to visual recognition; later papers explicitly broadened it to a learning paradigm that must reason, generalize, and adapt under open-vocabulary queries, multimodal inputs, and diverse contexts (Zhang et al., 15 Aug 2025).

This broader usage is now common. In open-world detectors, HOIL emphasizes structured reasoning and cross-modal grounding rather than closed-set label assignment (Zhang et al., 15 Aug 2025). In 4D reconstruction and imitation, HOIL further encompasses contact timing, object motion relative to the body, and physically plausible control (Wen et al., 30 Nov 2025, Liang et al., 8 Mar 2026). A plausible implication is that “HOIL” has become an umbrella term spanning recognition, reasoning, reconstruction, and control, unified by the requirement to model how humans engage objects rather than merely detect co-occurrence.

2. Core formulations and representational shifts

The standard image-based output remains the localized interaction triplet, but recent work has changed how that triplet is represented. Classical instance-centric pipelines score all human–object pairs using appearance features and coarse spatial encodings. “Learning Human-Object Interaction Detection using Interaction Points” argues that appearance features alone are insufficient for complex HOI scenes and reformulates HOI detection as a keypoint detection and grouping problem: the interaction point $p$ is defined as the midpoint between human and object centers, with vectors satisfying $p+v=h$ and $p-v=o$; grouping is then performed with soft geometric constraints rather than exhaustive pair scoring. This fully convolutional anchor-free formulation reached $\rangle$0 mAP_role on V-COCO and $\rangle$1 default full mAP on HICO-DET, while also showing that interaction boxes, corner-distance constraints, and center-pooling were responsible for an overall $\rangle$2 mAP_role gain over the baseline on V-COCO (Wang et al., 2020).

A complementary line treats pair discovery itself as a first-class problem. “Interactiveness Field in Human-Object Interactions” introduces an object-centric bimodal prior: for a fixed object, paired humans are usually either mostly non-interactive or mostly interactive, with the former much more frequent. It formalizes an “interactiveness field” with cardinality and difference constraints, then fuses the resulting interactiveness score $\rangle$3 with the verb score $\rangle$4. The method reaches $\rangle$5 full mAP on HICO-DET and $\rangle$6 role AP on V-COCO Scenario 1/2, while ablations show a $\rangle$7 mAP drop without the interactiveness field module (Liu et al., 2022). This reframing suggests that HOIL is not only about relation classification, but also about suppressing the combinatorial negative space created by human–object pairing.

Another representational shift occurs at the classifier rather than the detector. “The Overlooked Classifier in Human-Object Interaction Recognition” keeps the backbone untouched, initializes classifier weights with language embeddings of HOI classes, and replaces independent binary losses with LSE-Sign, $\rangle$8. In detection-free HOI classification it reaches $\rangle$9 mAP on HICO with ViT-B/16, and when transferred to instance-level HOI detection by attaching an off-the-shelf detector, it reaches $$(\text{Subject}, \text{Object}, \text{Locative}, \text{Preposition}, \text{Verb})$$0 mAP on HICO-DET without additional fine-tuning (Jin et al., 2022). The underlying claim is not that localization is unimportant, but that classifier geometry and long-tail multi-label learning are often the real bottlenecks.

Spatial context has become another explicit representational target. “ContextHOI: Spatial Context Learning for Human-Object Interaction Detection” adds a context branch parallel to the instance branch and supervises it through feature-level, region-level, and instance-level spatial contrastive losses, together with CLIP-guided semantic exploration. The method reaches $$(\text{Subject}, \text{Object}, \text{Locative}, \text{Preposition}, \text{Verb})$$1 full mAP on HICO-DET with ResNet-50 and $$(\text{Subject}, \text{Object}, \text{Locative}, \text{Preposition}, \text{Verb})$$2 mAP on the occlusion-heavy HICO-ambiguous subset, whereas several instance-centric baselines degrade sharply on that benchmark (Jia et al., 2024). This directly counters the common misconception that HOI is fully determined by cropped foreground appearance.

3. Supervision regimes, long-tail learning, and continual acquisition

A central difficulty in HOIL is annotation scarcity under a highly long-tailed label space. “Detecting Human-Object Interaction with Mixed Supervision” addresses the realistic regime where fully supervised and weakly supervised images coexist. Its MX-HOI pipeline combines a two-branch predicate predictor with momentum-independent learning, which uses separate momentum buffers for weak and full supervision so that region-level and image-level objectives do not contaminate each other. On HICO-DET, mixed supervision with WS/FS $$(\text{Subject}, \text{Object}, \text{Locative}, \text{Preposition}, \text{Verb})$$3 reaches $$(\text{Subject}, \text{Object}, \text{Locative}, \text{Preposition}, \text{Verb})$$4 full mAP, which is $$(\text{Subject}, \text{Object}, \text{Locative}, \text{Preposition}, \text{Verb})$$5 of the fully supervised accuracy of $$(\text{Subject}, \text{Object}, \text{Locative}, \text{Preposition}, \text{Verb})$$6, and the authors show that naïve mixing without momentum-independent learning can even underperform weak-only training (Kumaraswamy et al., 2020).

Synthetic or compositional data generation is another major strategy. “Detecting Human-Object Interaction via Fabricated Compositional Learning” introduces an object fabricator that generates feature-level object representations conditioned on the verb feature and noise, then composes them into synthetic HOI samples. On HICO-DET with the DRG detector, the method improves default full mAP from $$(\text{Subject}, \text{Object}, \text{Locative}, \text{Preposition}, \text{Verb})$$7 to $$(\text{Subject}, \text{Object}, \text{Locative}, \text{Preposition}, \text{Verb})$$8, and on unseen-composition zero-shot settings it raises unseen mAP from $$(\text{Subject}, \text{Object}, \text{Locative}, \text{Preposition}, \text{Verb})$$9 to $20\%$0 in the rare-first protocol (Hou et al., 2021). “Improving Human-Object Interaction Detection via Virtual Image Learning” instead creates virtual images through the MUSIC pipeline and uses teacher–student training with Adaptive Matching-and-Filtering; on HICO-DET, QPIC + VIL improves from $20\%$1 to $20\%$2 full mAP and yields the largest gains among the compared long-tail mitigation methods (Fang et al., 2023).

The long-tail problem also motivates explicit concept discovery. “Discovering Human-Object Interaction Concepts via Self-Compositional Learning” partitions the verb–object space into known concepts $20\%$3, unknown but reasonable concepts $20\%$4, and invalid concepts $20\%$5, then maintains an online concept confidence matrix $20\%$6 for self-training on unlabeled composites. On HICO-DET, unknown concept AP rises from $20\%$7 to $20\%$8; on V-COCO it rises from $20\%$9 to $3\%$0 (Hou et al., 2022). This suggests that HOIL increasingly targets not only recognition of annotated classes, but also discovery of plausible but unlabeled affordance-like compositions.

Continual and open-world acquisition extends this logic to non-stationary data streams. “Incremental Human-Object Interaction Detection with Invariant Relation Representation Learning” defines incremental HOI detection over phase-wise disjoint HOI subsets and identifies interaction drift as a specific continual-learning pathology: relation features degrade when the same relation is later seen with different objects. Its IRD framework freezes object detection, distills relation features through Momentum Feature Distillation and Concept Feature Distillation, and maintains per-relation queues of invariant prototypes. On HICO-DET with 5 phases, IRD reaches Old $3\%$1, Full $3\%$2, RID $3\%$3, and UC $3\%$4, improving over the strongest baseline on forgetting, drift robustness, and zero-shot compositional generalization (Wei et al., 30 Oct 2025). This indicates that in HOIL, continual learning cannot be reduced to standard class-incremental learning because the basic units are compositional relations rather than atomic categories.

4. Language grounding, foundation models, and reasoning-centric HOIL

Language has moved from auxiliary prior to core supervisory signal. The classifier-centric approach of language-embedding initialization and LSE-Sign already showed that HOI semantics can be injected directly into the classification head, with especially strong gains on few-shot HICO subsets and a transfer path to HICO-DET without detection fine-tuning (Jin et al., 2022). More recent foundation-model systems push this substantially further.

“Detecting Any Human-Object Interaction Relationship: Universal HOI Detector with Spatial Prompt Learning on Foundation Models” recasts HOI detection as HO-pair-level prompting over BLIP2, with HO spatial prompts $3\%$5 feeding an HO Prompt-guided Decoder that aligns frozen foundation-model features to specific human–object pairs. UniHOI supports both interaction phrases and GPT-generated interpretive sentences, reaches $3\%$6 full mAP on HICO-DET Default/Known Object in its large variant, and achieves strong zero-shot results such as $3\%$7 unseen mAP in the unseen-object protocol and $3\%$8 in the unseen-verb protocol (Cao et al., 2023). The key methodological point is pair-specific alignment: unlike image-level CLIP prompting, the relation token is conditioned on a particular human–object pair.

“Modeling the Multivariate Relationship with Contextualized Representations for Effective Human-Object Interaction Detection” extends the interaction unit itself from pairs to tool-mediated triplets. CRLN builds unary, binary, and ternary tokens, uses a tool knowledge bank to instantiate $3\%$9human, tool, object$39\%$0 structures, and enriches prompts with object-conditioned learnable tokens. Its best model reaches $39\%$1 full mAP on HICO-DET Default, $39\%$2 on Known Object, and $39\%$3 AP_role on V-COCO Scenario 1/2 (Li et al., 16 Sep 2025). This is significant because it treats affordance as a latent relation carried by auxiliary entities such as tools, rather than as a side comment on pair classification.

Reasoning has also become explicit in open-world detection. “HOID-R1: Reinforcement Learning for Open-World Human-Object Interaction Detection Reasoning with Multimodal LLM” combines CoT-guided supervised fine-tuning, Group Relative Policy Optimization, and an MLLM-as-a-judge that scores both step-level and group-level reasoning coherence. Under open-vocabulary protocols, HOID-R1 reaches $39\%$4 mAP on HICO-DET seen and $39\%$5 on unseen, and on SWIG-HOI it reaches $39\%$6 mAP on seen and $39\%$7 on unseen (Zhang et al., 15 Aug 2025). The paper explicitly argues that prompt-only VLM methods are brittle to phrasing and underuse the model’s reasoning and spatial grounding; whether one accepts that framing in full, it marks a clear shift from prompt engineering to policy optimization over multimodal reasoning traces.

Taken together, these systems imply that modern HOIL is no longer limited to predicting verbs from boxes. It increasingly learns a joint space in which free-form text, pair- or triplet-level visual grounding, and intermediate reasoning steps are all optimized together.

5. Temporal, 3D, 4D, and embodied extensions

Several recent works move beyond 2D image triplets to spatiotemporally grounded HOIL. “Efficient and Scalable Monocular Human-Object Interaction Motion Reconstruction” introduces 4DHOISolver and Open4DHOI, a monocular 4D reconstruction pipeline with sparse human-in-the-loop contact annotations. Open4DHOI contains $39\%$8 sequences and $39\%$9k frames, covering $43\%$0 object types and $43\%$1 action classes, with an average of $43\%$2 annotated points per video and annotation time of about $43\%$3 minutes per video. The reconstruction quality is strong enough to supervise physics-based control, yet a benchmark of 3D foundation models shows that precise HOI point-wise correspondence remains unsolved, with InteractVLM reaching only $43\%$4 recall (Wen et al., 30 Nov 2025). This directly challenges the assumption that 4D contact can already be recovered automatically at scale.

A complementary generative perspective appears in “Uni-HOI:A Unified framework for Learning the Joint distribution of Text and Human-Object Interaction.” Uni-HOI tokenizes human motion and object motion with separate VQ-VAEs, merges them with text in a unified vocabulary, and models the joint distribution $43\%$5 with Qwen3-8B. It supports text-driven HOI generation, object-motion-driven human generation, and human-motion-driven object prediction within a single autoregressive model. On FullBodyManipulation, its text-driven model reaches FID $43\%$6, Top-1 R-Precision $43\%$7, and Top-3 $43\%$8; in object-to-human generation with text, Uni-HOI-OT achieves HandJPE $43\%$9 and MPJPE $p$0 (Zhang et al., 30 Apr 2026). A plausible implication is that 4D HOIL is converging toward a token-based multimodal modeling paradigm analogous to recent language-driven motion generation.

HOIL has also been used as a feature-learning principle for perception tasks that are not usually framed as HOI detection. “Learning Human-Object Interaction for 3D Human Pose Estimation from LiDAR Point Clouds” introduces HOICL, CPPool, and optional contact-based temporal refinement. The full model reduces Waymo MPJPE from $p$1 to $p$2 and reaches $p$3 MPJPE on SLOPER4D, while specifically targeting ambiguity and class imbalance in interaction-frequent regions such as hands and feet (Jung et al., 17 Mar 2026). Here HOIL functions as an inductive bias for 3D pose estimation in interaction regions, rather than as a terminal labeling task.

At the control end of the spectrum, “InterReal: A Unified Physics-Based Imitation Framework for Learning Human-Object Interaction Skills” treats HOIL as contact-rich humanoid skill acquisition. It augments reference motions with hand–object contact constraints, learns reward allocation through a meta-policy, and deploys policies on the Unitree G1. InterReal reaches $p$4 task success on box-picking and $p$5 on box-pushing, outperforming recent baselines while preserving real-world deployability (Liang et al., 8 Mar 2026). This is a decisive shift in emphasis: interaction learning here is evaluated by tracking accuracy and task completion, not by mAP.

6. Benchmarks, metrics, and unresolved issues

The field now spans several benchmark families, each encoding a different operational definition of HOIL.

Benchmark family	Typical focus	Representative metrics
HICO-DET, V-COCO, HICO-ambiguous	Image-based HOI detection	mAP, AP_role
SWIG-HOI, open-vocabulary HICO-DET	Open-world reasoning and grounding	H-mIOU, O-mIOU, A-ACC, mAP
Open4DHOI, InterAct subset, FBM, BEHAVE, GRAB	4D reconstruction and generation	contact distance, projection IoU, FID, R-Precision, HandJPE, MPJPE
Waymo, SLOPER4D	Interaction-aware 3D pose estimation	MPJPE, PCK-3, PCK-5

For classical image-based HOI detection, HICO-DET and V-COCO remain central. HICO-DET is reported in Full, Rare, and Non-Rare splits; V-COCO emphasizes role AP; and the HICO-ambiguous subset makes explicit the robustness gap under occlusion and impaired foreground cues (Wang et al., 2024, Jia et al., 2024). Open-world systems add H-mIOU, O-mIOU, and A-ACC to separate localization and semantic correctness (Zhang et al., 15 Aug 2025). Temporal and 4D systems shift evaluation toward contact distance, projection IoU, FID, R-Precision, HandJPE, and MPJPE, while control-oriented systems further add task success rates (Wen et al., 30 Nov 2025, Zhang et al., 30 Apr 2026, Liang et al., 8 Mar 2026).

Several unresolved issues recur across these benchmarks. Long-tailed verbs and rare compositions remain difficult even for systems explicitly designed for them (Wang et al., 2020, Hou et al., 2022). Prompt-based open-vocabulary detectors improve semantic flexibility but can still be brittle to phrasing or semantically unfaithful in their intermediate reasoning (Zhang et al., 15 Aug 2025). Tool-dependent and multivariate interactions expose the inadequacy of pair-only representations (Li et al., 16 Sep 2025). Precise human–object contact correspondence is still an open challenge for current 3D foundation models (Wen et al., 30 Nov 2025). Continual settings reveal that relation representations can drift even when object detection remains stable (Wei et al., 30 Oct 2025). More generally, the survey literature identifies long-tail imbalance, polysemy, negative-pair explosion, occlusion, and spurious contextual correlations as persistent bottlenecks (Wang et al., 2024).

The trajectory of recent work suggests several converging directions rather than a single dominant solution. Spatially grounded pair discovery, compositional concept learning, multimodal reasoning, explicit context modeling, 4D contact reconstruction, and physics-based imitation are no longer isolated subfields; they increasingly define different slices of the same HOIL problem. The common objective is to learn not just that a human and an object are related, but where, how, under what semantic constraints, and with what physical consequences that relation occurs.