Towards Unconstrained Human-Object Interaction

Abstract: Human-Object Interaction (HOI) detection is a longstanding computer vision problem concerned with predicting the interaction between humans and objects. Current HOI models rely on a vocabulary of interactions at training and inference time, limiting their applicability to static environments. With the advent of Multimodal LLMs (MLLMs), it has become feasible to explore more flexible paradigms for interaction recognition. In this work, we revisit HOI detection through the lens of MLLMs and apply them to in-the-wild HOI detection. We define the Unconstrained HOI (U-HOI) task, a novel HOI domain that removes the requirement for a predefined list of interactions at both training and inference. We evaluate a range of MLLMs on this setting and introduce a pipeline that includes test-time inference and language-to-graph conversion to extract structured interactions from free-form text. Our findings highlight the limitations of current HOI detectors and the value of MLLMs for U-HOI. Code will be available at https://github.com/francescotonini/anyhoi

• The paper introduces the U-HOI task that removes fixed verb-object vocabularies, using MLLMs to extract free-form HOI triplets.
• The AnyHOI pipeline integrates off-the-shelf object detection with free-form MLLM generation and post-generation refinement to enhance interaction extraction.
• Empirical results demonstrate improved mAP and semantic recall, setting a new benchmark for dynamic, real-world HOI detection.

Unconstrained Human-Object Interaction Detection with Multimodal LLMs

Introduction

The paradigm of Human-Object Interaction (HOI) detection has historically relied on a closed or open but predetermined vocabulary of verb-object interactions during both training and inference. This constraint of operating within a fixed lexicon severely limits the capability of HOI models in practical, dynamic contexts such as assistive robotics, autonomous driving, and in-the-wild intention understanding. This paper introduces the Unconstrained HOI (U-HOI) setting, which eliminates the reliance on any fixed interaction vocabulary, and leverages the generative prowess of Multimodal LLMs (MLLMs) to extract interactions from free-form scene descriptions.

Figure 1: The U-HOI setting eliminates the need for a predefined interaction vocabulary and operates purely on unconstrained free-form MLLM outputs.

Problem Definition and Evaluation Protocol

The U-HOI task requires prediction of arbitrary human-object interaction triplets $$\langle \text{human}, \text{verb}, \text{object} \rangle$$ from an open vocabulary, with no prior knowledge of the set of valid verb-object combinations $\mathcal{M}$ or verb lexicon $\mathcal{V}$. Given the immense breadth of possible verbs (e.g., thousands in WordNet vs. a few hundred in typical datasets), lexical exact-matching is unsuitable. To this end, the paper adapts classical mean Average Precision (mAP) for evaluation by using semantic similarity mapping (BERT-based) between predicted and ground-truth verbs. Additionally, Semantic Recall (SR) is introduced, which measures whether a model’s predicted triplet semantically recovers ground-truth intent, independent of surface form.

Models are evaluated under two protocols: annotated-box (using ground-truth bounding boxes to isolate interaction interpretation) and computed-box (entailing integrated detection and interaction extraction). This protocol allows for a nuanced performance assessment on unconstrained triplet prediction.

The AnyHOI Pipeline

AnyHOI is presented as a reference pipeline for U-HOI, requiring no training and combining off-the-shelf MLLMs with structured post-generation refinement.

• Human-object pair detection: Using frozen object detectors (e.g., DETR for Chao2018, GroundingDINO for Wu2024), candidate pairs are extracted.
• Free-form generation: Each pair is cropped and submitted to the MLLM with a task-specific prompt (e.g., "What are the interactions between the person and the obj?"), yielding free-form text outputs.
• Post-generation refinement: Text-to-graph Transformers (FACTUAL) parse these outputs into subject-verb-object triplets. Irrelevant or non-HOI triplets (statements of attributes, positional relations) are pruned via heuristics.
• Test-time compute: Multiple generations are sampled with various seeds/temperatures, exploiting the diversity in MLLM outputs to improve recall and capture subtle interactions. Two strategies for triplet aggregation are examined: Top-k frequent responses and frequency-based sampling.

  Figure 2: AnyHOI pipeline: Object detection, scene cropping, prompted MLLM generation, text-to-triplet conversion, and filtering for HOI extraction.

  

  Figure 3: Test-time compute aggregates responses from multiple generations, sampling final predictions based on interaction frequency.

Empirical Results and Analysis

Quantitative Benchmarking

Comprehensive experiments are conducted on Chao2018 and Wu2024 datasets, under both annotation and detection protocols. Due to the unconstrained vocabulary, several prior methods are adapted for evaluation.

• Off-the-shelf MLLMs achieve mAP and SR scores comparable to SOTA HOI models in the unconstrained regime, though absolute values remain moderate, underlining the task difficulty.
• Incorporating AnyHOI notably boosts performance across all MLLM baselines, with even small models (e.g., LLaVA OV 0.5B) outperforming strong baselines when augmented with AnyHOI.
• Test-time compute yields substantial metric improvements, especially for smaller or less specialized models, with Top-k aggregation consistently outperforming sampling.

  Figure 4: Increasing the number of MLLM generations at test time consistently improves mAP and recall on Chao2018 under annotated-box protocol.

Ablation Studies

• Visual prompting strategies (red circle, reverse blur, crop, mask) are scrutinized; simple cropping outperforms masking strategies, demonstrating the importance of visual context between the human and object.
• Direct textual prompting yields superior results compared to descriptive, structured, or chain-of-thought prompts, emphasizing the criticality of prompt design in MLLM conditioning.
• Increasing the aggregation parameter $k$ improves recall and mAP, highlighting the flexibility afforded by storing multiple candidate triplets.

Qualitative and Topic-specific Evaluation

Analysis over verb categories grouped by topic (opposite, social, manipulation, activity, positional prepositions) reveals:

• MLLMs display similar behavior across topics; physical activities and object manipulations are most challenging.
• Social verbs and clear positional states have better detection rates.

  Figure 5: mAP breakdown by topic: MLLMs exhibit lower performance on physical activities and object manipulations.

  

  Figure 6: Performance on opposite verbs: MLLMs struggle with fine-grained distinctions such as 'pulling' vs. 'pushing'.

  

  Figure 7: Social interaction verbs are moderately better predicted, with certain models outperforming others in this category.

  

  Figure 8: For object manipulation verbs, actions like setting, spinning, wielding are universally difficult.

  

  Figure 9: Positional verbs show high success on simple states, but struggle with compositional prepositions.

Qualitative results further demonstrate that AnyHOI's triplets better reflect real scene semantics compared to legacy baselines.

Figure 10: AnyHOI and CLIP predictions: AnyHOI captures nuanced and scene-appropriate HOI triplets.

Practical and Theoretical Implications

The introduction of U-HOI fundamentally alters HOI detection and deployment strategy. It renders closed-set and open-vocabulary HOI obsolete for scenarios characterized by dynamic object/action distributions and unpredictable environments. The reliance of legacy HOI models on occurrence priors and co-occurrence matrices is empirically shown (see Supp. Fig. 1) to cause drastic performance drops when priors are omitted, revealing their lack of true generalization.

AnyHOI demonstrates that MLLMs, when properly prompted and refined, are highly adaptable for unconstrained triplet extraction without requiring retraining or explicit enumeration of interaction lexicons, facilitating practical applications in assistive robotics, vision-based human intention understanding, and real-world activity parsing.

On the theoretical side, U-HOI expands the feasible space for semantic action recognition, setting a new benchmark for free-form reasoning architectures and suggesting future directions in fine-tuning, grounding, and advanced test-time computation. The semantic recall metric is likely to become a standard for any open-set generative task where exhaustive annotation is impractical.

Conclusion

This paper proposes the U-HOI task and a rigorous evaluation protocol that eschews the limitations of fixed interaction vocabularies. The empirical results substantiate the utility of MLLMs, especially when augmented with the AnyHOI pipeline and test-time compute strategies, in extracting semantically accurate HOI triplets from unconstrained settings. The implications for the deployment of HOI systems in real-world environments are significant, and the future research landscape will transition toward further augmentation and object grounding in U-HOI, along with advanced multi-modal reasoning for open-set interaction detection.