HOID-R1: Open-World HOI Detection

• The paper introduces HOID-R1, a reinforcement learning-based framework that integrates chain-of-thought reasoning with multimodal LLMs for open-vocabulary HOI detection.
• It employs supervised fine-tuning and group relative policy optimization, achieving significant improvements in metrics such as mAP and IoU on challenging benchmarks.
• The approach provides interpretable, modular outputs that enhance spatial reasoning, enabling robust deployment in AR/VR and robotics applications.

HOID-R1 is a reinforcement learning-based framework for open-world human-object interaction (HOI) detection that leverages multimodal LLMs (MLLMs), chain-of-thought reasoning, and policy optimization to achieve generalization across unseen verbs, objects, and combinations. The approach is designed to address key challenges in open-vocabulary HOI detection, particularly the need for spatial reasoning and deep alignment between natural language and visual modalities, while providing interpretable predictions suited for deployment in AR/VR and robotics (Zhang et al., 15 Aug 2025).

1. Open-World HOI Detection Paradigm

HOID-R1 is formulated for open-vocabulary HOI detection, where the objective is, given an image $I$ and a language query $Q$, to localize bounding boxes for the human ($b_h$), the object ($b_o$), and to infer the verb ($v$) describing their interaction. Crucially, the label space is not restricted to a closed set known at training—the model must handle novel HOI classes at inference time.

Open-world HOI detection exposes two major technical challenges:

• Deep 3D Spatial Reasoning: Disambiguating fine-grained prepositions (e.g., "on", "under", "holding") from 2D imagery requires inferring occlusion, contact, and spatial relationships that are not explicitly annotated.
• Multimodal Alignment: Natural language queries can be ambiguous or compositional. Reliable mapping between free-form text and corresponding image regions is nontrivial, especially in under-constrained open-vocabulary settings.

Traditional approaches relying solely on visual-LLMs or prompt engineering lack mechanisms for stepwise logical inference or for pruning erroneous predictions under ambiguous language (Zhang et al., 15 Aug 2025).

2. Supervised Chain-of-Thought Fine-Tuning

The training pipeline begins with supervised fine-tuning (SFT) of a base MLLM (e.g., DeepSeek-R1) using data that includes not only ground-truth HOI triplets but also explicit stepwise rationales, or chain-of-thought (CoT) annotations. Each training sample comprises an image $I$, a language query $Q$, a ground-truth triplet $\langle b_h, v, b_o \rangle$, and a sequence of N chain-of-thought steps $\{ s_1, \ldots, s_N \}$ marked by a special > token for each reasoning stage.

Constraints on the reasoning structure include:

1. The first step must identify the human.
2. The second step must identify the object.
3. Verbs/relations are restricted to a curated, plausible HOI set.

The supervised objective is a standard next-token cross-entropy: $$\mathcal{L}_\mathrm{SFT} = -\sum_{t=1}^T \log \pi_\theta(w_t \mid w_{<t}, I, Q),$$ where $Q$0 are tokens corresponding to the chain-of-thought and the final HOI output. Auxiliary detection and classification heads provide additional losses on box regression and categorical outputs (Zhang et al., 15 Aug 2025).

This stage scaffolds logical spatial reasoning that pure prompt-based methods cannot capture, enforcing explicit interpretability in the model outputs.

3. Reinforcement Learning with Group Relative Policy Optimization (GRPO)

After SFT, the policy $Q$1 is further optimized using a reinforcement learning variant called group relative policy optimization (GRPO). The policy generates both the reasoning chain and the final HOI triplet in an autoregressive manner, with actions corresponding to selection of the next output token (CoT step, bounding box, or label).

The reward function is multi-faceted, with components:

• Format Reward ($Q$2): Enforces syntactic correctness of the output (proper triplet structure, CoT tags).
• Detection Reward ($Q$3): Measures detection fidelity via intersection-over-union (IoU) and $Q$4 error thresholds against ground-truth boxes.
• Interaction Reward ($Q$5): Rewards correct verb and object label predictions.
• CoT Reward ($Q$6): Quantifies the logical quality and veracity of the generated reasoning chain.

The overall reward for a rollout is the sum of these terms. GRPO maintains a group of $Q$7 candidate rollouts per sample. Each rollout's advantage is normalized against the group, and the learning signal is

$Q$8

with a Kullback–Leibler divergence penalty to keep the updated policy close to a reference actor. No separate critic is required, and credit assignment is distributed within each sample group. This efficiently enables credit assignment for complex sequence outputs and can scale to long CoT outputs (Zhang et al., 15 Aug 2025).

4. MLLM-as-a-Judge for Reasoning Quality Control

A distinguishing feature of HOID-R1 is the use of an auxiliary pretrained multimodal LLM as a "judge" to assess the quality of the generated chain of thought. This judge provides scalar or binary scores:

• Process Reward Model (PRM): Evaluates each reasoning step $Q$9 individually.
• Generalizable Reward Model (GRM): Aggregates and scores groups of consecutive steps.

The judge filters out hallucinated or logically flawed reasoning, providing feedback that directly shapes the reinforcement learning reward. The CoT reward is defined as

$b_h$0

where $b_h$1 and $b_h$2 are PRM and GRM scores, respectively. This mechanism targets generalization and robustness by disincentivizing spurious or inconsistent reasoning (Zhang et al., 15 Aug 2025).

5. Experimental Evaluation and Ablations

HOID-R1 has been evaluated on open-vocabulary HOI benchmarks including HICO-DET (600 classes, and a zero-shot split with 120 rare triplets) and SWIG-HOI (∼1800 unseen interactions). Key metrics include:

• H-mIOU: Mean IoU for human boxes.
• O-mIOU: Mean IoU for object boxes.
• A-ACC: Action (verb) classification accuracy.
• mAP: Mean average precision for correct simultaneous localization and classification.

Empirical results demonstrate state-of-the-art performance compared to established baselines:

• On HICO-DET open-vocabulary unseen split: H-mIOU ≈ 0.68, O-mIOU ≈ 0.68, A-ACC ≈ 0.72, mAP ≈ 50.0, each 5–7 points higher than previous published methods (Zhang et al., 15 Aug 2025).
• Ablation studies confirm that omitting any major component (e.g., format reward, detection reward, interaction reward, or CoT reward) degrades mAP by 4–6 points.

6. Interpretability, Modularity, and Deployment Considerations

A principal advantage of HOID-R1 is output interpretability: the model’s predictions are accompanied by an explicit, stepwise chain-of-thought rationale that can be reviewed by human operators. The modular structure—SFT for reasoning supervision, judge-based feedback, and RL for policy refinement—enables adaptation to new domains or label sets.

HOID-R1 is engineered to be directly applicable to AR/VR/robotics environments. Its CoT outputs can be audited or used for safe-fallback controls, e.g., triggering human intervention if the judge flags incoherent rationales. The group-relative RL structure operates efficiently on sequential outputs without a learned critic, making it tractable for high-dimensional reasoning tasks (Zhang et al., 15 Aug 2025).

7. Position Within HOI Detection Research

HOID-R1 represents the first HOI detection framework that fully unifies chain-of-thought reasoning supervision, group-based reinforcement learning optimization, and MLLM-based output judgment. Compared to pure language-prompting models or architectures lacking reasoner supervision, HOID-R1 provides enhanced spatial reasoning, generalization to novel compositions, and robust performance on open-vocabulary tasks. Its explicit multi-reward approach addresses the limitations of prior single-policy or purely discriminative models (Zhang et al., 15 Aug 2025).