HOID Reward Functions in HOI Detection

• HOID Reward Functions are formal evaluation metrics that define structured outputs for HOI detection, ensuring format, semantic, and spatial accuracy.
• They integrate multiple sub-rewards—format tag, label accuracy, and IoU measurements—into a unified reinforcement learning framework for policy optimization.
• Empirical results on benchmarks like HICO-DET demonstrate enhanced mAP and reduced reward hacking, validating the method’s effectiveness in open-world HOI contexts.

HOID Reward Functions (Human-Object Interaction Detection Reward Functions) are formal evaluation and learning objectives specifically constructed to align the outputs of multimodal LLMs (MLLMs) or vision-LLMs with the requirements of human-object interaction (HOI) detection tasks. These functions are designed to guide reinforcement learning (RL)–based fine-tuning on structured output tasks, where the end-goal is not just natural language or region grounding, but the precise, multi-part identification of human-object interaction instances—including both spatial (bounding box) localization and semantic (action/verb and object) correctness. In the context of recent work on open-world and language-centric HOI detection (Chen et al., 7 Oct 2025), HOID reward functions are central to aligning high-capacity models for robust HOI recognition in unconstrained scenarios.

1. Structural Overview of HOID Reward Functions

HOID reward functions are composed of multiple components, each targeting a distinct aspect of the desired output:

• Key format reward: Enforces rigid adherence to structured output templates for each HOI instance, such as required JSON keys (e.g., “human”, “object”, “object class”, “verb class”), specific tagging (e.g., > reasoning wrapper), and uniqueness across instances. > > - Object and verb label reward: Quantifies the semantic accuracy of predicted object classes and interaction verbs, employing “drop-on-match” matching to ensure one-to-one correspondence with ground truth and to penalize duplication. > > - HOI IoU reward: Measures location fidelity by matching predicted and ground-truth bounding boxes (both human and object) via the Hungarian algorithm, scoring with averaged Intersection over Union (IoU). > > - Aggregate reward: Combines all the above into a final signal that drives RL optimization. > > Each sub-reward uses indicator functions, set operations, or normalized scores to provide granular, interpretable feedback at the level of each predicted instance. The overall HOID reward is a sum (possibly weighted) of the format, label, and IoU sub-rewards. > > ## 2. Mathematical Formulation of Sub-Rewards > > Formally, let $\hat{y}$ denote a model’s text output prediction for an image, and let each HOI instance $\hat{y}_i$ be a dictionary containing keys, labels, and bounding box coordinates. If $\mathbb{I}$ denotes the indicator function and $C_o, C_{\text{gt},o}$ the object label sets, the main reward components are: > > | Reward Component | Mathematical Expression | Role | > |-------------------------|-----------------------------------------------------------------------------------------------|---------------------------------------------------| > | Format tag reward | $$r_{\text{tag}} = \mathbb{I}["\langle\text{think}\rangle" \in \hat{y}]$$ | Enforces <think> chain-of-thought wrapper | > | Box key reward | $$r_{b_i} = \mathbb{I}[\{ "human","object" \} \subseteq \text{keys}(\hat{y}_i) \land \ldots ]$$ | Ensures bounding box fields and no duplicates | > | Label reward (object) | $$r_{lo} = \frac{\sum_{i=1}^{\hat{N}_a} \alpha_i \mathbb{I}[\hat{c}^o_i \in C_{\text{gt},o}^{(i-1)}]}{\max(N_a,\hat{N}_a)}$$ | Assesses object label correctness and uniqueness | > | Label reward (verb) | $$r_{lv} = \frac{\sum_{i=1}^{\hat{N}_a} (\alpha_i/|\{ \hat{c}_i^v\}|)\sum_{j} \mathbb{I}[\hat{c}^v_{ij} \in C_{\text{gt},v}^{(i-1)}]}{\max(N_a,\hat{N}_a)}$$ | Assesses verb multi-label accuracy | > | IoU reward | $$r_{\text{IoU}} = \frac{1}{N_a} \sum_{(i,j) \in \mathcal{M}^*} s_{ij}$$ | Rewards spatial localization accuracy | > > Here, $\mathcal{M}^*$ is the optimal matching from predictions to ground truth via the Hungarian algorithm, $$s_{ij} = 0.5 [\text{IoU}(\hat{b}^h_i,b^h_j) + \text{IoU}(\hat{b}^o_i,b^o_j)]$$. > > Final reward: $$r = r_{\text{format}} + r_{lo} + r_{lv} + r_{\text{IoU}}$$. > > This composite structure ensures that models must generate text outputs that are simultaneously structurally correct, semantically accurate, and spatially aligned with the target HOI allocations. > > ## 3. Integration with Reinforcement Learning and Policy Optimization > > In contemporary HOI detection frameworks, the HOID reward is employed as the optimization target in RL-based fine-tuning of MLLMs. Policy optimization is performed via Group Relative Policy Optimization (GRPO), a variant of PPO adapted for text outputs: > > - Candidate outputs $\{o_1, ..., o_G\}$ are sampled from the model’s distribution. > > - For each, the total reward $r_i$ is computed as above. > > - The (normalized) advantage for each output is $$A_i = (r_i - \text{mean}(r_{1..G}))/\text{std}(r_{1..G})$$. > > - The surrogate loss incorporates advantage-weighted likelihood ratios and a KL-divergence penalty against a reference policy: > > $$\mathcal{J}_{\text{GRPO}} = - \mathbb{E}_{(x,q), \{o_i\}} \frac{1}{G} \sum_{i=1}^{G} \left[ \min(s_1 \cdot \hat{A}_i, s_2 \cdot \hat{A}_i) - \beta \mathcal{D}_{\text{KL}}[\pi_\theta || \pi_{\text{ref}}] \right]$$ > > where $s_1, s_2$ are clipped likelihood ratios to stabilise training. > > This RL paradigm enables the MLLM to adjust output generation strategies not only for conventional language quality but to optimize for end-to-end HOI detection performance. > > ## 4. Impact on Open-World HOI Detection and Generalization > > The use of HOID reward functions allows language-based or multimodal models to directly optimize not only semantic correctness (through label rewards) but also spatial and structural constraints that are rarely addressed in standard text-only RL frameworks. This is particularly critical in open-world HOI settings, where models face a combinatorial search space of possible object-action pairs and require robust output structuring to generalize to novel or long-tail interaction categories. > > Empirical ablation studies have demonstrated that each sub-reward makes distinct contributions: > > - Removing the IoU reward produces the largest decline in mean Average Precision (mAP), underscoring the importance of localization. > > - Omission of label rewards degrades semantic recognition accuracy. > > - Format termination and duplication constraints reduce “reward hacking” by penalizing redundant or improperly structured answers. > > This decomposition ensures that the total reward surfaces a sufficiently rich, fine-grained feedback landscape for sample-efficient RL fine-tuning, even in diverse or ambiguous images. > > ## 5. Comparison with Previous Reward Designs and Evaluation Paradigms > > Conventional reward designs in HOI detection—e.g., simple per-prediction accuracy, vanilla IoU thresholds, or LLM perplexity—fail to simultaneously incentivize all the constraints necessary for robust HOI instance output. The HOID reward unifies detection structure, semantics, and localization into a single function, reflecting the multi-task nature of HOI detection. > > Relative to prior approaches (e.g., vision–language pretraining with no explicit RL or modular detection pipelines with hand-crafted loss terms), HOID reward–driven RL demonstrates superior performance and generalization: > > | Design Aspect | Previous Approaches | HOID Reward Function Architecture | > |----------------------------------|-------------------------------------------------------|------------------------------------------| > | Format awareness | None or implicit | Explicitly enforced via format reward | > | Detection accuracy | Hard or soft IoU losses, not RL-based | RL using matched and normalized IoU | > | Semantic correctness | Softmax actions, cross-entropy, or prompt-based cues | RL with matched, one-to-one label reward | > | Output duplication handling | Limited | Explicit penalty terms (α_i factors) | > | Open-world generalization | Limited | Empirically doubled mAP on HICO-DET | > > This unified reward design is particularly effective when paired with algorithms like GRPO, which promote group-wise, sample-efficient exploration and stabilization via reward normalization and KL-regularization. > > ## 6. Practical Considerations and Potential Limitations > > While HOID reward functions provide a comprehensive and task-aligned optimization target, several practical issues must be considered: > > - Calculation of IoU rewards and optimal matchings via the Hungarian algorithm introduces additional computational overhead, especially for large batch sizes or images with many HOI instances. > > - The weighting and normalization of individual reward components (e.g., $w_{\text{tag}}, w_b, w_{ko}, w_{kv}$) must be tuned to prevent the RL policy from prioritizing trivial template completion over genuine detection accuracy. > > - The drop-on-match and one-to-one matching strategies assume that the ground truth is exhaustive and well-structured; in scenarios with ambiguous annotations or missing objects, this can reduce learning signal or create unfair penalization. > > - The generalizability of format- and structure-based rewards to other domains or datasets may be contingent on maintaining rigorous annotation standards. > > - A plausible implication is that, while HOID reward functions significantly improve structured detection and alignment in the HOI domain, analogous compound reward frameworks may be required when extending RL-based model alignment to other structured vision-language tasks. > > ## 7. Empirical Outcomes and Benchmark Performance > > On the HICO-DET dataset (38,118 training, 9,658 testing images, 600 categories), HOI-R1 (with SFT and 40 RL steps using HOID reward) achieved twice the mean Average Precision of a baseline MLLM model relying solely on prompt engineering. Ablation studies confirm that removing format, IoU, or label rewards consistently decreases mAP—in one cited case, eliminating the IoU term reduced Full mAP by 3.62 points, and dropping the label reward led to a 0.76 mAP decrease. > > Qualitative analyses reveal scenarios where the HOID reward penalizes incorrectly formatted or duplicate predictions and promotes dual detection of multiple HOI instances within a single image, thus aligning text-based output with the ground-truth instance structure. > > --- > > In sum, HOID reward functions operationalize a structured, multi-component reinforcement learning target for MLLMs (and related architectures) focused on human-object interaction detection. By simultaneously encoding strict output formatting, semantic correctness, and spatial localization into the reward, they enable substantial improvements in both task-aligned performance and open-world generalization, as evidenced on challenging benchmarks and through ablation analysis (Chen et al., 7 Oct 2025).