INP-CC: Interaction-Aware HOI Detection

• The paper presents INP-CC, an end-to-end framework that leverages dynamic, interaction-aware prompts and LLM-guided concept calibration to enhance HOI detection.
• It integrates a specialized prompt generator with a CLIP image encoder to direct focus on salient human-object interactions, yielding robust feature extraction.
• Empirical evaluations on SWIG-HOI and HICO-DET confirm significant mAP improvements through cluster-based concept grouping and targeted hard negative mining.

INteraction-aware Prompting with Concept Calibration (INP-CC) is an end-to-end open-vocabulary Human-Object Interaction (HOI) detection framework designed to address the limitations of previous vision-LLM-based HOI detectors. It achieves this by incorporating dynamically generated interaction-aware prompts and a concept calibration module guided by LLMs, resulting in robust generalization to novel interaction classes and enhanced discrimination of visually similar but semantically distinct HOIs. INP-CC has demonstrated significant state-of-the-art performance improvements on large-scale HOI datasets, notably SWIG-HOI and HICO-DET (Lei et al., 5 Aug 2025).

1. Architecture of the Interaction-Aware Prompt Generator

The INP-CC framework augments the CLIP image encoder with a specialized module that synthesizes “interaction-aware” prompts, effectively directing model attention to regions relevant for HOI detection. The module is constructed as follows:

• A single “common prompt” $P_C \in \mathbb{R}^{L \times D}$, shared across all interactions, where $L$ is prompt length and $D$ is the feature dimension.
• $M$ low-rank “interaction” basis prompts $\{\hat{P}^{i}_{IT}\}_{i=1}^M$, each defined as a rank-one matrix $$\hat{P}^{i}_{IT} = \hat{u}_i \hat{v}_i^\top \in \mathbb{R}^{L \times D}$$ and fused with the common prompt using Hadamard product: $P^{i}_{IT} = \hat{P}^{i}_{IT} \odot P_C$.
• A lightweight MLP $$g: \mathbb{R}^{L \times D} \rightarrow \mathbb{R}^D$$ generating a “prompt key” $k_{IT}^i = g(P_{IT}^i) \in \mathbb{R}^D$ per interaction prompt.
• For each scene, the CLIP encoder’s global feature $f_{scene} \in \mathbb{R}^D$ is used to compute cosine similarities $L$0, and the top-$L$1 prompts are selected to produce the final prompt:

$L$2

where $L$3 indexes the top-$L$4 keys.

These selected prompts, prepended to the CLIP image encoder, induce attention toward salient interaction patterns rather than generic scene-level semantics. The semantics of sharing—whereby several related HOI classes utilize the same interaction basis—are learned automatically, promoting prompt compactness and shared representations among similar HOIs.

2. Concept Calibration Module

INP-CC incorporates a LLM-guided concept calibration scheme to refine HOI category representations and encode nuanced visual semantics.

• For each HOI class (action-object pair), a detailed visual description $L$5 is generated using GPT-3.5.
• The description is embedded via a T5-based, instruction-finetuned encoder $L$6 to yield the “visual concept” vector $L$7.
• All $L$8 vectors are clustered into $L$9 groups using $D$0-means, grouping visually similar HOI categories.
• The final concept embedding is computed as:

$D$1

where $D$2 is the CLIP text encoder output, $D$3 a learnable projection, and $D$4 a learned gate via

$D$5

• An optional refinement adds the cluster centroid to $D$6 for further calibration.

This process leads to better discrimination of HOI categories, especially in cases involving complex or ambiguous visual semantics.

3. Negative Sampling and Hard Negative Mining

To explicitly encourage separability between visually similar but semantically distinct HOIs, the training procedure introduces a “hard negative” sampling strategy:

• In each batch, for all ground-truth HOI categories $D$7, the union of clusters $D$8 containing any $D$9 is computed.
• From $M$0, a set $M$1 of hard negative HOI categories is sampled.
• For each query, the classification loss is computed over the positive and these hard negative targets:

$M$2

with logits derived from cosine similarities to the calibrated concept vectors.

• Optionally, a margin-based push-away term $M$3 is included to further penalize undesired attractors in the representation space.

In practice, inclusion of hard negatives in the denominator of the cross-entropy is sufficient, enhancing the inter-modal contrastiveness of the learned representations.

4. Training Objectives and Optimization

The loss function used in INP-CC comprises three main components:

• $M$4: DETR-style $M$5 regression and IoU losses for human and object bounding boxes.
• $M$6: Cross-modal classification loss leveraging calibrated concept vectors and hard negative samples.
• $M$7 (optional): Margin-based penalty encouraging separation from hard negatives.

The overall loss is

$M$8

though $M$9 is often absorbed into $\{\hat{P}^{i}_{IT}\}_{i=1}^M$0. Training hyperparameters include use of CLIP ViT-B/16 backbone, prompt length $\{\hat{P}^{i}_{IT}\}_{i=1}^M$1 or $\{\hat{P}^{i}_{IT}\}_{i=1}^M$2 depending on dataset, $\{\hat{P}^{i}_{IT}\}_{i=1}^M$3 interaction prompts, $\{\hat{P}^{i}_{IT}\}_{i=1}^M$4 clusters, and sampling $\{\hat{P}^{i}_{IT}\}_{i=1}^M$5 hard negatives per batch. The typical regime uses 80 epochs, batch size 128, and AdamW optimizer on two NVIDIA 3090 GPUs.

5. Empirical Evaluation and Benchmark Performance

INP-CC’s effectiveness is validated on SWIG-HOI and HICO-DET under both standard and zero-shot splits. The key benchmarks and metrics include:

• HICO-DET:
  • 600 HOI classes; mAP@0.5 for unseen/seen/full: 17.38 / 24.74 / 23.12, compared to the prior open-vocabulary SOTA CMD-SE’s 16.70 / 23.95 / 22.35 ($\{\hat{P}^{i}_{IT}\}_{i=1}^M$6 mAP full).
• SWIG-HOI:
  • Yields mAP of 22.84 (non-rare), 16.74 (rare), 11.02 (unseen), and 16.74 (full), outperforming CMD-SE (21.46, 14.64, 10.70, 15.26; $\{\hat{P}^{i}_{IT}\}_{i=1}^M$7 mAP full, up to +2.1 rare).

Ablation studies show:

• Baseline (common prompt only): 14.43 mAP
• • Interaction-aware prompts: 15.54 mAP ($\{\hat{P}^{i}_{IT}\}_{i=1}^M$8)
• • Concept calibration: 15.30 mAP ($\{\hat{P}^{i}_{IT}\}_{i=1}^M$9)
• • Both modules (full): 16.74 mAP ($$\hat{P}^{i}_{IT} = \hat{u}_i \hat{v}_i^\top \in \mathbb{R}^{L \times D}$$0)

Further, use of the dynamic selection mechanism improves mAP by +0.76, cluster-based grouping by +0.81, and hard negative sampling by +0.59. Qualitative results display sharper model attention to human-object contact regions (e.g., painting, reading, hugging), a direct result of scene-adaptive and semantically calibrated representations.

6. Connections and Significance

INP-CC draws on prompt-learning paradigms exemplified by CoOp and CoCoOp, extending them with interaction specificity and cross-modal concept alignment. Unlike prior open-vocabulary HOI approaches, which utilize image-level prompts or simple textual embeddings, INP-CC’s interaction-aware prompts and its language-model guided calibration collectively yield compact, generalizable, and discrimination-enhanced context tokens.

This framework notably addresses the challenge of generalization to novel or infrequent HOI classes by leveraging both shared visual prompt bases and semantically grouped concept embeddings grounded in LLMs. The selective negative sampling scheme directly mitigates class confusion in visually similar but semantically distinct HOIs, a major source of error in prior detectors.

Overall, INP-CC represents a significant advancement in open-vocabulary HOI detection by unifying dynamic prompt generation, LLM-guided semantic calibration, and targeted negative mining, achieving consistent and substantial improvements over previous state-of-the-art models on challenging, large-scale HOI benchmarks (Lei et al., 5 Aug 2025).