Active-Passive Gap: Interaction-Aware Modeling

• Active-Passive Gap is the discrepancy between action-driven (active) and context-driven (passive) signals in human-object interaction tasks.
• Adaptive prompt generation and language-model guided concept calibration are used to enhance visual feature sensitivity and alignment.
• Empirical evaluations on benchmarks like HICO-DET and SWIG-HOI demonstrate significant mAP improvements, validating the approach.

Below is a unified, step‐by‐step description of INP-CC that weaves together all five of your requested components. Wherever possible, we give the precise network structures, the LaTeX formulas, and the experimental evidence for each module’s contribution.

1. Interaction-Aware Prompt Generator In INP-CC we augment the frozen CLIP image encoder E_V with a small, learnable “prompt bank” so that region features become sensitive to human–object contact patterns rather than only global scene statistics.

a) Network architecture ‑ Input: an image I is first embedded by E_V into a spatial feature map $F_{scene} = E_V(I)\in\mathbb R^{HW\times C}$. ‑ Prompt Bank: we maintain * A common prompt $\;P_C\in\mathbb R^{L\times D}$, * M interaction‐specific bases $\{\hat P_{IT}^i\}_{i=1}^M$, each a rank-one factorization $\hat P_{IT}^i = u^i_{IT}\,(v^i_{IT})^\top$, with $u^i_{IT}\in\mathbb R^L,\;v^i_{IT}\in\mathbb R^D$. ‑ Composition: each full interaction prompt is $$P_{IT}^i \;=\;\hat P_{IT}^i \odot P_C\,,\quad i=1\ldots M$$ where $\odot$ is element‐wise (Hadamard) multiplication. ‑ Prompt‐to‐Query Adapter: an MLP $g:\mathbb R^{L\times D}\to\mathbb R^D$ computes a key $k_{IT}^i=g(P_{IT}^i)$.

b) Adaptive selection We first extract a global “fingerprint” $f_I\in\mathbb R^D$ from $\;P_C\in\mathbb R^{L\times D}$0 (e.g.\ average‐pool + linear). Then for each $\;P_C\in\mathbb R^{L\times D}$1 we form $\;P_C\in\mathbb R^{L\times D}$2 select the top‐K prompts by descending $\;P_C\in\mathbb R^{L\times D}$3, call the selected set $\;P_C\in\mathbb R^{L\times D}$4, and finally form our scene‐specific prompt $\;P_C\in\mathbb R^{L\times D}$5 This “soft” top‐K mixture is then prepended to the patch embeddings before the transformer layers of the CLIP encoder, so that downstream region tokens attend to interaction‐aware context.

c) Prompt sharing among similar interactions Because $\;P_C\in\mathbb R^{L\times D}$6, many HOI categories must share the same basis $\;P_C\in\mathbb R^{L\times D}$7. In practice, we find that semantically or functionally related interactions (e.g.\ “hold cup” vs. “hold bottle”) gravitate to the same few prompt bases—thus encouraging the network to learn hand–object contact patterns that transfer across categories.

2. Concept Calibration Module While CLIP’s text encoder E_T gives us a first‐cut embedding $\;P_C\in\mathbb R^{L\times D}$8 for each HOI label $\;P_C\in\mathbb R^{L\times D}$9, these embeddings remain too coarse to distinguish visually similar actions. We therefore recalibrate them via a two‐stage, language‐model–guided procedure.

a) Intra-modal relationship modeling * We prompt GPT-3.5 to produce a fine‐grained visual description $\{\hat P_{IT}^i\}_{i=1}^M$0 for each HOI category. * We encode $\{\hat P_{IT}^i\}_{i=1}^M$1 using a T5‐based “Instructor” model E_{LM} to obtain $\{\hat P_{IT}^i\}_{i=1}^M$2 * We collect $\{\hat P_{IT}^i\}_{i=1}^M$3 over all categories and run K-means into J clusters $\{\hat P_{IT}^i\}_{i=1}^M$4. By construction, categories in the same $\{\hat P_{IT}^i\}_{i=1}^M$5 share strong visual attributes.

b) Calibration function We then learn a small calibration network $\{\hat P_{IT}^i\}_{i=1}^M$6 (e.g.\ two‐layer MLP) with parameters $\{\hat P_{IT}^i\}_{i=1}^M$7 that fuses the original CLIP text embedding $\{\hat P_{IT}^i\}_{i=1}^M$8 with its instruction embedding $\{\hat P_{IT}^i\}_{i=1}^M$9. Concretely, we set $\hat P_{IT}^i = u^i_{IT}\,(v^i_{IT})^\top$0 and use $\hat P_{IT}^i = u^i_{IT}\,(v^i_{IT})^\top$1 as our final, calibrated concept vector. During training we penalize misalignment between intra-cluster similarities in the text space and those in the visual‐description space: $\hat P_{IT}^i = u^i_{IT}\,(v^i_{IT})^\top$2

c) Visual similarity informs calibration By clustering the T5‐based embeddings $\hat P_{IT}^i = u^i_{IT}\,(v^i_{IT})^\top$3, we capture which HOI pairs appear visually similar (e.g.\ “throwing” vs. “pitching”). The calibration loss above then gently “pulls together” the corresponding $\hat P_{IT}^i = u^i_{IT}\,(v^i_{IT})^\top$4 vectors so that CLIP’s final alignments respect these fine‐grained distinctions.

3. Negative Sampling Strategy To further sharpen the network’s discrimination among closely related HOIs, we inject hard negatives into the classification loss.

a) Strategy * For each training minibatch we collect the ground‐truth HOI categories $\hat P_{IT}^i = u^i_{IT}\,(v^i_{IT})^\top$5. * We find all clusters $\hat P_{IT}^i = u^i_{IT}\,(v^i_{IT})^\top$6 that contain at least one $\hat P_{IT}^i = u^i_{IT}\,(v^i_{IT})^\top$7, call their union $\hat P_{IT}^i = u^i_{IT}\,(v^i_{IT})^\top$8. * We sample a fixed number $\hat P_{IT}^i = u^i_{IT}\,(v^i_{IT})^\top$9 of negative labels from $u^i_{IT}\in\mathbb R^L,\;v^i_{IT}\in\mathbb R^D$0.

b) Pseudo-code $\odot$5 c) Loss term Suppose our decoder produces N interaction features $u^i_{IT}\in\mathbb R^L,\;v^i_{IT}\in\mathbb R^D$1. Let $u^i_{IT}\in\mathbb R^L,\;v^i_{IT}\in\mathbb R^D$2 for each label $u^i_{IT}\in\mathbb R^L,\;v^i_{IT}\in\mathbb R^D$3. Then the classification loss over positives and sampled negatives is $u^i_{IT}\in\mathbb R^L,\;v^i_{IT}\in\mathbb R^D$4 where $u^i_{IT}\in\mathbb R^L,\;v^i_{IT}\in\mathbb R^D$5 is the ground‐truth HOI for query $u^i_{IT}\in\mathbb R^L,\;v^i_{IT}\in\mathbb R^D$6, $u^i_{IT}\in\mathbb R^L,\;v^i_{IT}\in\mathbb R^D$7 the sampled negatives, and $u^i_{IT}\in\mathbb R^L,\;v^i_{IT}\in\mathbb R^D$8 a temperature.

1. Training Objective and Loss Functions All modules are trained end-to-end (prompts, calibration net, decoder heads) via a unified loss $u^i_{IT}\in\mathbb R^L,\;v^i_{IT}\in\mathbb R^D$9 Here
   • $$P_{IT}^i \;=\;\hat P_{IT}^i \odot P_C\,,\quad i=1\ldots M$$0 is the standard ℓ1‐box regression loss over human/object corners,
   • $$P_{IT}^i \;=\;\hat P_{IT}^i \odot P_C\,,\quad i=1\ldots M$$1 is the IoU loss on the two predicted boxes,
   • $$P_{IT}^i \;=\;\hat P_{IT}^i \odot P_C\,,\quad i=1\ldots M$$2 is the cross‐entropy with hard negatives (above),
   • $$P_{IT}^i \;=\;\hat P_{IT}^i \odot P_C\,,\quad i=1\ldots M$$3 enforces intra‐cluster alignment of calibrated text vectors. All $$P_{IT}^i \;=\;\hat P_{IT}^i \odot P_C\,,\quad i=1\ldots M$$4-weights are chosen by grid search; for instance we use $$P_{IT}^i \;=\;\hat P_{IT}^i \odot P_C\,,\quad i=1\ldots M$$5, $$P_{IT}^i \;=\;\hat P_{IT}^i \odot P_C\,,\quad i=1\ldots M$$6, $$P_{IT}^i \;=\;\hat P_{IT}^i \odot P_C\,,\quad i=1\ldots M$$7, $$P_{IT}^i \;=\;\hat P_{IT}^i \odot P_C\,,\quad i=1\ldots M$$8.
2. Experimental Setup and Results

a) Datasets & Metrics * SWIG-HOI: ∼400 actions ×1000 objects with naturally occurring novel combinations. * HICO-DET: 117 actions×80 objects, with 120 rare triplets held out for zero‐shot evaluation. We report mean Average Precision (mAP) under the standard IoU>0.5 criterion.

b) Implementation details * Visual encoder: CLIP ViT-B/16, prompted by our learned $$P_{IT}^i \;=\;\hat P_{IT}^i \odot P_C\,,\quad i=1\ldots M$$9. * Prompt length $\odot$0 for HICO, $\odot$1 for SWIG; number of bases $\odot$2 (SWIG), $\odot$3 (HICO). * Selection top-K=2 prompts per image. * Clusters J=64; $\odot$4 hard negatives per batch. * Optimizer: AdamW, training for 80 epochs, batch size 128, on 2×3090 GPUs.

c) Main results * On HICO-DET (zero‐shot split), INP-CC achieves 23.12 mAP (full) vs. 22.35 mAP of the prior open-vocabulary state-of-the-art (CMD-SE). * On SWIG-HOI, INP-CC reaches 16.74 mAP (full), a +1.48 mAP gain over CMD-SE’s 15.26. Rare/unseen splits improve by up to +0.85 mAP.

d) Ablation studies * Turning off prompts entirely (common‐only) yields 14.43 mAP; adding just INP rises to 15.54. * Adding only Concept Calibration (no INP) gives 15.30 mAP; combining both yields 16.74 mAP. * Varying prompt size shows 128 tokens + selection is optimal; smaller or larger banks degrade performance. * Clustering in CLIP space vs. GPT-T5 space: only the latter (Instructor embeddings over GPT descriptions) gives the full +1.44 gain. * Negative sampling: “hard” (cluster‐based) negatives outperform “random” or “easy” by ∼0.6 mAP.

Taken together, these experiments confirm that (1) interaction-adaptive prompts bridge CLIP’s image‐level bias to fine‐grained HOI regions, (2) language‐model–guided calibration shapes a richer text–vision alignment, and (3) hard negative sampling further sharpens category boundaries—resulting in significant improvements on two challenging open-vocabulary HOI benchmarks.