---
title: 'Active-Passive Gap: Interaction-Aware Modeling'
url: https://www.emergentmind.com/topics/active-passive-gap
type: topic
---

# Active-Passive Gap: Interaction-Aware Modeling

Below is a unified, step‐by‐step description of INP-CC that weaves together all five of your requested components. Wherever possible, we give the precise network structures, the LaTeX formulas, and the experimental evidence for each module’s contribution.

1.  Interaction-Aware Prompt Generator  
   In INP-CC we augment the frozen CLIP image encoder E_V with a small, learnable “prompt bank” so that region features become sensitive to human–object contact patterns rather than only global scene statistics.  

   a) Network architecture  
   ‑ Input: an image I is first embedded by E_V into a spatial feature map  
       \(F_{scene} = E_V(I)\in\mathbb R^{HW\times C}\).  
   ‑ Prompt Bank: we maintain  
       • A common prompt  
         \(\;P_C\in\mathbb R^{L\times D}\),  
       • M interaction‐specific bases \(\{\hat P_{IT}^i\}_{i=1}^M\), each a rank-one factorization  
         \(\hat P_{IT}^i = u^i_{IT}\,(v^i_{IT})^\top\),  
         with \(u^i_{IT}\in\mathbb R^L,\;v^i_{IT}\in\mathbb R^D\).  
   ‑ Composition: each full interaction prompt is  
       \[
         P_{IT}^i \;=\;\hat P_{IT}^i \odot P_C\,,\quad i=1\ldots M
       \]  
       where \(\odot\) is element‐wise (Hadamard) multiplication.  
   ‑ Prompt‐to‐Query Adapter: an MLP \(g:\mathbb R^{L\times D}\to\mathbb R^D\) computes a key  
       \(k_{IT}^i=g(P_{IT}^i)\).  

   b) Adaptive selection  
   We first extract a global “fingerprint” \(f_I\in\mathbb R^D\) from \(F_{scene}\) (e.g.\ average‐pool + linear).  Then for each \(i\) we form  
       \[
         w_i \;=\;\cos\bigl(f_I,\;k_{IT}^i\bigr)\,,
       \]  
   select the top‐K prompts by descending \(w_i\), call the selected set \(\mathbb K_{IT}\), and finally form our scene‐specific prompt  
       \[
         P_{IA}
         \;=\;
         \sum_{i\in\mathbb K_{IT}} w_i\;P_{IT}^i\,.
       \]  
   This “soft” top‐K mixture is then prepended to the patch embeddings before the transformer layers of the CLIP encoder, so that downstream region tokens attend to interaction‐aware context.  

   c) Prompt sharing among similar interactions  
   Because \(M\ll|\mathbb A\times\mathbb O|\), many HOI categories must share the same basis \(\hat P_{IT}^i\).  In practice, we find that semantically or functionally related interactions (e.g.\ “hold cup” vs. “hold bottle”) gravitate to the same few prompt bases—thus encouraging the network to learn hand–object contact patterns that transfer across categories.

2.  Concept Calibration Module  
   While CLIP’s text encoder E_T gives us a first‐cut embedding \(T_{hoi}^{a,o}\) for each HOI label \((a,o)\), these embeddings remain too coarse to distinguish visually similar actions.  We therefore recalibrate them via a two‐stage, language‐model–guided procedure.  

   a) Intra-modal relationship modeling  
   – We prompt GPT-3.5 to produce a fine‐grained visual description \(d_{a,o}\) for each HOI category.  
   – We encode \(d_{a,o}\) using a T5‐based “Instructor” model E_{LM} to obtain  
       \[
         T_{vis}^{a,o}\;=\;E_{LM}(d_{a,o})\in\mathbb R^D.
       \]  
   – We collect \(\{T_{vis}^{a,o}\}\) over all categories and run K-means into J clusters  
       \(\mathbb C=\{C_1,\dots,C_J\}\).  By construction, categories in the same \(C_j\) share strong visual attributes.  

   b) Calibration function  
   We then learn a small calibration network \(h:\mathbb R^D\!\times\mathbb R^D\to\mathbb R^D\) (e.g.\ two‐layer MLP) with parameters \(\Theta_{calib}\) that fuses the original CLIP text embedding \(T_{hoi}^{a,o}\) with its instruction embedding \(T_{vis}^{a,o}\).  Concretely, we set  
       \[
         \widetilde T_{hoi}^{a,o}
         \;=\;
         T_{hoi}^{a,o}
         \;+\;
         h\bigl(T_{hoi}^{a,o},\,T_{vis}^{a,o};\,\Theta_{calib}\bigr),
       \]  
   and use \(\widetilde T_{hoi}^{a,o}\) as our final, calibrated concept vector.  During training we penalize misalignment between intra-cluster similarities in the text space and those in the visual‐description space:  
       \[
         \mathcal L_{calibration}
         \;=\;
         \sum_{j=1}^J\!\sum_{(a,o),(a',o')\in C_j}
         \Bigl\|\cos(\widetilde T_{hoi}^{a,o},\,\widetilde T_{hoi}^{a',o'})
                  \;-\;\cos(T_{vis}^{a,o},\,T_{vis}^{a',o'})\Bigr\|_2^2.
       \]  

   c) Visual similarity informs calibration  
   By clustering the T5‐based embeddings \(\{T_{vis}\}\), we capture which HOI pairs appear visually similar (e.g.\ “throwing” vs. “pitching”).  The calibration loss above then gently “pulls together” the corresponding \(\widetilde T_{hoi}\) vectors so that CLIP’s final alignments respect these fine‐grained distinctions.

3.  Negative Sampling Strategy  
   To further sharpen the network’s discrimination among closely related HOIs, we inject hard negatives into the classification loss.  

   a) Strategy  
   – For each training minibatch we collect the ground‐truth HOI categories \(\mathcal G\).  
   – We find all clusters \(C_j\) that contain at least one \(g\in\mathcal G\), call their union \(\mathcal N_{\mathrm{cand}}\).  
   – We sample a fixed number \(N_{neg}\) of negative labels from \(\mathcal N_{\mathrm{cand}}\setminus\mathcal G\).  

   b) Pseudo-code  
     ```
     Given batch GT set G
     clusters_used = { j | ∃ g∈G with g∈C_j }
     neg_cands = ⋃_{j∈clusters_used} C_j \ G
     NegSamples = random_sample(neg_cands, N_neg)
     ```
   c) Loss term  
   Suppose our decoder produces N interaction features \(h_i\).  Let \(s_{i,c}=\cos(h_i,\widetilde T_{hoi}^c)\) for each label \(c\).  Then the classification loss over positives and sampled negatives is  
       \[
         \mathcal L_{cls}
         \;=\;
         -\frac1N\sum_{i=1}^N
           \Bigl[
             \log\frac{\exp(s_{i,y_i}/\tau)}{\sum_{p\in\{y_i\}\cup Neg_i}\exp(s_{i,p}/\tau)}
           \Bigr],
       \]  
   where \(y_i\) is the ground‐truth HOI for query \(i\), \(Neg_i\) the sampled negatives, and \(\tau\) a temperature.

4.  Training Objective and Loss Functions  
   All modules are trained end-to-end (prompts, calibration net, decoder heads) via a unified loss  
     \[
       \mathcal L_{total}
       \;=\;
       \lambda_b\,\mathcal L_{b}
       \;+\;
       \lambda_{iou}\,\mathcal L_{iou}
       \;+\;
       \lambda_{cls}\,\mathcal L_{cls}
       \;+\;
       \lambda_{calib}\,\mathcal L_{calibration}.
     \]  
   Here  
   – \(\mathcal L_{b}\) is the standard ℓ1‐box regression loss over human/object corners,  
   – \(\mathcal L_{iou}\) is the IoU loss on the two predicted boxes,  
   – \(\mathcal L_{cls}\) is the cross‐entropy with hard negatives (above),  
   – \(\mathcal L_{calibration}\) enforces intra‐cluster alignment of calibrated text vectors.  
   All \(\lambda\)-weights are chosen by grid search; for instance we use \(\lambda_b\!=\!5\), \(\lambda_{iou}\!=\!2\), \(\lambda_{cls}\!=\!1\), \(\lambda_{calib}\!=\!0.1\).

5.  Experimental Setup and Results  

   a) Datasets & Metrics  
   • SWIG-HOI: ∼400 actions ×1000 objects with naturally occurring novel combinations.  
   • HICO-DET: 117 actions×80 objects, with 120 rare triplets held out for zero‐shot evaluation.  
   We report mean Average Precision (mAP) under the standard IoU>0.5 criterion.  

   b) Implementation details  
   • Visual encoder: CLIP ViT-B/16, prompted by our learned \(P_{IA}\).  
   • Prompt length \(L=8\) for HICO, \(L=128\) for SWIG; number of bases \(M=128\) (SWIG), \(M=8\) (HICO).  
   • Selection top-K=2 prompts per image.  
   • Clusters J=64; \(N_{neg}=10\) hard negatives per batch.  
   • Optimizer: AdamW, training for 80 epochs, batch size 128, on 2×3090 GPUs.  

   c) Main results  
   – On HICO-DET (zero‐shot split), INP-CC achieves 23.12 mAP (full) vs. 22.35 mAP of the prior open-vocabulary state-of-the-art (CMD-SE).  
   – On SWIG-HOI, INP-CC reaches 16.74 mAP (full), a +1.48 mAP gain over CMD-SE’s 15.26.  Rare/unseen splits improve by up to +0.85 mAP.  

   d) Ablation studies  
   • Turning off prompts entirely (common‐only) yields 14.43 mAP; adding just INP rises to 15.54.  
   • Adding only Concept Calibration (no INP) gives 15.30 mAP; combining both yields 16.74 mAP.  
   • Varying prompt size shows 128 tokens + selection is optimal; smaller or larger banks degrade performance.  
   • Clustering in CLIP space vs. GPT-T5 space: only the latter (Instructor embeddings over GPT descriptions) gives the full +1.44 gain.  
   • Negative sampling: “hard” (cluster‐based) negatives outperform “random” or “easy” by ∼0.6 mAP.  

Taken together, these experiments confirm that (1) interaction-adaptive prompts bridge CLIP’s image‐level bias to fine‐grained HOI regions, (2) language‐model–guided calibration shapes a richer text–vision alignment, and (3) hard negative sampling further sharpens category boundaries—resulting in significant improvements on two challenging open-vocabulary HOI benchmarks.

Source: https://www.emergentmind.com/topics/active-passive-gap