Funnel-HOI: Zero-Shot HOI Detection

• The paper introduces Funnel-HOI, a zero-shot HOI detection framework that anticipates object cues and refines action cues at the encoder stage.
• It employs dual asymmetric co-attention modules (OSACA and OVACA) to fuse visual and semantic features, improving detection in long-tail and unseen regimes.
• The study demonstrates significant gains on HICO-DET and V-COCO by incorporating an ORDis loss that modulates overconfident negatives and enhances overall interaction prediction.

Searching arXiv for the cited Funnel-HOI paper and related naming variants to ground the article. Funnel-HOI is a zero-shot human–object interaction detection framework that imposes a top-down perception paradigm on HOI detection: it first anticipates well-defined object cues and then probes object-conditional action cues, with the goal of strengthening interaction representations already at the encoder stage rather than relying primarily on decoder refinement (Sarma et al., 16 Jul 2025). The method was introduced for HOI detection, where an interaction is a triplet $(h,o,a)$ consisting of human and object bounding boxes and an action label, and where an HOI class is written as $y=\langle a,o\rangle$, with $a \in \mathcal{A}=\{a_1,\ldots,a_M\}$ and $o \in \mathcal{O}=\{o_1,\ldots,o_N\}$ (Sarma et al., 16 Jul 2025). Its design directly targets two persistent difficulties in HOID: long-tail class distributions and zero-shot regimes in which some $\langle a,o\rangle$ combinations, or even entire verbs or objects, are unseen during training.

1. Problem setting and conceptual motivation

HOI detection aims to localize interactive human–object pairs and classify their interactions. In the formulation used by Funnel-HOI, the dataset contains seen classes $\mathcal{Y}^s$ and unseen classes $\mathcal{Y}^u$, with $\mathcal{Y}=\mathcal{Y}^s \cup \mathcal{Y}^u$ and $\mathcal{Y}^s \cap \mathcal{Y}^u=\varnothing$. Training data consist only of seen triplets: $$\mathcal{D}^{train}=\{ I_k, \{ H^k_p \}_{p=1}^{N_h}, \{ O^k_q \}_{q=1}^{N_o}, \{ y^k_r \}_{r=1}^{N_i} \}_{k=1}^{K}, \quad y^k_r \in \mathcal{Y}^s.$$ This setting makes the zero-shot problem explicit: models must generalize beyond the interaction combinations observed during training (Sarma et al., 16 Jul 2025).

The method is motivated by two criticisms of prevailing DETR-based zero-shot HOID systems. First, many such systems concentrate on decoder design to disentangle or refine interaction features while leaving the encoder vanilla and purely visual. Second, semantic knowledge is often injected mainly at final classification, even though zero-shot generalization depends on visual–semantic associations throughout the pipeline. Funnel-HOI argues that HOI-specific cues should be anticipated at the encoder stage itself. Its object-first design is presented as being inspired by the human tendency to grasp well-defined concepts first and then associate them with abstract concepts during scene understanding; objects are treated as well-defined concepts, while actions are treated as more abstract and polysemous concepts (Sarma et al., 16 Jul 2025).

A central implication of this design is that scene interpretation is not deferred to later interaction decoding. Instead, the encoder is tasked with shaping the feature space around object compatibility and object-conditioned action plausibility before the decoders consume those features. This suggests a deliberate shift from decoder-centric HOID toward encoder-level multimodal structuring.

2. Top-down architecture and encoder-level interaction formation

Funnel-HOI uses a CNN backbone and a CLIP-based multimodal stack. A ResNet-50 backbone extracts visual features $y=\langle a,o\rangle$0 with $y=\langle a,o\rangle$1, while a CLIP ViT-B/32 image encoder produces a global visual embedding $y=\langle a,o\rangle$2 with $y=\langle a,o\rangle$3. CLIP text encoders generate embeddings for both objects and verbs (Sarma et al., 16 Jul 2025).

The architecture is organized around a top-down encoder with two sequential probing stages. The first stage, Object-Specific Asymmetric Co-Attention (OSACA), nominates $y=\langle a,o\rangle$4 objects, explicitly including “person,” and produces an object-probed feature map $y=\langle a,o\rangle$5. The second stage, Object-conditional Verb-specific Asymmetric Co-Attention (OVACA), nominates $y=\langle a,o\rangle$6 verbs using object–verb relatedness and scene alignment, then computes a verb-probed feature map $y=\langle a,o\rangle$7 conditioned on the object evidence. Interaction encoding is then formed directly at the encoder level by fusing the verb-probed features with the vanilla Transformer encoder output: $y=\langle a,o\rangle$8 This encoder output is passed to an instance decoder and an interaction decoder, following HOICLIP, to produce interaction embeddings and HOI scores. DETR-style query slots specialize over the scene, with $y=\langle a,o\rangle$9 learnable queries (Sarma et al., 16 Jul 2025).

This arrangement is notable because the encoder is no longer a generic visual preprocessor. It becomes the site where multimodal prior knowledge, scene-level evidence, and HOI-specific anticipation are fused. A plausible implication is that the decoder receives features that are already biased toward plausible human–object–action structure, reducing the burden on later stages to recover interaction semantics from unstructured visual tokens.

3. Asymmetric co-attention and multimodal probing

The asymmetric co-attention mechanism operates between per-location visual features and nominated semantic embeddings. Let $a \in \mathcal{A}=\{a_1,\ldots,a_M\}$0 be the reshaped backbone features and $a \in \mathcal{A}=\{a_1,\ldots,a_M\}$1 the nominated object embeddings, where $a \in \mathcal{A}=\{a_1,\ldots,a_M\}$2. Funnel-HOI projects both into a shared latent space of dimension $a \in \mathcal{A}=\{a_1,\ldots,a_M\}$3: $a \in \mathcal{A}=\{a_1,\ldots,a_M\}$4 It then constructs a rectangular affinity matrix

$a \in \mathcal{A}=\{a_1,\ldots,a_M\}$5

which captures simultaneous affinities between visual locations and multiple nominated object concepts (Sarma et al., 16 Jul 2025).

Two co-attention directions are computed. In the text-to-visual direction,

$a \in \mathcal{A}=\{a_1,\ldots,a_M\}$6

and in the visual-to-text direction,

$a \in \mathcal{A}=\{a_1,\ldots,a_M\}$7

The influence of $a \in \mathcal{A}=\{a_1,\ldots,a_M\}$8 is injected into attention via concatenation or conditioning, and the resulting streams are fused through

$a \in \mathcal{A}=\{a_1,\ldots,a_M\}$9

$o \in \mathcal{O}=\{o_1,\ldots,o_N\}$0

yielding the object-probed feature map

$o \in \mathcal{O}=\{o_1,\ldots,o_N\}$1

For action probing, the visual input is first conditioned on object evidence,

$o \in \mathcal{O}=\{o_1,\ldots,o_N\}$2

after which the same asymmetric co-attention pattern is applied using nominated verbs to produce $o \in \mathcal{O}=\{o_1,\ldots,o_N\}$3 (Sarma et al., 16 Jul 2025).

The nomination mechanism is itself multimodal. Object nomination is based on scene–object similarity using CLIP: $o \in \mathcal{O}=\{o_1,\ldots,o_N\}$4 where object text embeddings are built from prompts of the form “A photo of a/an [OBJ].” Verb embeddings use prompts of the form “A photo of a person [VERB]-ing.” Object–Verb Relatedness (OVeR) is defined as

$o \in \mathcal{O}=\{o_1,\ldots,o_N\}$5

For each nominated object, top verbs are selected from $o \in \mathcal{O}=\{o_1,\ldots,o_N\}$6 and then re-scored by scene alignment and object confidence: $o \in \mathcal{O}=\{o_1,\ldots,o_N\}$7 The top-$o \in \mathcal{O}=\{o_1,\ldots,o_N\}$8 verbs become the nominated verb set $o \in \mathcal{O}=\{o_1,\ldots,o_N\}$9 (Sarma et al., 16 Jul 2025).

This object-first then action-second funnel is the central operational idea of the model. OSACA steers the encoder toward likely object regions; OVACA then constrains verb probing by object compatibility, making action prediction conditional rather than globally unconstrained.

4. Object–action relatedness and the ORDis loss

Funnel-HOI supplements its encoder design with a new interaction-classification loss, Object-Regulated Discrepancy (ORDis), motivated by the claim that standard focal loss ignores object–verb relatedness and inter-class discrepancy among HOI labels. ORDis introduces three multiplicative regulators: an object perception factor $\langle a,o\rangle$0, a relatedness factor $\langle a,o\rangle$1, and an inter-class discrepancy term $\langle a,o\rangle$2 (Sarma et al., 16 Jul 2025).

The object perception factor is higher when the Hungarian matching cost of an optimal slot is lower: $\langle a,o\rangle$3 with $\langle a,o\rangle$4 and $\langle a,o\rangle$5. The relatedness factor is instantiated from OVeR: $\langle a,o\rangle$6 For negative classes, the discrepancy factor compares a negative class probability with the mean positive probability: $\langle a,o\rangle$7 while $\langle a,o\rangle$8 for positive classes (Sarma et al., 16 Jul 2025).

The full ORDis loss scales focal loss by a sigmoidal actuator: $\langle a,o\rangle$9 with $\mathcal{Y}^s$0. In the description accompanying the formula, this actuator increases the penalty for negative predictions that are both incompatible and overconfident, while modulating the penalty by object association confidence. The stated purpose is to better handle class imbalance and zero-shot bias toward seen classes by explicitly leveraging object–verb relatedness and inter-class discrepancies (Sarma et al., 16 Jul 2025).

Training follows DETR-style bipartite matching via the Hungarian algorithm. The total loss is

$\mathcal{Y}^s$1

where $\mathcal{Y}^s$2 is human/object box regression, $\mathcal{Y}^s$3 is generalized IoU, and $\mathcal{Y}^s$4 is object classification. The reported coefficients are $\mathcal{Y}^s$5, $\mathcal{Y}^s$6, and $\mathcal{Y}^s$7. Training uses 90 epochs, batch size 8, initial learning rate $\mathcal{Y}^s$8, and a decay factor of $\mathcal{Y}^s$9 at epochs 30 and 60. During zero-shot training, the interaction classifier output dimension is $\mathcal{Y}^u$0; at HICO-DET evaluation, it is 600 (Sarma et al., 16 Jul 2025).

At inference, the model performs end-to-end set prediction: each query predicts at most one human–object pair and its HOI scores, and no NMS is required. HOI triplets and confidences are read directly from the per-slot predictions.

5. Datasets, zero-shot settings, and reported empirical results

Funnel-HOI is evaluated on HICO-DET and V-COCO. HICO-DET contains 47,776 images and 600 HOI classes formed from 80 objects and 117 verbs; 138 HOIs are rare, defined as having fewer than 10 training instances, and 462 are non-rare. Evaluation uses the “Default” and “Known Object” protocols, reporting mAP overall and separately for rare and non-rare classes. V-COCO contains 10,346 images, 80 objects, and 29 actions, and reports Role AP in two scenarios: $\mathcal{Y}^u$1, which includes actions without objects, and $\mathcal{Y}^u$2, which excludes them (Sarma et al., 16 Jul 2025).

For HICO-DET zero-shot evaluation, six settings are used. UC holds out 120 of the 600 $\mathcal{Y}^u$3 combinations while keeping all verbs and objects seen, using five random 480/120 splits. RF-UC chooses unseen combinations from rare HOIs; NF-UC chooses them from non-rare HOIs. UO makes 12 objects unseen during training, producing 500 seen and 100 unseen HOIs. UV uses 20 unseen verbs under the GEN-VLKT protocol. UA uses 22 unseen verbs under the ConsNet protocol (Sarma et al., 16 Jul 2025).

The reported zero-shot HICO-DET results place Funnel-HOI at 24.50$\mathcal{Y}^u$41.39 unseen, 31.89$\mathcal{Y}^u$50.82 seen, and 30.41$\mathcal{Y}^u$60.47 full mAP in UC, surpassing both EoID and GEN-VLKT. In RF-UC it reports 25.90 unseen, 34.43 seen, and 32.70 full, with the best unseen and second-best seen and full. In NF-UC it reports 26.63 unseen, 28.32 seen, and 28.00 full, with the best seen and full and second-best unseen behind EoID. In UO it reports 15.28 unseen, 31.50 seen, and 28.66 full, improving over HOICLIP with up to 12.4% relative gain on unseen. In UA it reports 24.11 unseen, 31.85 seen, and 30.56 full; in UV, 26.10 unseen, 32.56 seen, and 31.66 full (Sarma et al., 16 Jul 2025).

Under supervised HICO-DET evaluation, Funnel-HOI reports 34.36 full, 31.07 rare, and 35.33 non-rare mAP in the Default setting, and 37.24 full, 33.76 rare, and 38.27 non-rare in Known Object. The paper states gains of up to 8.4% for rare HOIs over recent one-stage baselines, using a ResNet-50 backbone for fair comparison. On V-COCO, the supervised results are $\mathcal{Y}^u$7 and $\mathcal{Y}^u$8, described as comparable to recent transformer-based methods such as GEN-VLKT at 62.4 and 64.4 (Sarma et al., 16 Jul 2025).

Ablations attribute performance gains to the complementarity of the three main components. In NF-UC full mAP, OSACA only gives 27.15, OSACA+OVACA gives 27.66, OSACA+ORDis gives 27.73, ORDis only gives 27.77, and the full configuration OSACA+OVACA+ORDis gives 28.00. Factor-wise analysis of ORDis shows $\mathcal{Y}^u$9 only at 27.07, $\mathcal{Y}=\mathcal{Y}^s \cup \mathcal{Y}^u$0 only at 27.40, $\mathcal{Y}=\mathcal{Y}^s \cup \mathcal{Y}^u$1 only at 27.40, and the full $\mathcal{Y}=\mathcal{Y}^s \cup \mathcal{Y}^u$2 combination at 28.00. The paper interprets these effects as follows: $\mathcal{Y}=\mathcal{Y}^s \cup \mathcal{Y}^u$3 improves unseen performance through global compatibility, $\mathcal{Y}=\mathcal{Y}^s \cup \mathcal{Y}^u$4 reduces overconfident negatives, and $\mathcal{Y}=\mathcal{Y}^s \cup \mathcal{Y}^u$5 boosts confidence via object association. Sensitivity studies show best full mAP at $\mathcal{Y}=\mathcal{Y}^s \cup \mathcal{Y}^u$6 and $\mathcal{Y}=\mathcal{Y}^s \cup \mathcal{Y}^u$7, with degradation when either is made smaller or larger (Sarma et al., 16 Jul 2025).

Efficiency measurements indicate modest overhead relative to HOICLIP: 66.8M parameters versus 66.1M, 88.49 GFLOPs versus 88.23, and 56.59 ms per image versus 55.52. The same section reports sensitivity to the pretrained VLM: CLIP ViT-B/16 improves NF-UC performance by approximately 8% relative over ViT-B/32, while ALIP degrades performance by approximately 11%, indicating dependence on pretrained VLM quality.

6. Qualitative behavior, limitations, and terminological distinction

The qualitative analyses describe a progressive sharpening of attention over interaction regions. OSACA steers attention toward likely objects, OVACA refines action cues conditioned on those object cues, and ORDis further concentrates attention within detection boxes by penalizing incompatible overconfident negatives. Reported success cases include rare interactions such as cut orange and wash bicycle with only 7 training images, small objects such as toothbrush and tie under degradation, and partially occluded objects such as skis. In comparison examples, Funnel-HOI often localizes both interacting instances where HOICLIP misses one (Sarma et al., 16 Jul 2025).

The reported limitations are explicit. Performance depends on pretrained VLM quality, with weaker VLMs reducing gains. Fixed verb prompts of the form “[VERB]-ing” may not capture all semantic nuances, creating a form of prompt rigidity in the presence of verb polysemy. The $\mathcal{Y}=\mathcal{Y}^s \cup \mathcal{Y}^u$8 and $\mathcal{Y}=\mathcal{Y}^s \cup \mathcal{Y}^u$9 nomination scheme keeps probing tractable, but scaling to much larger object and verb vocabularies may increase compute; dynamic nomination or retrieval is suggested as a possible mitigation. Long-tail distributions and labeling noise can still bias predictions, and the discussion points to curriculum or debiasing strategies as possible future directions. Proposed future work includes extending top-down multimodal probing to videos, incorporating richer knowledge graphs, and exploring adaptive co-attention heads for finer-grained affordances (Sarma et al., 16 Jul 2025).

A terminological distinction is necessary because the label “F-HOI” appears in separate literature. In the 2024 paper “F-HOI: Toward Fine-grained Semantic-Aligned 3D Human-Object Interactions,” the name “F-HOI” denotes “Fine-grained” HOI and refers to a unified, instruction-driven framework for 3D HOI understanding, reasoning, generation, and reconstruction on the Semantic-HOI dataset, not to the zero-shot top-down HOI detector described above (Yang et al., 2024). The term “Funnel-HOI” does not appear in that paper. This distinction matters because the two works address different problem settings: one is a 2D zero-shot HOI detection system built around encoder-level multimodal probing and ORDis; the other is an MLLM-centered framework for fine-grained semantic alignment of 3D HOI states and transitions.