HOLa Zero-Shot HOI Detection

• HOLa is a zero-shot human-object interaction detection method that decomposes VLM text features into a class-shared basis and adaptable per-class weights.
• The approach refines action discrimination using lightweight weight adaptation, human-object tokens, and LLM-derived action regularization to effectively distinguish unseen actions.
• Empirical results on HICO-DET demonstrate state-of-the-art performance with an unseen mAP of 27.91, validating its robust generalization to novel HOI classes.

HOLa is a zero-shot human-object interaction detection method that addresses the problem of detecting interactions such as “person-ride-bike” when training annotations cover only a subset of HOI classes and test-time evaluation includes novel classes. It uses Vision-LLMs to access knowledge beyond the training data, but departs from prior approaches by decomposing VLM text features into a class-shared basis and per-class adaptable weights, refining only the weights for class discrimination, enriching the visual encoder with human-object tokens, and guiding unseen-action distinction with LLM-derived action regularization. On HICO-DET, HOLa sets a new state-of-the-art across zero-shot HOI settings, including an unseen-class mAP of 27.91 in the unseen-verb setting, while using only approximately 4 M trainable parameters (Lei et al., 21 Jul 2025).

1. Problem setting and design objective

Human-Object Interaction detection requires locating humans and objects in an image and predicting their pairwise interaction. In the zero-shot setting, during training the model only sees annotations for a subset of HOI classes, denoted “seen,” yet at test time it must detect instances of novel “unseen” HOIs. Three common zero-shot splits on HICO-DET are used to structure this evaluation: unseen-verb, where some verbs are held out entirely; unseen-composition, where certain action-object pairs never appear together in train; and unseen-object, where some objects are held out (Lei et al., 21 Jul 2025).

The central difficulty identified for zero-shot HOI detection is twofold. Existing methods that tap VLMs to access knowledge beyond the training data either struggle to distinguish actions involving the same object or demonstrate limited generalization to unseen classes. HOLa is designed explicitly around this tension. Its low-rank decomposition is intended to preserve information shared across seen and unseen HOIs, while its weight adaptation and human-object token design target action distinction. This suggests that the method treats generalization and discrimination not as separate subproblems, but as coupled constraints on the text and visual representations.

2. Low-rank decomposed HOI text representation

HOLa begins from frozen VLM text embeddings for the HOI class descriptions. Given $N$ HOI class descriptions, the frozen text encoder produces

$F = [f_1,\dots,f_N]^\top\in\mathbb{R}^{N\times d}.$

The method learns a basis matrix

$B=[b_1,\dots,b_m]\in\mathbb{R}^{d\times m},$

and per-class weights

$W=[w_1,\dots,w_N]^\top\in\mathbb{R}^{N\times m},$

with $m\ll d$. The decomposition is trained by minimizing the reconstruction error

$L_{\rm recon}^1 \;=\;\|F - W\,B^\top\|_2^2$

subject to the sparsity term

$L_{\rm sparse}^1 = \sum_{i=1}^N\|w_i\|_1$

and the orthogonality term

$L_{\rm ort} = \sum_{i\neq j} (b_i^\top b_j).$

At inference, each reconstructed text embedding is

$\hat f_i = w_i\,B^\top.$

The decomposition yields a compact HOI representation in which the basis $B$ is class-shared and the weights $F = [f_1,\dots,f_N]^\top\in\mathbb{R}^{N\times d}.$0 are class-specific. During training, both $F = [f_1,\dots,f_N]^\top\in\mathbb{R}^{N\times d}.$1 and $F = [f_1,\dots,f_N]^\top\in\mathbb{R}^{N\times d}.$2 are updated under the decomposition losses. However, to preserve unseen-class information in $F = [f_1,\dots,f_N]^\top\in\mathbb{R}^{N\times d}.$3, HOLa does not apply the HOI classification loss on $F = [f_1,\dots,f_N]^\top\in\mathbb{R}^{N\times d}.$4; it applies the classification loss only on $F = [f_1,\dots,f_N]^\top\in\mathbb{R}^{N\times d}.$5 (Lei et al., 21 Jul 2025).

This design encodes a specific inductive bias. The basis is intended to preserve shared information across classes, including information relevant to unseen HOIs, whereas the weights carry the adaptable part required for class-level separation. A plausible implication is that the low-rank representation acts as a factorization between transferable semantic structure and discriminative class assignment.

3. Weight adaptation and visual interaction encoding

After the low-rank factorization, HOLa refines action distinction through a lightweight text adapter that updates only the weight matrix $F = [f_1,\dots,f_N]^\top\in\mathbb{R}^{N\times d}.$6, not the basis $F = [f_1,\dots,f_N]^\top\in\mathbb{R}^{N\times d}.$7. The adaptation uses self- and cross-attention with image features. By restricting refinement to $F = [f_1,\dots,f_N]^\top\in\mathbb{R}^{N\times d}.$8, the method attempts to increase class discrimination while retaining the basis as the shared carrier of unseen-class information (Lei et al., 21 Jul 2025).

On the visual side, HOLa introduces human-object tokens into the VLM visual encoder. For each detected human $F = [f_1,\dots,f_N]^\top\in\mathbb{R}^{N\times d}.$9 and object $B=[b_1,\dots,b_m]\in\mathbb{R}^{d\times m},$0, detected via a DETR detector, the model extracts appearance features $B=[b_1,\dots,b_m]\in\mathbb{R}^{d\times m},$1 and $B=[b_1,\dots,b_m]\in\mathbb{R}^{d\times m},$2. It also computes a spatial feature from center offsets, IoU, and relative size through an MLP. These are combined into a human-object token. Optionally, an LLM-generated interaction prior for the pair $B=[b_1,\dots,b_m]\in\mathbb{R}^{d\times m},$3 is encoded by the VLM text encoder and fused into the token via cross-attention. All human-object tokens are injected alongside image patches into the VLM visual transformer with adapter layers. The resulting token outputs and the global feature map are then used for HOI prediction.

The role of these tokens is to enrich visual interaction representations beyond standalone human or object appearance. This is especially relevant for distinguishing actions that involve the same object category. The architecture therefore links the refined text-side weights to a visual pipeline that is explicitly pairwise and interaction-centric rather than purely object-centric.

4. LLM-derived action regularization and training losses

To further distinguish unseen actions, HOLa uses LLM-derived action regularization. The method generates LLM descriptions for each atomic action $B=[b_1,\dots,b_m]\in\mathbb{R}^{d\times m},$4, encodes them into

$B=[b_1,\dots,b_m]\in\mathbb{R}^{d\times m},$5

and decomposes them with

$B=[b_1,\dots,b_m]\in\mathbb{R}^{d\times m},$6

where $B=[b_1,\dots,b_m]\in\mathbb{R}^{d\times m},$7 are the action-related basis rows, yielding $B=[b_1,\dots,b_m]\in\mathbb{R}^{d\times m},$8. HOLa then selects the corresponding subset $B=[b_1,\dots,b_m]\in\mathbb{R}^{d\times m},$9 aligned with action basis indices and imposes

$W=[w_1,\dots,w_N]^\top\in\mathbb{R}^{N\times m},$0

This KL divergence is used to guide HOI weight adaptation toward LLM-derived action distributions, with the stated purpose of improving unseen-action distinction (Lei et al., 21 Jul 2025).

The full training objective combines decomposition, classification, and semantic regularization. The final action logits $W=[w_1,\dots,w_N]^\top\in\mathbb{R}^{N\times m},$1 are computed from cosine similarity between union-region visual features and reconstructed text features, between human-object tokens and reconstructed text features, and between fused human+object visual features and action-object text features. The classification term is a focal loss,

$W=[w_1,\dots,w_N]^\top\in\mathbb{R}^{N\times m},$2

and the semantic term $W=[w_1,\dots,w_N]^\top\in\mathbb{R}^{N\times m},$3 is a KL divergence between pairwise similarities in $W=[w_1,\dots,w_N]^\top\in\mathbb{R}^{N\times m},$4 and the original $W=[w_1,\dots,w_N]^\top\in\mathbb{R}^{N\times m},$5 among same-object HOIs.

$W=[w_1,\dots,w_N]^\top\in\mathbb{R}^{N\times m},$6

$W=[w_1,\dots,w_N]^\top\in\mathbb{R}^{N\times m},$7

A compact view of the loss structure is given below.

Term	Definition	Stated function
$W=[w_1,\dots,w_N]^\top\in\mathbb{R}^{N\times m},$8	Reconstruction losses for HOI and action text features	Low-rank decomposition fidelity
$W=[w_1,\dots,w_N]^\top\in\mathbb{R}^{N\times m},$9	$m\ll d$0 sparsity on weights	Sparse weight structure
$m\ll d$1	Orthogonality among basis vectors	Basis disentanglement
$m\ll d$2	$m\ll d$3	LLM-guided action regularization
$m\ll d$4	Focal classification loss	HOI supervision
$m\ll d$5	KL between pairwise similarities in $m\ll d$6 and $m\ll d$7	Same-object semantic preservation

Taken together, the objective formalizes the method’s two primary goals: preservation of shared semantic structure through decomposition and sharper action discrimination through weight adaptation and action-aware regularization.

5. Backbone configuration and computational profile

The implementation uses DETR with ResNet50 as the visual backbone for human/object detection, fine-tuned on HICO-DET, and CLIP with a ViT-B/32 backbone as the VLM. The decomposition ranks are $m\ll d$8 for HOI and $m\ll d$9 for action. The loss weights are

$L_{\rm recon}^1 \;=\;\|F - W\,B^\top\|_2^2$0

Optimization uses AdamW with learning rate $L_{\rm recon}^1 \;=\;\|F - W\,B^\top\|_2^2$1 and batch size $L_{\rm recon}^1 \;=\;\|F - W\,B^\top\|_2^2$2 on $L_{\rm recon}^1 \;=\;\|F - W\,B^\top\|_2^2$3A5000 GPUs. The reported training time is approximately 7 h, inference takes 82 ms per image, and the trainable parameter count is approximately 4 M. Zero-shot splits follow standard HICO-DET UV, NF-UC, RF-UC, and UO settings (Lei et al., 21 Jul 2025).

These implementation choices are consistent with the paper’s emphasis on lightweight adaptation rather than full backbone tuning. The combination of frozen VLM backbones with low-rank decomposition, restricted weight adaptation, and compact adapters is intended to preserve prior knowledge while keeping the trainable footprint small.

6. Empirical performance on HICO-DET

HOLa reports state-of-the-art zero-shot performance on HICO-DET. On the unseen-verb split, the method achieves unseen mAP $L_{\rm recon}^1 \;=\;\|F - W\,B^\top\|_2^2$4, seen mAP $L_{\rm recon}^1 \;=\;\|F - W\,B^\top\|_2^2$5, and harmonic mean $L_{\rm recon}^1 \;=\;\|F - W\,B^\top\|_2^2$6. The paper further states that HOLa sets new state-of-the-art on NF-UC, RF-UC, and UO splits as well (Lei et al., 21 Jul 2025).

Split	Metric	Result
UV	Unseen mAP	27.91
UV	Seen mAP	35.09
UV	HM	31.09

The unseen-verb result is especially significant because this split removes some verbs entirely from training, making action generalization the dominant challenge. In that setting, HOLa’s combination of class-shared low-rank structure and LLM-guided action regularization is presented as the mechanism by which unseen-action discrimination improves without sacrificing overall zero-shot transfer.

7. Ablation evidence, interpretation, and extensions

The ablation study under the unseen-verb split isolates the contribution of each major component. Low-rank feature decomposition alone raises unseen mAP from $L_{\rm recon}^1 \;=\;\|F - W\,B^\top\|_2^2$7 to $L_{\rm recon}^1 \;=\;\|F - W\,B^\top\|_2^2$8, a gain of $L_{\rm recon}^1 \;=\;\|F - W\,B^\top\|_2^2$9. Adding weight adaptation by fine-tuning $L_{\rm sparse}^1 = \sum_{i=1}^N\|w_i\|_1$0 via $L_{\rm sparse}^1 = \sum_{i=1}^N\|w_i\|_1$1 boosts seen performance to $L_{\rm sparse}^1 = \sum_{i=1}^N\|w_i\|_1$2 but slightly harms unseen performance. LLM action regularization recovers unseen performance by $L_{\rm sparse}^1 = \sum_{i=1}^N\|w_i\|_1$3, reaching $L_{\rm sparse}^1 = \sum_{i=1}^N\|w_i\|_1$4 mAP. Text fusion of action and object adds $L_{\rm sparse}^1 = \sum_{i=1}^N\|w_i\|_1$5 unseen mAP. Human-object tokens add $L_{\rm sparse}^1 = \sum_{i=1}^N\|w_i\|_1$6 unseen and $L_{\rm sparse}^1 = \sum_{i=1}^N\|w_i\|_1$7 full mAP, yielding the final $L_{\rm sparse}^1 = \sum_{i=1}^N\|w_i\|_1$8 result (Lei et al., 21 Jul 2025).

Additional ablations further characterize the method. Updating $L_{\rm sparse}^1 = \sum_{i=1}^N\|w_i\|_1$9 with $L_{\rm ort} = \sum_{i\neq j} (b_i^\top b_j).$0 versus keeping it fixed under $L_{\rm ort} = \sum_{i\neq j} (b_i^\top b_j).$1 shows that fixing $L_{\rm ort} = \sum_{i\neq j} (b_i^\top b_j).$2 is essential to preserve unseen information. Rank choices $L_{\rm ort} = \sum_{i\neq j} (b_i^\top b_j).$3 identify $L_{\rm ort} = \sum_{i\neq j} (b_i^\top b_j).$4 as the best balance. Orthogonality and sparsity constraints on $L_{\rm ort} = \sum_{i\neq j} (b_i^\top b_j).$5 and $L_{\rm ort} = \sum_{i\neq j} (b_i^\top b_j).$6 each yield approximately $L_{\rm ort} = \sum_{i\neq j} (b_i^\top b_j).$7 mAP gain. The semantic loss $L_{\rm ort} = \sum_{i\neq j} (b_i^\top b_j).$8 contributes $L_{\rm ort} = \sum_{i\neq j} (b_i^\top b_j).$9 unseen mAP. Among human, object, and union feature combinations, $\hat f_i = w_i\,B^\top.$0 is the best configuration.

The paper concludes with several potential extensions: open-vocabulary HOI beyond fixed unseen lists, applying low-rank decomposition to open-vocabulary object features or visual feature adaptation under partial annotation, and scaling to larger VLM backbones such as ViT-L/14 or multi-modal instruction-tuned encoders such as BLIP-2. These directions follow directly from the method’s reliance on decomposed language-aligned representations and lightweight adaptation, suggesting that HOLa is positioned as a zero-shot HOI framework whose core mechanisms could be generalized beyond the fixed-class HICO-DET protocol.