Cascade Disentangling Network for HOI Detection

• The paper introduces a transformer-based method that disentangles human–object detection from interaction classification using a cascade of specialized decoders.
• It combines the strengths of two-stage and one-stage HOI detection, achieving higher mAP and notable improvements on rare interaction classes.
• The approach leverages query transfer, dynamic re-weighting, and pair-wise non-maximal suppression to optimize end-to-end training and precise localization.

Cascade Disentangling Network (CDN) is a transformer-based, one-stage Human–Object Interaction (HOI) detector introduced in "Mining the Benefits of Two-stage and One-stage HOI Detection" (Zhang et al., 2021). Its defining design principle is the disentangling of human–object pair detection from interaction classification, with the two tasks arranged in a cascade. The model is intended to combine the advantages of conventional two-stage HOI detectors—explicit task separation between localization and semantic classification—with the advantages of one-stage detectors, namely end-to-end prediction, efficient direct generation of interactive pairs, and improved positive-pair positioning. In the formulation used by CDN, HOI detection predicts a set of triplets

$$S = \{(b^h_i, b^o_i, a_i), i \in \{1,2,\dots,K\}\},$$

where $b^h_i$ is a human bounding box, $b^o_i$ is an object bounding box, and $a_i$ is an interaction category.

1. Problem setting and motivation

HOI detection aims to infer human activities from a static image by localizing humans and objects and classifying their interactions. A prediction is correct when both human and object boxes have IoU $> 0.5$ with respect to ground truth, the object category is correct, and the action category is correct (Zhang et al., 2021). This definition makes the task intrinsically joint: geometric localization and relational semantics must both be satisfied.

The paper situates CDN against two established HOI paradigms. Classical two-stage methods such as iCAN, DRG, and VCL first perform instance detection and then enumerate all human–object pairs for interaction classification. In this decomposition, an instance detector $T_d(\cdot)$ produces $M$ human boxes and $N$ object boxes, and a classifier $T_c(\cdot)$ evaluates all $M \times N$ candidate pairs. The structural advantage is clear task disentangling, but the cost is severe positive/negative imbalance, combinatorial pairing complexity $b^h_i$0, and reliance on local region features that are not optimized for interaction semantics.

Recent one-stage methods such as PPDM, UnionDet, HOI-Trans, QPIC, AS-Net, and HOTR instead detect HOI triplets directly. In set-based transformer formulations, each query predicts $b^h_i$1 jointly by optimizing

$b^h_i$2

where $b^h_i$3 denotes visual features. These methods are end-to-end, efficient, and comparatively effective at positioning interactive pairs. However, the paper identifies a multi-task coupling problem: detection and interaction classification demand different cues—object boundaries, shape, and location for detection versus human pose, contact regions, and broader context for interaction recognition.

The motivating diagnosis is therefore asymmetric. Two-stage methods are effective at task disentangling but poor at positioning positive interactive pairs, whereas one-stage methods are effective at directly locating interactive pairs but make an unfavorable trade-off in multi-task learning. CDN is built around the question of whether one can retain one-stage pair generation while reinstating explicit disentangling between detection and interaction classification.

2. Cascade disentangling principle

CDN answers that question by decomposing HOI prediction into two cascaded conditional problems:

$b^h_i$4

for human–object pair detection, and

$b^h_i$5

for interaction classification (Zhang et al., 2021). This is the core conceptual distinction between CDN and joint-query one-stage detectors such as QPIC or HOI-Trans, where each query directly predicts boxes and actions in a single shared decoding process.

Operationally, CDN begins from a transformer-based one-stage HOI detector and removes the interaction classification head to create a human–object pair generator. This first decoder, termed the Human–Object Pair Decoder (HO-PD), predicts human boxes, object boxes, object classes, and an interaction score indicating whether a pair is interactive or non-interactive. A second decoder, with independent parameters, is then dedicated exclusively to action classification.

The cascade is realized through query transfer. The interaction decoder is initialized with the output queries of HO-PD, denoted $b^h_i$6, establishing a one-to-one correspondence between detected pairs and action predictions. Formally, if the HO-PD output is

$b^h_i$7

then the interaction decoder computes

$b^h_i$8

where $b^h_i$9 is the sequenced encoder memory and $b^o_i$0 is positional encoding. This design preserves end-to-end training while structurally separating localization from interaction semantics.

A central implication of this formulation is that the interaction decoder is no longer burdened with bounding-box regression. The paper’s qualitative analysis reports that HO-PD and a conventional multi-task decoder such as QPIC attend to human/object boundaries and contact points, whereas CDN’s interaction decoder shifts attention toward human pose, hands or body parts, and broader relational context. This suggests that cascade disentangling induces feature specialization rather than merely adding depth.

3. Architecture and inference pipeline

The input image $b^o_i$1 is processed by a visual feature extractor composed of a CNN backbone and a DETR-style transformer encoder (Zhang et al., 2021). The backbone is either ResNet-50 or ResNet-101 and produces a feature map of shape $b^o_i$2. A $b^o_i$3 convolution projects the channel dimension to $b^o_i$4, after which the spatial dimensions are flattened to obtain

$b^o_i$5

Positional encodings are added, and a transformer encoder produces the global memory

$b^o_i$6

Both decoders are DETR-like and have independent weights. Given query embeddings $b^o_i$7, each decoder layer applies self-attention among queries, cross-attention between queries and $b^o_i$8, and feed-forward prediction heads. In compact notation,

$b^o_i$9

All queries within a decoder share the same feed-forward heads, and intermediate supervision is applied at every layer.

The HO-PD uses learnable HO queries

$a_i$0

The number of queries is dataset-specific: $a_i$1 on HICO-Det and $a_i$2 on V-COCO. For each query, HO-PD predicts a human box $a_i$3, an object box $a_i$4, an object class distribution $a_i$5, and an interaction score $a_i$6. The human and object box heads are 3-layer MLPs with ReLU, while the object classification and interactive-score heads are 1-layer linear classifiers.

The interaction decoder takes

$a_i$7

as its queries and predicts action probabilities $a_i$8 through a 1-layer linear action classifier. The resulting per-query HOI output consists of a pair hypothesis and an associated action distribution. Three variants are defined: CDN-S uses ResNet-50 with a 6-layer encoder, a 3-layer HO-PD, and a 3-layer interaction decoder; CDN-B uses ResNet-50 with 6-layer decoders; CDN-L uses ResNet-101 with a 6-layer encoder and 6-layer decoders.

At inference, the model gathers human boxes $a_i$9, object boxes $> 0.5$0, object scores $> 0.5$1, interaction scores $> 0.5$2, and action scores $> 0.5$3 for all queries. For each query, the HOI confidence is computed as

$> 0.5$4

Triplets are sorted by $> 0.5$5, the top $> 0.5$6 candidates are retained, and Pair-wise Non-Maximal Suppression (PNMS) is applied. For two HOI triplets $> 0.5$7 and $> 0.5$8, the pair-wise IoU is

$> 0.5$9

with default parameters threshold $T_d(\cdot)$0, $T_d(\cdot)$1, and $T_d(\cdot)$2. The choice $T_d(\cdot)$3 reflects the paper’s statement that human-box overlap is more impactful than object-box overlap.

4. Objective function and training strategy

CDN is trained with a set-based loss and Hungarian bipartite matching, following the general training scheme of DETR-style HOI detectors (Zhang et al., 2021). Predictions from both decoders are matched one-to-one to ground-truth HOI triplets, and unmatched predictions are treated as background. The loss contains five components: box regression $T_d(\cdot)$4, generalized IoU $T_d(\cdot)$5, interactive score loss $T_d(\cdot)$6, object classification loss $T_d(\cdot)$7, and action classification loss $T_d(\cdot)$8. The overall objective is

$T_d(\cdot)$9

The reported loss weights are $M$0, $M$1, $M$2, $M$3, and $M$4, following QPIC.

A notable component of the method is its decoupling-oriented fine-tuning strategy for long-tailed HOI distributions. Training proceeds in two phases. In the first phase, the entire network is trained jointly for 90 epochs using static class weights. In the second phase, the visual feature extractor—CNN backbone plus transformer encoder—is frozen, and both decoders together with their FFNs are fine-tuned for 10 epochs at a smaller learning rate. Dynamic re-weighting is applied to object and action classification on HICO-Det, and only to action classification on V-COCO.

Dynamic re-weighting uses memory-bank queues $M$5 and $M$6 to accumulate positive sample counts over recent iterations. If $M$7 is the number of positive action samples for class $M$8 and $M$9 is the number of background samples, then the action weights are

$N$0

For objects,

$N$1

The final class weights interpolate between static dataset-derived weights and online dynamic weights:

$N$2

with

$N$3

The intended effect is straightforward: rare classes receive larger weights, while frequent classes are comparatively down-weighted.

The implementation is initialized from DETR pretrained on MS-COCO for object detection. Optimization uses AdamW with weight decay $N$4. Stage 1 uses learning rate $N$5 for 90 epochs with a $N$6 decay at epoch 60. Stage 2 uses learning rate $N$7 for 10 epochs. The reported hardware configuration is 8 Tesla V100 GPUs with CUDA 10.2 and batch size 16. Re-weighting hyperparameters are $N$8 and queue length $N$9, where $T_c(\cdot)$0 is the number of training samples.

5. Empirical performance and ablation evidence

The principal evaluations are conducted on HICO-Det and V-COCO (Zhang et al., 2021). HICO-Det contains 38,118 training images and 9,658 test images, with 80 object categories, 117 actions, and 600 HOI classes. Performance is reported as mean AP on the full set, a rare subset of 138 classes with fewer than 10 training instances, and a non-rare subset of 462 classes. V-COCO contains 5,400 trainval images and 4,946 test images, with 29 actions, 80 object categories, and role-mAP metrics $T_c(\cdot)$1 and $T_c(\cdot)$2.

On HICO-Det with ResNet-50 and no extra pose or language information, the reported one-stage baselines are HOI-Trans at 23.46 mAP, AS-Net at 28.87 mAP, and QPIC at 29.07 mAP. CDN reports 31.44 mAP for CDN-S, 31.78 mAP for CDN-B, and 32.07 mAP for CDN-L. The detailed scores are 31.44/27.39/32.64 for CDN-S on full/rare/non-rare, 31.78/27.55/33.05 for CDN-B, and 32.07/27.19/33.53 for CDN-L. Against QPIC with the same backbone, CDN-B improves full mAP from 29.07 to 31.78, corresponding to $T_c(\cdot)$3 absolute and $T_c(\cdot)$4 relative. On the rare split, the improvement is from 21.85 to 27.55, reported as $T_c(\cdot)$5 relative.

On V-COCO, QPIC reports $T_c(\cdot)$6 and $T_c(\cdot)$7, while CDN-B reports $T_c(\cdot)$8 and $T_c(\cdot)$9, respectively. CDN-L reaches $M \times N$0 on Scenario 1 and $M \times N$1 on Scenario 2. The relative gains listed for CDN-B over QPIC are approximately $M \times N$2 in Scenario 1 and $M \times N$3 in Scenario 2.

The ablation studies are central to interpreting why the architecture works. A base CDN-B without re-weighting and without PNMS attains 31.06 mAP, already above QPIC’s 29.07. Adding dynamic re-weighting raises this to 31.38, and adding PNMS yields the full 31.78. A more granular analysis of re-weighting shows that decoupled training alone gives 30.90, adding static weights gives 31.25, and full dynamic re-weighting with queue length $M \times N$4 and $M \times N$5 gives 31.38.

The paper also isolates pair generation quality. Replacing the pair proposals in iCAN with CDN’s HO-PD output raises mAP from 15.37 to 24.05, indicating that the one-stage pair generator is markedly better at producing high-quality interactive pairs. A separate comparison between QPIC and a simplified “CDN-S base” reports 29.07 versus 30.96 on full mAP, with rare performance improving from 21.85 to approximately 27.00+, which the paper describes as about $M \times N$6 improvement. This supports the claim that the disentangled interaction decoder materially improves interaction classification, especially in the long tail.

6. Relation to prior HOI detection paradigms, misconceptions, and limitations

CDN occupies an intermediate but structurally distinct position between conventional two-stage HOI detection and joint-query one-stage detection (Zhang et al., 2021). Relative to two-stage methods such as iCAN, DRG, VCL, FCMNet, PD-Net, DJ-RN, and IDN, it avoids exhaustive $M \times N$7 human–object pairing and the resulting negative-pair explosion. Relative to one-stage methods such as PPDM, UnionDet, DIRV, HOI-Trans, GG-Net, ATL, HOTR, AS-Net, and QPIC, it rejects the assumption that one shared decoder should jointly optimize detection and interaction recognition.

A common misconception would be to treat CDN as merely a two-stage detector recast in transformer terminology. That characterization is not accurate in the formulation reported in the paper. The system remains one-stage and end-to-end: pair generation is query-based, not proposal-enumerative; interaction decoding is conditioned on query states produced within the same forward computation; and both decoders are trained jointly under a single set-prediction loss. The cascade therefore does not reintroduce the classical two-stage pipeline of detecting all instances and then classifying all pair combinations.

A second misconception would be to interpret CDN’s gains as arising only from auxiliary training heuristics. The ablation results indicate otherwise. Dynamic re-weighting and PNMS improve performance, but the base disentangled architecture already exceeds the QPIC baseline, and the pair-generation substitution experiment further attributes a substantial portion of the gain to the HO-PD stage. This suggests that the principal contribution is architectural disentangling rather than post hoc score adjustment.

The limitations identified in the paper are correspondingly specific. CDN is presented in one transformer-based instantiation aligned with DETR and QPIC/HOI-Trans-style set prediction. The authors state that the idea could be generalized to other one-stage frameworks and combined with graph reasoning, language, or pose modules. They also note continuing HOI challenges, including long-tailed class distributions, complex multi-human and multi-object interactions, and domain generalization. A plausible implication is that CDN should be understood less as a final architecture than as a design pattern for task-separated HOI decoding within end-to-end set prediction.

In that sense, CDN’s broader significance lies in how it reframes the long-standing trade-off between two-stage and one-stage HOI detection. Rather than choosing between explicit task separation and direct interactive-pair localization, it factorizes the HOI problem into a cascaded pair detector and action classifier, preserving end-to-end optimization while allowing decoder-level specialization.